<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2004.11714] Residual Energy-Based Models for Text Generation</title><meta property="og:description" content="Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation.
The dominant parametric approach is based on locally normalized models which predict one word at a time. While t…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Residual Energy-Based Models for Text Generation">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Residual Energy-Based Models for Text Generation">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2004.11714">

<!--Generated on Sat Mar  2 11:59:17 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Residual Energy-Based Models for 
<br class="ltx_break">Text Generation</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Yuntian Deng<sup id="id8.8.id1" class="ltx_sup">1</sup>, Anton Bakhtin<sup id="id9.9.id2" class="ltx_sup">2</sup>, Myle Ott<sup id="id10.10.id3" class="ltx_sup">2</sup>, Arthur Szlam<sup id="id11.11.id4" class="ltx_sup">2</sup>, Marc’Aurelio Ranzato<sup id="id12.12.id5" class="ltx_sup">2</sup> 
<br class="ltx_break">Harvard University<sup id="id13.13.id6" class="ltx_sup">1</sup>
<br class="ltx_break">Facebook AI Research<sup id="id14.14.id7" class="ltx_sup">2</sup>
<br class="ltx_break"><span id="id15.15.id8" class="ltx_text ltx_font_typewriter">dengyuntian@seas.harvard.edu {yolo,myleott,aszlam,ranzato}@fb.com</span> 
<br class="ltx_break">
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id16.id1" class="ltx_p">Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation.
The dominant parametric approach is based on locally normalized models which predict one word at a time. While these
work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process.
In this work, we investigate <span id="id16.id1.1" class="ltx_text ltx_font_italic">un-normalized</span> energy-based models (EBMs) which operate not at the token but at the sequence
level. In order to make training tractable, we first work in the residual of a pretrained locally normalized
language model and second we train using noise contrastive estimation. Furthermore, since the EBM works at the
sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa.
Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity
compared to locally normalized baselines. Moreover, generation via importance sampling is very efficient and of higher quality
than the baseline models according to human evaluation.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">The dominant approach to parametric text generation is based on large neural auto-regressive
models <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib27" title="" class="ltx_ref">2019</a>)</cite>. These models can be trained efficiently via maximum likelihood
and they can efficiently generate samples of remarkable quality.
Key to their success is <span id="S1.p1.1.1" class="ltx_text ltx_font_italic">local normalization</span>, i.e. they are defined in terms of a product of conditional
distributions, one for each token in the sequence. Such distributions are relatively cheap to compute with modern hardware given
the limited vocabulary size of common sub-word units like BPE <cite class="ltx_cite ltx_citemacro_citep">(Sennrich et al., <a href="#bib.bib31" title="" class="ltx_ref">2015</a>)</cite>.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">Unfortunately, local normalization also brings some drawbacks. First, the designer of the model needs to specify the order
in which tokens are generated. Second, at training time the model is conditioned on ground truth context while at test
time it is conditioned on its own generations, a discrepancy referred to as exposure bias <cite class="ltx_cite ltx_citemacro_citep">(Ranzato et al., <a href="#bib.bib29" title="" class="ltx_ref">2016</a>)</cite>.
Finally, while heuristics like beam search somewhat help rescore at the sequence level,
generation generally lacks long-range coherency because it is produced by the greedy selection of one token at a time
without lookahead.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">Energy-based models (EBMs) <cite class="ltx_cite ltx_citemacro_citep">(Hinton, <a href="#bib.bib12" title="" class="ltx_ref">2002</a>; LeCun et al., <a href="#bib.bib18" title="" class="ltx_ref">2006</a>; Ranzato et al., <a href="#bib.bib28" title="" class="ltx_ref">2007</a>)</cite> are a more general framework which
potentially address all these issues, as they do not require any local normalization. They only require the definition
of an energy function defined over the whole input sequence. Training aims at shaping the energy function such that
regions of high density of training data points have lower energy than elsewhere.
In principle, EBMs are ideal for modeling text as they can score the whole input at once,
they are not prone to label bias <cite class="ltx_cite ltx_citemacro_citep">(Bottou, <a href="#bib.bib3" title="" class="ltx_ref">1991</a>)</cite> and they may enable generation of large chunks of text,
which should help improve coherency.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">However, so far EBMs had limited application in text generation, because
sampling from the model is intractable, and so is maximum likelihood training. The problem is that shaping the energy function
is accomplished by updating the model parameters such that the energy is decreased at the training data points (a.k.a. positive
examples) and increased at other data points (a.k.a. negative examples). In maximum likelihood training negatives are generated
from the model, but in text application we cannot use gradient-based MCMC methods <cite class="ltx_cite ltx_citemacro_citep">(Teh et al., <a href="#bib.bib33" title="" class="ltx_ref">2003</a>; Du &amp; Mordatch, <a href="#bib.bib6" title="" class="ltx_ref">2019</a>)</cite> and
Gibbs sampling <cite class="ltx_cite ltx_citemacro_citep">(Welling et al., <a href="#bib.bib40" title="" class="ltx_ref">2005</a>)</cite> is
too slow to be practical. Generating negatives by local perturbations of the ground truth would be efficient but hardly useful for
generation purposes, when at test time the model needs to generate from scratch.</p>
</div>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">Recently, <cite class="ltx_cite ltx_citemacro_citet">Bakhtin et al. (<a href="#bib.bib2" title="" class="ltx_ref">2019</a>)</cite> carefully studied the problem of training a discriminator to distinguish human written
text from language model generations. They experimented with different language model and discriminator architectures,
training/test time corpora and concluded that the discriminator can generalize rather well to weaker language
models when the training/test corpora match.
<cite class="ltx_cite ltx_citemacro_citet">Bakhtin et al. (<a href="#bib.bib2" title="" class="ltx_ref">2019</a>)</cite> found that the learned discriminator is not robust to random perturbations,
and argued that the discriminator operates in the “residual” space of the language model.</p>
</div>
<div id="S1.p6" class="ltx_para ltx_noindent">
<p id="S1.p6.1" class="ltx_p">Concurrently, <cite class="ltx_cite ltx_citemacro_citet">Grover et al. (<a href="#bib.bib10" title="" class="ltx_ref">2019</a>)</cite> proposed a general approach to “de-bias” a generator,
by simply training a discriminator and using its output for importance sampling.</p>
</div>
<div id="S1.p7" class="ltx_para ltx_noindent">
<p id="S1.p7.5" class="ltx_p">In this work, we build upon these two works. First, we formalize the residual interpretation by <cite class="ltx_cite ltx_citemacro_citet">Bakhtin et al. (<a href="#bib.bib2" title="" class="ltx_ref">2019</a>)</cite>
and use a generative model of the form:</p>
<table id="S1.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S1.E1.m1.5" class="ltx_Math" alttext="P_{\theta}(x)\propto P_{LM}(x)\exp(-E_{\theta}(x))" display="block"><semantics id="S1.E1.m1.5a"><mrow id="S1.E1.m1.5.5" xref="S1.E1.m1.5.5.cmml"><mrow id="S1.E1.m1.5.5.3" xref="S1.E1.m1.5.5.3.cmml"><msub id="S1.E1.m1.5.5.3.2" xref="S1.E1.m1.5.5.3.2.cmml"><mi id="S1.E1.m1.5.5.3.2.2" xref="S1.E1.m1.5.5.3.2.2.cmml">P</mi><mi id="S1.E1.m1.5.5.3.2.3" xref="S1.E1.m1.5.5.3.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S1.E1.m1.5.5.3.1" xref="S1.E1.m1.5.5.3.1.cmml">​</mo><mrow id="S1.E1.m1.5.5.3.3.2" xref="S1.E1.m1.5.5.3.cmml"><mo stretchy="false" id="S1.E1.m1.5.5.3.3.2.1" xref="S1.E1.m1.5.5.3.cmml">(</mo><mi id="S1.E1.m1.1.1" xref="S1.E1.m1.1.1.cmml">x</mi><mo stretchy="false" id="S1.E1.m1.5.5.3.3.2.2" xref="S1.E1.m1.5.5.3.cmml">)</mo></mrow></mrow><mo id="S1.E1.m1.5.5.2" xref="S1.E1.m1.5.5.2.cmml">∝</mo><mrow id="S1.E1.m1.5.5.1" xref="S1.E1.m1.5.5.1.cmml"><msub id="S1.E1.m1.5.5.1.3" xref="S1.E1.m1.5.5.1.3.cmml"><mi id="S1.E1.m1.5.5.1.3.2" xref="S1.E1.m1.5.5.1.3.2.cmml">P</mi><mrow id="S1.E1.m1.5.5.1.3.3" xref="S1.E1.m1.5.5.1.3.3.cmml"><mi id="S1.E1.m1.5.5.1.3.3.2" xref="S1.E1.m1.5.5.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S1.E1.m1.5.5.1.3.3.1" xref="S1.E1.m1.5.5.1.3.3.1.cmml">​</mo><mi id="S1.E1.m1.5.5.1.3.3.3" xref="S1.E1.m1.5.5.1.3.3.3.cmml">M</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S1.E1.m1.5.5.1.2" xref="S1.E1.m1.5.5.1.2.cmml">​</mo><mrow id="S1.E1.m1.5.5.1.4.2" xref="S1.E1.m1.5.5.1.cmml"><mo stretchy="false" id="S1.E1.m1.5.5.1.4.2.1" xref="S1.E1.m1.5.5.1.cmml">(</mo><mi id="S1.E1.m1.2.2" xref="S1.E1.m1.2.2.cmml">x</mi><mo stretchy="false" id="S1.E1.m1.5.5.1.4.2.2" xref="S1.E1.m1.5.5.1.cmml">)</mo></mrow><mo lspace="0.167em" rspace="0em" id="S1.E1.m1.5.5.1.2a" xref="S1.E1.m1.5.5.1.2.cmml">​</mo><mrow id="S1.E1.m1.5.5.1.1.1" xref="S1.E1.m1.5.5.1.1.2.cmml"><mi id="S1.E1.m1.4.4" xref="S1.E1.m1.4.4.cmml">exp</mi><mo id="S1.E1.m1.5.5.1.1.1a" xref="S1.E1.m1.5.5.1.1.2.cmml">⁡</mo><mrow id="S1.E1.m1.5.5.1.1.1.1" xref="S1.E1.m1.5.5.1.1.2.cmml"><mo stretchy="false" id="S1.E1.m1.5.5.1.1.1.1.2" xref="S1.E1.m1.5.5.1.1.2.cmml">(</mo><mrow id="S1.E1.m1.5.5.1.1.1.1.1" xref="S1.E1.m1.5.5.1.1.1.1.1.cmml"><mo id="S1.E1.m1.5.5.1.1.1.1.1a" xref="S1.E1.m1.5.5.1.1.1.1.1.cmml">−</mo><mrow id="S1.E1.m1.5.5.1.1.1.1.1.2" xref="S1.E1.m1.5.5.1.1.1.1.1.2.cmml"><msub id="S1.E1.m1.5.5.1.1.1.1.1.2.2" xref="S1.E1.m1.5.5.1.1.1.1.1.2.2.cmml"><mi id="S1.E1.m1.5.5.1.1.1.1.1.2.2.2" xref="S1.E1.m1.5.5.1.1.1.1.1.2.2.2.cmml">E</mi><mi id="S1.E1.m1.5.5.1.1.1.1.1.2.2.3" xref="S1.E1.m1.5.5.1.1.1.1.1.2.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S1.E1.m1.5.5.1.1.1.1.1.2.1" xref="S1.E1.m1.5.5.1.1.1.1.1.2.1.cmml">​</mo><mrow id="S1.E1.m1.5.5.1.1.1.1.1.2.3.2" xref="S1.E1.m1.5.5.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S1.E1.m1.5.5.1.1.1.1.1.2.3.2.1" xref="S1.E1.m1.5.5.1.1.1.1.1.2.cmml">(</mo><mi id="S1.E1.m1.3.3" xref="S1.E1.m1.3.3.cmml">x</mi><mo stretchy="false" id="S1.E1.m1.5.5.1.1.1.1.1.2.3.2.2" xref="S1.E1.m1.5.5.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S1.E1.m1.5.5.1.1.1.1.3" xref="S1.E1.m1.5.5.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.E1.m1.5b"><apply id="S1.E1.m1.5.5.cmml" xref="S1.E1.m1.5.5"><csymbol cd="latexml" id="S1.E1.m1.5.5.2.cmml" xref="S1.E1.m1.5.5.2">proportional-to</csymbol><apply id="S1.E1.m1.5.5.3.cmml" xref="S1.E1.m1.5.5.3"><times id="S1.E1.m1.5.5.3.1.cmml" xref="S1.E1.m1.5.5.3.1"></times><apply id="S1.E1.m1.5.5.3.2.cmml" xref="S1.E1.m1.5.5.3.2"><csymbol cd="ambiguous" id="S1.E1.m1.5.5.3.2.1.cmml" xref="S1.E1.m1.5.5.3.2">subscript</csymbol><ci id="S1.E1.m1.5.5.3.2.2.cmml" xref="S1.E1.m1.5.5.3.2.2">𝑃</ci><ci id="S1.E1.m1.5.5.3.2.3.cmml" xref="S1.E1.m1.5.5.3.2.3">𝜃</ci></apply><ci id="S1.E1.m1.1.1.cmml" xref="S1.E1.m1.1.1">𝑥</ci></apply><apply id="S1.E1.m1.5.5.1.cmml" xref="S1.E1.m1.5.5.1"><times id="S1.E1.m1.5.5.1.2.cmml" xref="S1.E1.m1.5.5.1.2"></times><apply id="S1.E1.m1.5.5.1.3.cmml" xref="S1.E1.m1.5.5.1.3"><csymbol cd="ambiguous" id="S1.E1.m1.5.5.1.3.1.cmml" xref="S1.E1.m1.5.5.1.3">subscript</csymbol><ci id="S1.E1.m1.5.5.1.3.2.cmml" xref="S1.E1.m1.5.5.1.3.2">𝑃</ci><apply id="S1.E1.m1.5.5.1.3.3.cmml" xref="S1.E1.m1.5.5.1.3.3"><times id="S1.E1.m1.5.5.1.3.3.1.cmml" xref="S1.E1.m1.5.5.1.3.3.1"></times><ci id="S1.E1.m1.5.5.1.3.3.2.cmml" xref="S1.E1.m1.5.5.1.3.3.2">𝐿</ci><ci id="S1.E1.m1.5.5.1.3.3.3.cmml" xref="S1.E1.m1.5.5.1.3.3.3">𝑀</ci></apply></apply><ci id="S1.E1.m1.2.2.cmml" xref="S1.E1.m1.2.2">𝑥</ci><apply id="S1.E1.m1.5.5.1.1.2.cmml" xref="S1.E1.m1.5.5.1.1.1"><exp id="S1.E1.m1.4.4.cmml" xref="S1.E1.m1.4.4"></exp><apply id="S1.E1.m1.5.5.1.1.1.1.1.cmml" xref="S1.E1.m1.5.5.1.1.1.1.1"><minus id="S1.E1.m1.5.5.1.1.1.1.1.1.cmml" xref="S1.E1.m1.5.5.1.1.1.1.1"></minus><apply id="S1.E1.m1.5.5.1.1.1.1.1.2.cmml" xref="S1.E1.m1.5.5.1.1.1.1.1.2"><times id="S1.E1.m1.5.5.1.1.1.1.1.2.1.cmml" xref="S1.E1.m1.5.5.1.1.1.1.1.2.1"></times><apply id="S1.E1.m1.5.5.1.1.1.1.1.2.2.cmml" xref="S1.E1.m1.5.5.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S1.E1.m1.5.5.1.1.1.1.1.2.2.1.cmml" xref="S1.E1.m1.5.5.1.1.1.1.1.2.2">subscript</csymbol><ci id="S1.E1.m1.5.5.1.1.1.1.1.2.2.2.cmml" xref="S1.E1.m1.5.5.1.1.1.1.1.2.2.2">𝐸</ci><ci id="S1.E1.m1.5.5.1.1.1.1.1.2.2.3.cmml" xref="S1.E1.m1.5.5.1.1.1.1.1.2.2.3">𝜃</ci></apply><ci id="S1.E1.m1.3.3.cmml" xref="S1.E1.m1.3.3">𝑥</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.E1.m1.5c">P_{\theta}(x)\propto P_{LM}(x)\exp(-E_{\theta}(x))</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S1.p7.4" class="ltx_p">where <math id="S1.p7.1.m1.1" class="ltx_Math" alttext="P_{LM}(x)" display="inline"><semantics id="S1.p7.1.m1.1a"><mrow id="S1.p7.1.m1.1.2" xref="S1.p7.1.m1.1.2.cmml"><msub id="S1.p7.1.m1.1.2.2" xref="S1.p7.1.m1.1.2.2.cmml"><mi id="S1.p7.1.m1.1.2.2.2" xref="S1.p7.1.m1.1.2.2.2.cmml">P</mi><mrow id="S1.p7.1.m1.1.2.2.3" xref="S1.p7.1.m1.1.2.2.3.cmml"><mi id="S1.p7.1.m1.1.2.2.3.2" xref="S1.p7.1.m1.1.2.2.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S1.p7.1.m1.1.2.2.3.1" xref="S1.p7.1.m1.1.2.2.3.1.cmml">​</mo><mi id="S1.p7.1.m1.1.2.2.3.3" xref="S1.p7.1.m1.1.2.2.3.3.cmml">M</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S1.p7.1.m1.1.2.1" xref="S1.p7.1.m1.1.2.1.cmml">​</mo><mrow id="S1.p7.1.m1.1.2.3.2" xref="S1.p7.1.m1.1.2.cmml"><mo stretchy="false" id="S1.p7.1.m1.1.2.3.2.1" xref="S1.p7.1.m1.1.2.cmml">(</mo><mi id="S1.p7.1.m1.1.1" xref="S1.p7.1.m1.1.1.cmml">x</mi><mo stretchy="false" id="S1.p7.1.m1.1.2.3.2.2" xref="S1.p7.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.p7.1.m1.1b"><apply id="S1.p7.1.m1.1.2.cmml" xref="S1.p7.1.m1.1.2"><times id="S1.p7.1.m1.1.2.1.cmml" xref="S1.p7.1.m1.1.2.1"></times><apply id="S1.p7.1.m1.1.2.2.cmml" xref="S1.p7.1.m1.1.2.2"><csymbol cd="ambiguous" id="S1.p7.1.m1.1.2.2.1.cmml" xref="S1.p7.1.m1.1.2.2">subscript</csymbol><ci id="S1.p7.1.m1.1.2.2.2.cmml" xref="S1.p7.1.m1.1.2.2.2">𝑃</ci><apply id="S1.p7.1.m1.1.2.2.3.cmml" xref="S1.p7.1.m1.1.2.2.3"><times id="S1.p7.1.m1.1.2.2.3.1.cmml" xref="S1.p7.1.m1.1.2.2.3.1"></times><ci id="S1.p7.1.m1.1.2.2.3.2.cmml" xref="S1.p7.1.m1.1.2.2.3.2">𝐿</ci><ci id="S1.p7.1.m1.1.2.2.3.3.cmml" xref="S1.p7.1.m1.1.2.2.3.3">𝑀</ci></apply></apply><ci id="S1.p7.1.m1.1.1.cmml" xref="S1.p7.1.m1.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p7.1.m1.1c">P_{LM}(x)</annotation></semantics></math> is a locally normalized language model which is fixed during training, and <math id="S1.p7.2.m2.1" class="ltx_Math" alttext="E_{\theta}" display="inline"><semantics id="S1.p7.2.m2.1a"><msub id="S1.p7.2.m2.1.1" xref="S1.p7.2.m2.1.1.cmml"><mi id="S1.p7.2.m2.1.1.2" xref="S1.p7.2.m2.1.1.2.cmml">E</mi><mi id="S1.p7.2.m2.1.1.3" xref="S1.p7.2.m2.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S1.p7.2.m2.1b"><apply id="S1.p7.2.m2.1.1.cmml" xref="S1.p7.2.m2.1.1"><csymbol cd="ambiguous" id="S1.p7.2.m2.1.1.1.cmml" xref="S1.p7.2.m2.1.1">subscript</csymbol><ci id="S1.p7.2.m2.1.1.2.cmml" xref="S1.p7.2.m2.1.1.2">𝐸</ci><ci id="S1.p7.2.m2.1.1.3.cmml" xref="S1.p7.2.m2.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p7.2.m2.1c">E_{\theta}</annotation></semantics></math> is the
energy function parameterized by <math id="S1.p7.3.m3.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S1.p7.3.m3.1a"><mi id="S1.p7.3.m3.1.1" xref="S1.p7.3.m3.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S1.p7.3.m3.1b"><ci id="S1.p7.3.m3.1.1.cmml" xref="S1.p7.3.m3.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p7.3.m3.1c">\theta</annotation></semantics></math>. The resulting model <math id="S1.p7.4.m4.1" class="ltx_Math" alttext="P_{\theta}(x)" display="inline"><semantics id="S1.p7.4.m4.1a"><mrow id="S1.p7.4.m4.1.2" xref="S1.p7.4.m4.1.2.cmml"><msub id="S1.p7.4.m4.1.2.2" xref="S1.p7.4.m4.1.2.2.cmml"><mi id="S1.p7.4.m4.1.2.2.2" xref="S1.p7.4.m4.1.2.2.2.cmml">P</mi><mi id="S1.p7.4.m4.1.2.2.3" xref="S1.p7.4.m4.1.2.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S1.p7.4.m4.1.2.1" xref="S1.p7.4.m4.1.2.1.cmml">​</mo><mrow id="S1.p7.4.m4.1.2.3.2" xref="S1.p7.4.m4.1.2.cmml"><mo stretchy="false" id="S1.p7.4.m4.1.2.3.2.1" xref="S1.p7.4.m4.1.2.cmml">(</mo><mi id="S1.p7.4.m4.1.1" xref="S1.p7.4.m4.1.1.cmml">x</mi><mo stretchy="false" id="S1.p7.4.m4.1.2.3.2.2" xref="S1.p7.4.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.p7.4.m4.1b"><apply id="S1.p7.4.m4.1.2.cmml" xref="S1.p7.4.m4.1.2"><times id="S1.p7.4.m4.1.2.1.cmml" xref="S1.p7.4.m4.1.2.1"></times><apply id="S1.p7.4.m4.1.2.2.cmml" xref="S1.p7.4.m4.1.2.2"><csymbol cd="ambiguous" id="S1.p7.4.m4.1.2.2.1.cmml" xref="S1.p7.4.m4.1.2.2">subscript</csymbol><ci id="S1.p7.4.m4.1.2.2.2.cmml" xref="S1.p7.4.m4.1.2.2.2">𝑃</ci><ci id="S1.p7.4.m4.1.2.2.3.cmml" xref="S1.p7.4.m4.1.2.2.3">𝜃</ci></apply><ci id="S1.p7.4.m4.1.1.cmml" xref="S1.p7.4.m4.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p7.4.m4.1c">P_{\theta}(x)</annotation></semantics></math> is globally normalized due to the energy term. Note that the same residual formulation was also used in <cite class="ltx_cite ltx_citemacro_citet">Rosenfeld et al. (<a href="#bib.bib30" title="" class="ltx_ref">2001</a>); Wang &amp; Ou (<a href="#bib.bib37" title="" class="ltx_ref">2018b</a>); Parshakova et al. (<a href="#bib.bib25" title="" class="ltx_ref">2019</a>)</cite>.</p>
</div>
<div id="S1.p8" class="ltx_para ltx_noindent">
<p id="S1.p8.1" class="ltx_p">This formulation has multi-fold benefits. First, by incorporating a locally normalized language model,
we can leverage recent advancements in locally normalized language modeling. Second, the language model provides a natural proposal distribution for training  <cite class="ltx_cite ltx_citemacro_citep">(Bakhtin et al., <a href="#bib.bib2" title="" class="ltx_ref">2019</a>)</cite>, and training can be made efficient by using
the conditional noise contrastive
estimation objective <cite class="ltx_cite ltx_citemacro_citep">(Gutmann &amp; Hyvärinen, <a href="#bib.bib11" title="" class="ltx_ref">2010</a>)</cite>
as we shall see in §<a href="#S3" title="3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.
Lastly, this formulation enables efficient evaluation and generation via importance sampling <cite class="ltx_cite ltx_citemacro_citep">(Horvitz &amp; Thompson, <a href="#bib.bib16" title="" class="ltx_ref">1952</a>; Grover et al., <a href="#bib.bib10" title="" class="ltx_ref">2019</a>)</cite>.</p>
</div>
<div id="S1.p9" class="ltx_para ltx_noindent">
<p id="S1.p9.1" class="ltx_p">In some sense, this last point is perhaps the central contribution of the paper, as it allows estimating perplexity of the residual EBM, and thus allows these EBMs to be compared in a standard way to other models. Indeed, in §<a href="#S4" title="4 Experiments ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> we show that our joint model decreases perplexity
on two large datasets, when compared
to various auto-regressive language model baselines. Finally, the EBM generations are significantly preferred by humans
according to our qualitative evaluation. To the best of our knowledge, this is the first time that an EBM has
demonstrated improved generation ability against very strong auto-regressive baselines, both in terms of estimated perplexity and through human evaluation.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p">Energy-based models have a long history in machine learning <cite class="ltx_cite ltx_citemacro_citep">(Hopfield, <a href="#bib.bib15" title="" class="ltx_ref">1982</a>; Hinton, <a href="#bib.bib12" title="" class="ltx_ref">2002</a>; LeCun et al., <a href="#bib.bib18" title="" class="ltx_ref">2006</a>; Ranzato et al., <a href="#bib.bib28" title="" class="ltx_ref">2007</a>)</cite>.
The key challenge of training is mining for good negatives. This can be accomplished explicitly by fantasizing inputs where
the energy should be increased or implicitly via global constraints such as sparsity <cite class="ltx_cite ltx_citemacro_citep">(Ranzato et al., <a href="#bib.bib28" title="" class="ltx_ref">2007</a>)</cite>.
Methods attempting at maximizing the likelihood of the data require to sample from the distribution induced by the model.
Unfortunately, gradient-based MCMC approaches like Hybrid Monte Carlo <cite class="ltx_cite ltx_citemacro_citep">(Teh et al., <a href="#bib.bib33" title="" class="ltx_ref">2003</a>)</cite> and Langevyn dynamics <cite class="ltx_cite ltx_citemacro_citep">(Ranzato et al., <a href="#bib.bib28" title="" class="ltx_ref">2007</a>; Du &amp; Mordatch, <a href="#bib.bib6" title="" class="ltx_ref">2019</a>; Xie et al., <a href="#bib.bib41" title="" class="ltx_ref">2016</a>; <a href="#bib.bib42" title="" class="ltx_ref">2017</a>; <a href="#bib.bib44" title="" class="ltx_ref">2019</a>; <a href="#bib.bib43" title="" class="ltx_ref">2018</a>; Gao et al., <a href="#bib.bib8" title="" class="ltx_ref">2018</a>; Nijkamp et al., <a href="#bib.bib22" title="" class="ltx_ref">2019</a>)</cite> are not applicable when the input is discrete like in text applications. Other approaches like Gibbs sampling <cite class="ltx_cite ltx_citemacro_citep">(Hinton, <a href="#bib.bib12" title="" class="ltx_ref">2002</a>)</cite> were applied to binary inputs but do not scale well to large dictionaries once the energy function is a large
bidirectional transformer model like the one used in this work. Several variants of auto-encoders have also been investigated
for representing and generating text <cite class="ltx_cite ltx_citemacro_citep">(Bowman et al., <a href="#bib.bib4" title="" class="ltx_ref">2016</a>; Zhao et al., <a href="#bib.bib45" title="" class="ltx_ref">2018</a>)</cite>, but they have not shown significant improvements in terms of perplexity
and they have so far been applied to relatively small datasets only.</p>
</div>
<div id="S2.p2" class="ltx_para ltx_noindent">
<p id="S2.p2.1" class="ltx_p">Our approach appears similar to discriminative reranking approaches used in the parsing and machine translation
community <cite class="ltx_cite ltx_citemacro_citep">(Shen et al., <a href="#bib.bib32" title="" class="ltx_ref">2004</a>)</cite>. However, our approach provides a generative model, and parameters/hyper-parameters
are directly tuned to close the gap between the model distribution and the data distribution,
rather than relying on surrogate ranking losses. This approach is also related to other sequence level training
objectives <cite class="ltx_cite ltx_citemacro_citep">(Edunov et al., <a href="#bib.bib7" title="" class="ltx_ref">2018</a>)</cite>, with the major difference that in those works training aims at improving the baseline model, but generation at test time is still greedy.</p>
</div>
<div id="S2.p3" class="ltx_para ltx_noindent">
<p id="S2.p3.1" class="ltx_p">Energy Networks have been used for sequence modeling <cite class="ltx_cite ltx_citemacro_citep">(Rosenfeld et al., <a href="#bib.bib30" title="" class="ltx_ref">2001</a>; Wang et al., <a href="#bib.bib38" title="" class="ltx_ref">2015</a>; <a href="#bib.bib39" title="" class="ltx_ref">2017</a>; Wang &amp; Ou, <a href="#bib.bib35" title="" class="ltx_ref">2017</a>; <a href="#bib.bib36" title="" class="ltx_ref">2018a</a>; Parshakova et al., <a href="#bib.bib25" title="" class="ltx_ref">2019</a>)</cite>. In particular, our residual modeling form and the training algorithm is the same as in <cite class="ltx_cite ltx_citemacro_citet">Wang &amp; Ou (<a href="#bib.bib37" title="" class="ltx_ref">2018b</a>)</cite>, where they used an LSTM as the generator and a CNN-LSTM as the energy function, and showed significant gains compared to LSTM baselines in speech recognition. Our work builds on these prior works and develops new lower and upper bounds for the log-probability under the joint model, which makes it possible to show that the residual EBM approach gets better perplexity. We also develop an importance weighting sampling scheme used at generation time, which is focused on conditional generation as opposed to rescoring in speech recognition <cite class="ltx_cite ltx_citemacro_citep">(Wang &amp; Ou, <a href="#bib.bib37" title="" class="ltx_ref">2018b</a>)</cite>. The residual EBM formalism makes it very natural to use BERT for language modeling, and we show that empirically this type of approach can outperform modern state-of-the-art language modeling baselines, both in terms of perplexity, and through human evaluation.</p>
</div>
<div id="S2.p4" class="ltx_para ltx_noindent">
<p id="S2.p4.1" class="ltx_p">Generative Adversarial Networks <cite class="ltx_cite ltx_citemacro_citep">(Goodfellow et al., <a href="#bib.bib9" title="" class="ltx_ref">2014</a>)</cite> also relate to EBMs, except that in EBMs the generator is implicit and negatives
samples are produced by the discriminator itself. In our work, the pretrained locally normalized language model can be seen as
a fixed generator, like in <cite class="ltx_cite ltx_citemacro_citet">Bakhtin et al. (<a href="#bib.bib2" title="" class="ltx_ref">2019</a>)</cite>.
<cite class="ltx_cite ltx_citemacro_citet">Azadi et al. (<a href="#bib.bib1" title="" class="ltx_ref">2018</a>)</cite> also share our same goal but their generator is not locally normalized and they
propose to improve the sampling from the generator by using the discriminator for rejection sampling. Similar to our work,
<cite class="ltx_cite ltx_citemacro_citet">Grover et al. (<a href="#bib.bib10" title="" class="ltx_ref">2019</a>)</cite> propose to use the discriminator to de-bias the pretrained generator using importance sampling.
We adapt this work to the application of text generation. In particular, we adopt the conditional noise contrastive estimation
(NCE) objective <cite class="ltx_cite ltx_citemacro_citep">(Ma &amp; Collins, <a href="#bib.bib20" title="" class="ltx_ref">2018</a>; Gutmann &amp; Hyvärinen, <a href="#bib.bib11" title="" class="ltx_ref">2010</a>)</cite>
to our residual model energy function and then sample from the joint model using importance sampling. We want to note that the same formulation has been proposed in <cite class="ltx_cite ltx_citemacro_citep">(Wang &amp; Ou, <a href="#bib.bib37" title="" class="ltx_ref">2018b</a>; Parshakova et al., <a href="#bib.bib25" title="" class="ltx_ref">2019</a>)</cite>. While <cite class="ltx_cite ltx_citemacro_citet">Ma &amp; Collins (<a href="#bib.bib20" title="" class="ltx_ref">2018</a>)</cite> used
conditional NCE to predict the next word in a sequence, we apply it to produce a whole sequence at once with
the pretrained auto-regressive language model as the noise distribution.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Residual Energy-Based Models</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.4" class="ltx_p">We study the problem of conditional generation of discrete sequences.
Given a prefix <math id="S3.p1.1.m1.3" class="ltx_Math" alttext="x_{1},\cdots,x_{p}" display="inline"><semantics id="S3.p1.1.m1.3a"><mrow id="S3.p1.1.m1.3.3.2" xref="S3.p1.1.m1.3.3.3.cmml"><msub id="S3.p1.1.m1.2.2.1.1" xref="S3.p1.1.m1.2.2.1.1.cmml"><mi id="S3.p1.1.m1.2.2.1.1.2" xref="S3.p1.1.m1.2.2.1.1.2.cmml">x</mi><mn id="S3.p1.1.m1.2.2.1.1.3" xref="S3.p1.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S3.p1.1.m1.3.3.2.3" xref="S3.p1.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">⋯</mi><mo id="S3.p1.1.m1.3.3.2.4" xref="S3.p1.1.m1.3.3.3.cmml">,</mo><msub id="S3.p1.1.m1.3.3.2.2" xref="S3.p1.1.m1.3.3.2.2.cmml"><mi id="S3.p1.1.m1.3.3.2.2.2" xref="S3.p1.1.m1.3.3.2.2.2.cmml">x</mi><mi id="S3.p1.1.m1.3.3.2.2.3" xref="S3.p1.1.m1.3.3.2.2.3.cmml">p</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.3b"><list id="S3.p1.1.m1.3.3.3.cmml" xref="S3.p1.1.m1.3.3.2"><apply id="S3.p1.1.m1.2.2.1.1.cmml" xref="S3.p1.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S3.p1.1.m1.2.2.1.1.1.cmml" xref="S3.p1.1.m1.2.2.1.1">subscript</csymbol><ci id="S3.p1.1.m1.2.2.1.1.2.cmml" xref="S3.p1.1.m1.2.2.1.1.2">𝑥</ci><cn type="integer" id="S3.p1.1.m1.2.2.1.1.3.cmml" xref="S3.p1.1.m1.2.2.1.1.3">1</cn></apply><ci id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">⋯</ci><apply id="S3.p1.1.m1.3.3.2.2.cmml" xref="S3.p1.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S3.p1.1.m1.3.3.2.2.1.cmml" xref="S3.p1.1.m1.3.3.2.2">subscript</csymbol><ci id="S3.p1.1.m1.3.3.2.2.2.cmml" xref="S3.p1.1.m1.3.3.2.2.2">𝑥</ci><ci id="S3.p1.1.m1.3.3.2.2.3.cmml" xref="S3.p1.1.m1.3.3.2.2.3">𝑝</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.3c">x_{1},\cdots,x_{p}</annotation></semantics></math> with <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="x_{j}\in V" display="inline"><semantics id="S3.p1.2.m2.1a"><mrow id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml"><msub id="S3.p1.2.m2.1.1.2" xref="S3.p1.2.m2.1.1.2.cmml"><mi id="S3.p1.2.m2.1.1.2.2" xref="S3.p1.2.m2.1.1.2.2.cmml">x</mi><mi id="S3.p1.2.m2.1.1.2.3" xref="S3.p1.2.m2.1.1.2.3.cmml">j</mi></msub><mo id="S3.p1.2.m2.1.1.1" xref="S3.p1.2.m2.1.1.1.cmml">∈</mo><mi id="S3.p1.2.m2.1.1.3" xref="S3.p1.2.m2.1.1.3.cmml">V</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><apply id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1"><in id="S3.p1.2.m2.1.1.1.cmml" xref="S3.p1.2.m2.1.1.1"></in><apply id="S3.p1.2.m2.1.1.2.cmml" xref="S3.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.2.1.cmml" xref="S3.p1.2.m2.1.1.2">subscript</csymbol><ci id="S3.p1.2.m2.1.1.2.2.cmml" xref="S3.p1.2.m2.1.1.2.2">𝑥</ci><ci id="S3.p1.2.m2.1.1.2.3.cmml" xref="S3.p1.2.m2.1.1.2.3">𝑗</ci></apply><ci id="S3.p1.2.m2.1.1.3.cmml" xref="S3.p1.2.m2.1.1.3">𝑉</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">x_{j}\in V</annotation></semantics></math> where <math id="S3.p1.3.m3.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S3.p1.3.m3.1a"><mi id="S3.p1.3.m3.1.1" xref="S3.p1.3.m3.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S3.p1.3.m3.1b"><ci id="S3.p1.3.m3.1.1.cmml" xref="S3.p1.3.m3.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m3.1c">V</annotation></semantics></math> is the vocabulary,
we want to model the probabilities of generating a sequence of total length
<math id="S3.p1.4.m4.1" class="ltx_Math" alttext="T&gt;p" display="inline"><semantics id="S3.p1.4.m4.1a"><mrow id="S3.p1.4.m4.1.1" xref="S3.p1.4.m4.1.1.cmml"><mi id="S3.p1.4.m4.1.1.2" xref="S3.p1.4.m4.1.1.2.cmml">T</mi><mo id="S3.p1.4.m4.1.1.1" xref="S3.p1.4.m4.1.1.1.cmml">&gt;</mo><mi id="S3.p1.4.m4.1.1.3" xref="S3.p1.4.m4.1.1.3.cmml">p</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.4.m4.1b"><apply id="S3.p1.4.m4.1.1.cmml" xref="S3.p1.4.m4.1.1"><gt id="S3.p1.4.m4.1.1.1.cmml" xref="S3.p1.4.m4.1.1.1"></gt><ci id="S3.p1.4.m4.1.1.2.cmml" xref="S3.p1.4.m4.1.1.2">𝑇</ci><ci id="S3.p1.4.m4.1.1.3.cmml" xref="S3.p1.4.m4.1.1.3">𝑝</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.4.m4.1c">T&gt;p</annotation></semantics></math><span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>We assume a fixed <math id="footnote1.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="footnote1.m1.1b"><mi id="footnote1.m1.1.1" xref="footnote1.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="footnote1.m1.1c"><ci id="footnote1.m1.1.1.cmml" xref="footnote1.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote1.m1.1d">T</annotation></semantics></math> for simplicity of analysis and implementation,
but our method generalizes to varying length generation with an end-of-sequence symbol.</span></span></span>.
The generative model is:</p>
<table id="S3.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E2.m1.14" class="ltx_Math" alttext="P_{\theta}(x_{p+1},\cdots,x_{T}|x_{1},\cdots,x_{p})=\frac{P_{LM}(x_{p+1},\cdots,x_{T}|x_{1},\cdots,x_{p})\exp(-E_{\theta}(x_{1},\cdots,x_{T}))}{Z_{\theta}(x_{1},\cdots,x_{p})}" display="block"><semantics id="S3.E2.m1.14a"><mrow id="S3.E2.m1.14.14" xref="S3.E2.m1.14.14.cmml"><mrow id="S3.E2.m1.14.14.2" xref="S3.E2.m1.14.14.2.cmml"><msub id="S3.E2.m1.14.14.2.4" xref="S3.E2.m1.14.14.2.4.cmml"><mi id="S3.E2.m1.14.14.2.4.2" xref="S3.E2.m1.14.14.2.4.2.cmml">P</mi><mi id="S3.E2.m1.14.14.2.4.3" xref="S3.E2.m1.14.14.2.4.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E2.m1.14.14.2.3" xref="S3.E2.m1.14.14.2.3.cmml">​</mo><mrow id="S3.E2.m1.14.14.2.2.2" xref="S3.E2.m1.14.14.2.2.3.cmml"><mo stretchy="false" id="S3.E2.m1.14.14.2.2.2.3" xref="S3.E2.m1.14.14.2.2.3.cmml">(</mo><msub id="S3.E2.m1.13.13.1.1.1.1" xref="S3.E2.m1.13.13.1.1.1.1.cmml"><mi id="S3.E2.m1.13.13.1.1.1.1.2" xref="S3.E2.m1.13.13.1.1.1.1.2.cmml">x</mi><mrow id="S3.E2.m1.13.13.1.1.1.1.3" xref="S3.E2.m1.13.13.1.1.1.1.3.cmml"><mi id="S3.E2.m1.13.13.1.1.1.1.3.2" xref="S3.E2.m1.13.13.1.1.1.1.3.2.cmml">p</mi><mo id="S3.E2.m1.13.13.1.1.1.1.3.1" xref="S3.E2.m1.13.13.1.1.1.1.3.1.cmml">+</mo><mn id="S3.E2.m1.13.13.1.1.1.1.3.3" xref="S3.E2.m1.13.13.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S3.E2.m1.14.14.2.2.2.4" xref="S3.E2.m1.14.14.2.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.E2.m1.12.12" xref="S3.E2.m1.12.12.cmml">⋯</mi><mo id="S3.E2.m1.14.14.2.2.2.5" xref="S3.E2.m1.14.14.2.2.3.cmml">,</mo><mrow id="S3.E2.m1.14.14.2.2.2.2" xref="S3.E2.m1.14.14.2.2.2.2.cmml"><msub id="S3.E2.m1.14.14.2.2.2.2.4" xref="S3.E2.m1.14.14.2.2.2.2.4.cmml"><mi id="S3.E2.m1.14.14.2.2.2.2.4.2" xref="S3.E2.m1.14.14.2.2.2.2.4.2.cmml">x</mi><mi id="S3.E2.m1.14.14.2.2.2.2.4.3" xref="S3.E2.m1.14.14.2.2.2.2.4.3.cmml">T</mi></msub><mo fence="false" id="S3.E2.m1.14.14.2.2.2.2.3" xref="S3.E2.m1.14.14.2.2.2.2.3.cmml">|</mo><mrow id="S3.E2.m1.14.14.2.2.2.2.2.2" xref="S3.E2.m1.14.14.2.2.2.2.2.3.cmml"><msub id="S3.E2.m1.14.14.2.2.2.2.1.1.1" xref="S3.E2.m1.14.14.2.2.2.2.1.1.1.cmml"><mi id="S3.E2.m1.14.14.2.2.2.2.1.1.1.2" xref="S3.E2.m1.14.14.2.2.2.2.1.1.1.2.cmml">x</mi><mn id="S3.E2.m1.14.14.2.2.2.2.1.1.1.3" xref="S3.E2.m1.14.14.2.2.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.E2.m1.14.14.2.2.2.2.2.2.3" xref="S3.E2.m1.14.14.2.2.2.2.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.E2.m1.11.11" xref="S3.E2.m1.11.11.cmml">⋯</mi><mo id="S3.E2.m1.14.14.2.2.2.2.2.2.4" xref="S3.E2.m1.14.14.2.2.2.2.2.3.cmml">,</mo><msub id="S3.E2.m1.14.14.2.2.2.2.2.2.2" xref="S3.E2.m1.14.14.2.2.2.2.2.2.2.cmml"><mi id="S3.E2.m1.14.14.2.2.2.2.2.2.2.2" xref="S3.E2.m1.14.14.2.2.2.2.2.2.2.2.cmml">x</mi><mi id="S3.E2.m1.14.14.2.2.2.2.2.2.2.3" xref="S3.E2.m1.14.14.2.2.2.2.2.2.2.3.cmml">p</mi></msub></mrow></mrow><mo stretchy="false" id="S3.E2.m1.14.14.2.2.2.6" xref="S3.E2.m1.14.14.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E2.m1.14.14.3" xref="S3.E2.m1.14.14.3.cmml">=</mo><mfrac id="S3.E2.m1.10.10" xref="S3.E2.m1.10.10.cmml"><mrow id="S3.E2.m1.7.7.7" xref="S3.E2.m1.7.7.7.cmml"><msub id="S3.E2.m1.7.7.7.9" xref="S3.E2.m1.7.7.7.9.cmml"><mi id="S3.E2.m1.7.7.7.9.2" xref="S3.E2.m1.7.7.7.9.2.cmml">P</mi><mrow id="S3.E2.m1.7.7.7.9.3" xref="S3.E2.m1.7.7.7.9.3.cmml"><mi id="S3.E2.m1.7.7.7.9.3.2" xref="S3.E2.m1.7.7.7.9.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.7.7.7.9.3.1" xref="S3.E2.m1.7.7.7.9.3.1.cmml">​</mo><mi id="S3.E2.m1.7.7.7.9.3.3" xref="S3.E2.m1.7.7.7.9.3.3.cmml">M</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.E2.m1.7.7.7.8" xref="S3.E2.m1.7.7.7.8.cmml">​</mo><mrow id="S3.E2.m1.6.6.6.6.2" xref="S3.E2.m1.6.6.6.6.3.cmml"><mo stretchy="false" id="S3.E2.m1.6.6.6.6.2.3" xref="S3.E2.m1.6.6.6.6.3.cmml">(</mo><msub id="S3.E2.m1.5.5.5.5.1.1" xref="S3.E2.m1.5.5.5.5.1.1.cmml"><mi id="S3.E2.m1.5.5.5.5.1.1.2" xref="S3.E2.m1.5.5.5.5.1.1.2.cmml">x</mi><mrow id="S3.E2.m1.5.5.5.5.1.1.3" xref="S3.E2.m1.5.5.5.5.1.1.3.cmml"><mi id="S3.E2.m1.5.5.5.5.1.1.3.2" xref="S3.E2.m1.5.5.5.5.1.1.3.2.cmml">p</mi><mo id="S3.E2.m1.5.5.5.5.1.1.3.1" xref="S3.E2.m1.5.5.5.5.1.1.3.1.cmml">+</mo><mn id="S3.E2.m1.5.5.5.5.1.1.3.3" xref="S3.E2.m1.5.5.5.5.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S3.E2.m1.6.6.6.6.2.4" xref="S3.E2.m1.6.6.6.6.3.cmml">,</mo><mi mathvariant="normal" id="S3.E2.m1.2.2.2.2" xref="S3.E2.m1.2.2.2.2.cmml">⋯</mi><mo id="S3.E2.m1.6.6.6.6.2.5" xref="S3.E2.m1.6.6.6.6.3.cmml">,</mo><mrow id="S3.E2.m1.6.6.6.6.2.2" xref="S3.E2.m1.6.6.6.6.2.2.cmml"><msub id="S3.E2.m1.6.6.6.6.2.2.4" xref="S3.E2.m1.6.6.6.6.2.2.4.cmml"><mi id="S3.E2.m1.6.6.6.6.2.2.4.2" xref="S3.E2.m1.6.6.6.6.2.2.4.2.cmml">x</mi><mi id="S3.E2.m1.6.6.6.6.2.2.4.3" xref="S3.E2.m1.6.6.6.6.2.2.4.3.cmml">T</mi></msub><mo fence="false" id="S3.E2.m1.6.6.6.6.2.2.3" xref="S3.E2.m1.6.6.6.6.2.2.3.cmml">|</mo><mrow id="S3.E2.m1.6.6.6.6.2.2.2.2" xref="S3.E2.m1.6.6.6.6.2.2.2.3.cmml"><msub id="S3.E2.m1.6.6.6.6.2.2.1.1.1" xref="S3.E2.m1.6.6.6.6.2.2.1.1.1.cmml"><mi id="S3.E2.m1.6.6.6.6.2.2.1.1.1.2" xref="S3.E2.m1.6.6.6.6.2.2.1.1.1.2.cmml">x</mi><mn id="S3.E2.m1.6.6.6.6.2.2.1.1.1.3" xref="S3.E2.m1.6.6.6.6.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.E2.m1.6.6.6.6.2.2.2.2.3" xref="S3.E2.m1.6.6.6.6.2.2.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.E2.m1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.cmml">⋯</mi><mo id="S3.E2.m1.6.6.6.6.2.2.2.2.4" xref="S3.E2.m1.6.6.6.6.2.2.2.3.cmml">,</mo><msub id="S3.E2.m1.6.6.6.6.2.2.2.2.2" xref="S3.E2.m1.6.6.6.6.2.2.2.2.2.cmml"><mi id="S3.E2.m1.6.6.6.6.2.2.2.2.2.2" xref="S3.E2.m1.6.6.6.6.2.2.2.2.2.2.cmml">x</mi><mi id="S3.E2.m1.6.6.6.6.2.2.2.2.2.3" xref="S3.E2.m1.6.6.6.6.2.2.2.2.2.3.cmml">p</mi></msub></mrow></mrow><mo stretchy="false" id="S3.E2.m1.6.6.6.6.2.6" xref="S3.E2.m1.6.6.6.6.3.cmml">)</mo></mrow><mo lspace="0.167em" rspace="0em" id="S3.E2.m1.7.7.7.8a" xref="S3.E2.m1.7.7.7.8.cmml">​</mo><mrow id="S3.E2.m1.7.7.7.7.1" xref="S3.E2.m1.7.7.7.7.2.cmml"><mi id="S3.E2.m1.4.4.4.4" xref="S3.E2.m1.4.4.4.4.cmml">exp</mi><mo id="S3.E2.m1.7.7.7.7.1a" xref="S3.E2.m1.7.7.7.7.2.cmml">⁡</mo><mrow id="S3.E2.m1.7.7.7.7.1.1" xref="S3.E2.m1.7.7.7.7.2.cmml"><mo stretchy="false" id="S3.E2.m1.7.7.7.7.1.1.2" xref="S3.E2.m1.7.7.7.7.2.cmml">(</mo><mrow id="S3.E2.m1.7.7.7.7.1.1.1" xref="S3.E2.m1.7.7.7.7.1.1.1.cmml"><mo id="S3.E2.m1.7.7.7.7.1.1.1a" xref="S3.E2.m1.7.7.7.7.1.1.1.cmml">−</mo><mrow id="S3.E2.m1.7.7.7.7.1.1.1.2" xref="S3.E2.m1.7.7.7.7.1.1.1.2.cmml"><msub id="S3.E2.m1.7.7.7.7.1.1.1.2.4" xref="S3.E2.m1.7.7.7.7.1.1.1.2.4.cmml"><mi id="S3.E2.m1.7.7.7.7.1.1.1.2.4.2" xref="S3.E2.m1.7.7.7.7.1.1.1.2.4.2.cmml">E</mi><mi id="S3.E2.m1.7.7.7.7.1.1.1.2.4.3" xref="S3.E2.m1.7.7.7.7.1.1.1.2.4.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E2.m1.7.7.7.7.1.1.1.2.3" xref="S3.E2.m1.7.7.7.7.1.1.1.2.3.cmml">​</mo><mrow id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.3" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.3.cmml">(</mo><msub id="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1" xref="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1.cmml"><mi id="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1.2" xref="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1.2.cmml">x</mi><mn id="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1.3" xref="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.4" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.E2.m1.3.3.3.3" xref="S3.E2.m1.3.3.3.3.cmml">⋯</mi><mo id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.5" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.3.cmml">,</mo><msub id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2.cmml"><mi id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2.2" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2.2.cmml">x</mi><mi id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2.3" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2.3.cmml">T</mi></msub><mo stretchy="false" id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.6" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.E2.m1.7.7.7.7.1.1.3" xref="S3.E2.m1.7.7.7.7.2.cmml">)</mo></mrow></mrow></mrow><mrow id="S3.E2.m1.10.10.10" xref="S3.E2.m1.10.10.10.cmml"><msub id="S3.E2.m1.10.10.10.5" xref="S3.E2.m1.10.10.10.5.cmml"><mi id="S3.E2.m1.10.10.10.5.2" xref="S3.E2.m1.10.10.10.5.2.cmml">Z</mi><mi id="S3.E2.m1.10.10.10.5.3" xref="S3.E2.m1.10.10.10.5.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E2.m1.10.10.10.4" xref="S3.E2.m1.10.10.10.4.cmml">​</mo><mrow id="S3.E2.m1.10.10.10.3.2" xref="S3.E2.m1.10.10.10.3.3.cmml"><mo stretchy="false" id="S3.E2.m1.10.10.10.3.2.3" xref="S3.E2.m1.10.10.10.3.3.cmml">(</mo><msub id="S3.E2.m1.9.9.9.2.1.1" xref="S3.E2.m1.9.9.9.2.1.1.cmml"><mi id="S3.E2.m1.9.9.9.2.1.1.2" xref="S3.E2.m1.9.9.9.2.1.1.2.cmml">x</mi><mn id="S3.E2.m1.9.9.9.2.1.1.3" xref="S3.E2.m1.9.9.9.2.1.1.3.cmml">1</mn></msub><mo id="S3.E2.m1.10.10.10.3.2.4" xref="S3.E2.m1.10.10.10.3.3.cmml">,</mo><mi mathvariant="normal" id="S3.E2.m1.8.8.8.1" xref="S3.E2.m1.8.8.8.1.cmml">⋯</mi><mo id="S3.E2.m1.10.10.10.3.2.5" xref="S3.E2.m1.10.10.10.3.3.cmml">,</mo><msub id="S3.E2.m1.10.10.10.3.2.2" xref="S3.E2.m1.10.10.10.3.2.2.cmml"><mi id="S3.E2.m1.10.10.10.3.2.2.2" xref="S3.E2.m1.10.10.10.3.2.2.2.cmml">x</mi><mi id="S3.E2.m1.10.10.10.3.2.2.3" xref="S3.E2.m1.10.10.10.3.2.2.3.cmml">p</mi></msub><mo stretchy="false" id="S3.E2.m1.10.10.10.3.2.6" xref="S3.E2.m1.10.10.10.3.3.cmml">)</mo></mrow></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m1.14b"><apply id="S3.E2.m1.14.14.cmml" xref="S3.E2.m1.14.14"><eq id="S3.E2.m1.14.14.3.cmml" xref="S3.E2.m1.14.14.3"></eq><apply id="S3.E2.m1.14.14.2.cmml" xref="S3.E2.m1.14.14.2"><times id="S3.E2.m1.14.14.2.3.cmml" xref="S3.E2.m1.14.14.2.3"></times><apply id="S3.E2.m1.14.14.2.4.cmml" xref="S3.E2.m1.14.14.2.4"><csymbol cd="ambiguous" id="S3.E2.m1.14.14.2.4.1.cmml" xref="S3.E2.m1.14.14.2.4">subscript</csymbol><ci id="S3.E2.m1.14.14.2.4.2.cmml" xref="S3.E2.m1.14.14.2.4.2">𝑃</ci><ci id="S3.E2.m1.14.14.2.4.3.cmml" xref="S3.E2.m1.14.14.2.4.3">𝜃</ci></apply><vector id="S3.E2.m1.14.14.2.2.3.cmml" xref="S3.E2.m1.14.14.2.2.2"><apply id="S3.E2.m1.13.13.1.1.1.1.cmml" xref="S3.E2.m1.13.13.1.1.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.13.13.1.1.1.1.1.cmml" xref="S3.E2.m1.13.13.1.1.1.1">subscript</csymbol><ci id="S3.E2.m1.13.13.1.1.1.1.2.cmml" xref="S3.E2.m1.13.13.1.1.1.1.2">𝑥</ci><apply id="S3.E2.m1.13.13.1.1.1.1.3.cmml" xref="S3.E2.m1.13.13.1.1.1.1.3"><plus id="S3.E2.m1.13.13.1.1.1.1.3.1.cmml" xref="S3.E2.m1.13.13.1.1.1.1.3.1"></plus><ci id="S3.E2.m1.13.13.1.1.1.1.3.2.cmml" xref="S3.E2.m1.13.13.1.1.1.1.3.2">𝑝</ci><cn type="integer" id="S3.E2.m1.13.13.1.1.1.1.3.3.cmml" xref="S3.E2.m1.13.13.1.1.1.1.3.3">1</cn></apply></apply><ci id="S3.E2.m1.12.12.cmml" xref="S3.E2.m1.12.12">⋯</ci><apply id="S3.E2.m1.14.14.2.2.2.2.cmml" xref="S3.E2.m1.14.14.2.2.2.2"><csymbol cd="latexml" id="S3.E2.m1.14.14.2.2.2.2.3.cmml" xref="S3.E2.m1.14.14.2.2.2.2.3">conditional</csymbol><apply id="S3.E2.m1.14.14.2.2.2.2.4.cmml" xref="S3.E2.m1.14.14.2.2.2.2.4"><csymbol cd="ambiguous" id="S3.E2.m1.14.14.2.2.2.2.4.1.cmml" xref="S3.E2.m1.14.14.2.2.2.2.4">subscript</csymbol><ci id="S3.E2.m1.14.14.2.2.2.2.4.2.cmml" xref="S3.E2.m1.14.14.2.2.2.2.4.2">𝑥</ci><ci id="S3.E2.m1.14.14.2.2.2.2.4.3.cmml" xref="S3.E2.m1.14.14.2.2.2.2.4.3">𝑇</ci></apply><list id="S3.E2.m1.14.14.2.2.2.2.2.3.cmml" xref="S3.E2.m1.14.14.2.2.2.2.2.2"><apply id="S3.E2.m1.14.14.2.2.2.2.1.1.1.cmml" xref="S3.E2.m1.14.14.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.14.14.2.2.2.2.1.1.1.1.cmml" xref="S3.E2.m1.14.14.2.2.2.2.1.1.1">subscript</csymbol><ci id="S3.E2.m1.14.14.2.2.2.2.1.1.1.2.cmml" xref="S3.E2.m1.14.14.2.2.2.2.1.1.1.2">𝑥</ci><cn type="integer" id="S3.E2.m1.14.14.2.2.2.2.1.1.1.3.cmml" xref="S3.E2.m1.14.14.2.2.2.2.1.1.1.3">1</cn></apply><ci id="S3.E2.m1.11.11.cmml" xref="S3.E2.m1.11.11">⋯</ci><apply id="S3.E2.m1.14.14.2.2.2.2.2.2.2.cmml" xref="S3.E2.m1.14.14.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.E2.m1.14.14.2.2.2.2.2.2.2.1.cmml" xref="S3.E2.m1.14.14.2.2.2.2.2.2.2">subscript</csymbol><ci id="S3.E2.m1.14.14.2.2.2.2.2.2.2.2.cmml" xref="S3.E2.m1.14.14.2.2.2.2.2.2.2.2">𝑥</ci><ci id="S3.E2.m1.14.14.2.2.2.2.2.2.2.3.cmml" xref="S3.E2.m1.14.14.2.2.2.2.2.2.2.3">𝑝</ci></apply></list></apply></vector></apply><apply id="S3.E2.m1.10.10.cmml" xref="S3.E2.m1.10.10"><divide id="S3.E2.m1.10.10.11.cmml" xref="S3.E2.m1.10.10"></divide><apply id="S3.E2.m1.7.7.7.cmml" xref="S3.E2.m1.7.7.7"><times id="S3.E2.m1.7.7.7.8.cmml" xref="S3.E2.m1.7.7.7.8"></times><apply id="S3.E2.m1.7.7.7.9.cmml" xref="S3.E2.m1.7.7.7.9"><csymbol cd="ambiguous" id="S3.E2.m1.7.7.7.9.1.cmml" xref="S3.E2.m1.7.7.7.9">subscript</csymbol><ci id="S3.E2.m1.7.7.7.9.2.cmml" xref="S3.E2.m1.7.7.7.9.2">𝑃</ci><apply id="S3.E2.m1.7.7.7.9.3.cmml" xref="S3.E2.m1.7.7.7.9.3"><times id="S3.E2.m1.7.7.7.9.3.1.cmml" xref="S3.E2.m1.7.7.7.9.3.1"></times><ci id="S3.E2.m1.7.7.7.9.3.2.cmml" xref="S3.E2.m1.7.7.7.9.3.2">𝐿</ci><ci id="S3.E2.m1.7.7.7.9.3.3.cmml" xref="S3.E2.m1.7.7.7.9.3.3">𝑀</ci></apply></apply><vector id="S3.E2.m1.6.6.6.6.3.cmml" xref="S3.E2.m1.6.6.6.6.2"><apply id="S3.E2.m1.5.5.5.5.1.1.cmml" xref="S3.E2.m1.5.5.5.5.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.5.5.5.5.1.1.1.cmml" xref="S3.E2.m1.5.5.5.5.1.1">subscript</csymbol><ci id="S3.E2.m1.5.5.5.5.1.1.2.cmml" xref="S3.E2.m1.5.5.5.5.1.1.2">𝑥</ci><apply id="S3.E2.m1.5.5.5.5.1.1.3.cmml" xref="S3.E2.m1.5.5.5.5.1.1.3"><plus id="S3.E2.m1.5.5.5.5.1.1.3.1.cmml" xref="S3.E2.m1.5.5.5.5.1.1.3.1"></plus><ci id="S3.E2.m1.5.5.5.5.1.1.3.2.cmml" xref="S3.E2.m1.5.5.5.5.1.1.3.2">𝑝</ci><cn type="integer" id="S3.E2.m1.5.5.5.5.1.1.3.3.cmml" xref="S3.E2.m1.5.5.5.5.1.1.3.3">1</cn></apply></apply><ci id="S3.E2.m1.2.2.2.2.cmml" xref="S3.E2.m1.2.2.2.2">⋯</ci><apply id="S3.E2.m1.6.6.6.6.2.2.cmml" xref="S3.E2.m1.6.6.6.6.2.2"><csymbol cd="latexml" id="S3.E2.m1.6.6.6.6.2.2.3.cmml" xref="S3.E2.m1.6.6.6.6.2.2.3">conditional</csymbol><apply id="S3.E2.m1.6.6.6.6.2.2.4.cmml" xref="S3.E2.m1.6.6.6.6.2.2.4"><csymbol cd="ambiguous" id="S3.E2.m1.6.6.6.6.2.2.4.1.cmml" xref="S3.E2.m1.6.6.6.6.2.2.4">subscript</csymbol><ci id="S3.E2.m1.6.6.6.6.2.2.4.2.cmml" xref="S3.E2.m1.6.6.6.6.2.2.4.2">𝑥</ci><ci id="S3.E2.m1.6.6.6.6.2.2.4.3.cmml" xref="S3.E2.m1.6.6.6.6.2.2.4.3">𝑇</ci></apply><list id="S3.E2.m1.6.6.6.6.2.2.2.3.cmml" xref="S3.E2.m1.6.6.6.6.2.2.2.2"><apply id="S3.E2.m1.6.6.6.6.2.2.1.1.1.cmml" xref="S3.E2.m1.6.6.6.6.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.6.6.6.6.2.2.1.1.1.1.cmml" xref="S3.E2.m1.6.6.6.6.2.2.1.1.1">subscript</csymbol><ci id="S3.E2.m1.6.6.6.6.2.2.1.1.1.2.cmml" xref="S3.E2.m1.6.6.6.6.2.2.1.1.1.2">𝑥</ci><cn type="integer" id="S3.E2.m1.6.6.6.6.2.2.1.1.1.3.cmml" xref="S3.E2.m1.6.6.6.6.2.2.1.1.1.3">1</cn></apply><ci id="S3.E2.m1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1">⋯</ci><apply id="S3.E2.m1.6.6.6.6.2.2.2.2.2.cmml" xref="S3.E2.m1.6.6.6.6.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.E2.m1.6.6.6.6.2.2.2.2.2.1.cmml" xref="S3.E2.m1.6.6.6.6.2.2.2.2.2">subscript</csymbol><ci id="S3.E2.m1.6.6.6.6.2.2.2.2.2.2.cmml" xref="S3.E2.m1.6.6.6.6.2.2.2.2.2.2">𝑥</ci><ci id="S3.E2.m1.6.6.6.6.2.2.2.2.2.3.cmml" xref="S3.E2.m1.6.6.6.6.2.2.2.2.2.3">𝑝</ci></apply></list></apply></vector><apply id="S3.E2.m1.7.7.7.7.2.cmml" xref="S3.E2.m1.7.7.7.7.1"><exp id="S3.E2.m1.4.4.4.4.cmml" xref="S3.E2.m1.4.4.4.4"></exp><apply id="S3.E2.m1.7.7.7.7.1.1.1.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1"><minus id="S3.E2.m1.7.7.7.7.1.1.1.3.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1"></minus><apply id="S3.E2.m1.7.7.7.7.1.1.1.2.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.2"><times id="S3.E2.m1.7.7.7.7.1.1.1.2.3.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.2.3"></times><apply id="S3.E2.m1.7.7.7.7.1.1.1.2.4.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.2.4"><csymbol cd="ambiguous" id="S3.E2.m1.7.7.7.7.1.1.1.2.4.1.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.2.4">subscript</csymbol><ci id="S3.E2.m1.7.7.7.7.1.1.1.2.4.2.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.2.4.2">𝐸</ci><ci id="S3.E2.m1.7.7.7.7.1.1.1.2.4.3.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.2.4.3">𝜃</ci></apply><vector id="S3.E2.m1.7.7.7.7.1.1.1.2.2.3.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.2"><apply id="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1.2.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1.2">𝑥</ci><cn type="integer" id="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1.3.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.1.1.1.1.3">1</cn></apply><ci id="S3.E2.m1.3.3.3.3.cmml" xref="S3.E2.m1.3.3.3.3">⋯</ci><apply id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2.1.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2.2.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2.2">𝑥</ci><ci id="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2.3.cmml" xref="S3.E2.m1.7.7.7.7.1.1.1.2.2.2.2.3">𝑇</ci></apply></vector></apply></apply></apply></apply><apply id="S3.E2.m1.10.10.10.cmml" xref="S3.E2.m1.10.10.10"><times id="S3.E2.m1.10.10.10.4.cmml" xref="S3.E2.m1.10.10.10.4"></times><apply id="S3.E2.m1.10.10.10.5.cmml" xref="S3.E2.m1.10.10.10.5"><csymbol cd="ambiguous" id="S3.E2.m1.10.10.10.5.1.cmml" xref="S3.E2.m1.10.10.10.5">subscript</csymbol><ci id="S3.E2.m1.10.10.10.5.2.cmml" xref="S3.E2.m1.10.10.10.5.2">𝑍</ci><ci id="S3.E2.m1.10.10.10.5.3.cmml" xref="S3.E2.m1.10.10.10.5.3">𝜃</ci></apply><vector id="S3.E2.m1.10.10.10.3.3.cmml" xref="S3.E2.m1.10.10.10.3.2"><apply id="S3.E2.m1.9.9.9.2.1.1.cmml" xref="S3.E2.m1.9.9.9.2.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.9.9.9.2.1.1.1.cmml" xref="S3.E2.m1.9.9.9.2.1.1">subscript</csymbol><ci id="S3.E2.m1.9.9.9.2.1.1.2.cmml" xref="S3.E2.m1.9.9.9.2.1.1.2">𝑥</ci><cn type="integer" id="S3.E2.m1.9.9.9.2.1.1.3.cmml" xref="S3.E2.m1.9.9.9.2.1.1.3">1</cn></apply><ci id="S3.E2.m1.8.8.8.1.cmml" xref="S3.E2.m1.8.8.8.1">⋯</ci><apply id="S3.E2.m1.10.10.10.3.2.2.cmml" xref="S3.E2.m1.10.10.10.3.2.2"><csymbol cd="ambiguous" id="S3.E2.m1.10.10.10.3.2.2.1.cmml" xref="S3.E2.m1.10.10.10.3.2.2">subscript</csymbol><ci id="S3.E2.m1.10.10.10.3.2.2.2.cmml" xref="S3.E2.m1.10.10.10.3.2.2.2">𝑥</ci><ci id="S3.E2.m1.10.10.10.3.2.2.3.cmml" xref="S3.E2.m1.10.10.10.3.2.2.3">𝑝</ci></apply></vector></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.14c">P_{\theta}(x_{p+1},\cdots,x_{T}|x_{1},\cdots,x_{p})=\frac{P_{LM}(x_{p+1},\cdots,x_{T}|x_{1},\cdots,x_{p})\exp(-E_{\theta}(x_{1},\cdots,x_{T}))}{Z_{\theta}(x_{1},\cdots,x_{p})}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S3.p1.9" class="ltx_p">where <math id="S3.p1.5.m1.3" class="ltx_Math" alttext="Z_{\theta}(x_{1},\cdots,x_{p})" display="inline"><semantics id="S3.p1.5.m1.3a"><mrow id="S3.p1.5.m1.3.3" xref="S3.p1.5.m1.3.3.cmml"><msub id="S3.p1.5.m1.3.3.4" xref="S3.p1.5.m1.3.3.4.cmml"><mi id="S3.p1.5.m1.3.3.4.2" xref="S3.p1.5.m1.3.3.4.2.cmml">Z</mi><mi id="S3.p1.5.m1.3.3.4.3" xref="S3.p1.5.m1.3.3.4.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.p1.5.m1.3.3.3" xref="S3.p1.5.m1.3.3.3.cmml">​</mo><mrow id="S3.p1.5.m1.3.3.2.2" xref="S3.p1.5.m1.3.3.2.3.cmml"><mo stretchy="false" id="S3.p1.5.m1.3.3.2.2.3" xref="S3.p1.5.m1.3.3.2.3.cmml">(</mo><msub id="S3.p1.5.m1.2.2.1.1.1" xref="S3.p1.5.m1.2.2.1.1.1.cmml"><mi id="S3.p1.5.m1.2.2.1.1.1.2" xref="S3.p1.5.m1.2.2.1.1.1.2.cmml">x</mi><mn id="S3.p1.5.m1.2.2.1.1.1.3" xref="S3.p1.5.m1.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.p1.5.m1.3.3.2.2.4" xref="S3.p1.5.m1.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.p1.5.m1.1.1" xref="S3.p1.5.m1.1.1.cmml">⋯</mi><mo id="S3.p1.5.m1.3.3.2.2.5" xref="S3.p1.5.m1.3.3.2.3.cmml">,</mo><msub id="S3.p1.5.m1.3.3.2.2.2" xref="S3.p1.5.m1.3.3.2.2.2.cmml"><mi id="S3.p1.5.m1.3.3.2.2.2.2" xref="S3.p1.5.m1.3.3.2.2.2.2.cmml">x</mi><mi id="S3.p1.5.m1.3.3.2.2.2.3" xref="S3.p1.5.m1.3.3.2.2.2.3.cmml">p</mi></msub><mo stretchy="false" id="S3.p1.5.m1.3.3.2.2.6" xref="S3.p1.5.m1.3.3.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.5.m1.3b"><apply id="S3.p1.5.m1.3.3.cmml" xref="S3.p1.5.m1.3.3"><times id="S3.p1.5.m1.3.3.3.cmml" xref="S3.p1.5.m1.3.3.3"></times><apply id="S3.p1.5.m1.3.3.4.cmml" xref="S3.p1.5.m1.3.3.4"><csymbol cd="ambiguous" id="S3.p1.5.m1.3.3.4.1.cmml" xref="S3.p1.5.m1.3.3.4">subscript</csymbol><ci id="S3.p1.5.m1.3.3.4.2.cmml" xref="S3.p1.5.m1.3.3.4.2">𝑍</ci><ci id="S3.p1.5.m1.3.3.4.3.cmml" xref="S3.p1.5.m1.3.3.4.3">𝜃</ci></apply><vector id="S3.p1.5.m1.3.3.2.3.cmml" xref="S3.p1.5.m1.3.3.2.2"><apply id="S3.p1.5.m1.2.2.1.1.1.cmml" xref="S3.p1.5.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.p1.5.m1.2.2.1.1.1.1.cmml" xref="S3.p1.5.m1.2.2.1.1.1">subscript</csymbol><ci id="S3.p1.5.m1.2.2.1.1.1.2.cmml" xref="S3.p1.5.m1.2.2.1.1.1.2">𝑥</ci><cn type="integer" id="S3.p1.5.m1.2.2.1.1.1.3.cmml" xref="S3.p1.5.m1.2.2.1.1.1.3">1</cn></apply><ci id="S3.p1.5.m1.1.1.cmml" xref="S3.p1.5.m1.1.1">⋯</ci><apply id="S3.p1.5.m1.3.3.2.2.2.cmml" xref="S3.p1.5.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.p1.5.m1.3.3.2.2.2.1.cmml" xref="S3.p1.5.m1.3.3.2.2.2">subscript</csymbol><ci id="S3.p1.5.m1.3.3.2.2.2.2.cmml" xref="S3.p1.5.m1.3.3.2.2.2.2">𝑥</ci><ci id="S3.p1.5.m1.3.3.2.2.2.3.cmml" xref="S3.p1.5.m1.3.3.2.2.2.3">𝑝</ci></apply></vector></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.5.m1.3c">Z_{\theta}(x_{1},\cdots,x_{p})</annotation></semantics></math> is a normalizing factor known as partition function. Computing the partition function is intractable
in our case since it involves a sum over <math id="S3.p1.6.m2.1" class="ltx_Math" alttext="|V|^{T-p}" display="inline"><semantics id="S3.p1.6.m2.1a"><msup id="S3.p1.6.m2.1.2" xref="S3.p1.6.m2.1.2.cmml"><mrow id="S3.p1.6.m2.1.2.2.2" xref="S3.p1.6.m2.1.2.2.1.cmml"><mo stretchy="false" id="S3.p1.6.m2.1.2.2.2.1" xref="S3.p1.6.m2.1.2.2.1.1.cmml">|</mo><mi id="S3.p1.6.m2.1.1" xref="S3.p1.6.m2.1.1.cmml">V</mi><mo stretchy="false" id="S3.p1.6.m2.1.2.2.2.2" xref="S3.p1.6.m2.1.2.2.1.1.cmml">|</mo></mrow><mrow id="S3.p1.6.m2.1.2.3" xref="S3.p1.6.m2.1.2.3.cmml"><mi id="S3.p1.6.m2.1.2.3.2" xref="S3.p1.6.m2.1.2.3.2.cmml">T</mi><mo id="S3.p1.6.m2.1.2.3.1" xref="S3.p1.6.m2.1.2.3.1.cmml">−</mo><mi id="S3.p1.6.m2.1.2.3.3" xref="S3.p1.6.m2.1.2.3.3.cmml">p</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.p1.6.m2.1b"><apply id="S3.p1.6.m2.1.2.cmml" xref="S3.p1.6.m2.1.2"><csymbol cd="ambiguous" id="S3.p1.6.m2.1.2.1.cmml" xref="S3.p1.6.m2.1.2">superscript</csymbol><apply id="S3.p1.6.m2.1.2.2.1.cmml" xref="S3.p1.6.m2.1.2.2.2"><abs id="S3.p1.6.m2.1.2.2.1.1.cmml" xref="S3.p1.6.m2.1.2.2.2.1"></abs><ci id="S3.p1.6.m2.1.1.cmml" xref="S3.p1.6.m2.1.1">𝑉</ci></apply><apply id="S3.p1.6.m2.1.2.3.cmml" xref="S3.p1.6.m2.1.2.3"><minus id="S3.p1.6.m2.1.2.3.1.cmml" xref="S3.p1.6.m2.1.2.3.1"></minus><ci id="S3.p1.6.m2.1.2.3.2.cmml" xref="S3.p1.6.m2.1.2.3.2">𝑇</ci><ci id="S3.p1.6.m2.1.2.3.3.cmml" xref="S3.p1.6.m2.1.2.3.3">𝑝</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.6.m2.1c">|V|^{T-p}</annotation></semantics></math> terms which grow exponentially with the sequence length: in our
experiments the size of the vocabulary is 50,096 and the length of the generation is 40 tokens.
We call <math id="S3.p1.7.m3.1" class="ltx_Math" alttext="P_{\theta}" display="inline"><semantics id="S3.p1.7.m3.1a"><msub id="S3.p1.7.m3.1.1" xref="S3.p1.7.m3.1.1.cmml"><mi id="S3.p1.7.m3.1.1.2" xref="S3.p1.7.m3.1.1.2.cmml">P</mi><mi id="S3.p1.7.m3.1.1.3" xref="S3.p1.7.m3.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p1.7.m3.1b"><apply id="S3.p1.7.m3.1.1.cmml" xref="S3.p1.7.m3.1.1"><csymbol cd="ambiguous" id="S3.p1.7.m3.1.1.1.cmml" xref="S3.p1.7.m3.1.1">subscript</csymbol><ci id="S3.p1.7.m3.1.1.2.cmml" xref="S3.p1.7.m3.1.1.2">𝑃</ci><ci id="S3.p1.7.m3.1.1.3.cmml" xref="S3.p1.7.m3.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.7.m3.1c">P_{\theta}</annotation></semantics></math> the joint model, and <math id="S3.p1.8.m4.1" class="ltx_Math" alttext="E_{\theta}" display="inline"><semantics id="S3.p1.8.m4.1a"><msub id="S3.p1.8.m4.1.1" xref="S3.p1.8.m4.1.1.cmml"><mi id="S3.p1.8.m4.1.1.2" xref="S3.p1.8.m4.1.1.2.cmml">E</mi><mi id="S3.p1.8.m4.1.1.3" xref="S3.p1.8.m4.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p1.8.m4.1b"><apply id="S3.p1.8.m4.1.1.cmml" xref="S3.p1.8.m4.1.1"><csymbol cd="ambiguous" id="S3.p1.8.m4.1.1.1.cmml" xref="S3.p1.8.m4.1.1">subscript</csymbol><ci id="S3.p1.8.m4.1.1.2.cmml" xref="S3.p1.8.m4.1.1.2">𝐸</ci><ci id="S3.p1.8.m4.1.1.3.cmml" xref="S3.p1.8.m4.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.8.m4.1c">E_{\theta}</annotation></semantics></math>
the residual energy function since <math id="S3.p1.9.m5.1" class="ltx_Math" alttext="P_{LM}" display="inline"><semantics id="S3.p1.9.m5.1a"><msub id="S3.p1.9.m5.1.1" xref="S3.p1.9.m5.1.1.cmml"><mi id="S3.p1.9.m5.1.1.2" xref="S3.p1.9.m5.1.1.2.cmml">P</mi><mrow id="S3.p1.9.m5.1.1.3" xref="S3.p1.9.m5.1.1.3.cmml"><mi id="S3.p1.9.m5.1.1.3.2" xref="S3.p1.9.m5.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.p1.9.m5.1.1.3.1" xref="S3.p1.9.m5.1.1.3.1.cmml">​</mo><mi id="S3.p1.9.m5.1.1.3.3" xref="S3.p1.9.m5.1.1.3.3.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.p1.9.m5.1b"><apply id="S3.p1.9.m5.1.1.cmml" xref="S3.p1.9.m5.1.1"><csymbol cd="ambiguous" id="S3.p1.9.m5.1.1.1.cmml" xref="S3.p1.9.m5.1.1">subscript</csymbol><ci id="S3.p1.9.m5.1.1.2.cmml" xref="S3.p1.9.m5.1.1.2">𝑃</ci><apply id="S3.p1.9.m5.1.1.3.cmml" xref="S3.p1.9.m5.1.1.3"><times id="S3.p1.9.m5.1.1.3.1.cmml" xref="S3.p1.9.m5.1.1.3.1"></times><ci id="S3.p1.9.m5.1.1.3.2.cmml" xref="S3.p1.9.m5.1.1.3.2">𝐿</ci><ci id="S3.p1.9.m5.1.1.3.3.cmml" xref="S3.p1.9.m5.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.9.m5.1c">P_{LM}</annotation></semantics></math> is fixed throughout training. The goal of training is to learn the parameters
of the energy function such that the joint model distribution gets close to the data distribution.
For the sake of reducing clutter in the notation, we will drop the conditioning variables in the following discussion.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Training</h3>

<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.1" class="ltx_p">When the partition function is intractable, Maximum Likelihood Estimation (MLE) requires samples from the model distribution,
which is usually approximated with Monte Carlo sampling or mean field inference <cite class="ltx_cite ltx_citemacro_citep">(Hinton, <a href="#bib.bib13" title="" class="ltx_ref">2012</a>; LeCun et al., <a href="#bib.bib18" title="" class="ltx_ref">2006</a>)</cite> for globally normalized models.
Unfortunately, both approaches are too computationally expensive for text applications when using large bidirectional transformer models.
For instance, if we were to employ Gibbs sampling exactly, we would need to perform at every position as many forward passes
as words in the dictionary to compute each conditional distribution.
On large datasets where training locally normalized models on multiple machines already takes days,
having such additional overhead means that the model would learn from much less data for the same amount of time,
and this is seldom a beneficial strategy for learning models that generalize well.
Therefore, we do not use either MCMC nor mean field methods, as the latter would introduce additional variational parameters
or an inference network which anyway yields an approximation to MLE learning.
</p>
</div>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS1.p2.4" class="ltx_p">Instead, we train our residual energy function using Noise Contrastive Estimation (NCE) <cite class="ltx_cite ltx_citemacro_citep">(Gutmann &amp; Hyvärinen, <a href="#bib.bib11" title="" class="ltx_ref">2010</a>)</cite>, and
more specifically its conditional version <cite class="ltx_cite ltx_citemacro_citep">(Ma &amp; Collins, <a href="#bib.bib20" title="" class="ltx_ref">2018</a>)</cite>. NCE requires two distributions: The model distribution and a noise
distribution. In our case, the model distribution is the joint model of Eq. <a href="#S3.E2" title="In 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="P_{\theta}" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><msub id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml"><mi id="S3.SS1.p2.1.m1.1.1.2" xref="S3.SS1.p2.1.m1.1.1.2.cmml">P</mi><mi id="S3.SS1.p2.1.m1.1.1.3" xref="S3.SS1.p2.1.m1.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><apply id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.1.m1.1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p2.1.m1.1.1.2.cmml" xref="S3.SS1.p2.1.m1.1.1.2">𝑃</ci><ci id="S3.SS1.p2.1.m1.1.1.3.cmml" xref="S3.SS1.p2.1.m1.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">P_{\theta}</annotation></semantics></math>, while the noise distribution
is the pretrained language model, <math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="P_{LM}" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><msub id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml"><mi id="S3.SS1.p2.2.m2.1.1.2" xref="S3.SS1.p2.2.m2.1.1.2.cmml">P</mi><mrow id="S3.SS1.p2.2.m2.1.1.3" xref="S3.SS1.p2.2.m2.1.1.3.cmml"><mi id="S3.SS1.p2.2.m2.1.1.3.2" xref="S3.SS1.p2.2.m2.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p2.2.m2.1.1.3.1" xref="S3.SS1.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="S3.SS1.p2.2.m2.1.1.3.3" xref="S3.SS1.p2.2.m2.1.1.3.3.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><apply id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.2.m2.1.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p2.2.m2.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.2">𝑃</ci><apply id="S3.SS1.p2.2.m2.1.1.3.cmml" xref="S3.SS1.p2.2.m2.1.1.3"><times id="S3.SS1.p2.2.m2.1.1.3.1.cmml" xref="S3.SS1.p2.2.m2.1.1.3.1"></times><ci id="S3.SS1.p2.2.m2.1.1.3.2.cmml" xref="S3.SS1.p2.2.m2.1.1.3.2">𝐿</ci><ci id="S3.SS1.p2.2.m2.1.1.3.3.cmml" xref="S3.SS1.p2.2.m2.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">P_{LM}</annotation></semantics></math>. NCE then trains a binary classifier on the difference of log-probability scores
of these two models. Since our joint model is the product of the energy function (whose parameters we want to learn) with
<math id="S3.SS1.p2.3.m3.1" class="ltx_Math" alttext="P_{LM}" display="inline"><semantics id="S3.SS1.p2.3.m3.1a"><msub id="S3.SS1.p2.3.m3.1.1" xref="S3.SS1.p2.3.m3.1.1.cmml"><mi id="S3.SS1.p2.3.m3.1.1.2" xref="S3.SS1.p2.3.m3.1.1.2.cmml">P</mi><mrow id="S3.SS1.p2.3.m3.1.1.3" xref="S3.SS1.p2.3.m3.1.1.3.cmml"><mi id="S3.SS1.p2.3.m3.1.1.3.2" xref="S3.SS1.p2.3.m3.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p2.3.m3.1.1.3.1" xref="S3.SS1.p2.3.m3.1.1.3.1.cmml">​</mo><mi id="S3.SS1.p2.3.m3.1.1.3.3" xref="S3.SS1.p2.3.m3.1.1.3.3.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.3.m3.1b"><apply id="S3.SS1.p2.3.m3.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.3.m3.1.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1">subscript</csymbol><ci id="S3.SS1.p2.3.m3.1.1.2.cmml" xref="S3.SS1.p2.3.m3.1.1.2">𝑃</ci><apply id="S3.SS1.p2.3.m3.1.1.3.cmml" xref="S3.SS1.p2.3.m3.1.1.3"><times id="S3.SS1.p2.3.m3.1.1.3.1.cmml" xref="S3.SS1.p2.3.m3.1.1.3.1"></times><ci id="S3.SS1.p2.3.m3.1.1.3.2.cmml" xref="S3.SS1.p2.3.m3.1.1.3.2">𝐿</ci><ci id="S3.SS1.p2.3.m3.1.1.3.3.cmml" xref="S3.SS1.p2.3.m3.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.3.m3.1c">P_{LM}</annotation></semantics></math>, the difference reduces to: <math id="S3.SS1.p2.4.m4.1" class="ltx_Math" alttext="\log P_{\theta}-\log P_{LM}=-E_{\theta}" display="inline"><semantics id="S3.SS1.p2.4.m4.1a"><mrow id="S3.SS1.p2.4.m4.1.1" xref="S3.SS1.p2.4.m4.1.1.cmml"><mrow id="S3.SS1.p2.4.m4.1.1.2" xref="S3.SS1.p2.4.m4.1.1.2.cmml"><mrow id="S3.SS1.p2.4.m4.1.1.2.2" xref="S3.SS1.p2.4.m4.1.1.2.2.cmml"><mi id="S3.SS1.p2.4.m4.1.1.2.2.1" xref="S3.SS1.p2.4.m4.1.1.2.2.1.cmml">log</mi><mo lspace="0.167em" id="S3.SS1.p2.4.m4.1.1.2.2a" xref="S3.SS1.p2.4.m4.1.1.2.2.cmml">⁡</mo><msub id="S3.SS1.p2.4.m4.1.1.2.2.2" xref="S3.SS1.p2.4.m4.1.1.2.2.2.cmml"><mi id="S3.SS1.p2.4.m4.1.1.2.2.2.2" xref="S3.SS1.p2.4.m4.1.1.2.2.2.2.cmml">P</mi><mi id="S3.SS1.p2.4.m4.1.1.2.2.2.3" xref="S3.SS1.p2.4.m4.1.1.2.2.2.3.cmml">θ</mi></msub></mrow><mo id="S3.SS1.p2.4.m4.1.1.2.1" xref="S3.SS1.p2.4.m4.1.1.2.1.cmml">−</mo><mrow id="S3.SS1.p2.4.m4.1.1.2.3" xref="S3.SS1.p2.4.m4.1.1.2.3.cmml"><mi id="S3.SS1.p2.4.m4.1.1.2.3.1" xref="S3.SS1.p2.4.m4.1.1.2.3.1.cmml">log</mi><mo lspace="0.167em" id="S3.SS1.p2.4.m4.1.1.2.3a" xref="S3.SS1.p2.4.m4.1.1.2.3.cmml">⁡</mo><msub id="S3.SS1.p2.4.m4.1.1.2.3.2" xref="S3.SS1.p2.4.m4.1.1.2.3.2.cmml"><mi id="S3.SS1.p2.4.m4.1.1.2.3.2.2" xref="S3.SS1.p2.4.m4.1.1.2.3.2.2.cmml">P</mi><mrow id="S3.SS1.p2.4.m4.1.1.2.3.2.3" xref="S3.SS1.p2.4.m4.1.1.2.3.2.3.cmml"><mi id="S3.SS1.p2.4.m4.1.1.2.3.2.3.2" xref="S3.SS1.p2.4.m4.1.1.2.3.2.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p2.4.m4.1.1.2.3.2.3.1" xref="S3.SS1.p2.4.m4.1.1.2.3.2.3.1.cmml">​</mo><mi id="S3.SS1.p2.4.m4.1.1.2.3.2.3.3" xref="S3.SS1.p2.4.m4.1.1.2.3.2.3.3.cmml">M</mi></mrow></msub></mrow></mrow><mo id="S3.SS1.p2.4.m4.1.1.1" xref="S3.SS1.p2.4.m4.1.1.1.cmml">=</mo><mrow id="S3.SS1.p2.4.m4.1.1.3" xref="S3.SS1.p2.4.m4.1.1.3.cmml"><mo id="S3.SS1.p2.4.m4.1.1.3a" xref="S3.SS1.p2.4.m4.1.1.3.cmml">−</mo><msub id="S3.SS1.p2.4.m4.1.1.3.2" xref="S3.SS1.p2.4.m4.1.1.3.2.cmml"><mi id="S3.SS1.p2.4.m4.1.1.3.2.2" xref="S3.SS1.p2.4.m4.1.1.3.2.2.cmml">E</mi><mi id="S3.SS1.p2.4.m4.1.1.3.2.3" xref="S3.SS1.p2.4.m4.1.1.3.2.3.cmml">θ</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.4.m4.1b"><apply id="S3.SS1.p2.4.m4.1.1.cmml" xref="S3.SS1.p2.4.m4.1.1"><eq id="S3.SS1.p2.4.m4.1.1.1.cmml" xref="S3.SS1.p2.4.m4.1.1.1"></eq><apply id="S3.SS1.p2.4.m4.1.1.2.cmml" xref="S3.SS1.p2.4.m4.1.1.2"><minus id="S3.SS1.p2.4.m4.1.1.2.1.cmml" xref="S3.SS1.p2.4.m4.1.1.2.1"></minus><apply id="S3.SS1.p2.4.m4.1.1.2.2.cmml" xref="S3.SS1.p2.4.m4.1.1.2.2"><log id="S3.SS1.p2.4.m4.1.1.2.2.1.cmml" xref="S3.SS1.p2.4.m4.1.1.2.2.1"></log><apply id="S3.SS1.p2.4.m4.1.1.2.2.2.cmml" xref="S3.SS1.p2.4.m4.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.SS1.p2.4.m4.1.1.2.2.2.1.cmml" xref="S3.SS1.p2.4.m4.1.1.2.2.2">subscript</csymbol><ci id="S3.SS1.p2.4.m4.1.1.2.2.2.2.cmml" xref="S3.SS1.p2.4.m4.1.1.2.2.2.2">𝑃</ci><ci id="S3.SS1.p2.4.m4.1.1.2.2.2.3.cmml" xref="S3.SS1.p2.4.m4.1.1.2.2.2.3">𝜃</ci></apply></apply><apply id="S3.SS1.p2.4.m4.1.1.2.3.cmml" xref="S3.SS1.p2.4.m4.1.1.2.3"><log id="S3.SS1.p2.4.m4.1.1.2.3.1.cmml" xref="S3.SS1.p2.4.m4.1.1.2.3.1"></log><apply id="S3.SS1.p2.4.m4.1.1.2.3.2.cmml" xref="S3.SS1.p2.4.m4.1.1.2.3.2"><csymbol cd="ambiguous" id="S3.SS1.p2.4.m4.1.1.2.3.2.1.cmml" xref="S3.SS1.p2.4.m4.1.1.2.3.2">subscript</csymbol><ci id="S3.SS1.p2.4.m4.1.1.2.3.2.2.cmml" xref="S3.SS1.p2.4.m4.1.1.2.3.2.2">𝑃</ci><apply id="S3.SS1.p2.4.m4.1.1.2.3.2.3.cmml" xref="S3.SS1.p2.4.m4.1.1.2.3.2.3"><times id="S3.SS1.p2.4.m4.1.1.2.3.2.3.1.cmml" xref="S3.SS1.p2.4.m4.1.1.2.3.2.3.1"></times><ci id="S3.SS1.p2.4.m4.1.1.2.3.2.3.2.cmml" xref="S3.SS1.p2.4.m4.1.1.2.3.2.3.2">𝐿</ci><ci id="S3.SS1.p2.4.m4.1.1.2.3.2.3.3.cmml" xref="S3.SS1.p2.4.m4.1.1.2.3.2.3.3">𝑀</ci></apply></apply></apply></apply><apply id="S3.SS1.p2.4.m4.1.1.3.cmml" xref="S3.SS1.p2.4.m4.1.1.3"><minus id="S3.SS1.p2.4.m4.1.1.3.1.cmml" xref="S3.SS1.p2.4.m4.1.1.3"></minus><apply id="S3.SS1.p2.4.m4.1.1.3.2.cmml" xref="S3.SS1.p2.4.m4.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS1.p2.4.m4.1.1.3.2.1.cmml" xref="S3.SS1.p2.4.m4.1.1.3.2">subscript</csymbol><ci id="S3.SS1.p2.4.m4.1.1.3.2.2.cmml" xref="S3.SS1.p2.4.m4.1.1.3.2.2">𝐸</ci><ci id="S3.SS1.p2.4.m4.1.1.3.2.3.cmml" xref="S3.SS1.p2.4.m4.1.1.3.2.3">𝜃</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.4.m4.1c">\log P_{\theta}-\log P_{LM}=-E_{\theta}</annotation></semantics></math>. Therefore, under these modeling
assumptions of residual learning and noise model, the objective function becomes:</p>
<table id="S3.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E3.m1.4" class="ltx_Math" alttext="\max\mathbb{E}_{x_{+}\sim P_{data}}\log\frac{1}{1+\exp(E_{\theta}(x_{+}))}+\mathbb{E}_{x_{-}\sim P_{LM}}\log\frac{1}{1+\exp(-E_{\theta}(x_{-}))}" display="block"><semantics id="S3.E3.m1.4a"><mrow id="S3.E3.m1.4.5" xref="S3.E3.m1.4.5.cmml"><mrow id="S3.E3.m1.4.5.2" xref="S3.E3.m1.4.5.2.cmml"><mi id="S3.E3.m1.4.5.2.1" xref="S3.E3.m1.4.5.2.1.cmml">max</mi><mo lspace="0.167em" id="S3.E3.m1.4.5.2a" xref="S3.E3.m1.4.5.2.cmml">⁡</mo><mrow id="S3.E3.m1.4.5.2.2" xref="S3.E3.m1.4.5.2.2.cmml"><msub id="S3.E3.m1.4.5.2.2.2" xref="S3.E3.m1.4.5.2.2.2.cmml"><mi id="S3.E3.m1.4.5.2.2.2.2" xref="S3.E3.m1.4.5.2.2.2.2.cmml">𝔼</mi><mrow id="S3.E3.m1.4.5.2.2.2.3" xref="S3.E3.m1.4.5.2.2.2.3.cmml"><msub id="S3.E3.m1.4.5.2.2.2.3.2" xref="S3.E3.m1.4.5.2.2.2.3.2.cmml"><mi id="S3.E3.m1.4.5.2.2.2.3.2.2" xref="S3.E3.m1.4.5.2.2.2.3.2.2.cmml">x</mi><mo id="S3.E3.m1.4.5.2.2.2.3.2.3" xref="S3.E3.m1.4.5.2.2.2.3.2.3.cmml">+</mo></msub><mo id="S3.E3.m1.4.5.2.2.2.3.1" xref="S3.E3.m1.4.5.2.2.2.3.1.cmml">∼</mo><msub id="S3.E3.m1.4.5.2.2.2.3.3" xref="S3.E3.m1.4.5.2.2.2.3.3.cmml"><mi id="S3.E3.m1.4.5.2.2.2.3.3.2" xref="S3.E3.m1.4.5.2.2.2.3.3.2.cmml">P</mi><mrow id="S3.E3.m1.4.5.2.2.2.3.3.3" xref="S3.E3.m1.4.5.2.2.2.3.3.3.cmml"><mi id="S3.E3.m1.4.5.2.2.2.3.3.3.2" xref="S3.E3.m1.4.5.2.2.2.3.3.3.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.4.5.2.2.2.3.3.3.1" xref="S3.E3.m1.4.5.2.2.2.3.3.3.1.cmml">​</mo><mi id="S3.E3.m1.4.5.2.2.2.3.3.3.3" xref="S3.E3.m1.4.5.2.2.2.3.3.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.4.5.2.2.2.3.3.3.1a" xref="S3.E3.m1.4.5.2.2.2.3.3.3.1.cmml">​</mo><mi id="S3.E3.m1.4.5.2.2.2.3.3.3.4" xref="S3.E3.m1.4.5.2.2.2.3.3.3.4.cmml">t</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.4.5.2.2.2.3.3.3.1b" xref="S3.E3.m1.4.5.2.2.2.3.3.3.1.cmml">​</mo><mi id="S3.E3.m1.4.5.2.2.2.3.3.3.5" xref="S3.E3.m1.4.5.2.2.2.3.3.3.5.cmml">a</mi></mrow></msub></mrow></msub><mo lspace="0.167em" rspace="0em" id="S3.E3.m1.4.5.2.2.1" xref="S3.E3.m1.4.5.2.2.1.cmml">​</mo><mrow id="S3.E3.m1.4.5.2.2.3" xref="S3.E3.m1.4.5.2.2.3.cmml"><mi id="S3.E3.m1.4.5.2.2.3.1" xref="S3.E3.m1.4.5.2.2.3.1.cmml">log</mi><mo lspace="0.167em" id="S3.E3.m1.4.5.2.2.3a" xref="S3.E3.m1.4.5.2.2.3.cmml">⁡</mo><mfrac id="S3.E3.m1.2.2" xref="S3.E3.m1.2.2.cmml"><mn id="S3.E3.m1.2.2.4" xref="S3.E3.m1.2.2.4.cmml">1</mn><mrow id="S3.E3.m1.2.2.2" xref="S3.E3.m1.2.2.2.cmml"><mn id="S3.E3.m1.2.2.2.4" xref="S3.E3.m1.2.2.2.4.cmml">1</mn><mo id="S3.E3.m1.2.2.2.3" xref="S3.E3.m1.2.2.2.3.cmml">+</mo><mrow id="S3.E3.m1.2.2.2.2.1" xref="S3.E3.m1.2.2.2.2.2.cmml"><mi id="S3.E3.m1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.cmml">exp</mi><mo id="S3.E3.m1.2.2.2.2.1a" xref="S3.E3.m1.2.2.2.2.2.cmml">⁡</mo><mrow id="S3.E3.m1.2.2.2.2.1.1" xref="S3.E3.m1.2.2.2.2.2.cmml"><mo stretchy="false" id="S3.E3.m1.2.2.2.2.1.1.2" xref="S3.E3.m1.2.2.2.2.2.cmml">(</mo><mrow id="S3.E3.m1.2.2.2.2.1.1.1" xref="S3.E3.m1.2.2.2.2.1.1.1.cmml"><msub id="S3.E3.m1.2.2.2.2.1.1.1.3" xref="S3.E3.m1.2.2.2.2.1.1.1.3.cmml"><mi id="S3.E3.m1.2.2.2.2.1.1.1.3.2" xref="S3.E3.m1.2.2.2.2.1.1.1.3.2.cmml">E</mi><mi id="S3.E3.m1.2.2.2.2.1.1.1.3.3" xref="S3.E3.m1.2.2.2.2.1.1.1.3.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E3.m1.2.2.2.2.1.1.1.2" xref="S3.E3.m1.2.2.2.2.1.1.1.2.cmml">​</mo><mrow id="S3.E3.m1.2.2.2.2.1.1.1.1.1" xref="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.2.2.2.2.1.1.1.1.1.2" xref="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.cmml">(</mo><msub id="S3.E3.m1.2.2.2.2.1.1.1.1.1.1" xref="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.2" xref="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.2.cmml">x</mi><mo id="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.3" xref="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.3.cmml">+</mo></msub><mo stretchy="false" id="S3.E3.m1.2.2.2.2.1.1.1.1.1.3" xref="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.E3.m1.2.2.2.2.1.1.3" xref="S3.E3.m1.2.2.2.2.2.cmml">)</mo></mrow></mrow></mrow></mfrac></mrow></mrow></mrow><mo id="S3.E3.m1.4.5.1" xref="S3.E3.m1.4.5.1.cmml">+</mo><mrow id="S3.E3.m1.4.5.3" xref="S3.E3.m1.4.5.3.cmml"><msub id="S3.E3.m1.4.5.3.2" xref="S3.E3.m1.4.5.3.2.cmml"><mi id="S3.E3.m1.4.5.3.2.2" xref="S3.E3.m1.4.5.3.2.2.cmml">𝔼</mi><mrow id="S3.E3.m1.4.5.3.2.3" xref="S3.E3.m1.4.5.3.2.3.cmml"><msub id="S3.E3.m1.4.5.3.2.3.2" xref="S3.E3.m1.4.5.3.2.3.2.cmml"><mi id="S3.E3.m1.4.5.3.2.3.2.2" xref="S3.E3.m1.4.5.3.2.3.2.2.cmml">x</mi><mo id="S3.E3.m1.4.5.3.2.3.2.3" xref="S3.E3.m1.4.5.3.2.3.2.3.cmml">−</mo></msub><mo id="S3.E3.m1.4.5.3.2.3.1" xref="S3.E3.m1.4.5.3.2.3.1.cmml">∼</mo><msub id="S3.E3.m1.4.5.3.2.3.3" xref="S3.E3.m1.4.5.3.2.3.3.cmml"><mi id="S3.E3.m1.4.5.3.2.3.3.2" xref="S3.E3.m1.4.5.3.2.3.3.2.cmml">P</mi><mrow id="S3.E3.m1.4.5.3.2.3.3.3" xref="S3.E3.m1.4.5.3.2.3.3.3.cmml"><mi id="S3.E3.m1.4.5.3.2.3.3.3.2" xref="S3.E3.m1.4.5.3.2.3.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.4.5.3.2.3.3.3.1" xref="S3.E3.m1.4.5.3.2.3.3.3.1.cmml">​</mo><mi id="S3.E3.m1.4.5.3.2.3.3.3.3" xref="S3.E3.m1.4.5.3.2.3.3.3.3.cmml">M</mi></mrow></msub></mrow></msub><mo lspace="0.167em" rspace="0em" id="S3.E3.m1.4.5.3.1" xref="S3.E3.m1.4.5.3.1.cmml">​</mo><mrow id="S3.E3.m1.4.5.3.3" xref="S3.E3.m1.4.5.3.3.cmml"><mi id="S3.E3.m1.4.5.3.3.1" xref="S3.E3.m1.4.5.3.3.1.cmml">log</mi><mo lspace="0.167em" id="S3.E3.m1.4.5.3.3a" xref="S3.E3.m1.4.5.3.3.cmml">⁡</mo><mfrac id="S3.E3.m1.4.4" xref="S3.E3.m1.4.4.cmml"><mn id="S3.E3.m1.4.4.4" xref="S3.E3.m1.4.4.4.cmml">1</mn><mrow id="S3.E3.m1.4.4.2" xref="S3.E3.m1.4.4.2.cmml"><mn id="S3.E3.m1.4.4.2.4" xref="S3.E3.m1.4.4.2.4.cmml">1</mn><mo id="S3.E3.m1.4.4.2.3" xref="S3.E3.m1.4.4.2.3.cmml">+</mo><mrow id="S3.E3.m1.4.4.2.2.1" xref="S3.E3.m1.4.4.2.2.2.cmml"><mi id="S3.E3.m1.3.3.1.1" xref="S3.E3.m1.3.3.1.1.cmml">exp</mi><mo id="S3.E3.m1.4.4.2.2.1a" xref="S3.E3.m1.4.4.2.2.2.cmml">⁡</mo><mrow id="S3.E3.m1.4.4.2.2.1.1" xref="S3.E3.m1.4.4.2.2.2.cmml"><mo stretchy="false" id="S3.E3.m1.4.4.2.2.1.1.2" xref="S3.E3.m1.4.4.2.2.2.cmml">(</mo><mrow id="S3.E3.m1.4.4.2.2.1.1.1" xref="S3.E3.m1.4.4.2.2.1.1.1.cmml"><mo id="S3.E3.m1.4.4.2.2.1.1.1a" xref="S3.E3.m1.4.4.2.2.1.1.1.cmml">−</mo><mrow id="S3.E3.m1.4.4.2.2.1.1.1.1" xref="S3.E3.m1.4.4.2.2.1.1.1.1.cmml"><msub id="S3.E3.m1.4.4.2.2.1.1.1.1.3" xref="S3.E3.m1.4.4.2.2.1.1.1.1.3.cmml"><mi id="S3.E3.m1.4.4.2.2.1.1.1.1.3.2" xref="S3.E3.m1.4.4.2.2.1.1.1.1.3.2.cmml">E</mi><mi id="S3.E3.m1.4.4.2.2.1.1.1.1.3.3" xref="S3.E3.m1.4.4.2.2.1.1.1.1.3.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E3.m1.4.4.2.2.1.1.1.1.2" xref="S3.E3.m1.4.4.2.2.1.1.1.1.2.cmml">​</mo><mrow id="S3.E3.m1.4.4.2.2.1.1.1.1.1.1" xref="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.2" xref="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1" xref="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.2" xref="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.2.cmml">x</mi><mo id="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.3" xref="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.3.cmml">−</mo></msub><mo stretchy="false" id="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.3" xref="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.E3.m1.4.4.2.2.1.1.3" xref="S3.E3.m1.4.4.2.2.2.cmml">)</mo></mrow></mrow></mrow></mfrac></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.4b"><apply id="S3.E3.m1.4.5.cmml" xref="S3.E3.m1.4.5"><plus id="S3.E3.m1.4.5.1.cmml" xref="S3.E3.m1.4.5.1"></plus><apply id="S3.E3.m1.4.5.2.cmml" xref="S3.E3.m1.4.5.2"><max id="S3.E3.m1.4.5.2.1.cmml" xref="S3.E3.m1.4.5.2.1"></max><apply id="S3.E3.m1.4.5.2.2.cmml" xref="S3.E3.m1.4.5.2.2"><times id="S3.E3.m1.4.5.2.2.1.cmml" xref="S3.E3.m1.4.5.2.2.1"></times><apply id="S3.E3.m1.4.5.2.2.2.cmml" xref="S3.E3.m1.4.5.2.2.2"><csymbol cd="ambiguous" id="S3.E3.m1.4.5.2.2.2.1.cmml" xref="S3.E3.m1.4.5.2.2.2">subscript</csymbol><ci id="S3.E3.m1.4.5.2.2.2.2.cmml" xref="S3.E3.m1.4.5.2.2.2.2">𝔼</ci><apply id="S3.E3.m1.4.5.2.2.2.3.cmml" xref="S3.E3.m1.4.5.2.2.2.3"><csymbol cd="latexml" id="S3.E3.m1.4.5.2.2.2.3.1.cmml" xref="S3.E3.m1.4.5.2.2.2.3.1">similar-to</csymbol><apply id="S3.E3.m1.4.5.2.2.2.3.2.cmml" xref="S3.E3.m1.4.5.2.2.2.3.2"><csymbol cd="ambiguous" id="S3.E3.m1.4.5.2.2.2.3.2.1.cmml" xref="S3.E3.m1.4.5.2.2.2.3.2">subscript</csymbol><ci id="S3.E3.m1.4.5.2.2.2.3.2.2.cmml" xref="S3.E3.m1.4.5.2.2.2.3.2.2">𝑥</ci><plus id="S3.E3.m1.4.5.2.2.2.3.2.3.cmml" xref="S3.E3.m1.4.5.2.2.2.3.2.3"></plus></apply><apply id="S3.E3.m1.4.5.2.2.2.3.3.cmml" xref="S3.E3.m1.4.5.2.2.2.3.3"><csymbol cd="ambiguous" id="S3.E3.m1.4.5.2.2.2.3.3.1.cmml" xref="S3.E3.m1.4.5.2.2.2.3.3">subscript</csymbol><ci id="S3.E3.m1.4.5.2.2.2.3.3.2.cmml" xref="S3.E3.m1.4.5.2.2.2.3.3.2">𝑃</ci><apply id="S3.E3.m1.4.5.2.2.2.3.3.3.cmml" xref="S3.E3.m1.4.5.2.2.2.3.3.3"><times id="S3.E3.m1.4.5.2.2.2.3.3.3.1.cmml" xref="S3.E3.m1.4.5.2.2.2.3.3.3.1"></times><ci id="S3.E3.m1.4.5.2.2.2.3.3.3.2.cmml" xref="S3.E3.m1.4.5.2.2.2.3.3.3.2">𝑑</ci><ci id="S3.E3.m1.4.5.2.2.2.3.3.3.3.cmml" xref="S3.E3.m1.4.5.2.2.2.3.3.3.3">𝑎</ci><ci id="S3.E3.m1.4.5.2.2.2.3.3.3.4.cmml" xref="S3.E3.m1.4.5.2.2.2.3.3.3.4">𝑡</ci><ci id="S3.E3.m1.4.5.2.2.2.3.3.3.5.cmml" xref="S3.E3.m1.4.5.2.2.2.3.3.3.5">𝑎</ci></apply></apply></apply></apply><apply id="S3.E3.m1.4.5.2.2.3.cmml" xref="S3.E3.m1.4.5.2.2.3"><log id="S3.E3.m1.4.5.2.2.3.1.cmml" xref="S3.E3.m1.4.5.2.2.3.1"></log><apply id="S3.E3.m1.2.2.cmml" xref="S3.E3.m1.2.2"><divide id="S3.E3.m1.2.2.3.cmml" xref="S3.E3.m1.2.2"></divide><cn type="integer" id="S3.E3.m1.2.2.4.cmml" xref="S3.E3.m1.2.2.4">1</cn><apply id="S3.E3.m1.2.2.2.cmml" xref="S3.E3.m1.2.2.2"><plus id="S3.E3.m1.2.2.2.3.cmml" xref="S3.E3.m1.2.2.2.3"></plus><cn type="integer" id="S3.E3.m1.2.2.2.4.cmml" xref="S3.E3.m1.2.2.2.4">1</cn><apply id="S3.E3.m1.2.2.2.2.2.cmml" xref="S3.E3.m1.2.2.2.2.1"><exp id="S3.E3.m1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1"></exp><apply id="S3.E3.m1.2.2.2.2.1.1.1.cmml" xref="S3.E3.m1.2.2.2.2.1.1.1"><times id="S3.E3.m1.2.2.2.2.1.1.1.2.cmml" xref="S3.E3.m1.2.2.2.2.1.1.1.2"></times><apply id="S3.E3.m1.2.2.2.2.1.1.1.3.cmml" xref="S3.E3.m1.2.2.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.2.2.1.1.1.3.1.cmml" xref="S3.E3.m1.2.2.2.2.1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.2.2.2.2.1.1.1.3.2.cmml" xref="S3.E3.m1.2.2.2.2.1.1.1.3.2">𝐸</ci><ci id="S3.E3.m1.2.2.2.2.1.1.1.3.3.cmml" xref="S3.E3.m1.2.2.2.2.1.1.1.3.3">𝜃</ci></apply><apply id="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.cmml" xref="S3.E3.m1.2.2.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.2.2.2.2.1.1.1.1.1">subscript</csymbol><ci id="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.2">𝑥</ci><plus id="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.2.2.2.2.1.1.1.1.1.1.3"></plus></apply></apply></apply></apply></apply></apply></apply></apply><apply id="S3.E3.m1.4.5.3.cmml" xref="S3.E3.m1.4.5.3"><times id="S3.E3.m1.4.5.3.1.cmml" xref="S3.E3.m1.4.5.3.1"></times><apply id="S3.E3.m1.4.5.3.2.cmml" xref="S3.E3.m1.4.5.3.2"><csymbol cd="ambiguous" id="S3.E3.m1.4.5.3.2.1.cmml" xref="S3.E3.m1.4.5.3.2">subscript</csymbol><ci id="S3.E3.m1.4.5.3.2.2.cmml" xref="S3.E3.m1.4.5.3.2.2">𝔼</ci><apply id="S3.E3.m1.4.5.3.2.3.cmml" xref="S3.E3.m1.4.5.3.2.3"><csymbol cd="latexml" id="S3.E3.m1.4.5.3.2.3.1.cmml" xref="S3.E3.m1.4.5.3.2.3.1">similar-to</csymbol><apply id="S3.E3.m1.4.5.3.2.3.2.cmml" xref="S3.E3.m1.4.5.3.2.3.2"><csymbol cd="ambiguous" id="S3.E3.m1.4.5.3.2.3.2.1.cmml" xref="S3.E3.m1.4.5.3.2.3.2">subscript</csymbol><ci id="S3.E3.m1.4.5.3.2.3.2.2.cmml" xref="S3.E3.m1.4.5.3.2.3.2.2">𝑥</ci><minus id="S3.E3.m1.4.5.3.2.3.2.3.cmml" xref="S3.E3.m1.4.5.3.2.3.2.3"></minus></apply><apply id="S3.E3.m1.4.5.3.2.3.3.cmml" xref="S3.E3.m1.4.5.3.2.3.3"><csymbol cd="ambiguous" id="S3.E3.m1.4.5.3.2.3.3.1.cmml" xref="S3.E3.m1.4.5.3.2.3.3">subscript</csymbol><ci id="S3.E3.m1.4.5.3.2.3.3.2.cmml" xref="S3.E3.m1.4.5.3.2.3.3.2">𝑃</ci><apply id="S3.E3.m1.4.5.3.2.3.3.3.cmml" xref="S3.E3.m1.4.5.3.2.3.3.3"><times id="S3.E3.m1.4.5.3.2.3.3.3.1.cmml" xref="S3.E3.m1.4.5.3.2.3.3.3.1"></times><ci id="S3.E3.m1.4.5.3.2.3.3.3.2.cmml" xref="S3.E3.m1.4.5.3.2.3.3.3.2">𝐿</ci><ci id="S3.E3.m1.4.5.3.2.3.3.3.3.cmml" xref="S3.E3.m1.4.5.3.2.3.3.3.3">𝑀</ci></apply></apply></apply></apply><apply id="S3.E3.m1.4.5.3.3.cmml" xref="S3.E3.m1.4.5.3.3"><log id="S3.E3.m1.4.5.3.3.1.cmml" xref="S3.E3.m1.4.5.3.3.1"></log><apply id="S3.E3.m1.4.4.cmml" xref="S3.E3.m1.4.4"><divide id="S3.E3.m1.4.4.3.cmml" xref="S3.E3.m1.4.4"></divide><cn type="integer" id="S3.E3.m1.4.4.4.cmml" xref="S3.E3.m1.4.4.4">1</cn><apply id="S3.E3.m1.4.4.2.cmml" xref="S3.E3.m1.4.4.2"><plus id="S3.E3.m1.4.4.2.3.cmml" xref="S3.E3.m1.4.4.2.3"></plus><cn type="integer" id="S3.E3.m1.4.4.2.4.cmml" xref="S3.E3.m1.4.4.2.4">1</cn><apply id="S3.E3.m1.4.4.2.2.2.cmml" xref="S3.E3.m1.4.4.2.2.1"><exp id="S3.E3.m1.3.3.1.1.cmml" xref="S3.E3.m1.3.3.1.1"></exp><apply id="S3.E3.m1.4.4.2.2.1.1.1.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1"><minus id="S3.E3.m1.4.4.2.2.1.1.1.2.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1"></minus><apply id="S3.E3.m1.4.4.2.2.1.1.1.1.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1.1"><times id="S3.E3.m1.4.4.2.2.1.1.1.1.2.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1.1.2"></times><apply id="S3.E3.m1.4.4.2.2.1.1.1.1.3.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.4.4.2.2.1.1.1.1.3.1.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.4.4.2.2.1.1.1.1.3.2.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1.1.3.2">𝐸</ci><ci id="S3.E3.m1.4.4.2.2.1.1.1.1.3.3.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1.1.3.3">𝜃</ci></apply><apply id="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.2">𝑥</ci><minus id="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.4.4.2.2.1.1.1.1.1.1.1.3"></minus></apply></apply></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.4c">\max\mathbb{E}_{x_{+}\sim P_{data}}\log\frac{1}{1+\exp(E_{\theta}(x_{+}))}+\mathbb{E}_{x_{-}\sim P_{LM}}\log\frac{1}{1+\exp(-E_{\theta}(x_{-}))}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p2.7" class="ltx_p">where <math id="S3.SS1.p2.5.m1.1" class="ltx_Math" alttext="x_{+}" display="inline"><semantics id="S3.SS1.p2.5.m1.1a"><msub id="S3.SS1.p2.5.m1.1.1" xref="S3.SS1.p2.5.m1.1.1.cmml"><mi id="S3.SS1.p2.5.m1.1.1.2" xref="S3.SS1.p2.5.m1.1.1.2.cmml">x</mi><mo id="S3.SS1.p2.5.m1.1.1.3" xref="S3.SS1.p2.5.m1.1.1.3.cmml">+</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.5.m1.1b"><apply id="S3.SS1.p2.5.m1.1.1.cmml" xref="S3.SS1.p2.5.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.5.m1.1.1.1.cmml" xref="S3.SS1.p2.5.m1.1.1">subscript</csymbol><ci id="S3.SS1.p2.5.m1.1.1.2.cmml" xref="S3.SS1.p2.5.m1.1.1.2">𝑥</ci><plus id="S3.SS1.p2.5.m1.1.1.3.cmml" xref="S3.SS1.p2.5.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.5.m1.1c">x_{+}</annotation></semantics></math> is a positive sequence taken from the human generated training set, and <math id="S3.SS1.p2.6.m2.1" class="ltx_Math" alttext="x_{-}" display="inline"><semantics id="S3.SS1.p2.6.m2.1a"><msub id="S3.SS1.p2.6.m2.1.1" xref="S3.SS1.p2.6.m2.1.1.cmml"><mi id="S3.SS1.p2.6.m2.1.1.2" xref="S3.SS1.p2.6.m2.1.1.2.cmml">x</mi><mo id="S3.SS1.p2.6.m2.1.1.3" xref="S3.SS1.p2.6.m2.1.1.3.cmml">−</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.6.m2.1b"><apply id="S3.SS1.p2.6.m2.1.1.cmml" xref="S3.SS1.p2.6.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.6.m2.1.1.1.cmml" xref="S3.SS1.p2.6.m2.1.1">subscript</csymbol><ci id="S3.SS1.p2.6.m2.1.1.2.cmml" xref="S3.SS1.p2.6.m2.1.1.2">𝑥</ci><minus id="S3.SS1.p2.6.m2.1.1.3.cmml" xref="S3.SS1.p2.6.m2.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.6.m2.1c">x_{-}</annotation></semantics></math> is a negative sequence drawn
from <math id="S3.SS1.p2.7.m3.1" class="ltx_Math" alttext="P_{LM}" display="inline"><semantics id="S3.SS1.p2.7.m3.1a"><msub id="S3.SS1.p2.7.m3.1.1" xref="S3.SS1.p2.7.m3.1.1.cmml"><mi id="S3.SS1.p2.7.m3.1.1.2" xref="S3.SS1.p2.7.m3.1.1.2.cmml">P</mi><mrow id="S3.SS1.p2.7.m3.1.1.3" xref="S3.SS1.p2.7.m3.1.1.3.cmml"><mi id="S3.SS1.p2.7.m3.1.1.3.2" xref="S3.SS1.p2.7.m3.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p2.7.m3.1.1.3.1" xref="S3.SS1.p2.7.m3.1.1.3.1.cmml">​</mo><mi id="S3.SS1.p2.7.m3.1.1.3.3" xref="S3.SS1.p2.7.m3.1.1.3.3.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.7.m3.1b"><apply id="S3.SS1.p2.7.m3.1.1.cmml" xref="S3.SS1.p2.7.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.7.m3.1.1.1.cmml" xref="S3.SS1.p2.7.m3.1.1">subscript</csymbol><ci id="S3.SS1.p2.7.m3.1.1.2.cmml" xref="S3.SS1.p2.7.m3.1.1.2">𝑃</ci><apply id="S3.SS1.p2.7.m3.1.1.3.cmml" xref="S3.SS1.p2.7.m3.1.1.3"><times id="S3.SS1.p2.7.m3.1.1.3.1.cmml" xref="S3.SS1.p2.7.m3.1.1.3.1"></times><ci id="S3.SS1.p2.7.m3.1.1.3.2.cmml" xref="S3.SS1.p2.7.m3.1.1.3.2">𝐿</ci><ci id="S3.SS1.p2.7.m3.1.1.3.3.cmml" xref="S3.SS1.p2.7.m3.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.7.m3.1c">P_{LM}</annotation></semantics></math> (for a given ground truth prefix). In other words, training the energy function reduces to training a binary classifier to discriminate between real text and text
generated by an auto-regressive language model. The aim of training is to assign as negative energy as possible to real data,
and as positive energy as possible to machine generated data. Interestingly, the role of positive and negative samples is totally
symmetric in this loss function, §<a href="#S5" title="5 Limitations ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> will discuss the consequences of this.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS1.p3.1" class="ltx_p">With the theoretical guarantee of NCE, we can show that the optimum of the above objective is reached at data
distribution with infinite amount of data and model with enough capacity, which is also proved in <cite class="ltx_cite ltx_citemacro_citet">Ma &amp; Collins (<a href="#bib.bib20" title="" class="ltx_ref">2018</a>)</cite><span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>From <cite class="ltx_cite ltx_citemacro_citet">Ma &amp; Collins (<a href="#bib.bib20" title="" class="ltx_ref">2018</a>)</cite> Assumption 2, for conditional NCE the model needs to be flexible enough such that the self-normalizing property can be satisfied conditioned on any prefix.</span></span></span>.</p>
</div>
<div id="Thmtheorem1" class="ltx_theorem ltx_theorem_theorem">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmtheorem1.1.1.1" class="ltx_text ltx_font_bold">Theorem 1</span></span><span id="Thmtheorem1.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmtheorem1.p1" class="ltx_para">
<p id="Thmtheorem1.p1.4" class="ltx_p"><span id="Thmtheorem1.p1.4.4" class="ltx_text ltx_font_italic">If <math id="Thmtheorem1.p1.1.1.m1.1" class="ltx_Math" alttext="P_{LM}" display="inline"><semantics id="Thmtheorem1.p1.1.1.m1.1a"><msub id="Thmtheorem1.p1.1.1.m1.1.1" xref="Thmtheorem1.p1.1.1.m1.1.1.cmml"><mi id="Thmtheorem1.p1.1.1.m1.1.1.2" xref="Thmtheorem1.p1.1.1.m1.1.1.2.cmml">P</mi><mrow id="Thmtheorem1.p1.1.1.m1.1.1.3" xref="Thmtheorem1.p1.1.1.m1.1.1.3.cmml"><mi id="Thmtheorem1.p1.1.1.m1.1.1.3.2" xref="Thmtheorem1.p1.1.1.m1.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.1.1.m1.1.1.3.1" xref="Thmtheorem1.p1.1.1.m1.1.1.3.1.cmml">​</mo><mi id="Thmtheorem1.p1.1.1.m1.1.1.3.3" xref="Thmtheorem1.p1.1.1.m1.1.1.3.3.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="Thmtheorem1.p1.1.1.m1.1b"><apply id="Thmtheorem1.p1.1.1.m1.1.1.cmml" xref="Thmtheorem1.p1.1.1.m1.1.1"><csymbol cd="ambiguous" id="Thmtheorem1.p1.1.1.m1.1.1.1.cmml" xref="Thmtheorem1.p1.1.1.m1.1.1">subscript</csymbol><ci id="Thmtheorem1.p1.1.1.m1.1.1.2.cmml" xref="Thmtheorem1.p1.1.1.m1.1.1.2">𝑃</ci><apply id="Thmtheorem1.p1.1.1.m1.1.1.3.cmml" xref="Thmtheorem1.p1.1.1.m1.1.1.3"><times id="Thmtheorem1.p1.1.1.m1.1.1.3.1.cmml" xref="Thmtheorem1.p1.1.1.m1.1.1.3.1"></times><ci id="Thmtheorem1.p1.1.1.m1.1.1.3.2.cmml" xref="Thmtheorem1.p1.1.1.m1.1.1.3.2">𝐿</ci><ci id="Thmtheorem1.p1.1.1.m1.1.1.3.3.cmml" xref="Thmtheorem1.p1.1.1.m1.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem1.p1.1.1.m1.1c">P_{LM}</annotation></semantics></math> has the same support as <math id="Thmtheorem1.p1.2.2.m2.1" class="ltx_Math" alttext="P_{data}" display="inline"><semantics id="Thmtheorem1.p1.2.2.m2.1a"><msub id="Thmtheorem1.p1.2.2.m2.1.1" xref="Thmtheorem1.p1.2.2.m2.1.1.cmml"><mi id="Thmtheorem1.p1.2.2.m2.1.1.2" xref="Thmtheorem1.p1.2.2.m2.1.1.2.cmml">P</mi><mrow id="Thmtheorem1.p1.2.2.m2.1.1.3" xref="Thmtheorem1.p1.2.2.m2.1.1.3.cmml"><mi id="Thmtheorem1.p1.2.2.m2.1.1.3.2" xref="Thmtheorem1.p1.2.2.m2.1.1.3.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.2.2.m2.1.1.3.1" xref="Thmtheorem1.p1.2.2.m2.1.1.3.1.cmml">​</mo><mi id="Thmtheorem1.p1.2.2.m2.1.1.3.3" xref="Thmtheorem1.p1.2.2.m2.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.2.2.m2.1.1.3.1a" xref="Thmtheorem1.p1.2.2.m2.1.1.3.1.cmml">​</mo><mi id="Thmtheorem1.p1.2.2.m2.1.1.3.4" xref="Thmtheorem1.p1.2.2.m2.1.1.3.4.cmml">t</mi><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.2.2.m2.1.1.3.1b" xref="Thmtheorem1.p1.2.2.m2.1.1.3.1.cmml">​</mo><mi id="Thmtheorem1.p1.2.2.m2.1.1.3.5" xref="Thmtheorem1.p1.2.2.m2.1.1.3.5.cmml">a</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="Thmtheorem1.p1.2.2.m2.1b"><apply id="Thmtheorem1.p1.2.2.m2.1.1.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1"><csymbol cd="ambiguous" id="Thmtheorem1.p1.2.2.m2.1.1.1.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1">subscript</csymbol><ci id="Thmtheorem1.p1.2.2.m2.1.1.2.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.2">𝑃</ci><apply id="Thmtheorem1.p1.2.2.m2.1.1.3.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.3"><times id="Thmtheorem1.p1.2.2.m2.1.1.3.1.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.3.1"></times><ci id="Thmtheorem1.p1.2.2.m2.1.1.3.2.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.3.2">𝑑</ci><ci id="Thmtheorem1.p1.2.2.m2.1.1.3.3.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.3.3">𝑎</ci><ci id="Thmtheorem1.p1.2.2.m2.1.1.3.4.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.3.4">𝑡</ci><ci id="Thmtheorem1.p1.2.2.m2.1.1.3.5.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.3.5">𝑎</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem1.p1.2.2.m2.1c">P_{data}</annotation></semantics></math>, then the objective function in Eq. <a href="#S3.E3" title="In 3.1 Training ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> reaches its maximum at <math id="Thmtheorem1.p1.3.3.m3.2" class="ltx_Math" alttext="\log P_{LM}(x)-E_{\theta}(x)=\log P_{data}" display="inline"><semantics id="Thmtheorem1.p1.3.3.m3.2a"><mrow id="Thmtheorem1.p1.3.3.m3.2.3" xref="Thmtheorem1.p1.3.3.m3.2.3.cmml"><mrow id="Thmtheorem1.p1.3.3.m3.2.3.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.cmml"><mrow id="Thmtheorem1.p1.3.3.m3.2.3.2.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.cmml"><mrow id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.cmml"><mi id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.1" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.1.cmml">log</mi><mo lspace="0.167em" id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2a" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.cmml">⁡</mo><msub id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.cmml"><mi id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.2.cmml">P</mi><mrow id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.cmml"><mi id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.1" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.1.cmml">​</mo><mi id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.3" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.3.cmml">M</mi></mrow></msub></mrow><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.3.3.m3.2.3.2.2.1" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.1.cmml">​</mo><mrow id="Thmtheorem1.p1.3.3.m3.2.3.2.2.3.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.cmml"><mo stretchy="false" id="Thmtheorem1.p1.3.3.m3.2.3.2.2.3.2.1" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.cmml">(</mo><mi id="Thmtheorem1.p1.3.3.m3.1.1" xref="Thmtheorem1.p1.3.3.m3.1.1.cmml">x</mi><mo stretchy="false" id="Thmtheorem1.p1.3.3.m3.2.3.2.2.3.2.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.cmml">)</mo></mrow></mrow><mo id="Thmtheorem1.p1.3.3.m3.2.3.2.1" xref="Thmtheorem1.p1.3.3.m3.2.3.2.1.cmml">−</mo><mrow id="Thmtheorem1.p1.3.3.m3.2.3.2.3" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.cmml"><msub id="Thmtheorem1.p1.3.3.m3.2.3.2.3.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.2.cmml"><mi id="Thmtheorem1.p1.3.3.m3.2.3.2.3.2.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.2.2.cmml">E</mi><mi id="Thmtheorem1.p1.3.3.m3.2.3.2.3.2.3" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.3.3.m3.2.3.2.3.1" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.1.cmml">​</mo><mrow id="Thmtheorem1.p1.3.3.m3.2.3.2.3.3.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.cmml"><mo stretchy="false" id="Thmtheorem1.p1.3.3.m3.2.3.2.3.3.2.1" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.cmml">(</mo><mi id="Thmtheorem1.p1.3.3.m3.2.2" xref="Thmtheorem1.p1.3.3.m3.2.2.cmml">x</mi><mo stretchy="false" id="Thmtheorem1.p1.3.3.m3.2.3.2.3.3.2.2" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.cmml">)</mo></mrow></mrow></mrow><mo id="Thmtheorem1.p1.3.3.m3.2.3.1" xref="Thmtheorem1.p1.3.3.m3.2.3.1.cmml">=</mo><mrow id="Thmtheorem1.p1.3.3.m3.2.3.3" xref="Thmtheorem1.p1.3.3.m3.2.3.3.cmml"><mi id="Thmtheorem1.p1.3.3.m3.2.3.3.1" xref="Thmtheorem1.p1.3.3.m3.2.3.3.1.cmml">log</mi><mo lspace="0.167em" id="Thmtheorem1.p1.3.3.m3.2.3.3a" xref="Thmtheorem1.p1.3.3.m3.2.3.3.cmml">⁡</mo><msub id="Thmtheorem1.p1.3.3.m3.2.3.3.2" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.cmml"><mi id="Thmtheorem1.p1.3.3.m3.2.3.3.2.2" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.2.cmml">P</mi><mrow id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.cmml"><mi id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.2" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.1" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.1.cmml">​</mo><mi id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.3" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.1a" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.1.cmml">​</mo><mi id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.4" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.4.cmml">t</mi><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.1b" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.1.cmml">​</mo><mi id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.5" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.5.cmml">a</mi></mrow></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem1.p1.3.3.m3.2b"><apply id="Thmtheorem1.p1.3.3.m3.2.3.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3"><eq id="Thmtheorem1.p1.3.3.m3.2.3.1.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.1"></eq><apply id="Thmtheorem1.p1.3.3.m3.2.3.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2"><minus id="Thmtheorem1.p1.3.3.m3.2.3.2.1.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.1"></minus><apply id="Thmtheorem1.p1.3.3.m3.2.3.2.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2"><times id="Thmtheorem1.p1.3.3.m3.2.3.2.2.1.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.1"></times><apply id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2"><log id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.1.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.1"></log><apply id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2"><csymbol cd="ambiguous" id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.1.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2">subscript</csymbol><ci id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.2">𝑃</ci><apply id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3"><times id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.1.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.1"></times><ci id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.2">𝐿</ci><ci id="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.3.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.2.2.2.3.3">𝑀</ci></apply></apply></apply><ci id="Thmtheorem1.p1.3.3.m3.1.1.cmml" xref="Thmtheorem1.p1.3.3.m3.1.1">𝑥</ci></apply><apply id="Thmtheorem1.p1.3.3.m3.2.3.2.3.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3"><times id="Thmtheorem1.p1.3.3.m3.2.3.2.3.1.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.1"></times><apply id="Thmtheorem1.p1.3.3.m3.2.3.2.3.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.2"><csymbol cd="ambiguous" id="Thmtheorem1.p1.3.3.m3.2.3.2.3.2.1.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.2">subscript</csymbol><ci id="Thmtheorem1.p1.3.3.m3.2.3.2.3.2.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.2.2">𝐸</ci><ci id="Thmtheorem1.p1.3.3.m3.2.3.2.3.2.3.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.2.3.2.3">𝜃</ci></apply><ci id="Thmtheorem1.p1.3.3.m3.2.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.2">𝑥</ci></apply></apply><apply id="Thmtheorem1.p1.3.3.m3.2.3.3.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.3"><log id="Thmtheorem1.p1.3.3.m3.2.3.3.1.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.3.1"></log><apply id="Thmtheorem1.p1.3.3.m3.2.3.3.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2"><csymbol cd="ambiguous" id="Thmtheorem1.p1.3.3.m3.2.3.3.2.1.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2">subscript</csymbol><ci id="Thmtheorem1.p1.3.3.m3.2.3.3.2.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.2">𝑃</ci><apply id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3"><times id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.1.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.1"></times><ci id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.2.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.2">𝑑</ci><ci id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.3.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.3">𝑎</ci><ci id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.4.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.4">𝑡</ci><ci id="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.5.cmml" xref="Thmtheorem1.p1.3.3.m3.2.3.3.2.3.5">𝑎</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem1.p1.3.3.m3.2c">\log P_{LM}(x)-E_{\theta}(x)=\log P_{data}</annotation></semantics></math>, if there exists such <math id="Thmtheorem1.p1.4.4.m4.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="Thmtheorem1.p1.4.4.m4.1a"><mi id="Thmtheorem1.p1.4.4.m4.1.1" xref="Thmtheorem1.p1.4.4.m4.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="Thmtheorem1.p1.4.4.m4.1b"><ci id="Thmtheorem1.p1.4.4.m4.1.1.cmml" xref="Thmtheorem1.p1.4.4.m4.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem1.p1.4.4.m4.1c">\theta</annotation></semantics></math>.</span></p>
</div>
</div>
<div id="S3.SS1.1" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="S3.SS1.1.p1" class="ltx_para">
<p id="S3.SS1.1.p1.3" class="ltx_p">This theorem directly follows from the proof in <cite class="ltx_cite ltx_citemacro_citet">Gutmann &amp; Hyvärinen (<a href="#bib.bib11" title="" class="ltx_ref">2010</a>)</cite>. Note that at optimum, <math id="S3.SS1.1.p1.1.m1.4" class="ltx_Math" alttext="P_{LM}(x)\exp(-E_{\theta}(x))" display="inline"><semantics id="S3.SS1.1.p1.1.m1.4a"><mrow id="S3.SS1.1.p1.1.m1.4.4" xref="S3.SS1.1.p1.1.m1.4.4.cmml"><msub id="S3.SS1.1.p1.1.m1.4.4.3" xref="S3.SS1.1.p1.1.m1.4.4.3.cmml"><mi id="S3.SS1.1.p1.1.m1.4.4.3.2" xref="S3.SS1.1.p1.1.m1.4.4.3.2.cmml">P</mi><mrow id="S3.SS1.1.p1.1.m1.4.4.3.3" xref="S3.SS1.1.p1.1.m1.4.4.3.3.cmml"><mi id="S3.SS1.1.p1.1.m1.4.4.3.3.2" xref="S3.SS1.1.p1.1.m1.4.4.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.1.p1.1.m1.4.4.3.3.1" xref="S3.SS1.1.p1.1.m1.4.4.3.3.1.cmml">​</mo><mi id="S3.SS1.1.p1.1.m1.4.4.3.3.3" xref="S3.SS1.1.p1.1.m1.4.4.3.3.3.cmml">M</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.SS1.1.p1.1.m1.4.4.2" xref="S3.SS1.1.p1.1.m1.4.4.2.cmml">​</mo><mrow id="S3.SS1.1.p1.1.m1.4.4.4.2" xref="S3.SS1.1.p1.1.m1.4.4.cmml"><mo stretchy="false" id="S3.SS1.1.p1.1.m1.4.4.4.2.1" xref="S3.SS1.1.p1.1.m1.4.4.cmml">(</mo><mi id="S3.SS1.1.p1.1.m1.1.1" xref="S3.SS1.1.p1.1.m1.1.1.cmml">x</mi><mo stretchy="false" id="S3.SS1.1.p1.1.m1.4.4.4.2.2" xref="S3.SS1.1.p1.1.m1.4.4.cmml">)</mo></mrow><mo lspace="0.167em" rspace="0em" id="S3.SS1.1.p1.1.m1.4.4.2a" xref="S3.SS1.1.p1.1.m1.4.4.2.cmml">​</mo><mrow id="S3.SS1.1.p1.1.m1.4.4.1.1" xref="S3.SS1.1.p1.1.m1.4.4.1.2.cmml"><mi id="S3.SS1.1.p1.1.m1.3.3" xref="S3.SS1.1.p1.1.m1.3.3.cmml">exp</mi><mo id="S3.SS1.1.p1.1.m1.4.4.1.1a" xref="S3.SS1.1.p1.1.m1.4.4.1.2.cmml">⁡</mo><mrow id="S3.SS1.1.p1.1.m1.4.4.1.1.1" xref="S3.SS1.1.p1.1.m1.4.4.1.2.cmml"><mo stretchy="false" id="S3.SS1.1.p1.1.m1.4.4.1.1.1.2" xref="S3.SS1.1.p1.1.m1.4.4.1.2.cmml">(</mo><mrow id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.cmml"><mo id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1a" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.cmml">−</mo><mrow id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.cmml"><msub id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2.cmml"><mi id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2.2" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2.2.cmml">E</mi><mi id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2.3" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.1" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.1.cmml">​</mo><mrow id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.3.2" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.3.2.1" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.cmml">(</mo><mi id="S3.SS1.1.p1.1.m1.2.2" xref="S3.SS1.1.p1.1.m1.2.2.cmml">x</mi><mo stretchy="false" id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.3.2.2" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.SS1.1.p1.1.m1.4.4.1.1.1.3" xref="S3.SS1.1.p1.1.m1.4.4.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.1.p1.1.m1.4b"><apply id="S3.SS1.1.p1.1.m1.4.4.cmml" xref="S3.SS1.1.p1.1.m1.4.4"><times id="S3.SS1.1.p1.1.m1.4.4.2.cmml" xref="S3.SS1.1.p1.1.m1.4.4.2"></times><apply id="S3.SS1.1.p1.1.m1.4.4.3.cmml" xref="S3.SS1.1.p1.1.m1.4.4.3"><csymbol cd="ambiguous" id="S3.SS1.1.p1.1.m1.4.4.3.1.cmml" xref="S3.SS1.1.p1.1.m1.4.4.3">subscript</csymbol><ci id="S3.SS1.1.p1.1.m1.4.4.3.2.cmml" xref="S3.SS1.1.p1.1.m1.4.4.3.2">𝑃</ci><apply id="S3.SS1.1.p1.1.m1.4.4.3.3.cmml" xref="S3.SS1.1.p1.1.m1.4.4.3.3"><times id="S3.SS1.1.p1.1.m1.4.4.3.3.1.cmml" xref="S3.SS1.1.p1.1.m1.4.4.3.3.1"></times><ci id="S3.SS1.1.p1.1.m1.4.4.3.3.2.cmml" xref="S3.SS1.1.p1.1.m1.4.4.3.3.2">𝐿</ci><ci id="S3.SS1.1.p1.1.m1.4.4.3.3.3.cmml" xref="S3.SS1.1.p1.1.m1.4.4.3.3.3">𝑀</ci></apply></apply><ci id="S3.SS1.1.p1.1.m1.1.1.cmml" xref="S3.SS1.1.p1.1.m1.1.1">𝑥</ci><apply id="S3.SS1.1.p1.1.m1.4.4.1.2.cmml" xref="S3.SS1.1.p1.1.m1.4.4.1.1"><exp id="S3.SS1.1.p1.1.m1.3.3.cmml" xref="S3.SS1.1.p1.1.m1.3.3"></exp><apply id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.cmml" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1"><minus id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.1.cmml" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1"></minus><apply id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.cmml" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2"><times id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.1.cmml" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.1"></times><apply id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2.cmml" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2.1.cmml" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2">subscript</csymbol><ci id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2.2.cmml" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2.2">𝐸</ci><ci id="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2.3.cmml" xref="S3.SS1.1.p1.1.m1.4.4.1.1.1.1.2.2.3">𝜃</ci></apply><ci id="S3.SS1.1.p1.1.m1.2.2.cmml" xref="S3.SS1.1.p1.1.m1.2.2">𝑥</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.1.p1.1.m1.4c">P_{LM}(x)\exp(-E_{\theta}(x))</annotation></semantics></math> is self-normalizing: instead of <math id="S3.SS1.1.p1.2.m2.5" class="ltx_Math" alttext="P_{\theta}(x)\propto P_{LM}(x)\exp(-E_{\theta}(x))" display="inline"><semantics id="S3.SS1.1.p1.2.m2.5a"><mrow id="S3.SS1.1.p1.2.m2.5.5" xref="S3.SS1.1.p1.2.m2.5.5.cmml"><mrow id="S3.SS1.1.p1.2.m2.5.5.3" xref="S3.SS1.1.p1.2.m2.5.5.3.cmml"><msub id="S3.SS1.1.p1.2.m2.5.5.3.2" xref="S3.SS1.1.p1.2.m2.5.5.3.2.cmml"><mi id="S3.SS1.1.p1.2.m2.5.5.3.2.2" xref="S3.SS1.1.p1.2.m2.5.5.3.2.2.cmml">P</mi><mi id="S3.SS1.1.p1.2.m2.5.5.3.2.3" xref="S3.SS1.1.p1.2.m2.5.5.3.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS1.1.p1.2.m2.5.5.3.1" xref="S3.SS1.1.p1.2.m2.5.5.3.1.cmml">​</mo><mrow id="S3.SS1.1.p1.2.m2.5.5.3.3.2" xref="S3.SS1.1.p1.2.m2.5.5.3.cmml"><mo stretchy="false" id="S3.SS1.1.p1.2.m2.5.5.3.3.2.1" xref="S3.SS1.1.p1.2.m2.5.5.3.cmml">(</mo><mi id="S3.SS1.1.p1.2.m2.1.1" xref="S3.SS1.1.p1.2.m2.1.1.cmml">x</mi><mo stretchy="false" id="S3.SS1.1.p1.2.m2.5.5.3.3.2.2" xref="S3.SS1.1.p1.2.m2.5.5.3.cmml">)</mo></mrow></mrow><mo id="S3.SS1.1.p1.2.m2.5.5.2" xref="S3.SS1.1.p1.2.m2.5.5.2.cmml">∝</mo><mrow id="S3.SS1.1.p1.2.m2.5.5.1" xref="S3.SS1.1.p1.2.m2.5.5.1.cmml"><msub id="S3.SS1.1.p1.2.m2.5.5.1.3" xref="S3.SS1.1.p1.2.m2.5.5.1.3.cmml"><mi id="S3.SS1.1.p1.2.m2.5.5.1.3.2" xref="S3.SS1.1.p1.2.m2.5.5.1.3.2.cmml">P</mi><mrow id="S3.SS1.1.p1.2.m2.5.5.1.3.3" xref="S3.SS1.1.p1.2.m2.5.5.1.3.3.cmml"><mi id="S3.SS1.1.p1.2.m2.5.5.1.3.3.2" xref="S3.SS1.1.p1.2.m2.5.5.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.1.p1.2.m2.5.5.1.3.3.1" xref="S3.SS1.1.p1.2.m2.5.5.1.3.3.1.cmml">​</mo><mi id="S3.SS1.1.p1.2.m2.5.5.1.3.3.3" xref="S3.SS1.1.p1.2.m2.5.5.1.3.3.3.cmml">M</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.SS1.1.p1.2.m2.5.5.1.2" xref="S3.SS1.1.p1.2.m2.5.5.1.2.cmml">​</mo><mrow id="S3.SS1.1.p1.2.m2.5.5.1.4.2" xref="S3.SS1.1.p1.2.m2.5.5.1.cmml"><mo stretchy="false" id="S3.SS1.1.p1.2.m2.5.5.1.4.2.1" xref="S3.SS1.1.p1.2.m2.5.5.1.cmml">(</mo><mi id="S3.SS1.1.p1.2.m2.2.2" xref="S3.SS1.1.p1.2.m2.2.2.cmml">x</mi><mo stretchy="false" id="S3.SS1.1.p1.2.m2.5.5.1.4.2.2" xref="S3.SS1.1.p1.2.m2.5.5.1.cmml">)</mo></mrow><mo lspace="0.167em" rspace="0em" id="S3.SS1.1.p1.2.m2.5.5.1.2a" xref="S3.SS1.1.p1.2.m2.5.5.1.2.cmml">​</mo><mrow id="S3.SS1.1.p1.2.m2.5.5.1.1.1" xref="S3.SS1.1.p1.2.m2.5.5.1.1.2.cmml"><mi id="S3.SS1.1.p1.2.m2.4.4" xref="S3.SS1.1.p1.2.m2.4.4.cmml">exp</mi><mo id="S3.SS1.1.p1.2.m2.5.5.1.1.1a" xref="S3.SS1.1.p1.2.m2.5.5.1.1.2.cmml">⁡</mo><mrow id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1" xref="S3.SS1.1.p1.2.m2.5.5.1.1.2.cmml"><mo stretchy="false" id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.2" xref="S3.SS1.1.p1.2.m2.5.5.1.1.2.cmml">(</mo><mrow id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.cmml"><mo id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1a" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.cmml">−</mo><mrow id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.cmml"><msub id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2.cmml"><mi id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2.2" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2.2.cmml">E</mi><mi id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2.3" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.1" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.1.cmml">​</mo><mrow id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.3.2" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.3.2.1" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.cmml">(</mo><mi id="S3.SS1.1.p1.2.m2.3.3" xref="S3.SS1.1.p1.2.m2.3.3.cmml">x</mi><mo stretchy="false" id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.3.2.2" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.3" xref="S3.SS1.1.p1.2.m2.5.5.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.1.p1.2.m2.5b"><apply id="S3.SS1.1.p1.2.m2.5.5.cmml" xref="S3.SS1.1.p1.2.m2.5.5"><csymbol cd="latexml" id="S3.SS1.1.p1.2.m2.5.5.2.cmml" xref="S3.SS1.1.p1.2.m2.5.5.2">proportional-to</csymbol><apply id="S3.SS1.1.p1.2.m2.5.5.3.cmml" xref="S3.SS1.1.p1.2.m2.5.5.3"><times id="S3.SS1.1.p1.2.m2.5.5.3.1.cmml" xref="S3.SS1.1.p1.2.m2.5.5.3.1"></times><apply id="S3.SS1.1.p1.2.m2.5.5.3.2.cmml" xref="S3.SS1.1.p1.2.m2.5.5.3.2"><csymbol cd="ambiguous" id="S3.SS1.1.p1.2.m2.5.5.3.2.1.cmml" xref="S3.SS1.1.p1.2.m2.5.5.3.2">subscript</csymbol><ci id="S3.SS1.1.p1.2.m2.5.5.3.2.2.cmml" xref="S3.SS1.1.p1.2.m2.5.5.3.2.2">𝑃</ci><ci id="S3.SS1.1.p1.2.m2.5.5.3.2.3.cmml" xref="S3.SS1.1.p1.2.m2.5.5.3.2.3">𝜃</ci></apply><ci id="S3.SS1.1.p1.2.m2.1.1.cmml" xref="S3.SS1.1.p1.2.m2.1.1">𝑥</ci></apply><apply id="S3.SS1.1.p1.2.m2.5.5.1.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1"><times id="S3.SS1.1.p1.2.m2.5.5.1.2.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.2"></times><apply id="S3.SS1.1.p1.2.m2.5.5.1.3.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.3"><csymbol cd="ambiguous" id="S3.SS1.1.p1.2.m2.5.5.1.3.1.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.3">subscript</csymbol><ci id="S3.SS1.1.p1.2.m2.5.5.1.3.2.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.3.2">𝑃</ci><apply id="S3.SS1.1.p1.2.m2.5.5.1.3.3.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.3.3"><times id="S3.SS1.1.p1.2.m2.5.5.1.3.3.1.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.3.3.1"></times><ci id="S3.SS1.1.p1.2.m2.5.5.1.3.3.2.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.3.3.2">𝐿</ci><ci id="S3.SS1.1.p1.2.m2.5.5.1.3.3.3.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.3.3.3">𝑀</ci></apply></apply><ci id="S3.SS1.1.p1.2.m2.2.2.cmml" xref="S3.SS1.1.p1.2.m2.2.2">𝑥</ci><apply id="S3.SS1.1.p1.2.m2.5.5.1.1.2.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1"><exp id="S3.SS1.1.p1.2.m2.4.4.cmml" xref="S3.SS1.1.p1.2.m2.4.4"></exp><apply id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1"><minus id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.1.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1"></minus><apply id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2"><times id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.1.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.1"></times><apply id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2.1.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2">subscript</csymbol><ci id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2.2.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2.2">𝐸</ci><ci id="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2.3.cmml" xref="S3.SS1.1.p1.2.m2.5.5.1.1.1.1.1.2.2.3">𝜃</ci></apply><ci id="S3.SS1.1.p1.2.m2.3.3.cmml" xref="S3.SS1.1.p1.2.m2.3.3">𝑥</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.1.p1.2.m2.5c">P_{\theta}(x)\propto P_{LM}(x)\exp(-E_{\theta}(x))</annotation></semantics></math>, we have <math id="S3.SS1.1.p1.3.m3.5" class="ltx_Math" alttext="P_{\theta}(x)=P_{LM}(x)\exp(-E_{\theta}(x))" display="inline"><semantics id="S3.SS1.1.p1.3.m3.5a"><mrow id="S3.SS1.1.p1.3.m3.5.5" xref="S3.SS1.1.p1.3.m3.5.5.cmml"><mrow id="S3.SS1.1.p1.3.m3.5.5.3" xref="S3.SS1.1.p1.3.m3.5.5.3.cmml"><msub id="S3.SS1.1.p1.3.m3.5.5.3.2" xref="S3.SS1.1.p1.3.m3.5.5.3.2.cmml"><mi id="S3.SS1.1.p1.3.m3.5.5.3.2.2" xref="S3.SS1.1.p1.3.m3.5.5.3.2.2.cmml">P</mi><mi id="S3.SS1.1.p1.3.m3.5.5.3.2.3" xref="S3.SS1.1.p1.3.m3.5.5.3.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS1.1.p1.3.m3.5.5.3.1" xref="S3.SS1.1.p1.3.m3.5.5.3.1.cmml">​</mo><mrow id="S3.SS1.1.p1.3.m3.5.5.3.3.2" xref="S3.SS1.1.p1.3.m3.5.5.3.cmml"><mo stretchy="false" id="S3.SS1.1.p1.3.m3.5.5.3.3.2.1" xref="S3.SS1.1.p1.3.m3.5.5.3.cmml">(</mo><mi id="S3.SS1.1.p1.3.m3.1.1" xref="S3.SS1.1.p1.3.m3.1.1.cmml">x</mi><mo stretchy="false" id="S3.SS1.1.p1.3.m3.5.5.3.3.2.2" xref="S3.SS1.1.p1.3.m3.5.5.3.cmml">)</mo></mrow></mrow><mo id="S3.SS1.1.p1.3.m3.5.5.2" xref="S3.SS1.1.p1.3.m3.5.5.2.cmml">=</mo><mrow id="S3.SS1.1.p1.3.m3.5.5.1" xref="S3.SS1.1.p1.3.m3.5.5.1.cmml"><msub id="S3.SS1.1.p1.3.m3.5.5.1.3" xref="S3.SS1.1.p1.3.m3.5.5.1.3.cmml"><mi id="S3.SS1.1.p1.3.m3.5.5.1.3.2" xref="S3.SS1.1.p1.3.m3.5.5.1.3.2.cmml">P</mi><mrow id="S3.SS1.1.p1.3.m3.5.5.1.3.3" xref="S3.SS1.1.p1.3.m3.5.5.1.3.3.cmml"><mi id="S3.SS1.1.p1.3.m3.5.5.1.3.3.2" xref="S3.SS1.1.p1.3.m3.5.5.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.1.p1.3.m3.5.5.1.3.3.1" xref="S3.SS1.1.p1.3.m3.5.5.1.3.3.1.cmml">​</mo><mi id="S3.SS1.1.p1.3.m3.5.5.1.3.3.3" xref="S3.SS1.1.p1.3.m3.5.5.1.3.3.3.cmml">M</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.SS1.1.p1.3.m3.5.5.1.2" xref="S3.SS1.1.p1.3.m3.5.5.1.2.cmml">​</mo><mrow id="S3.SS1.1.p1.3.m3.5.5.1.4.2" xref="S3.SS1.1.p1.3.m3.5.5.1.cmml"><mo stretchy="false" id="S3.SS1.1.p1.3.m3.5.5.1.4.2.1" xref="S3.SS1.1.p1.3.m3.5.5.1.cmml">(</mo><mi id="S3.SS1.1.p1.3.m3.2.2" xref="S3.SS1.1.p1.3.m3.2.2.cmml">x</mi><mo stretchy="false" id="S3.SS1.1.p1.3.m3.5.5.1.4.2.2" xref="S3.SS1.1.p1.3.m3.5.5.1.cmml">)</mo></mrow><mo lspace="0.167em" rspace="0em" id="S3.SS1.1.p1.3.m3.5.5.1.2a" xref="S3.SS1.1.p1.3.m3.5.5.1.2.cmml">​</mo><mrow id="S3.SS1.1.p1.3.m3.5.5.1.1.1" xref="S3.SS1.1.p1.3.m3.5.5.1.1.2.cmml"><mi id="S3.SS1.1.p1.3.m3.4.4" xref="S3.SS1.1.p1.3.m3.4.4.cmml">exp</mi><mo id="S3.SS1.1.p1.3.m3.5.5.1.1.1a" xref="S3.SS1.1.p1.3.m3.5.5.1.1.2.cmml">⁡</mo><mrow id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1" xref="S3.SS1.1.p1.3.m3.5.5.1.1.2.cmml"><mo stretchy="false" id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.2" xref="S3.SS1.1.p1.3.m3.5.5.1.1.2.cmml">(</mo><mrow id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.cmml"><mo id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1a" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.cmml">−</mo><mrow id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.cmml"><msub id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2.cmml"><mi id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2.2" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2.2.cmml">E</mi><mi id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2.3" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.1" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.1.cmml">​</mo><mrow id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.3.2" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.3.2.1" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.cmml">(</mo><mi id="S3.SS1.1.p1.3.m3.3.3" xref="S3.SS1.1.p1.3.m3.3.3.cmml">x</mi><mo stretchy="false" id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.3.2.2" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.3" xref="S3.SS1.1.p1.3.m3.5.5.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.1.p1.3.m3.5b"><apply id="S3.SS1.1.p1.3.m3.5.5.cmml" xref="S3.SS1.1.p1.3.m3.5.5"><eq id="S3.SS1.1.p1.3.m3.5.5.2.cmml" xref="S3.SS1.1.p1.3.m3.5.5.2"></eq><apply id="S3.SS1.1.p1.3.m3.5.5.3.cmml" xref="S3.SS1.1.p1.3.m3.5.5.3"><times id="S3.SS1.1.p1.3.m3.5.5.3.1.cmml" xref="S3.SS1.1.p1.3.m3.5.5.3.1"></times><apply id="S3.SS1.1.p1.3.m3.5.5.3.2.cmml" xref="S3.SS1.1.p1.3.m3.5.5.3.2"><csymbol cd="ambiguous" id="S3.SS1.1.p1.3.m3.5.5.3.2.1.cmml" xref="S3.SS1.1.p1.3.m3.5.5.3.2">subscript</csymbol><ci id="S3.SS1.1.p1.3.m3.5.5.3.2.2.cmml" xref="S3.SS1.1.p1.3.m3.5.5.3.2.2">𝑃</ci><ci id="S3.SS1.1.p1.3.m3.5.5.3.2.3.cmml" xref="S3.SS1.1.p1.3.m3.5.5.3.2.3">𝜃</ci></apply><ci id="S3.SS1.1.p1.3.m3.1.1.cmml" xref="S3.SS1.1.p1.3.m3.1.1">𝑥</ci></apply><apply id="S3.SS1.1.p1.3.m3.5.5.1.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1"><times id="S3.SS1.1.p1.3.m3.5.5.1.2.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.2"></times><apply id="S3.SS1.1.p1.3.m3.5.5.1.3.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.3"><csymbol cd="ambiguous" id="S3.SS1.1.p1.3.m3.5.5.1.3.1.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.3">subscript</csymbol><ci id="S3.SS1.1.p1.3.m3.5.5.1.3.2.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.3.2">𝑃</ci><apply id="S3.SS1.1.p1.3.m3.5.5.1.3.3.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.3.3"><times id="S3.SS1.1.p1.3.m3.5.5.1.3.3.1.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.3.3.1"></times><ci id="S3.SS1.1.p1.3.m3.5.5.1.3.3.2.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.3.3.2">𝐿</ci><ci id="S3.SS1.1.p1.3.m3.5.5.1.3.3.3.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.3.3.3">𝑀</ci></apply></apply><ci id="S3.SS1.1.p1.3.m3.2.2.cmml" xref="S3.SS1.1.p1.3.m3.2.2">𝑥</ci><apply id="S3.SS1.1.p1.3.m3.5.5.1.1.2.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1"><exp id="S3.SS1.1.p1.3.m3.4.4.cmml" xref="S3.SS1.1.p1.3.m3.4.4"></exp><apply id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1"><minus id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.1.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1"></minus><apply id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2"><times id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.1.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.1"></times><apply id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2.1.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2">subscript</csymbol><ci id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2.2.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2.2">𝐸</ci><ci id="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2.3.cmml" xref="S3.SS1.1.p1.3.m3.5.5.1.1.1.1.1.2.2.3">𝜃</ci></apply><ci id="S3.SS1.1.p1.3.m3.3.3.cmml" xref="S3.SS1.1.p1.3.m3.3.3">𝑥</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.1.p1.3.m3.5c">P_{\theta}(x)=P_{LM}(x)\exp(-E_{\theta}(x))</annotation></semantics></math>. However, we still need to estimate the partition function throughout the rest of this paper, since we cannot guarantee that this optimum can be reached.
∎</p>
</div>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Evaluation</h3>

<div id="S3.SS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.p1.3" class="ltx_p">A commonly used protocol for evaluating generative sequence models, especially language models, is perplexity (PPL), which is
equal to <math id="S3.SS2.p1.1.m1.2" class="ltx_Math" alttext="2^{-\frac{1}{T-p}\sum_{i=p+1}^{T}\log_{2}P(x_{i}|x_{i-1},\cdots,x_{1})}" display="inline"><semantics id="S3.SS2.p1.1.m1.2a"><msup id="S3.SS2.p1.1.m1.2.3" xref="S3.SS2.p1.1.m1.2.3.cmml"><mn id="S3.SS2.p1.1.m1.2.3.2" xref="S3.SS2.p1.1.m1.2.3.2.cmml">2</mn><mrow id="S3.SS2.p1.1.m1.2.2.2" xref="S3.SS2.p1.1.m1.2.2.2.cmml"><mo id="S3.SS2.p1.1.m1.2.2.2a" xref="S3.SS2.p1.1.m1.2.2.2.cmml">−</mo><mrow id="S3.SS2.p1.1.m1.2.2.2.2" xref="S3.SS2.p1.1.m1.2.2.2.2.cmml"><mfrac id="S3.SS2.p1.1.m1.2.2.2.2.3" xref="S3.SS2.p1.1.m1.2.2.2.2.3.cmml"><mn id="S3.SS2.p1.1.m1.2.2.2.2.3.2" xref="S3.SS2.p1.1.m1.2.2.2.2.3.2.cmml">1</mn><mrow id="S3.SS2.p1.1.m1.2.2.2.2.3.3" xref="S3.SS2.p1.1.m1.2.2.2.2.3.3.cmml"><mi id="S3.SS2.p1.1.m1.2.2.2.2.3.3.2" xref="S3.SS2.p1.1.m1.2.2.2.2.3.3.2.cmml">T</mi><mo id="S3.SS2.p1.1.m1.2.2.2.2.3.3.1" xref="S3.SS2.p1.1.m1.2.2.2.2.3.3.1.cmml">−</mo><mi id="S3.SS2.p1.1.m1.2.2.2.2.3.3.3" xref="S3.SS2.p1.1.m1.2.2.2.2.3.3.3.cmml">p</mi></mrow></mfrac><mo lspace="0em" rspace="0em" id="S3.SS2.p1.1.m1.2.2.2.2.2" xref="S3.SS2.p1.1.m1.2.2.2.2.2.cmml">​</mo><mrow id="S3.SS2.p1.1.m1.2.2.2.2.1" xref="S3.SS2.p1.1.m1.2.2.2.2.1.cmml"><mstyle displaystyle="false" id="S3.SS2.p1.1.m1.2.2.2.2.1.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.cmml"><msubsup id="S3.SS2.p1.1.m1.2.2.2.2.1.2a" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.cmml"><mo id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.2.cmml">∑</mo><mrow id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.cmml"><mi id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.2.cmml">i</mi><mo id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.1" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.1.cmml">=</mo><mrow id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.cmml"><mi id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.2.cmml">p</mi><mo id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.1" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.1.cmml">+</mo><mn id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.3.cmml">1</mn></mrow></mrow><mi id="S3.SS2.p1.1.m1.2.2.2.2.1.2.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.3.cmml">T</mi></msubsup></mstyle><mrow id="S3.SS2.p1.1.m1.2.2.2.2.1.1" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.cmml"><mrow id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.cmml"><msub id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1.cmml"><mi id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1.2.cmml">log</mi><mn id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1.3.cmml">2</mn></msub><mo lspace="0.167em" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3a" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.cmml">⁡</mo><mi id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.2.cmml">P</mi></mrow><mo lspace="0em" rspace="0em" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.cmml"><msub id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4.cmml"><mi id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4.2.cmml">x</mi><mi id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4.3.cmml">i</mi></msub><mo fence="false" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.3.cmml">|</mo><mrow id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.3.cmml"><msub id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.2.cmml">x</mi><mrow id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.2.cmml">i</mi><mo id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.1" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.1.cmml">−</mo><mn id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.SS2.p1.1.m1.1.1.1.1" xref="S3.SS2.p1.1.m1.1.1.1.1.cmml">⋯</mi><mo id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.4" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.3.cmml">,</mo><msub id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2.cmml"><mi id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2.2" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2.2.cmml">x</mi><mn id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2.3.cmml">1</mn></msub></mrow></mrow><mo stretchy="false" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.3" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.2b"><apply id="S3.SS2.p1.1.m1.2.3.cmml" xref="S3.SS2.p1.1.m1.2.3"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.2.3.1.cmml" xref="S3.SS2.p1.1.m1.2.3">superscript</csymbol><cn type="integer" id="S3.SS2.p1.1.m1.2.3.2.cmml" xref="S3.SS2.p1.1.m1.2.3.2">2</cn><apply id="S3.SS2.p1.1.m1.2.2.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2"><minus id="S3.SS2.p1.1.m1.2.2.2.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2"></minus><apply id="S3.SS2.p1.1.m1.2.2.2.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2"><times id="S3.SS2.p1.1.m1.2.2.2.2.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.2"></times><apply id="S3.SS2.p1.1.m1.2.2.2.2.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.3"><divide id="S3.SS2.p1.1.m1.2.2.2.2.3.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.3"></divide><cn type="integer" id="S3.SS2.p1.1.m1.2.2.2.2.3.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.3.2">1</cn><apply id="S3.SS2.p1.1.m1.2.2.2.2.3.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.3.3"><minus id="S3.SS2.p1.1.m1.2.2.2.2.3.3.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.3.3.1"></minus><ci id="S3.SS2.p1.1.m1.2.2.2.2.3.3.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.3.3.2">𝑇</ci><ci id="S3.SS2.p1.1.m1.2.2.2.2.3.3.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.3.3.3">𝑝</ci></apply></apply><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1"><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.2.2.2.2.1.2.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2">superscript</csymbol><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2">subscript</csymbol><sum id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.2"></sum><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3"><eq id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.1"></eq><ci id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.2">𝑖</ci><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3"><plus id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.1"></plus><ci id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.2">𝑝</ci><cn type="integer" id="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.2.3.3.3">1</cn></apply></apply></apply><ci id="S3.SS2.p1.1.m1.2.2.2.2.1.2.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.2.3">𝑇</ci></apply><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1"><times id="S3.SS2.p1.1.m1.2.2.2.2.1.1.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.2"></times><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3"><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1">subscript</csymbol><log id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1.2"></log><cn type="integer" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.1.3">2</cn></apply><ci id="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.3.2">𝑃</ci></apply><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1"><csymbol cd="latexml" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.3">conditional</csymbol><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4">subscript</csymbol><ci id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4.2">𝑥</ci><ci id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.4.3">𝑖</ci></apply><list id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2"><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.2">𝑥</ci><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3"><minus id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.1"></minus><ci id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.3">1</cn></apply></apply><ci id="S3.SS2.p1.1.m1.1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1.1.1">⋯</ci><apply id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2.1.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2">subscript</csymbol><ci id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2.2.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2.2">𝑥</ci><cn type="integer" id="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2.3.cmml" xref="S3.SS2.p1.1.m1.2.2.2.2.1.1.1.1.1.2.2.2.3">1</cn></apply></list></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.2c">2^{-\frac{1}{T-p}\sum_{i=p+1}^{T}\log_{2}P(x_{i}|x_{i-1},\cdots,x_{1})}</annotation></semantics></math>. PPL can be interpreted as the average number of
tokens the model is uncertain of at every time step.
Since the log-likelihood required by PPL relies on estimating the partition function <math id="S3.SS2.p1.2.m2.7" class="ltx_Math" alttext="Z_{\theta}=\sum_{x}P_{LM}(x)\exp(-E_{\theta}(x))=\mathbb{E}_{x\sim P_{LM}}\exp(-E_{\theta}(x))" display="inline"><semantics id="S3.SS2.p1.2.m2.7a"><mrow id="S3.SS2.p1.2.m2.7.7" xref="S3.SS2.p1.2.m2.7.7.cmml"><msub id="S3.SS2.p1.2.m2.7.7.4" xref="S3.SS2.p1.2.m2.7.7.4.cmml"><mi id="S3.SS2.p1.2.m2.7.7.4.2" xref="S3.SS2.p1.2.m2.7.7.4.2.cmml">Z</mi><mi id="S3.SS2.p1.2.m2.7.7.4.3" xref="S3.SS2.p1.2.m2.7.7.4.3.cmml">θ</mi></msub><mo rspace="0.111em" id="S3.SS2.p1.2.m2.7.7.5" xref="S3.SS2.p1.2.m2.7.7.5.cmml">=</mo><mrow id="S3.SS2.p1.2.m2.6.6.1" xref="S3.SS2.p1.2.m2.6.6.1.cmml"><msub id="S3.SS2.p1.2.m2.6.6.1.2" xref="S3.SS2.p1.2.m2.6.6.1.2.cmml"><mo id="S3.SS2.p1.2.m2.6.6.1.2.2" xref="S3.SS2.p1.2.m2.6.6.1.2.2.cmml">∑</mo><mi id="S3.SS2.p1.2.m2.6.6.1.2.3" xref="S3.SS2.p1.2.m2.6.6.1.2.3.cmml">x</mi></msub><mrow id="S3.SS2.p1.2.m2.6.6.1.1" xref="S3.SS2.p1.2.m2.6.6.1.1.cmml"><msub id="S3.SS2.p1.2.m2.6.6.1.1.3" xref="S3.SS2.p1.2.m2.6.6.1.1.3.cmml"><mi id="S3.SS2.p1.2.m2.6.6.1.1.3.2" xref="S3.SS2.p1.2.m2.6.6.1.1.3.2.cmml">P</mi><mrow id="S3.SS2.p1.2.m2.6.6.1.1.3.3" xref="S3.SS2.p1.2.m2.6.6.1.1.3.3.cmml"><mi id="S3.SS2.p1.2.m2.6.6.1.1.3.3.2" xref="S3.SS2.p1.2.m2.6.6.1.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.6.6.1.1.3.3.1" xref="S3.SS2.p1.2.m2.6.6.1.1.3.3.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.6.6.1.1.3.3.3" xref="S3.SS2.p1.2.m2.6.6.1.1.3.3.3.cmml">M</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.6.6.1.1.2" xref="S3.SS2.p1.2.m2.6.6.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.2.m2.6.6.1.1.4.2" xref="S3.SS2.p1.2.m2.6.6.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.2.m2.6.6.1.1.4.2.1" xref="S3.SS2.p1.2.m2.6.6.1.1.cmml">(</mo><mi id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml">x</mi><mo stretchy="false" id="S3.SS2.p1.2.m2.6.6.1.1.4.2.2" xref="S3.SS2.p1.2.m2.6.6.1.1.cmml">)</mo></mrow><mo lspace="0.167em" rspace="0em" id="S3.SS2.p1.2.m2.6.6.1.1.2a" xref="S3.SS2.p1.2.m2.6.6.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.2.m2.6.6.1.1.1.1" xref="S3.SS2.p1.2.m2.6.6.1.1.1.2.cmml"><mi id="S3.SS2.p1.2.m2.3.3" xref="S3.SS2.p1.2.m2.3.3.cmml">exp</mi><mo id="S3.SS2.p1.2.m2.6.6.1.1.1.1a" xref="S3.SS2.p1.2.m2.6.6.1.1.1.2.cmml">⁡</mo><mrow id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1" xref="S3.SS2.p1.2.m2.6.6.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.2" xref="S3.SS2.p1.2.m2.6.6.1.1.1.2.cmml">(</mo><mrow id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.cmml"><mo id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1a" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.cmml">−</mo><mrow id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.cmml"><msub id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2.cmml"><mi id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2.2" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2.2.cmml">E</mi><mi id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2.3" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.1" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.1.cmml">​</mo><mrow id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.3.2" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.3.2.1" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.cmml">(</mo><mi id="S3.SS2.p1.2.m2.2.2" xref="S3.SS2.p1.2.m2.2.2.cmml">x</mi><mo stretchy="false" id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.3.2.2" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.3" xref="S3.SS2.p1.2.m2.6.6.1.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><mo id="S3.SS2.p1.2.m2.7.7.6" xref="S3.SS2.p1.2.m2.7.7.6.cmml">=</mo><mrow id="S3.SS2.p1.2.m2.7.7.2" xref="S3.SS2.p1.2.m2.7.7.2.cmml"><msub id="S3.SS2.p1.2.m2.7.7.2.3" xref="S3.SS2.p1.2.m2.7.7.2.3.cmml"><mi id="S3.SS2.p1.2.m2.7.7.2.3.2" xref="S3.SS2.p1.2.m2.7.7.2.3.2.cmml">𝔼</mi><mrow id="S3.SS2.p1.2.m2.7.7.2.3.3" xref="S3.SS2.p1.2.m2.7.7.2.3.3.cmml"><mi id="S3.SS2.p1.2.m2.7.7.2.3.3.2" xref="S3.SS2.p1.2.m2.7.7.2.3.3.2.cmml">x</mi><mo id="S3.SS2.p1.2.m2.7.7.2.3.3.1" xref="S3.SS2.p1.2.m2.7.7.2.3.3.1.cmml">∼</mo><msub id="S3.SS2.p1.2.m2.7.7.2.3.3.3" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3.cmml"><mi id="S3.SS2.p1.2.m2.7.7.2.3.3.3.2" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3.2.cmml">P</mi><mrow id="S3.SS2.p1.2.m2.7.7.2.3.3.3.3" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.cmml"><mi id="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.2" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.1" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.3" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.3.cmml">M</mi></mrow></msub></mrow></msub><mo lspace="0.167em" rspace="0em" id="S3.SS2.p1.2.m2.7.7.2.2" xref="S3.SS2.p1.2.m2.7.7.2.2.cmml">​</mo><mrow id="S3.SS2.p1.2.m2.7.7.2.1.1" xref="S3.SS2.p1.2.m2.7.7.2.1.2.cmml"><mi id="S3.SS2.p1.2.m2.5.5" xref="S3.SS2.p1.2.m2.5.5.cmml">exp</mi><mo id="S3.SS2.p1.2.m2.7.7.2.1.1a" xref="S3.SS2.p1.2.m2.7.7.2.1.2.cmml">⁡</mo><mrow id="S3.SS2.p1.2.m2.7.7.2.1.1.1" xref="S3.SS2.p1.2.m2.7.7.2.1.2.cmml"><mo stretchy="false" id="S3.SS2.p1.2.m2.7.7.2.1.1.1.2" xref="S3.SS2.p1.2.m2.7.7.2.1.2.cmml">(</mo><mrow id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.cmml"><mo id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1a" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.cmml">−</mo><mrow id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.cmml"><msub id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2.cmml"><mi id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2.2" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2.2.cmml">E</mi><mi id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2.3" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.1" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.1.cmml">​</mo><mrow id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.3.2" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.3.2.1" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.cmml">(</mo><mi id="S3.SS2.p1.2.m2.4.4" xref="S3.SS2.p1.2.m2.4.4.cmml">x</mi><mo stretchy="false" id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.3.2.2" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.SS2.p1.2.m2.7.7.2.1.1.1.3" xref="S3.SS2.p1.2.m2.7.7.2.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.7b"><apply id="S3.SS2.p1.2.m2.7.7.cmml" xref="S3.SS2.p1.2.m2.7.7"><and id="S3.SS2.p1.2.m2.7.7a.cmml" xref="S3.SS2.p1.2.m2.7.7"></and><apply id="S3.SS2.p1.2.m2.7.7b.cmml" xref="S3.SS2.p1.2.m2.7.7"><eq id="S3.SS2.p1.2.m2.7.7.5.cmml" xref="S3.SS2.p1.2.m2.7.7.5"></eq><apply id="S3.SS2.p1.2.m2.7.7.4.cmml" xref="S3.SS2.p1.2.m2.7.7.4"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.7.7.4.1.cmml" xref="S3.SS2.p1.2.m2.7.7.4">subscript</csymbol><ci id="S3.SS2.p1.2.m2.7.7.4.2.cmml" xref="S3.SS2.p1.2.m2.7.7.4.2">𝑍</ci><ci id="S3.SS2.p1.2.m2.7.7.4.3.cmml" xref="S3.SS2.p1.2.m2.7.7.4.3">𝜃</ci></apply><apply id="S3.SS2.p1.2.m2.6.6.1.cmml" xref="S3.SS2.p1.2.m2.6.6.1"><apply id="S3.SS2.p1.2.m2.6.6.1.2.cmml" xref="S3.SS2.p1.2.m2.6.6.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.6.6.1.2.1.cmml" xref="S3.SS2.p1.2.m2.6.6.1.2">subscript</csymbol><sum id="S3.SS2.p1.2.m2.6.6.1.2.2.cmml" xref="S3.SS2.p1.2.m2.6.6.1.2.2"></sum><ci id="S3.SS2.p1.2.m2.6.6.1.2.3.cmml" xref="S3.SS2.p1.2.m2.6.6.1.2.3">𝑥</ci></apply><apply id="S3.SS2.p1.2.m2.6.6.1.1.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1"><times id="S3.SS2.p1.2.m2.6.6.1.1.2.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.2"></times><apply id="S3.SS2.p1.2.m2.6.6.1.1.3.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.6.6.1.1.3.1.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.3">subscript</csymbol><ci id="S3.SS2.p1.2.m2.6.6.1.1.3.2.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.3.2">𝑃</ci><apply id="S3.SS2.p1.2.m2.6.6.1.1.3.3.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.3.3"><times id="S3.SS2.p1.2.m2.6.6.1.1.3.3.1.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.3.3.1"></times><ci id="S3.SS2.p1.2.m2.6.6.1.1.3.3.2.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.3.3.2">𝐿</ci><ci id="S3.SS2.p1.2.m2.6.6.1.1.3.3.3.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.3.3.3">𝑀</ci></apply></apply><ci id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">𝑥</ci><apply id="S3.SS2.p1.2.m2.6.6.1.1.1.2.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1"><exp id="S3.SS2.p1.2.m2.3.3.cmml" xref="S3.SS2.p1.2.m2.3.3"></exp><apply id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1"><minus id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1"></minus><apply id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2"><times id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.1.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.1"></times><apply id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2.1.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2.2.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2.2">𝐸</ci><ci id="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2.3.cmml" xref="S3.SS2.p1.2.m2.6.6.1.1.1.1.1.1.2.2.3">𝜃</ci></apply><ci id="S3.SS2.p1.2.m2.2.2.cmml" xref="S3.SS2.p1.2.m2.2.2">𝑥</ci></apply></apply></apply></apply></apply></apply><apply id="S3.SS2.p1.2.m2.7.7c.cmml" xref="S3.SS2.p1.2.m2.7.7"><eq id="S3.SS2.p1.2.m2.7.7.6.cmml" xref="S3.SS2.p1.2.m2.7.7.6"></eq><share href="#S3.SS2.p1.2.m2.6.6.1.cmml" id="S3.SS2.p1.2.m2.7.7d.cmml" xref="S3.SS2.p1.2.m2.7.7"></share><apply id="S3.SS2.p1.2.m2.7.7.2.cmml" xref="S3.SS2.p1.2.m2.7.7.2"><times id="S3.SS2.p1.2.m2.7.7.2.2.cmml" xref="S3.SS2.p1.2.m2.7.7.2.2"></times><apply id="S3.SS2.p1.2.m2.7.7.2.3.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.7.7.2.3.1.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3">subscript</csymbol><ci id="S3.SS2.p1.2.m2.7.7.2.3.2.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3.2">𝔼</ci><apply id="S3.SS2.p1.2.m2.7.7.2.3.3.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3.3"><csymbol cd="latexml" id="S3.SS2.p1.2.m2.7.7.2.3.3.1.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3.3.1">similar-to</csymbol><ci id="S3.SS2.p1.2.m2.7.7.2.3.3.2.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3.3.2">𝑥</ci><apply id="S3.SS2.p1.2.m2.7.7.2.3.3.3.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.7.7.2.3.3.3.1.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3">subscript</csymbol><ci id="S3.SS2.p1.2.m2.7.7.2.3.3.3.2.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3.2">𝑃</ci><apply id="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3.3"><times id="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.1.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.1"></times><ci id="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.2.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.2">𝐿</ci><ci id="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.3.cmml" xref="S3.SS2.p1.2.m2.7.7.2.3.3.3.3.3">𝑀</ci></apply></apply></apply></apply><apply id="S3.SS2.p1.2.m2.7.7.2.1.2.cmml" xref="S3.SS2.p1.2.m2.7.7.2.1.1"><exp id="S3.SS2.p1.2.m2.5.5.cmml" xref="S3.SS2.p1.2.m2.5.5"></exp><apply id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1"><minus id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1"></minus><apply id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.cmml" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2"><times id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.1.cmml" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.1"></times><apply id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2.cmml" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2.1.cmml" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2">subscript</csymbol><ci id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2.2.cmml" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2.2">𝐸</ci><ci id="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2.3.cmml" xref="S3.SS2.p1.2.m2.7.7.2.1.1.1.1.2.2.3">𝜃</ci></apply><ci id="S3.SS2.p1.2.m2.4.4.cmml" xref="S3.SS2.p1.2.m2.4.4">𝑥</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.7c">Z_{\theta}=\sum_{x}P_{LM}(x)\exp(-E_{\theta}(x))=\mathbb{E}_{x\sim P_{LM}}\exp(-E_{\theta}(x))</annotation></semantics></math>, we derive two estimators for the log-partition function <math id="S3.SS2.p1.3.m3.1" class="ltx_Math" alttext="\log Z_{\theta}" display="inline"><semantics id="S3.SS2.p1.3.m3.1a"><mrow id="S3.SS2.p1.3.m3.1.1" xref="S3.SS2.p1.3.m3.1.1.cmml"><mi id="S3.SS2.p1.3.m3.1.1.1" xref="S3.SS2.p1.3.m3.1.1.1.cmml">log</mi><mo lspace="0.167em" id="S3.SS2.p1.3.m3.1.1a" xref="S3.SS2.p1.3.m3.1.1.cmml">⁡</mo><msub id="S3.SS2.p1.3.m3.1.1.2" xref="S3.SS2.p1.3.m3.1.1.2.cmml"><mi id="S3.SS2.p1.3.m3.1.1.2.2" xref="S3.SS2.p1.3.m3.1.1.2.2.cmml">Z</mi><mi id="S3.SS2.p1.3.m3.1.1.2.3" xref="S3.SS2.p1.3.m3.1.1.2.3.cmml">θ</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.1b"><apply id="S3.SS2.p1.3.m3.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1"><log id="S3.SS2.p1.3.m3.1.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1.1"></log><apply id="S3.SS2.p1.3.m3.1.1.2.cmml" xref="S3.SS2.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.3.m3.1.1.2.1.cmml" xref="S3.SS2.p1.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS2.p1.3.m3.1.1.2.2.cmml" xref="S3.SS2.p1.3.m3.1.1.2.2">𝑍</ci><ci id="S3.SS2.p1.3.m3.1.1.2.3.cmml" xref="S3.SS2.p1.3.m3.1.1.2.3">𝜃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.1c">\log Z_{\theta}</annotation></semantics></math> based on the work of <cite class="ltx_cite ltx_citemacro_citet">Nowozin (<a href="#bib.bib23" title="" class="ltx_ref">2018</a>)</cite>.</p>
</div>
<div id="Thmtheorem2" class="ltx_theorem ltx_theorem_theorem">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmtheorem2.1.1.1" class="ltx_text ltx_font_bold">Theorem 2</span></span><span id="Thmtheorem2.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmtheorem2.p1" class="ltx_para">
<p id="Thmtheorem2.p1.8" class="ltx_p"><span id="Thmtheorem2.p1.8.8" class="ltx_text ltx_font_italic">Denote <math id="Thmtheorem2.p1.1.1.m1.1" class="ltx_Math" alttext="T_{n}" display="inline"><semantics id="Thmtheorem2.p1.1.1.m1.1a"><msub id="Thmtheorem2.p1.1.1.m1.1.1" xref="Thmtheorem2.p1.1.1.m1.1.1.cmml"><mi id="Thmtheorem2.p1.1.1.m1.1.1.2" xref="Thmtheorem2.p1.1.1.m1.1.1.2.cmml">T</mi><mi id="Thmtheorem2.p1.1.1.m1.1.1.3" xref="Thmtheorem2.p1.1.1.m1.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.1.1.m1.1b"><apply id="Thmtheorem2.p1.1.1.m1.1.1.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1"><csymbol cd="ambiguous" id="Thmtheorem2.p1.1.1.m1.1.1.1.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1">subscript</csymbol><ci id="Thmtheorem2.p1.1.1.m1.1.1.2.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.2">𝑇</ci><ci id="Thmtheorem2.p1.1.1.m1.1.1.3.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.1.1.m1.1c">T_{n}</annotation></semantics></math> as the empirical estimate of <math id="Thmtheorem2.p1.2.2.m2.3" class="ltx_Math" alttext="\log\mathbb{E}_{x\sim P_{LM}}\exp(-E(x))" display="inline"><semantics id="Thmtheorem2.p1.2.2.m2.3a"><mrow id="Thmtheorem2.p1.2.2.m2.3.3" xref="Thmtheorem2.p1.2.2.m2.3.3.cmml"><mi id="Thmtheorem2.p1.2.2.m2.3.3.2" xref="Thmtheorem2.p1.2.2.m2.3.3.2.cmml">log</mi><mo lspace="0.167em" id="Thmtheorem2.p1.2.2.m2.3.3a" xref="Thmtheorem2.p1.2.2.m2.3.3.cmml">⁡</mo><mrow id="Thmtheorem2.p1.2.2.m2.3.3.1" xref="Thmtheorem2.p1.2.2.m2.3.3.1.cmml"><msub id="Thmtheorem2.p1.2.2.m2.3.3.1.3" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.cmml"><mi id="Thmtheorem2.p1.2.2.m2.3.3.1.3.2" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.2.cmml">𝔼</mi><mrow id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.cmml"><mi id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.2" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.2.cmml">x</mi><mo id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.1" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.1.cmml">∼</mo><msub id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.cmml"><mi id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.2" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.2.cmml">P</mi><mrow id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.cmml"><mi id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.2" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.1" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.1.cmml">​</mo><mi id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.3" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.3.cmml">M</mi></mrow></msub></mrow></msub><mo lspace="0.167em" rspace="0em" id="Thmtheorem2.p1.2.2.m2.3.3.1.2" xref="Thmtheorem2.p1.2.2.m2.3.3.1.2.cmml">​</mo><mrow id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.2.cmml"><mi id="Thmtheorem2.p1.2.2.m2.2.2" xref="Thmtheorem2.p1.2.2.m2.2.2.cmml">exp</mi><mo id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1a" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.2.cmml">⁡</mo><mrow id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.2.cmml"><mo stretchy="false" id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.2" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.2.cmml">(</mo><mrow id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.cmml"><mo id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1a" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.cmml">−</mo><mrow id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.cmml"><mi id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.2" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.2.cmml">E</mi><mo lspace="0em" rspace="0em" id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.1" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.1.cmml">​</mo><mrow id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.3.2" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.cmml"><mo stretchy="false" id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.3.2.1" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.cmml">(</mo><mi id="Thmtheorem2.p1.2.2.m2.1.1" xref="Thmtheorem2.p1.2.2.m2.1.1.cmml">x</mi><mo stretchy="false" id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.3.2.2" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.3" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.2.2.m2.3b"><apply id="Thmtheorem2.p1.2.2.m2.3.3.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3"><log id="Thmtheorem2.p1.2.2.m2.3.3.2.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.2"></log><apply id="Thmtheorem2.p1.2.2.m2.3.3.1.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1"><times id="Thmtheorem2.p1.2.2.m2.3.3.1.2.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.2"></times><apply id="Thmtheorem2.p1.2.2.m2.3.3.1.3.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3"><csymbol cd="ambiguous" id="Thmtheorem2.p1.2.2.m2.3.3.1.3.1.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3">subscript</csymbol><ci id="Thmtheorem2.p1.2.2.m2.3.3.1.3.2.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.2">𝔼</ci><apply id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3"><csymbol cd="latexml" id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.1.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.1">similar-to</csymbol><ci id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.2.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.2">𝑥</ci><apply id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3"><csymbol cd="ambiguous" id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.1.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3">subscript</csymbol><ci id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.2.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.2">𝑃</ci><apply id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3"><times id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.1.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.1"></times><ci id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.2.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.2">𝐿</ci><ci id="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.3.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.3.3.3.3.3">𝑀</ci></apply></apply></apply></apply><apply id="Thmtheorem2.p1.2.2.m2.3.3.1.1.2.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1"><exp id="Thmtheorem2.p1.2.2.m2.2.2.cmml" xref="Thmtheorem2.p1.2.2.m2.2.2"></exp><apply id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1"><minus id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.1.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1"></minus><apply id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2"><times id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.1.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.1"></times><ci id="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.2.cmml" xref="Thmtheorem2.p1.2.2.m2.3.3.1.1.1.1.1.2.2">𝐸</ci><ci id="Thmtheorem2.p1.2.2.m2.1.1.cmml" xref="Thmtheorem2.p1.2.2.m2.1.1">𝑥</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.2.2.m2.3c">\log\mathbb{E}_{x\sim P_{LM}}\exp(-E(x))</annotation></semantics></math> with <math id="Thmtheorem2.p1.3.3.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="Thmtheorem2.p1.3.3.m3.1a"><mi id="Thmtheorem2.p1.3.3.m3.1.1" xref="Thmtheorem2.p1.3.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.3.3.m3.1b"><ci id="Thmtheorem2.p1.3.3.m3.1.1.cmml" xref="Thmtheorem2.p1.3.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.3.3.m3.1c">n</annotation></semantics></math> samples <math id="Thmtheorem2.p1.4.4.m4.4" class="ltx_Math" alttext="x_{i}\sim P_{LM}(i=1,\cdots,n)" display="inline"><semantics id="Thmtheorem2.p1.4.4.m4.4a"><mrow id="Thmtheorem2.p1.4.4.m4.4.4" xref="Thmtheorem2.p1.4.4.m4.4.4.cmml"><msub id="Thmtheorem2.p1.4.4.m4.4.4.3" xref="Thmtheorem2.p1.4.4.m4.4.4.3.cmml"><mi id="Thmtheorem2.p1.4.4.m4.4.4.3.2" xref="Thmtheorem2.p1.4.4.m4.4.4.3.2.cmml">x</mi><mi id="Thmtheorem2.p1.4.4.m4.4.4.3.3" xref="Thmtheorem2.p1.4.4.m4.4.4.3.3.cmml">i</mi></msub><mo id="Thmtheorem2.p1.4.4.m4.4.4.2" xref="Thmtheorem2.p1.4.4.m4.4.4.2.cmml">∼</mo><mrow id="Thmtheorem2.p1.4.4.m4.4.4.1" xref="Thmtheorem2.p1.4.4.m4.4.4.1.cmml"><msub id="Thmtheorem2.p1.4.4.m4.4.4.1.3" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3.cmml"><mi id="Thmtheorem2.p1.4.4.m4.4.4.1.3.2" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3.2.cmml">P</mi><mrow id="Thmtheorem2.p1.4.4.m4.4.4.1.3.3" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.cmml"><mi id="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.2" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.1" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.1.cmml">​</mo><mi id="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.3" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.3.cmml">M</mi></mrow></msub><mo lspace="0em" rspace="0em" id="Thmtheorem2.p1.4.4.m4.4.4.1.2" xref="Thmtheorem2.p1.4.4.m4.4.4.1.2.cmml">​</mo><mrow id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.cmml"><mo stretchy="false" id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.2" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.cmml">(</mo><mrow id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.cmml"><mi id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.2" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.2.cmml">i</mi><mo id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.1" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.1.cmml">=</mo><mrow id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.3.2" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.3.1.cmml"><mn id="Thmtheorem2.p1.4.4.m4.1.1" xref="Thmtheorem2.p1.4.4.m4.1.1.cmml">1</mn><mo id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.3.2.1" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.3.1.cmml">,</mo><mi mathvariant="normal" id="Thmtheorem2.p1.4.4.m4.2.2" xref="Thmtheorem2.p1.4.4.m4.2.2.cmml">⋯</mi><mo id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.3.2.2" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.3.1.cmml">,</mo><mi id="Thmtheorem2.p1.4.4.m4.3.3" xref="Thmtheorem2.p1.4.4.m4.3.3.cmml">n</mi></mrow></mrow><mo stretchy="false" id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.3" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.4.4.m4.4b"><apply id="Thmtheorem2.p1.4.4.m4.4.4.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4"><csymbol cd="latexml" id="Thmtheorem2.p1.4.4.m4.4.4.2.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.2">similar-to</csymbol><apply id="Thmtheorem2.p1.4.4.m4.4.4.3.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.3"><csymbol cd="ambiguous" id="Thmtheorem2.p1.4.4.m4.4.4.3.1.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.3">subscript</csymbol><ci id="Thmtheorem2.p1.4.4.m4.4.4.3.2.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.3.2">𝑥</ci><ci id="Thmtheorem2.p1.4.4.m4.4.4.3.3.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.3.3">𝑖</ci></apply><apply id="Thmtheorem2.p1.4.4.m4.4.4.1.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1"><times id="Thmtheorem2.p1.4.4.m4.4.4.1.2.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.2"></times><apply id="Thmtheorem2.p1.4.4.m4.4.4.1.3.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3"><csymbol cd="ambiguous" id="Thmtheorem2.p1.4.4.m4.4.4.1.3.1.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3">subscript</csymbol><ci id="Thmtheorem2.p1.4.4.m4.4.4.1.3.2.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3.2">𝑃</ci><apply id="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3.3"><times id="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.1.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.1"></times><ci id="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.2.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.2">𝐿</ci><ci id="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.3.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.3.3.3">𝑀</ci></apply></apply><apply id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1"><eq id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.1.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.1"></eq><ci id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.2.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.2">𝑖</ci><list id="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.3.1.cmml" xref="Thmtheorem2.p1.4.4.m4.4.4.1.1.1.1.3.2"><cn type="integer" id="Thmtheorem2.p1.4.4.m4.1.1.cmml" xref="Thmtheorem2.p1.4.4.m4.1.1">1</cn><ci id="Thmtheorem2.p1.4.4.m4.2.2.cmml" xref="Thmtheorem2.p1.4.4.m4.2.2">⋯</ci><ci id="Thmtheorem2.p1.4.4.m4.3.3.cmml" xref="Thmtheorem2.p1.4.4.m4.3.3">𝑛</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.4.4.m4.4c">x_{i}\sim P_{LM}(i=1,\cdots,n)</annotation></semantics></math>: <math id="Thmtheorem2.p1.5.5.m5.2" class="ltx_Math" alttext="T_{n}=\log\frac{1}{n}\sum_{i=1}^{n}\exp(-E(x_{i}))" display="inline"><semantics id="Thmtheorem2.p1.5.5.m5.2a"><mrow id="Thmtheorem2.p1.5.5.m5.2.2" xref="Thmtheorem2.p1.5.5.m5.2.2.cmml"><msub id="Thmtheorem2.p1.5.5.m5.2.2.3" xref="Thmtheorem2.p1.5.5.m5.2.2.3.cmml"><mi id="Thmtheorem2.p1.5.5.m5.2.2.3.2" xref="Thmtheorem2.p1.5.5.m5.2.2.3.2.cmml">T</mi><mi id="Thmtheorem2.p1.5.5.m5.2.2.3.3" xref="Thmtheorem2.p1.5.5.m5.2.2.3.3.cmml">n</mi></msub><mo id="Thmtheorem2.p1.5.5.m5.2.2.2" xref="Thmtheorem2.p1.5.5.m5.2.2.2.cmml">=</mo><mrow id="Thmtheorem2.p1.5.5.m5.2.2.1" xref="Thmtheorem2.p1.5.5.m5.2.2.1.cmml"><mrow id="Thmtheorem2.p1.5.5.m5.2.2.1.3" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3.cmml"><mi id="Thmtheorem2.p1.5.5.m5.2.2.1.3.1" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3.1.cmml">log</mi><mo lspace="0.167em" id="Thmtheorem2.p1.5.5.m5.2.2.1.3a" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3.cmml">⁡</mo><mfrac id="Thmtheorem2.p1.5.5.m5.2.2.1.3.2" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3.2.cmml"><mn id="Thmtheorem2.p1.5.5.m5.2.2.1.3.2.2" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3.2.2.cmml">1</mn><mi id="Thmtheorem2.p1.5.5.m5.2.2.1.3.2.3" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3.2.3.cmml">n</mi></mfrac></mrow><mo lspace="0em" rspace="0em" id="Thmtheorem2.p1.5.5.m5.2.2.1.2" xref="Thmtheorem2.p1.5.5.m5.2.2.1.2.cmml">​</mo><mrow id="Thmtheorem2.p1.5.5.m5.2.2.1.1" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.cmml"><msubsup id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.cmml"><mo id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.2" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.2.cmml">∑</mo><mrow id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.cmml"><mi id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.2" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.2.cmml">i</mi><mo id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.1" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.1.cmml">=</mo><mn id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.3" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.3" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.3.cmml">n</mi></msubsup><mrow id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.2.cmml"><mi id="Thmtheorem2.p1.5.5.m5.1.1" xref="Thmtheorem2.p1.5.5.m5.1.1.cmml">exp</mi><mo id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1a" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.2.cmml">⁡</mo><mrow id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.2.cmml"><mo stretchy="false" id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.2" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.2.cmml">(</mo><mrow id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.cmml"><mo id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1a" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.cmml">−</mo><mrow id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.cmml"><mi id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.3" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.2" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.2" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.2" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.3" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.3" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.3" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.5.5.m5.2b"><apply id="Thmtheorem2.p1.5.5.m5.2.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2"><eq id="Thmtheorem2.p1.5.5.m5.2.2.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.2"></eq><apply id="Thmtheorem2.p1.5.5.m5.2.2.3.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.3"><csymbol cd="ambiguous" id="Thmtheorem2.p1.5.5.m5.2.2.3.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.3">subscript</csymbol><ci id="Thmtheorem2.p1.5.5.m5.2.2.3.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.3.2">𝑇</ci><ci id="Thmtheorem2.p1.5.5.m5.2.2.3.3.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.3.3">𝑛</ci></apply><apply id="Thmtheorem2.p1.5.5.m5.2.2.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1"><times id="Thmtheorem2.p1.5.5.m5.2.2.1.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.2"></times><apply id="Thmtheorem2.p1.5.5.m5.2.2.1.3.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3"><log id="Thmtheorem2.p1.5.5.m5.2.2.1.3.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3.1"></log><apply id="Thmtheorem2.p1.5.5.m5.2.2.1.3.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3.2"><divide id="Thmtheorem2.p1.5.5.m5.2.2.1.3.2.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3.2"></divide><cn type="integer" id="Thmtheorem2.p1.5.5.m5.2.2.1.3.2.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3.2.2">1</cn><ci id="Thmtheorem2.p1.5.5.m5.2.2.1.3.2.3.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.3.2.3">𝑛</ci></apply></apply><apply id="Thmtheorem2.p1.5.5.m5.2.2.1.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1"><apply id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2"><csymbol cd="ambiguous" id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2">superscript</csymbol><apply id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2"><csymbol cd="ambiguous" id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2">subscript</csymbol><sum id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.2"></sum><apply id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3"><eq id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.1"></eq><ci id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.3.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.2.3.3">1</cn></apply></apply><ci id="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.3.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.2.3">𝑛</ci></apply><apply id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1"><exp id="Thmtheorem2.p1.5.5.m5.1.1.cmml" xref="Thmtheorem2.p1.5.5.m5.1.1"></exp><apply id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1"><minus id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1"></minus><apply id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1"><times id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.2"></times><ci id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.3.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.3">𝐸</ci><apply id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Thmtheorem2.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.5.5.m5.2c">T_{n}=\log\frac{1}{n}\sum_{i=1}^{n}\exp(-E(x_{i}))</annotation></semantics></math>, then <math id="Thmtheorem2.p1.6.6.m6.1" class="ltx_Math" alttext="\forall\epsilon&gt;0" display="inline"><semantics id="Thmtheorem2.p1.6.6.m6.1a"><mrow id="Thmtheorem2.p1.6.6.m6.1.1" xref="Thmtheorem2.p1.6.6.m6.1.1.cmml"><mrow id="Thmtheorem2.p1.6.6.m6.1.1.2" xref="Thmtheorem2.p1.6.6.m6.1.1.2.cmml"><mo rspace="0.167em" id="Thmtheorem2.p1.6.6.m6.1.1.2.1" xref="Thmtheorem2.p1.6.6.m6.1.1.2.1.cmml">∀</mo><mi id="Thmtheorem2.p1.6.6.m6.1.1.2.2" xref="Thmtheorem2.p1.6.6.m6.1.1.2.2.cmml">ϵ</mi></mrow><mo id="Thmtheorem2.p1.6.6.m6.1.1.1" xref="Thmtheorem2.p1.6.6.m6.1.1.1.cmml">&gt;</mo><mn id="Thmtheorem2.p1.6.6.m6.1.1.3" xref="Thmtheorem2.p1.6.6.m6.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.6.6.m6.1b"><apply id="Thmtheorem2.p1.6.6.m6.1.1.cmml" xref="Thmtheorem2.p1.6.6.m6.1.1"><gt id="Thmtheorem2.p1.6.6.m6.1.1.1.cmml" xref="Thmtheorem2.p1.6.6.m6.1.1.1"></gt><apply id="Thmtheorem2.p1.6.6.m6.1.1.2.cmml" xref="Thmtheorem2.p1.6.6.m6.1.1.2"><csymbol cd="latexml" id="Thmtheorem2.p1.6.6.m6.1.1.2.1.cmml" xref="Thmtheorem2.p1.6.6.m6.1.1.2.1">for-all</csymbol><ci id="Thmtheorem2.p1.6.6.m6.1.1.2.2.cmml" xref="Thmtheorem2.p1.6.6.m6.1.1.2.2">italic-ϵ</ci></apply><cn type="integer" id="Thmtheorem2.p1.6.6.m6.1.1.3.cmml" xref="Thmtheorem2.p1.6.6.m6.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.6.6.m6.1c">\forall\epsilon&gt;0</annotation></semantics></math>, <math id="Thmtheorem2.p1.7.7.m7.1" class="ltx_Math" alttext="\exists N&gt;0" display="inline"><semantics id="Thmtheorem2.p1.7.7.m7.1a"><mrow id="Thmtheorem2.p1.7.7.m7.1.1" xref="Thmtheorem2.p1.7.7.m7.1.1.cmml"><mrow id="Thmtheorem2.p1.7.7.m7.1.1.2" xref="Thmtheorem2.p1.7.7.m7.1.1.2.cmml"><mo rspace="0.167em" id="Thmtheorem2.p1.7.7.m7.1.1.2.1" xref="Thmtheorem2.p1.7.7.m7.1.1.2.1.cmml">∃</mo><mi id="Thmtheorem2.p1.7.7.m7.1.1.2.2" xref="Thmtheorem2.p1.7.7.m7.1.1.2.2.cmml">N</mi></mrow><mo id="Thmtheorem2.p1.7.7.m7.1.1.1" xref="Thmtheorem2.p1.7.7.m7.1.1.1.cmml">&gt;</mo><mn id="Thmtheorem2.p1.7.7.m7.1.1.3" xref="Thmtheorem2.p1.7.7.m7.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.7.7.m7.1b"><apply id="Thmtheorem2.p1.7.7.m7.1.1.cmml" xref="Thmtheorem2.p1.7.7.m7.1.1"><gt id="Thmtheorem2.p1.7.7.m7.1.1.1.cmml" xref="Thmtheorem2.p1.7.7.m7.1.1.1"></gt><apply id="Thmtheorem2.p1.7.7.m7.1.1.2.cmml" xref="Thmtheorem2.p1.7.7.m7.1.1.2"><exists id="Thmtheorem2.p1.7.7.m7.1.1.2.1.cmml" xref="Thmtheorem2.p1.7.7.m7.1.1.2.1"></exists><ci id="Thmtheorem2.p1.7.7.m7.1.1.2.2.cmml" xref="Thmtheorem2.p1.7.7.m7.1.1.2.2">𝑁</ci></apply><cn type="integer" id="Thmtheorem2.p1.7.7.m7.1.1.3.cmml" xref="Thmtheorem2.p1.7.7.m7.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.7.7.m7.1c">\exists N&gt;0</annotation></semantics></math> such that <math id="Thmtheorem2.p1.8.8.m8.1" class="ltx_Math" alttext="\forall n&gt;N" display="inline"><semantics id="Thmtheorem2.p1.8.8.m8.1a"><mrow id="Thmtheorem2.p1.8.8.m8.1.1" xref="Thmtheorem2.p1.8.8.m8.1.1.cmml"><mrow id="Thmtheorem2.p1.8.8.m8.1.1.2" xref="Thmtheorem2.p1.8.8.m8.1.1.2.cmml"><mo rspace="0.167em" id="Thmtheorem2.p1.8.8.m8.1.1.2.1" xref="Thmtheorem2.p1.8.8.m8.1.1.2.1.cmml">∀</mo><mi id="Thmtheorem2.p1.8.8.m8.1.1.2.2" xref="Thmtheorem2.p1.8.8.m8.1.1.2.2.cmml">n</mi></mrow><mo id="Thmtheorem2.p1.8.8.m8.1.1.1" xref="Thmtheorem2.p1.8.8.m8.1.1.1.cmml">&gt;</mo><mi id="Thmtheorem2.p1.8.8.m8.1.1.3" xref="Thmtheorem2.p1.8.8.m8.1.1.3.cmml">N</mi></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.8.8.m8.1b"><apply id="Thmtheorem2.p1.8.8.m8.1.1.cmml" xref="Thmtheorem2.p1.8.8.m8.1.1"><gt id="Thmtheorem2.p1.8.8.m8.1.1.1.cmml" xref="Thmtheorem2.p1.8.8.m8.1.1.1"></gt><apply id="Thmtheorem2.p1.8.8.m8.1.1.2.cmml" xref="Thmtheorem2.p1.8.8.m8.1.1.2"><csymbol cd="latexml" id="Thmtheorem2.p1.8.8.m8.1.1.2.1.cmml" xref="Thmtheorem2.p1.8.8.m8.1.1.2.1">for-all</csymbol><ci id="Thmtheorem2.p1.8.8.m8.1.1.2.2.cmml" xref="Thmtheorem2.p1.8.8.m8.1.1.2.2">𝑛</ci></apply><ci id="Thmtheorem2.p1.8.8.m8.1.1.3.cmml" xref="Thmtheorem2.p1.8.8.m8.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.8.8.m8.1c">\forall n&gt;N</annotation></semantics></math> we have</span></p>
<table id="S3.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E4.m1.2" class="ltx_Math" alttext="Z_{\theta}-\epsilon&lt;\mathbb{E}[T_{n}]&lt;Z_{\theta}&lt;\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]&lt;Z_{\theta}+\epsilon" display="block"><semantics id="S3.E4.m1.2a"><mrow id="S3.E4.m1.2.2" xref="S3.E4.m1.2.2.cmml"><mrow id="S3.E4.m1.2.2.4" xref="S3.E4.m1.2.2.4.cmml"><msub id="S3.E4.m1.2.2.4.2" xref="S3.E4.m1.2.2.4.2.cmml"><mi id="S3.E4.m1.2.2.4.2.2" xref="S3.E4.m1.2.2.4.2.2.cmml">Z</mi><mi id="S3.E4.m1.2.2.4.2.3" xref="S3.E4.m1.2.2.4.2.3.cmml">θ</mi></msub><mo id="S3.E4.m1.2.2.4.1" xref="S3.E4.m1.2.2.4.1.cmml">−</mo><mi id="S3.E4.m1.2.2.4.3" xref="S3.E4.m1.2.2.4.3.cmml">ϵ</mi></mrow><mo id="S3.E4.m1.2.2.5" xref="S3.E4.m1.2.2.5.cmml">&lt;</mo><mrow id="S3.E4.m1.1.1.1" xref="S3.E4.m1.1.1.1.cmml"><mi id="S3.E4.m1.1.1.1.3" xref="S3.E4.m1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.1.2" xref="S3.E4.m1.1.1.1.2.cmml">​</mo><mrow id="S3.E4.m1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.E4.m1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.2.1.cmml">[</mo><msub id="S3.E4.m1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.1.2.cmml">T</mi><mi id="S3.E4.m1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="S3.E4.m1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="S3.E4.m1.2.2.6" xref="S3.E4.m1.2.2.6.cmml">&lt;</mo><msub id="S3.E4.m1.2.2.7" xref="S3.E4.m1.2.2.7.cmml"><mi id="S3.E4.m1.2.2.7.2" xref="S3.E4.m1.2.2.7.2.cmml">Z</mi><mi id="S3.E4.m1.2.2.7.3" xref="S3.E4.m1.2.2.7.3.cmml">θ</mi></msub><mo id="S3.E4.m1.2.2.8" xref="S3.E4.m1.2.2.8.cmml">&lt;</mo><mrow id="S3.E4.m1.2.2.2" xref="S3.E4.m1.2.2.2.cmml"><mi id="S3.E4.m1.2.2.2.3" xref="S3.E4.m1.2.2.2.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.2.2.2.2" xref="S3.E4.m1.2.2.2.2.cmml">​</mo><mrow id="S3.E4.m1.2.2.2.1.1" xref="S3.E4.m1.2.2.2.1.2.cmml"><mo stretchy="false" id="S3.E4.m1.2.2.2.1.1.2" xref="S3.E4.m1.2.2.2.1.2.1.cmml">[</mo><mrow id="S3.E4.m1.2.2.2.1.1.1" xref="S3.E4.m1.2.2.2.1.1.1.cmml"><mrow id="S3.E4.m1.2.2.2.1.1.1.1" xref="S3.E4.m1.2.2.2.1.1.1.1.cmml"><mrow id="S3.E4.m1.2.2.2.1.1.1.1.1.1" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E4.m1.2.2.2.1.1.1.1.1.1.2" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.cmml"><mrow id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.cmml"><mn id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.2" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.1" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.3" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.3.cmml">n</mi></mrow><mo id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.1" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.1.cmml">−</mo><mn id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.3" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S3.E4.m1.2.2.2.1.1.1.1.1.1.3" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E4.m1.2.2.2.1.1.1.1.2" xref="S3.E4.m1.2.2.2.1.1.1.1.2.cmml">​</mo><msub id="S3.E4.m1.2.2.2.1.1.1.1.3" xref="S3.E4.m1.2.2.2.1.1.1.1.3.cmml"><mi id="S3.E4.m1.2.2.2.1.1.1.1.3.2" xref="S3.E4.m1.2.2.2.1.1.1.1.3.2.cmml">T</mi><mi id="S3.E4.m1.2.2.2.1.1.1.1.3.3" xref="S3.E4.m1.2.2.2.1.1.1.1.3.3.cmml">n</mi></msub></mrow><mo id="S3.E4.m1.2.2.2.1.1.1.3" xref="S3.E4.m1.2.2.2.1.1.1.3.cmml">−</mo><mrow id="S3.E4.m1.2.2.2.1.1.1.2" xref="S3.E4.m1.2.2.2.1.1.1.2.cmml"><mn id="S3.E4.m1.2.2.2.1.1.1.2.3" xref="S3.E4.m1.2.2.2.1.1.1.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="S3.E4.m1.2.2.2.1.1.1.2.2" xref="S3.E4.m1.2.2.2.1.1.1.2.2.cmml">​</mo><mrow id="S3.E4.m1.2.2.2.1.1.1.2.1.1" xref="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="S3.E4.m1.2.2.2.1.1.1.2.1.1.2" xref="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.cmml">(</mo><mrow id="S3.E4.m1.2.2.2.1.1.1.2.1.1.1" xref="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.cmml"><mi id="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.2" xref="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.2.cmml">n</mi><mo id="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.1" xref="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.1.cmml">−</mo><mn id="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.3" xref="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S3.E4.m1.2.2.2.1.1.1.2.1.1.3" xref="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E4.m1.2.2.2.1.1.1.2.2a" xref="S3.E4.m1.2.2.2.1.1.1.2.2.cmml">​</mo><msub id="S3.E4.m1.2.2.2.1.1.1.2.4" xref="S3.E4.m1.2.2.2.1.1.1.2.4.cmml"><mi id="S3.E4.m1.2.2.2.1.1.1.2.4.2" xref="S3.E4.m1.2.2.2.1.1.1.2.4.2.cmml">T</mi><mrow id="S3.E4.m1.2.2.2.1.1.1.2.4.3" xref="S3.E4.m1.2.2.2.1.1.1.2.4.3.cmml"><mi id="S3.E4.m1.2.2.2.1.1.1.2.4.3.2" xref="S3.E4.m1.2.2.2.1.1.1.2.4.3.2.cmml">n</mi><mo id="S3.E4.m1.2.2.2.1.1.1.2.4.3.1" xref="S3.E4.m1.2.2.2.1.1.1.2.4.3.1.cmml">−</mo><mn id="S3.E4.m1.2.2.2.1.1.1.2.4.3.3" xref="S3.E4.m1.2.2.2.1.1.1.2.4.3.3.cmml">1</mn></mrow></msub></mrow></mrow><mo stretchy="false" id="S3.E4.m1.2.2.2.1.1.3" xref="S3.E4.m1.2.2.2.1.2.1.cmml">]</mo></mrow></mrow><mo id="S3.E4.m1.2.2.9" xref="S3.E4.m1.2.2.9.cmml">&lt;</mo><mrow id="S3.E4.m1.2.2.10" xref="S3.E4.m1.2.2.10.cmml"><msub id="S3.E4.m1.2.2.10.2" xref="S3.E4.m1.2.2.10.2.cmml"><mi id="S3.E4.m1.2.2.10.2.2" xref="S3.E4.m1.2.2.10.2.2.cmml">Z</mi><mi id="S3.E4.m1.2.2.10.2.3" xref="S3.E4.m1.2.2.10.2.3.cmml">θ</mi></msub><mo id="S3.E4.m1.2.2.10.1" xref="S3.E4.m1.2.2.10.1.cmml">+</mo><mi id="S3.E4.m1.2.2.10.3" xref="S3.E4.m1.2.2.10.3.cmml">ϵ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.2b"><apply id="S3.E4.m1.2.2.cmml" xref="S3.E4.m1.2.2"><and id="S3.E4.m1.2.2a.cmml" xref="S3.E4.m1.2.2"></and><apply id="S3.E4.m1.2.2b.cmml" xref="S3.E4.m1.2.2"><lt id="S3.E4.m1.2.2.5.cmml" xref="S3.E4.m1.2.2.5"></lt><apply id="S3.E4.m1.2.2.4.cmml" xref="S3.E4.m1.2.2.4"><minus id="S3.E4.m1.2.2.4.1.cmml" xref="S3.E4.m1.2.2.4.1"></minus><apply id="S3.E4.m1.2.2.4.2.cmml" xref="S3.E4.m1.2.2.4.2"><csymbol cd="ambiguous" id="S3.E4.m1.2.2.4.2.1.cmml" xref="S3.E4.m1.2.2.4.2">subscript</csymbol><ci id="S3.E4.m1.2.2.4.2.2.cmml" xref="S3.E4.m1.2.2.4.2.2">𝑍</ci><ci id="S3.E4.m1.2.2.4.2.3.cmml" xref="S3.E4.m1.2.2.4.2.3">𝜃</ci></apply><ci id="S3.E4.m1.2.2.4.3.cmml" xref="S3.E4.m1.2.2.4.3">italic-ϵ</ci></apply><apply id="S3.E4.m1.1.1.1.cmml" xref="S3.E4.m1.1.1.1"><times id="S3.E4.m1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.2"></times><ci id="S3.E4.m1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.3">𝔼</ci><apply id="S3.E4.m1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E4.m1.1.1.1.1.2.1.cmml" xref="S3.E4.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.E4.m1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.2">𝑇</ci><ci id="S3.E4.m1.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></apply><apply id="S3.E4.m1.2.2c.cmml" xref="S3.E4.m1.2.2"><lt id="S3.E4.m1.2.2.6.cmml" xref="S3.E4.m1.2.2.6"></lt><share href="#S3.E4.m1.1.1.1.cmml" id="S3.E4.m1.2.2d.cmml" xref="S3.E4.m1.2.2"></share><apply id="S3.E4.m1.2.2.7.cmml" xref="S3.E4.m1.2.2.7"><csymbol cd="ambiguous" id="S3.E4.m1.2.2.7.1.cmml" xref="S3.E4.m1.2.2.7">subscript</csymbol><ci id="S3.E4.m1.2.2.7.2.cmml" xref="S3.E4.m1.2.2.7.2">𝑍</ci><ci id="S3.E4.m1.2.2.7.3.cmml" xref="S3.E4.m1.2.2.7.3">𝜃</ci></apply></apply><apply id="S3.E4.m1.2.2e.cmml" xref="S3.E4.m1.2.2"><lt id="S3.E4.m1.2.2.8.cmml" xref="S3.E4.m1.2.2.8"></lt><share href="#S3.E4.m1.2.2.7.cmml" id="S3.E4.m1.2.2f.cmml" xref="S3.E4.m1.2.2"></share><apply id="S3.E4.m1.2.2.2.cmml" xref="S3.E4.m1.2.2.2"><times id="S3.E4.m1.2.2.2.2.cmml" xref="S3.E4.m1.2.2.2.2"></times><ci id="S3.E4.m1.2.2.2.3.cmml" xref="S3.E4.m1.2.2.2.3">𝔼</ci><apply id="S3.E4.m1.2.2.2.1.2.cmml" xref="S3.E4.m1.2.2.2.1.1"><csymbol cd="latexml" id="S3.E4.m1.2.2.2.1.2.1.cmml" xref="S3.E4.m1.2.2.2.1.1.2">delimited-[]</csymbol><apply id="S3.E4.m1.2.2.2.1.1.1.cmml" xref="S3.E4.m1.2.2.2.1.1.1"><minus id="S3.E4.m1.2.2.2.1.1.1.3.cmml" xref="S3.E4.m1.2.2.2.1.1.1.3"></minus><apply id="S3.E4.m1.2.2.2.1.1.1.1.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1"><times id="S3.E4.m1.2.2.2.1.1.1.1.2.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.2"></times><apply id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1"><minus id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.1"></minus><apply id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2"><times id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.2">2</cn><ci id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.2.3">𝑛</ci></apply><cn type="integer" id="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.1.1.1.3">1</cn></apply><apply id="S3.E4.m1.2.2.2.1.1.1.1.3.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.2.2.2.1.1.1.1.3.1.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.3">subscript</csymbol><ci id="S3.E4.m1.2.2.2.1.1.1.1.3.2.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.3.2">𝑇</ci><ci id="S3.E4.m1.2.2.2.1.1.1.1.3.3.cmml" xref="S3.E4.m1.2.2.2.1.1.1.1.3.3">𝑛</ci></apply></apply><apply id="S3.E4.m1.2.2.2.1.1.1.2.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2"><times id="S3.E4.m1.2.2.2.1.1.1.2.2.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.2"></times><cn type="integer" id="S3.E4.m1.2.2.2.1.1.1.2.3.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.3">2</cn><apply id="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.1.1"><minus id="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.1.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.1"></minus><ci id="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.2.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.2">𝑛</ci><cn type="integer" id="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.3.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.1.1.1.3">1</cn></apply><apply id="S3.E4.m1.2.2.2.1.1.1.2.4.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.4"><csymbol cd="ambiguous" id="S3.E4.m1.2.2.2.1.1.1.2.4.1.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.4">subscript</csymbol><ci id="S3.E4.m1.2.2.2.1.1.1.2.4.2.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.4.2">𝑇</ci><apply id="S3.E4.m1.2.2.2.1.1.1.2.4.3.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.4.3"><minus id="S3.E4.m1.2.2.2.1.1.1.2.4.3.1.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.4.3.1"></minus><ci id="S3.E4.m1.2.2.2.1.1.1.2.4.3.2.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.4.3.2">𝑛</ci><cn type="integer" id="S3.E4.m1.2.2.2.1.1.1.2.4.3.3.cmml" xref="S3.E4.m1.2.2.2.1.1.1.2.4.3.3">1</cn></apply></apply></apply></apply></apply></apply></apply><apply id="S3.E4.m1.2.2g.cmml" xref="S3.E4.m1.2.2"><lt id="S3.E4.m1.2.2.9.cmml" xref="S3.E4.m1.2.2.9"></lt><share href="#S3.E4.m1.2.2.2.cmml" id="S3.E4.m1.2.2h.cmml" xref="S3.E4.m1.2.2"></share><apply id="S3.E4.m1.2.2.10.cmml" xref="S3.E4.m1.2.2.10"><plus id="S3.E4.m1.2.2.10.1.cmml" xref="S3.E4.m1.2.2.10.1"></plus><apply id="S3.E4.m1.2.2.10.2.cmml" xref="S3.E4.m1.2.2.10.2"><csymbol cd="ambiguous" id="S3.E4.m1.2.2.10.2.1.cmml" xref="S3.E4.m1.2.2.10.2">subscript</csymbol><ci id="S3.E4.m1.2.2.10.2.2.cmml" xref="S3.E4.m1.2.2.10.2.2">𝑍</ci><ci id="S3.E4.m1.2.2.10.2.3.cmml" xref="S3.E4.m1.2.2.10.2.3">𝜃</ci></apply><ci id="S3.E4.m1.2.2.10.3.cmml" xref="S3.E4.m1.2.2.10.3">italic-ϵ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.2c">Z_{\theta}-\epsilon&lt;\mathbb{E}[T_{n}]&lt;Z_{\theta}&lt;\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]&lt;Z_{\theta}+\epsilon</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
</div>
</div>
<div id="S3.SS2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS2.p2.1" class="ltx_p">The proof is given in Appendix <a href="#A1.SS2" title="A.2 Proof of Theorem 2 ‣ Appendix A Appendix ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.2</span></a>.</p>
</div>
<div id="S3.SS2.p3" class="ltx_para ltx_noindent">
<p id="S3.SS2.p3.2" class="ltx_p">We can use the above two estimators to estimate the lower and upper bounds of the partition function, but we want to emphasize that they are true only asymptotically (when <math id="S3.SS2.p3.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS2.p3.1.m1.1a"><mi id="S3.SS2.p3.1.m1.1.1" xref="S3.SS2.p3.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.1.m1.1b"><ci id="S3.SS2.p3.1.m1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.1.m1.1c">n</annotation></semantics></math> is sufficiently large).
We also want to note that to get lower variance estimates we use leave-one-out strategy to estimate <math id="S3.SS2.p3.2.m2.1" class="ltx_Math" alttext="T_{n-1}" display="inline"><semantics id="S3.SS2.p3.2.m2.1a"><msub id="S3.SS2.p3.2.m2.1.1" xref="S3.SS2.p3.2.m2.1.1.cmml"><mi id="S3.SS2.p3.2.m2.1.1.2" xref="S3.SS2.p3.2.m2.1.1.2.cmml">T</mi><mrow id="S3.SS2.p3.2.m2.1.1.3" xref="S3.SS2.p3.2.m2.1.1.3.cmml"><mi id="S3.SS2.p3.2.m2.1.1.3.2" xref="S3.SS2.p3.2.m2.1.1.3.2.cmml">n</mi><mo id="S3.SS2.p3.2.m2.1.1.3.1" xref="S3.SS2.p3.2.m2.1.1.3.1.cmml">−</mo><mn id="S3.SS2.p3.2.m2.1.1.3.3" xref="S3.SS2.p3.2.m2.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.2.m2.1b"><apply id="S3.SS2.p3.2.m2.1.1.cmml" xref="S3.SS2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p3.2.m2.1.1.1.cmml" xref="S3.SS2.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.p3.2.m2.1.1.2.cmml" xref="S3.SS2.p3.2.m2.1.1.2">𝑇</ci><apply id="S3.SS2.p3.2.m2.1.1.3.cmml" xref="S3.SS2.p3.2.m2.1.1.3"><minus id="S3.SS2.p3.2.m2.1.1.3.1.cmml" xref="S3.SS2.p3.2.m2.1.1.3.1"></minus><ci id="S3.SS2.p3.2.m2.1.1.3.2.cmml" xref="S3.SS2.p3.2.m2.1.1.3.2">𝑛</ci><cn type="integer" id="S3.SS2.p3.2.m2.1.1.3.3.cmml" xref="S3.SS2.p3.2.m2.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.2.m2.1c">T_{n-1}</annotation></semantics></math>.
See <cite class="ltx_cite ltx_citemacro_citet">Nowozin (<a href="#bib.bib23" title="" class="ltx_ref">2018</a>)</cite> for implementation details and methods to improve numeric stability.</p>
</div>
<div id="S3.SS2.p4" class="ltx_para ltx_noindent">
<p id="S3.SS2.p4.1" class="ltx_p">Similarly to locally normalized models, we can also factorize the probabilities of an entire sequence step by step, as <math id="S3.SS2.p4.1.m1.2" class="ltx_Math" alttext="P(x)=\prod_{t=1}^{T}P(x_{t}|x_{&lt;t})" display="inline"><semantics id="S3.SS2.p4.1.m1.2a"><mrow id="S3.SS2.p4.1.m1.2.2" xref="S3.SS2.p4.1.m1.2.2.cmml"><mrow id="S3.SS2.p4.1.m1.2.2.3" xref="S3.SS2.p4.1.m1.2.2.3.cmml"><mi id="S3.SS2.p4.1.m1.2.2.3.2" xref="S3.SS2.p4.1.m1.2.2.3.2.cmml">P</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p4.1.m1.2.2.3.1" xref="S3.SS2.p4.1.m1.2.2.3.1.cmml">​</mo><mrow id="S3.SS2.p4.1.m1.2.2.3.3.2" xref="S3.SS2.p4.1.m1.2.2.3.cmml"><mo stretchy="false" id="S3.SS2.p4.1.m1.2.2.3.3.2.1" xref="S3.SS2.p4.1.m1.2.2.3.cmml">(</mo><mi id="S3.SS2.p4.1.m1.1.1" xref="S3.SS2.p4.1.m1.1.1.cmml">x</mi><mo stretchy="false" id="S3.SS2.p4.1.m1.2.2.3.3.2.2" xref="S3.SS2.p4.1.m1.2.2.3.cmml">)</mo></mrow></mrow><mo rspace="0.111em" id="S3.SS2.p4.1.m1.2.2.2" xref="S3.SS2.p4.1.m1.2.2.2.cmml">=</mo><mrow id="S3.SS2.p4.1.m1.2.2.1" xref="S3.SS2.p4.1.m1.2.2.1.cmml"><msubsup id="S3.SS2.p4.1.m1.2.2.1.2" xref="S3.SS2.p4.1.m1.2.2.1.2.cmml"><mo id="S3.SS2.p4.1.m1.2.2.1.2.2.2" xref="S3.SS2.p4.1.m1.2.2.1.2.2.2.cmml">∏</mo><mrow id="S3.SS2.p4.1.m1.2.2.1.2.2.3" xref="S3.SS2.p4.1.m1.2.2.1.2.2.3.cmml"><mi id="S3.SS2.p4.1.m1.2.2.1.2.2.3.2" xref="S3.SS2.p4.1.m1.2.2.1.2.2.3.2.cmml">t</mi><mo id="S3.SS2.p4.1.m1.2.2.1.2.2.3.1" xref="S3.SS2.p4.1.m1.2.2.1.2.2.3.1.cmml">=</mo><mn id="S3.SS2.p4.1.m1.2.2.1.2.2.3.3" xref="S3.SS2.p4.1.m1.2.2.1.2.2.3.3.cmml">1</mn></mrow><mi id="S3.SS2.p4.1.m1.2.2.1.2.3" xref="S3.SS2.p4.1.m1.2.2.1.2.3.cmml">T</mi></msubsup><mrow id="S3.SS2.p4.1.m1.2.2.1.1" xref="S3.SS2.p4.1.m1.2.2.1.1.cmml"><mi id="S3.SS2.p4.1.m1.2.2.1.1.3" xref="S3.SS2.p4.1.m1.2.2.1.1.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p4.1.m1.2.2.1.1.2" xref="S3.SS2.p4.1.m1.2.2.1.1.2.cmml">​</mo><mrow id="S3.SS2.p4.1.m1.2.2.1.1.1.1" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p4.1.m1.2.2.1.1.1.1.2" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.cmml"><msub id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2.cmml"><mi id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2.2" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2.3" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2.3.cmml">t</mi></msub><mo fence="false" id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.1" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.1.cmml">|</mo><msub id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.cmml"><mi id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.2" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.2.cmml">x</mi><mrow id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.cmml"><mi id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.2" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.2.cmml"></mi><mo id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.1" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.3" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.3.cmml">t</mi></mrow></msub></mrow><mo stretchy="false" id="S3.SS2.p4.1.m1.2.2.1.1.1.1.3" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.1.m1.2b"><apply id="S3.SS2.p4.1.m1.2.2.cmml" xref="S3.SS2.p4.1.m1.2.2"><eq id="S3.SS2.p4.1.m1.2.2.2.cmml" xref="S3.SS2.p4.1.m1.2.2.2"></eq><apply id="S3.SS2.p4.1.m1.2.2.3.cmml" xref="S3.SS2.p4.1.m1.2.2.3"><times id="S3.SS2.p4.1.m1.2.2.3.1.cmml" xref="S3.SS2.p4.1.m1.2.2.3.1"></times><ci id="S3.SS2.p4.1.m1.2.2.3.2.cmml" xref="S3.SS2.p4.1.m1.2.2.3.2">𝑃</ci><ci id="S3.SS2.p4.1.m1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1">𝑥</ci></apply><apply id="S3.SS2.p4.1.m1.2.2.1.cmml" xref="S3.SS2.p4.1.m1.2.2.1"><apply id="S3.SS2.p4.1.m1.2.2.1.2.cmml" xref="S3.SS2.p4.1.m1.2.2.1.2"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.2.2.1.2.1.cmml" xref="S3.SS2.p4.1.m1.2.2.1.2">superscript</csymbol><apply id="S3.SS2.p4.1.m1.2.2.1.2.2.cmml" xref="S3.SS2.p4.1.m1.2.2.1.2"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.2.2.1.2.2.1.cmml" xref="S3.SS2.p4.1.m1.2.2.1.2">subscript</csymbol><csymbol cd="latexml" id="S3.SS2.p4.1.m1.2.2.1.2.2.2.cmml" xref="S3.SS2.p4.1.m1.2.2.1.2.2.2">product</csymbol><apply id="S3.SS2.p4.1.m1.2.2.1.2.2.3.cmml" xref="S3.SS2.p4.1.m1.2.2.1.2.2.3"><eq id="S3.SS2.p4.1.m1.2.2.1.2.2.3.1.cmml" xref="S3.SS2.p4.1.m1.2.2.1.2.2.3.1"></eq><ci id="S3.SS2.p4.1.m1.2.2.1.2.2.3.2.cmml" xref="S3.SS2.p4.1.m1.2.2.1.2.2.3.2">𝑡</ci><cn type="integer" id="S3.SS2.p4.1.m1.2.2.1.2.2.3.3.cmml" xref="S3.SS2.p4.1.m1.2.2.1.2.2.3.3">1</cn></apply></apply><ci id="S3.SS2.p4.1.m1.2.2.1.2.3.cmml" xref="S3.SS2.p4.1.m1.2.2.1.2.3">𝑇</ci></apply><apply id="S3.SS2.p4.1.m1.2.2.1.1.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1"><times id="S3.SS2.p4.1.m1.2.2.1.1.2.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.2"></times><ci id="S3.SS2.p4.1.m1.2.2.1.1.3.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.3">𝑃</ci><apply id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1"><csymbol cd="latexml" id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.1.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.1">conditional</csymbol><apply id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2.1.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2.2.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2.3.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.1.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.2.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.2">𝑥</ci><apply id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3"><lt id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.1.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.2.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.2">absent</csymbol><ci id="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.3.cmml" xref="S3.SS2.p4.1.m1.2.2.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.1.m1.2c">P(x)=\prod_{t=1}^{T}P(x_{t}|x_{&lt;t})</annotation></semantics></math>, and evaluate the PPL for each generation step. By marginalizing over the future, we can derive the following
per step probabilities:</p>
<table id="S3.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E5.m1.9" class="ltx_math_unparsed" alttext="P(x_{t}|x_{&lt;t})=P_{LM}(x_{t}|x_{&lt;t})\frac{\mathbb{E}_{x_{t+1}^{\prime},\cdots,x_{T}^{\prime}\sim P_{LM}(\cdot|x_{\leq t})}[\exp(-E_{\theta}(x_{\leq t},x_{t+1}^{\prime},\cdots,x_{T}^{\prime}))]}{\mathbb{E}_{x_{t}^{\prime},\cdots,x_{T}^{\prime}\sim P_{LM}(\cdot|x_{\leq t-1})}[\exp(-E_{\theta}(x_{\leq t-1},x_{t}^{\prime},\cdots,x_{T}^{\prime}))]}." display="block"><semantics id="S3.E5.m1.9a"><mrow id="S3.E5.m1.9.9.1"><mrow id="S3.E5.m1.9.9.1.1"><mrow id="S3.E5.m1.9.9.1.1.1"><mi id="S3.E5.m1.9.9.1.1.1.3">P</mi><mo lspace="0em" rspace="0em" id="S3.E5.m1.9.9.1.1.1.2">​</mo><mrow id="S3.E5.m1.9.9.1.1.1.1.1"><mo stretchy="false" id="S3.E5.m1.9.9.1.1.1.1.1.2">(</mo><mrow id="S3.E5.m1.9.9.1.1.1.1.1.1"><msub id="S3.E5.m1.9.9.1.1.1.1.1.1.2"><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.2.2">x</mi><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.2.3">t</mi></msub><mo fence="false" id="S3.E5.m1.9.9.1.1.1.1.1.1.1">|</mo><msub id="S3.E5.m1.9.9.1.1.1.1.1.1.3"><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.3.2">x</mi><mrow id="S3.E5.m1.9.9.1.1.1.1.1.1.3.3"><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.3.3.2"></mi><mo id="S3.E5.m1.9.9.1.1.1.1.1.1.3.3.1">&lt;</mo><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.3.3.3">t</mi></mrow></msub></mrow><mo stretchy="false" id="S3.E5.m1.9.9.1.1.1.1.1.3">)</mo></mrow></mrow><mo id="S3.E5.m1.9.9.1.1.3">=</mo><mrow id="S3.E5.m1.9.9.1.1.2"><msub id="S3.E5.m1.9.9.1.1.2.3"><mi id="S3.E5.m1.9.9.1.1.2.3.2">P</mi><mrow id="S3.E5.m1.9.9.1.1.2.3.3"><mi id="S3.E5.m1.9.9.1.1.2.3.3.2">L</mi><mo lspace="0em" rspace="0em" id="S3.E5.m1.9.9.1.1.2.3.3.1">​</mo><mi id="S3.E5.m1.9.9.1.1.2.3.3.3">M</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.E5.m1.9.9.1.1.2.2">​</mo><mrow id="S3.E5.m1.9.9.1.1.2.1.1"><mo stretchy="false" id="S3.E5.m1.9.9.1.1.2.1.1.2">(</mo><mrow id="S3.E5.m1.9.9.1.1.2.1.1.1"><msub id="S3.E5.m1.9.9.1.1.2.1.1.1.2"><mi id="S3.E5.m1.9.9.1.1.2.1.1.1.2.2">x</mi><mi id="S3.E5.m1.9.9.1.1.2.1.1.1.2.3">t</mi></msub><mo fence="false" id="S3.E5.m1.9.9.1.1.2.1.1.1.1">|</mo><msub id="S3.E5.m1.9.9.1.1.2.1.1.1.3"><mi id="S3.E5.m1.9.9.1.1.2.1.1.1.3.2">x</mi><mrow id="S3.E5.m1.9.9.1.1.2.1.1.1.3.3"><mi id="S3.E5.m1.9.9.1.1.2.1.1.1.3.3.2"></mi><mo id="S3.E5.m1.9.9.1.1.2.1.1.1.3.3.1">&lt;</mo><mi id="S3.E5.m1.9.9.1.1.2.1.1.1.3.3.3">t</mi></mrow></msub></mrow><mo stretchy="false" id="S3.E5.m1.9.9.1.1.2.1.1.3">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E5.m1.9.9.1.1.2.2a">​</mo><mfrac id="S3.E5.m1.8.8"><mrow id="S3.E5.m1.4.4.4"><msub id="S3.E5.m1.4.4.4.6"><mi id="S3.E5.m1.4.4.4.6.2">𝔼</mi><mrow id="S3.E5.m1.1.1.1.1.1"><msubsup id="S3.E5.m1.1.1.1.1.1.2"><mi id="S3.E5.m1.1.1.1.1.1.2.2.2">x</mi><mrow id="S3.E5.m1.1.1.1.1.1.2.2.3"><mi id="S3.E5.m1.1.1.1.1.1.2.2.3.2">t</mi><mo id="S3.E5.m1.1.1.1.1.1.2.2.3.1">+</mo><mn id="S3.E5.m1.1.1.1.1.1.2.2.3.3">1</mn></mrow><mo id="S3.E5.m1.1.1.1.1.1.2.3">′</mo></msubsup><mo id="S3.E5.m1.1.1.1.1.1.3">,</mo><mi mathvariant="normal" id="S3.E5.m1.1.1.1.1.1.1">⋯</mi><mo id="S3.E5.m1.1.1.1.1.1.4">,</mo><msubsup id="S3.E5.m1.1.1.1.1.1.5"><mi id="S3.E5.m1.1.1.1.1.1.5.2.2">x</mi><mi id="S3.E5.m1.1.1.1.1.1.5.2.3">T</mi><mo id="S3.E5.m1.1.1.1.1.1.5.3">′</mo></msubsup><mo id="S3.E5.m1.1.1.1.1.1.6">∼</mo><msub id="S3.E5.m1.1.1.1.1.1.7"><mi id="S3.E5.m1.1.1.1.1.1.7.2">P</mi><mrow id="S3.E5.m1.1.1.1.1.1.7.3"><mi id="S3.E5.m1.1.1.1.1.1.7.3.2">L</mi><mo lspace="0em" rspace="0em" id="S3.E5.m1.1.1.1.1.1.7.3.1">​</mo><mi id="S3.E5.m1.1.1.1.1.1.7.3.3">M</mi></mrow></msub><mrow id="S3.E5.m1.1.1.1.1.1.8"><mo stretchy="false" id="S3.E5.m1.1.1.1.1.1.8.1">(</mo><mo lspace="0em" rspace="0em" id="S3.E5.m1.1.1.1.1.1.8.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="S3.E5.m1.1.1.1.1.1.8.3">|</mo><msub id="S3.E5.m1.1.1.1.1.1.8.4"><mi id="S3.E5.m1.1.1.1.1.1.8.4.2">x</mi><mrow id="S3.E5.m1.1.1.1.1.1.8.4.3"><mi id="S3.E5.m1.1.1.1.1.1.8.4.3.2"></mi><mo id="S3.E5.m1.1.1.1.1.1.8.4.3.1">≤</mo><mi id="S3.E5.m1.1.1.1.1.1.8.4.3.3">t</mi></mrow></msub><mo stretchy="false" id="S3.E5.m1.1.1.1.1.1.8.5">)</mo></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="S3.E5.m1.4.4.4.5">​</mo><mrow id="S3.E5.m1.4.4.4.4.1"><mo stretchy="false" id="S3.E5.m1.4.4.4.4.1.2">[</mo><mrow id="S3.E5.m1.4.4.4.4.1.1.1"><mi id="S3.E5.m1.3.3.3.3">exp</mi><mo id="S3.E5.m1.4.4.4.4.1.1.1a">⁡</mo><mrow id="S3.E5.m1.4.4.4.4.1.1.1.1"><mo stretchy="false" id="S3.E5.m1.4.4.4.4.1.1.1.1.2">(</mo><mrow id="S3.E5.m1.4.4.4.4.1.1.1.1.1"><mo id="S3.E5.m1.4.4.4.4.1.1.1.1.1a">−</mo><mrow id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3"><msub id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.5"><mi id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.5.2">E</mi><mi id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.5.3">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.4">​</mo><mrow id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.3.3"><mo stretchy="false" id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.3.3.4">(</mo><msub id="S3.E5.m1.4.4.4.4.1.1.1.1.1.1.1.1.1"><mi id="S3.E5.m1.4.4.4.4.1.1.1.1.1.1.1.1.1.2">x</mi><mrow id="S3.E5.m1.4.4.4.4.1.1.1.1.1.1.1.1.1.3"><mi id="S3.E5.m1.4.4.4.4.1.1.1.1.1.1.1.1.1.3.2"></mi><mo id="S3.E5.m1.4.4.4.4.1.1.1.1.1.1.1.1.1.3.1">≤</mo><mi id="S3.E5.m1.4.4.4.4.1.1.1.1.1.1.1.1.1.3.3">t</mi></mrow></msub><mo id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.3.3.5">,</mo><msubsup id="S3.E5.m1.4.4.4.4.1.1.1.1.1.2.2.2.2"><mi id="S3.E5.m1.4.4.4.4.1.1.1.1.1.2.2.2.2.2.2">x</mi><mrow id="S3.E5.m1.4.4.4.4.1.1.1.1.1.2.2.2.2.2.3"><mi id="S3.E5.m1.4.4.4.4.1.1.1.1.1.2.2.2.2.2.3.2">t</mi><mo id="S3.E5.m1.4.4.4.4.1.1.1.1.1.2.2.2.2.2.3.1">+</mo><mn id="S3.E5.m1.4.4.4.4.1.1.1.1.1.2.2.2.2.2.3.3">1</mn></mrow><mo id="S3.E5.m1.4.4.4.4.1.1.1.1.1.2.2.2.2.3">′</mo></msubsup><mo id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.3.3.6">,</mo><mi mathvariant="normal" id="S3.E5.m1.2.2.2.2">⋯</mi><mo id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.3.3.7">,</mo><msubsup id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.3.3.3"><mi id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.3.3.3.2.2">x</mi><mi id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.3.3.3.2.3">T</mi><mo id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.3.3.3.3">′</mo></msubsup><mo stretchy="false" id="S3.E5.m1.4.4.4.4.1.1.1.1.1.3.3.3.8">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.E5.m1.4.4.4.4.1.1.1.1.3">)</mo></mrow></mrow><mo stretchy="false" id="S3.E5.m1.4.4.4.4.1.3">]</mo></mrow></mrow><mrow id="S3.E5.m1.8.8.8"><msub id="S3.E5.m1.8.8.8.6"><mi id="S3.E5.m1.8.8.8.6.2">𝔼</mi><mrow id="S3.E5.m1.5.5.5.1.1"><msubsup id="S3.E5.m1.5.5.5.1.1.2"><mi id="S3.E5.m1.5.5.5.1.1.2.2.2">x</mi><mi id="S3.E5.m1.5.5.5.1.1.2.2.3">t</mi><mo id="S3.E5.m1.5.5.5.1.1.2.3">′</mo></msubsup><mo id="S3.E5.m1.5.5.5.1.1.3">,</mo><mi mathvariant="normal" id="S3.E5.m1.5.5.5.1.1.1">⋯</mi><mo id="S3.E5.m1.5.5.5.1.1.4">,</mo><msubsup id="S3.E5.m1.5.5.5.1.1.5"><mi id="S3.E5.m1.5.5.5.1.1.5.2.2">x</mi><mi id="S3.E5.m1.5.5.5.1.1.5.2.3">T</mi><mo id="S3.E5.m1.5.5.5.1.1.5.3">′</mo></msubsup><mo id="S3.E5.m1.5.5.5.1.1.6">∼</mo><msub id="S3.E5.m1.5.5.5.1.1.7"><mi id="S3.E5.m1.5.5.5.1.1.7.2">P</mi><mrow id="S3.E5.m1.5.5.5.1.1.7.3"><mi id="S3.E5.m1.5.5.5.1.1.7.3.2">L</mi><mo lspace="0em" rspace="0em" id="S3.E5.m1.5.5.5.1.1.7.3.1">​</mo><mi id="S3.E5.m1.5.5.5.1.1.7.3.3">M</mi></mrow></msub><mrow id="S3.E5.m1.5.5.5.1.1.8"><mo stretchy="false" id="S3.E5.m1.5.5.5.1.1.8.1">(</mo><mo lspace="0em" rspace="0em" id="S3.E5.m1.5.5.5.1.1.8.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="S3.E5.m1.5.5.5.1.1.8.3">|</mo><msub id="S3.E5.m1.5.5.5.1.1.8.4"><mi id="S3.E5.m1.5.5.5.1.1.8.4.2">x</mi><mrow id="S3.E5.m1.5.5.5.1.1.8.4.3"><mi id="S3.E5.m1.5.5.5.1.1.8.4.3.2"></mi><mo id="S3.E5.m1.5.5.5.1.1.8.4.3.1">≤</mo><mrow id="S3.E5.m1.5.5.5.1.1.8.4.3.3"><mi id="S3.E5.m1.5.5.5.1.1.8.4.3.3.2">t</mi><mo id="S3.E5.m1.5.5.5.1.1.8.4.3.3.1">−</mo><mn id="S3.E5.m1.5.5.5.1.1.8.4.3.3.3">1</mn></mrow></mrow></msub><mo stretchy="false" id="S3.E5.m1.5.5.5.1.1.8.5">)</mo></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="S3.E5.m1.8.8.8.5">​</mo><mrow id="S3.E5.m1.8.8.8.4.1"><mo stretchy="false" id="S3.E5.m1.8.8.8.4.1.2">[</mo><mrow id="S3.E5.m1.8.8.8.4.1.1.1"><mi id="S3.E5.m1.7.7.7.3">exp</mi><mo id="S3.E5.m1.8.8.8.4.1.1.1a">⁡</mo><mrow id="S3.E5.m1.8.8.8.4.1.1.1.1"><mo stretchy="false" id="S3.E5.m1.8.8.8.4.1.1.1.1.2">(</mo><mrow id="S3.E5.m1.8.8.8.4.1.1.1.1.1"><mo id="S3.E5.m1.8.8.8.4.1.1.1.1.1a">−</mo><mrow id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3"><msub id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.5"><mi id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.5.2">E</mi><mi id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.5.3">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.4">​</mo><mrow id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.3.3"><mo stretchy="false" id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.3.3.4">(</mo><msub id="S3.E5.m1.8.8.8.4.1.1.1.1.1.1.1.1.1"><mi id="S3.E5.m1.8.8.8.4.1.1.1.1.1.1.1.1.1.2">x</mi><mrow id="S3.E5.m1.8.8.8.4.1.1.1.1.1.1.1.1.1.3"><mi id="S3.E5.m1.8.8.8.4.1.1.1.1.1.1.1.1.1.3.2"></mi><mo id="S3.E5.m1.8.8.8.4.1.1.1.1.1.1.1.1.1.3.1">≤</mo><mrow id="S3.E5.m1.8.8.8.4.1.1.1.1.1.1.1.1.1.3.3"><mi id="S3.E5.m1.8.8.8.4.1.1.1.1.1.1.1.1.1.3.3.2">t</mi><mo id="S3.E5.m1.8.8.8.4.1.1.1.1.1.1.1.1.1.3.3.1">−</mo><mn id="S3.E5.m1.8.8.8.4.1.1.1.1.1.1.1.1.1.3.3.3">1</mn></mrow></mrow></msub><mo id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.3.3.5">,</mo><msubsup id="S3.E5.m1.8.8.8.4.1.1.1.1.1.2.2.2.2"><mi id="S3.E5.m1.8.8.8.4.1.1.1.1.1.2.2.2.2.2.2">x</mi><mi id="S3.E5.m1.8.8.8.4.1.1.1.1.1.2.2.2.2.2.3">t</mi><mo id="S3.E5.m1.8.8.8.4.1.1.1.1.1.2.2.2.2.3">′</mo></msubsup><mo id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.3.3.6">,</mo><mi mathvariant="normal" id="S3.E5.m1.6.6.6.2">⋯</mi><mo id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.3.3.7">,</mo><msubsup id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.3.3.3"><mi id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.3.3.3.2.2">x</mi><mi id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.3.3.3.2.3">T</mi><mo id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.3.3.3.3">′</mo></msubsup><mo stretchy="false" id="S3.E5.m1.8.8.8.4.1.1.1.1.1.3.3.3.8">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.E5.m1.8.8.8.4.1.1.1.1.3">)</mo></mrow></mrow><mo stretchy="false" id="S3.E5.m1.8.8.8.4.1.3">]</mo></mrow></mrow></mfrac></mrow></mrow><mo lspace="0em" id="S3.E5.m1.9.9.1.2">.</mo></mrow><annotation encoding="application/x-tex" id="S3.E5.m1.9b">P(x_{t}|x_{&lt;t})=P_{LM}(x_{t}|x_{&lt;t})\frac{\mathbb{E}_{x_{t+1}^{\prime},\cdots,x_{T}^{\prime}\sim P_{LM}(\cdot|x_{\leq t})}[\exp(-E_{\theta}(x_{\leq t},x_{t+1}^{\prime},\cdots,x_{T}^{\prime}))]}{\mathbb{E}_{x_{t}^{\prime},\cdots,x_{T}^{\prime}\sim P_{LM}(\cdot|x_{\leq t-1})}[\exp(-E_{\theta}(x_{\leq t-1},x_{t}^{\prime},\cdots,x_{T}^{\prime}))]}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS2.p5" class="ltx_para ltx_noindent">
<p id="S3.SS2.p5.7" class="ltx_p">The step-wise probabilities in Eq. <a href="#S3.E5" title="In 3.2 Evaluation ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> are an instance of importance sampling <cite class="ltx_cite ltx_citemacro_citep">(Horvitz &amp; Thompson, <a href="#bib.bib16" title="" class="ltx_ref">1952</a>)</cite>.
The basic <math id="S3.SS2.p5.1.m1.1" class="ltx_Math" alttext="P_{LM}" display="inline"><semantics id="S3.SS2.p5.1.m1.1a"><msub id="S3.SS2.p5.1.m1.1.1" xref="S3.SS2.p5.1.m1.1.1.cmml"><mi id="S3.SS2.p5.1.m1.1.1.2" xref="S3.SS2.p5.1.m1.1.1.2.cmml">P</mi><mrow id="S3.SS2.p5.1.m1.1.1.3" xref="S3.SS2.p5.1.m1.1.1.3.cmml"><mi id="S3.SS2.p5.1.m1.1.1.3.2" xref="S3.SS2.p5.1.m1.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p5.1.m1.1.1.3.1" xref="S3.SS2.p5.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p5.1.m1.1.1.3.3" xref="S3.SS2.p5.1.m1.1.1.3.3.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.1.m1.1b"><apply id="S3.SS2.p5.1.m1.1.1.cmml" xref="S3.SS2.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.1.m1.1.1.1.cmml" xref="S3.SS2.p5.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p5.1.m1.1.1.2.cmml" xref="S3.SS2.p5.1.m1.1.1.2">𝑃</ci><apply id="S3.SS2.p5.1.m1.1.1.3.cmml" xref="S3.SS2.p5.1.m1.1.1.3"><times id="S3.SS2.p5.1.m1.1.1.3.1.cmml" xref="S3.SS2.p5.1.m1.1.1.3.1"></times><ci id="S3.SS2.p5.1.m1.1.1.3.2.cmml" xref="S3.SS2.p5.1.m1.1.1.3.2">𝐿</ci><ci id="S3.SS2.p5.1.m1.1.1.3.3.cmml" xref="S3.SS2.p5.1.m1.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.1.m1.1c">P_{LM}</annotation></semantics></math> distribution is adjusted by the probability assigned to token <math id="S3.SS2.p5.2.m2.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S3.SS2.p5.2.m2.1a"><msub id="S3.SS2.p5.2.m2.1.1" xref="S3.SS2.p5.2.m2.1.1.cmml"><mi id="S3.SS2.p5.2.m2.1.1.2" xref="S3.SS2.p5.2.m2.1.1.2.cmml">x</mi><mi id="S3.SS2.p5.2.m2.1.1.3" xref="S3.SS2.p5.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.2.m2.1b"><apply id="S3.SS2.p5.2.m2.1.1.cmml" xref="S3.SS2.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.2.m2.1.1.1.cmml" xref="S3.SS2.p5.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.p5.2.m2.1.1.2.cmml" xref="S3.SS2.p5.2.m2.1.1.2">𝑥</ci><ci id="S3.SS2.p5.2.m2.1.1.3.cmml" xref="S3.SS2.p5.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.2.m2.1c">x_{t}</annotation></semantics></math> by the energy function (numerator
is clamped at <math id="S3.SS2.p5.3.m3.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S3.SS2.p5.3.m3.1a"><msub id="S3.SS2.p5.3.m3.1.1" xref="S3.SS2.p5.3.m3.1.1.cmml"><mi id="S3.SS2.p5.3.m3.1.1.2" xref="S3.SS2.p5.3.m3.1.1.2.cmml">x</mi><mi id="S3.SS2.p5.3.m3.1.1.3" xref="S3.SS2.p5.3.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.3.m3.1b"><apply id="S3.SS2.p5.3.m3.1.1.cmml" xref="S3.SS2.p5.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.3.m3.1.1.1.cmml" xref="S3.SS2.p5.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.p5.3.m3.1.1.2.cmml" xref="S3.SS2.p5.3.m3.1.1.2">𝑥</ci><ci id="S3.SS2.p5.3.m3.1.1.3.cmml" xref="S3.SS2.p5.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.3.m3.1c">x_{t}</annotation></semantics></math> while denominator sums over all the possible values of the token at position t), with the additional
marginalization over all subsequent tokens up to the horizon <math id="S3.SS2.p5.4.m4.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS2.p5.4.m4.1a"><mi id="S3.SS2.p5.4.m4.1.1" xref="S3.SS2.p5.4.m4.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.4.m4.1b"><ci id="S3.SS2.p5.4.m4.1.1.cmml" xref="S3.SS2.p5.4.m4.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.4.m4.1c">T</annotation></semantics></math>. Since the summation involves exponentially many terms, unless
<math id="S3.SS2.p5.5.m5.1" class="ltx_Math" alttext="t=T" display="inline"><semantics id="S3.SS2.p5.5.m5.1a"><mrow id="S3.SS2.p5.5.m5.1.1" xref="S3.SS2.p5.5.m5.1.1.cmml"><mi id="S3.SS2.p5.5.m5.1.1.2" xref="S3.SS2.p5.5.m5.1.1.2.cmml">t</mi><mo id="S3.SS2.p5.5.m5.1.1.1" xref="S3.SS2.p5.5.m5.1.1.1.cmml">=</mo><mi id="S3.SS2.p5.5.m5.1.1.3" xref="S3.SS2.p5.5.m5.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.5.m5.1b"><apply id="S3.SS2.p5.5.m5.1.1.cmml" xref="S3.SS2.p5.5.m5.1.1"><eq id="S3.SS2.p5.5.m5.1.1.1.cmml" xref="S3.SS2.p5.5.m5.1.1.1"></eq><ci id="S3.SS2.p5.5.m5.1.1.2.cmml" xref="S3.SS2.p5.5.m5.1.1.2">𝑡</ci><ci id="S3.SS2.p5.5.m5.1.1.3.cmml" xref="S3.SS2.p5.5.m5.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.5.m5.1c">t=T</annotation></semantics></math>, this is approximated by samples drawn by <math id="S3.SS2.p5.6.m6.1" class="ltx_Math" alttext="P_{LM}" display="inline"><semantics id="S3.SS2.p5.6.m6.1a"><msub id="S3.SS2.p5.6.m6.1.1" xref="S3.SS2.p5.6.m6.1.1.cmml"><mi id="S3.SS2.p5.6.m6.1.1.2" xref="S3.SS2.p5.6.m6.1.1.2.cmml">P</mi><mrow id="S3.SS2.p5.6.m6.1.1.3" xref="S3.SS2.p5.6.m6.1.1.3.cmml"><mi id="S3.SS2.p5.6.m6.1.1.3.2" xref="S3.SS2.p5.6.m6.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p5.6.m6.1.1.3.1" xref="S3.SS2.p5.6.m6.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p5.6.m6.1.1.3.3" xref="S3.SS2.p5.6.m6.1.1.3.3.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.6.m6.1b"><apply id="S3.SS2.p5.6.m6.1.1.cmml" xref="S3.SS2.p5.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.6.m6.1.1.1.cmml" xref="S3.SS2.p5.6.m6.1.1">subscript</csymbol><ci id="S3.SS2.p5.6.m6.1.1.2.cmml" xref="S3.SS2.p5.6.m6.1.1.2">𝑃</ci><apply id="S3.SS2.p5.6.m6.1.1.3.cmml" xref="S3.SS2.p5.6.m6.1.1.3"><times id="S3.SS2.p5.6.m6.1.1.3.1.cmml" xref="S3.SS2.p5.6.m6.1.1.3.1"></times><ci id="S3.SS2.p5.6.m6.1.1.3.2.cmml" xref="S3.SS2.p5.6.m6.1.1.3.2">𝐿</ci><ci id="S3.SS2.p5.6.m6.1.1.3.3.cmml" xref="S3.SS2.p5.6.m6.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.6.m6.1c">P_{LM}</annotation></semantics></math>. Since both the numerator and the denominator take the same form as the partition function, we also use Eq. <a href="#S3.E4" title="In Theorem 2. ‣ 3.2 Evaluation ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> to estimate the upper and lower bounds. E.g., the lower bound of <math id="S3.SS2.p5.7.m7.1" class="ltx_Math" alttext="\log P(x_{t}|x_{&lt;t})" display="inline"><semantics id="S3.SS2.p5.7.m7.1a"><mrow id="S3.SS2.p5.7.m7.1.1" xref="S3.SS2.p5.7.m7.1.1.cmml"><mrow id="S3.SS2.p5.7.m7.1.1.3" xref="S3.SS2.p5.7.m7.1.1.3.cmml"><mi id="S3.SS2.p5.7.m7.1.1.3.1" xref="S3.SS2.p5.7.m7.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S3.SS2.p5.7.m7.1.1.3a" xref="S3.SS2.p5.7.m7.1.1.3.cmml">⁡</mo><mi id="S3.SS2.p5.7.m7.1.1.3.2" xref="S3.SS2.p5.7.m7.1.1.3.2.cmml">P</mi></mrow><mo lspace="0em" rspace="0em" id="S3.SS2.p5.7.m7.1.1.2" xref="S3.SS2.p5.7.m7.1.1.2.cmml">​</mo><mrow id="S3.SS2.p5.7.m7.1.1.1.1" xref="S3.SS2.p5.7.m7.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p5.7.m7.1.1.1.1.2" xref="S3.SS2.p5.7.m7.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p5.7.m7.1.1.1.1.1" xref="S3.SS2.p5.7.m7.1.1.1.1.1.cmml"><msub id="S3.SS2.p5.7.m7.1.1.1.1.1.2" xref="S3.SS2.p5.7.m7.1.1.1.1.1.2.cmml"><mi id="S3.SS2.p5.7.m7.1.1.1.1.1.2.2" xref="S3.SS2.p5.7.m7.1.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS2.p5.7.m7.1.1.1.1.1.2.3" xref="S3.SS2.p5.7.m7.1.1.1.1.1.2.3.cmml">t</mi></msub><mo fence="false" id="S3.SS2.p5.7.m7.1.1.1.1.1.1" xref="S3.SS2.p5.7.m7.1.1.1.1.1.1.cmml">|</mo><msub id="S3.SS2.p5.7.m7.1.1.1.1.1.3" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3.cmml"><mi id="S3.SS2.p5.7.m7.1.1.1.1.1.3.2" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3.2.cmml">x</mi><mrow id="S3.SS2.p5.7.m7.1.1.1.1.1.3.3" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.cmml"><mi id="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.2" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.2.cmml"></mi><mo id="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.1" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.3" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.3.cmml">t</mi></mrow></msub></mrow><mo stretchy="false" id="S3.SS2.p5.7.m7.1.1.1.1.3" xref="S3.SS2.p5.7.m7.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.7.m7.1b"><apply id="S3.SS2.p5.7.m7.1.1.cmml" xref="S3.SS2.p5.7.m7.1.1"><times id="S3.SS2.p5.7.m7.1.1.2.cmml" xref="S3.SS2.p5.7.m7.1.1.2"></times><apply id="S3.SS2.p5.7.m7.1.1.3.cmml" xref="S3.SS2.p5.7.m7.1.1.3"><log id="S3.SS2.p5.7.m7.1.1.3.1.cmml" xref="S3.SS2.p5.7.m7.1.1.3.1"></log><ci id="S3.SS2.p5.7.m7.1.1.3.2.cmml" xref="S3.SS2.p5.7.m7.1.1.3.2">𝑃</ci></apply><apply id="S3.SS2.p5.7.m7.1.1.1.1.1.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1"><csymbol cd="latexml" id="S3.SS2.p5.7.m7.1.1.1.1.1.1.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.1">conditional</csymbol><apply id="S3.SS2.p5.7.m7.1.1.1.1.1.2.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p5.7.m7.1.1.1.1.1.2.1.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p5.7.m7.1.1.1.1.1.2.2.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS2.p5.7.m7.1.1.1.1.1.2.3.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S3.SS2.p5.7.m7.1.1.1.1.1.3.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p5.7.m7.1.1.1.1.1.3.1.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p5.7.m7.1.1.1.1.1.3.2.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3.2">𝑥</ci><apply id="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3.3"><lt id="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.1.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.2.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.2">absent</csymbol><ci id="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.3.cmml" xref="S3.SS2.p5.7.m7.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.7.m7.1c">\log P(x_{t}|x_{&lt;t})</annotation></semantics></math> can be obtained by using the lower bound of the numerator and the upper bound of the denominator.</p>
</div>
<div id="S3.SS2.p6" class="ltx_para ltx_noindent">
<p id="S3.SS2.p6.1" class="ltx_p">For <math id="S3.SS2.p6.1.m1.1" class="ltx_Math" alttext="t=T" display="inline"><semantics id="S3.SS2.p6.1.m1.1a"><mrow id="S3.SS2.p6.1.m1.1.1" xref="S3.SS2.p6.1.m1.1.1.cmml"><mi id="S3.SS2.p6.1.m1.1.1.2" xref="S3.SS2.p6.1.m1.1.1.2.cmml">t</mi><mo id="S3.SS2.p6.1.m1.1.1.1" xref="S3.SS2.p6.1.m1.1.1.1.cmml">=</mo><mi id="S3.SS2.p6.1.m1.1.1.3" xref="S3.SS2.p6.1.m1.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p6.1.m1.1b"><apply id="S3.SS2.p6.1.m1.1.1.cmml" xref="S3.SS2.p6.1.m1.1.1"><eq id="S3.SS2.p6.1.m1.1.1.1.cmml" xref="S3.SS2.p6.1.m1.1.1.1"></eq><ci id="S3.SS2.p6.1.m1.1.1.2.cmml" xref="S3.SS2.p6.1.m1.1.1.2">𝑡</ci><ci id="S3.SS2.p6.1.m1.1.1.3.cmml" xref="S3.SS2.p6.1.m1.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p6.1.m1.1c">t=T</annotation></semantics></math>, we can calculate the log probability by exhaustive enumeration. This gives us an idea of the true performance
of our model at the last step, and it also provides a sanity-check of the tightness of our estimators.</p>
</div>
<figure id="alg1" class="ltx_float ltx_algorithm">
<div id="alg1.1" class="ltx_listing ltx_lst_numbers_left ltx_listing">
<div id="alg1.1.1" class="ltx_listingline">






















































<div id="alg1.1.1.1" class="ltx_listing ltx_listing">
<div id="alg1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span>  <span id="alg1.l1.1" class="ltx_text ltx_font_bold">Input:</span> number of samples <math id="alg1.l1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="alg1.l1.m1.1a"><mi id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><ci id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">n</annotation></semantics></math> drawn from <math id="alg1.l1.m2.1" class="ltx_Math" alttext="P_{LM}" display="inline"><semantics id="alg1.l1.m2.1a"><msub id="alg1.l1.m2.1.1" xref="alg1.l1.m2.1.1.cmml"><mi id="alg1.l1.m2.1.1.2" xref="alg1.l1.m2.1.1.2.cmml">P</mi><mrow id="alg1.l1.m2.1.1.3" xref="alg1.l1.m2.1.1.3.cmml"><mi id="alg1.l1.m2.1.1.3.2" xref="alg1.l1.m2.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m2.1.1.3.1" xref="alg1.l1.m2.1.1.3.1.cmml">​</mo><mi id="alg1.l1.m2.1.1.3.3" xref="alg1.l1.m2.1.1.3.3.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="alg1.l1.m2.1b"><apply id="alg1.l1.m2.1.1.cmml" xref="alg1.l1.m2.1.1"><csymbol cd="ambiguous" id="alg1.l1.m2.1.1.1.cmml" xref="alg1.l1.m2.1.1">subscript</csymbol><ci id="alg1.l1.m2.1.1.2.cmml" xref="alg1.l1.m2.1.1.2">𝑃</ci><apply id="alg1.l1.m2.1.1.3.cmml" xref="alg1.l1.m2.1.1.3"><times id="alg1.l1.m2.1.1.3.1.cmml" xref="alg1.l1.m2.1.1.3.1"></times><ci id="alg1.l1.m2.1.1.3.2.cmml" xref="alg1.l1.m2.1.1.3.2">𝐿</ci><ci id="alg1.l1.m2.1.1.3.3.cmml" xref="alg1.l1.m2.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m2.1c">P_{LM}</annotation></semantics></math>, value of <math id="alg1.l1.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="alg1.l1.m3.1a"><mi id="alg1.l1.m3.1.1" xref="alg1.l1.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m3.1b"><ci id="alg1.l1.m3.1.1.cmml" xref="alg1.l1.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m3.1c">k</annotation></semantics></math> in top-k

</div>
<div id="alg1.1.1.1.1" class="ltx_listingline">
<span id="alg1.1.1.1.1.2" class="ltx_text ltx_font_typewriter">// </span><span id="alg1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Get a set of samples from <math id="alg1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="P_{LM}" display="inline"><semantics id="alg1.1.1.1.1.1.m1.1a"><msub id="alg1.1.1.1.1.1.m1.1.1" xref="alg1.1.1.1.1.1.m1.1.1.cmml"><mi id="alg1.1.1.1.1.1.m1.1.1.2" xref="alg1.1.1.1.1.1.m1.1.1.2.cmml">P</mi><mrow id="alg1.1.1.1.1.1.m1.1.1.3" xref="alg1.1.1.1.1.1.m1.1.1.3.cmml"><mi id="alg1.1.1.1.1.1.m1.1.1.3.2" xref="alg1.1.1.1.1.1.m1.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="alg1.1.1.1.1.1.m1.1.1.3.1" xref="alg1.1.1.1.1.1.m1.1.1.3.1.cmml">​</mo><mi id="alg1.1.1.1.1.1.m1.1.1.3.3" xref="alg1.1.1.1.1.1.m1.1.1.3.3.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="alg1.1.1.1.1.1.m1.1b"><apply id="alg1.1.1.1.1.1.m1.1.1.cmml" xref="alg1.1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="alg1.1.1.1.1.1.m1.1.1.1.cmml" xref="alg1.1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="alg1.1.1.1.1.1.m1.1.1.2.cmml" xref="alg1.1.1.1.1.1.m1.1.1.2">𝑃</ci><apply id="alg1.1.1.1.1.1.m1.1.1.3.cmml" xref="alg1.1.1.1.1.1.m1.1.1.3"><times id="alg1.1.1.1.1.1.m1.1.1.3.1.cmml" xref="alg1.1.1.1.1.1.m1.1.1.3.1"></times><ci id="alg1.1.1.1.1.1.m1.1.1.3.2.cmml" xref="alg1.1.1.1.1.1.m1.1.1.3.2">𝐿</ci><ci id="alg1.1.1.1.1.1.m1.1.1.3.3.cmml" xref="alg1.1.1.1.1.1.m1.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.1.1.1.1.1.m1.1c">P_{LM}</annotation></semantics></math><span id="alg1.1.1.1.1.1.1" class="ltx_text" style="float:right;"></span></span>
</div>
<div id="alg1.1.1.1.2" class="ltx_listingline">

</div>
<div id="alg1.l2" class="ltx_listingline">  sample <math id="alg1.l2.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="alg1.l2.m1.1a"><mi id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><ci id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">n</annotation></semantics></math> samples <math id="alg1.l2.m2.3" class="ltx_Math" alttext="\{x^{1},\cdots,x^{n}\}" display="inline"><semantics id="alg1.l2.m2.3a"><mrow id="alg1.l2.m2.3.3.2" xref="alg1.l2.m2.3.3.3.cmml"><mo stretchy="false" id="alg1.l2.m2.3.3.2.3" xref="alg1.l2.m2.3.3.3.cmml">{</mo><msup id="alg1.l2.m2.2.2.1.1" xref="alg1.l2.m2.2.2.1.1.cmml"><mi id="alg1.l2.m2.2.2.1.1.2" xref="alg1.l2.m2.2.2.1.1.2.cmml">x</mi><mn id="alg1.l2.m2.2.2.1.1.3" xref="alg1.l2.m2.2.2.1.1.3.cmml">1</mn></msup><mo id="alg1.l2.m2.3.3.2.4" xref="alg1.l2.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="alg1.l2.m2.1.1" xref="alg1.l2.m2.1.1.cmml">⋯</mi><mo id="alg1.l2.m2.3.3.2.5" xref="alg1.l2.m2.3.3.3.cmml">,</mo><msup id="alg1.l2.m2.3.3.2.2" xref="alg1.l2.m2.3.3.2.2.cmml"><mi id="alg1.l2.m2.3.3.2.2.2" xref="alg1.l2.m2.3.3.2.2.2.cmml">x</mi><mi id="alg1.l2.m2.3.3.2.2.3" xref="alg1.l2.m2.3.3.2.2.3.cmml">n</mi></msup><mo stretchy="false" id="alg1.l2.m2.3.3.2.6" xref="alg1.l2.m2.3.3.3.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="alg1.l2.m2.3b"><set id="alg1.l2.m2.3.3.3.cmml" xref="alg1.l2.m2.3.3.2"><apply id="alg1.l2.m2.2.2.1.1.cmml" xref="alg1.l2.m2.2.2.1.1"><csymbol cd="ambiguous" id="alg1.l2.m2.2.2.1.1.1.cmml" xref="alg1.l2.m2.2.2.1.1">superscript</csymbol><ci id="alg1.l2.m2.2.2.1.1.2.cmml" xref="alg1.l2.m2.2.2.1.1.2">𝑥</ci><cn type="integer" id="alg1.l2.m2.2.2.1.1.3.cmml" xref="alg1.l2.m2.2.2.1.1.3">1</cn></apply><ci id="alg1.l2.m2.1.1.cmml" xref="alg1.l2.m2.1.1">⋯</ci><apply id="alg1.l2.m2.3.3.2.2.cmml" xref="alg1.l2.m2.3.3.2.2"><csymbol cd="ambiguous" id="alg1.l2.m2.3.3.2.2.1.cmml" xref="alg1.l2.m2.3.3.2.2">superscript</csymbol><ci id="alg1.l2.m2.3.3.2.2.2.cmml" xref="alg1.l2.m2.3.3.2.2.2">𝑥</ci><ci id="alg1.l2.m2.3.3.2.2.3.cmml" xref="alg1.l2.m2.3.3.2.2.3">𝑛</ci></apply></set></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m2.3c">\{x^{1},\cdots,x^{n}\}</annotation></semantics></math> from <math id="alg1.l2.m3.1" class="ltx_Math" alttext="P_{LM}" display="inline"><semantics id="alg1.l2.m3.1a"><msub id="alg1.l2.m3.1.1" xref="alg1.l2.m3.1.1.cmml"><mi id="alg1.l2.m3.1.1.2" xref="alg1.l2.m3.1.1.2.cmml">P</mi><mrow id="alg1.l2.m3.1.1.3" xref="alg1.l2.m3.1.1.3.cmml"><mi id="alg1.l2.m3.1.1.3.2" xref="alg1.l2.m3.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m3.1.1.3.1" xref="alg1.l2.m3.1.1.3.1.cmml">​</mo><mi id="alg1.l2.m3.1.1.3.3" xref="alg1.l2.m3.1.1.3.3.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="alg1.l2.m3.1b"><apply id="alg1.l2.m3.1.1.cmml" xref="alg1.l2.m3.1.1"><csymbol cd="ambiguous" id="alg1.l2.m3.1.1.1.cmml" xref="alg1.l2.m3.1.1">subscript</csymbol><ci id="alg1.l2.m3.1.1.2.cmml" xref="alg1.l2.m3.1.1.2">𝑃</ci><apply id="alg1.l2.m3.1.1.3.cmml" xref="alg1.l2.m3.1.1.3"><times id="alg1.l2.m3.1.1.3.1.cmml" xref="alg1.l2.m3.1.1.3.1"></times><ci id="alg1.l2.m3.1.1.3.2.cmml" xref="alg1.l2.m3.1.1.3.2">𝐿</ci><ci id="alg1.l2.m3.1.1.3.3.cmml" xref="alg1.l2.m3.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m3.1c">P_{LM}</annotation></semantics></math> with top-k sampling

</div>
<div id="alg1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span>  calculate energies <math id="alg1.l3.m1.1" class="ltx_Math" alttext="s^{i}=E_{\theta}(x^{i})" display="inline"><semantics id="alg1.l3.m1.1a"><mrow id="alg1.l3.m1.1.1" xref="alg1.l3.m1.1.1.cmml"><msup id="alg1.l3.m1.1.1.3" xref="alg1.l3.m1.1.1.3.cmml"><mi id="alg1.l3.m1.1.1.3.2" xref="alg1.l3.m1.1.1.3.2.cmml">s</mi><mi id="alg1.l3.m1.1.1.3.3" xref="alg1.l3.m1.1.1.3.3.cmml">i</mi></msup><mo id="alg1.l3.m1.1.1.2" xref="alg1.l3.m1.1.1.2.cmml">=</mo><mrow id="alg1.l3.m1.1.1.1" xref="alg1.l3.m1.1.1.1.cmml"><msub id="alg1.l3.m1.1.1.1.3" xref="alg1.l3.m1.1.1.1.3.cmml"><mi id="alg1.l3.m1.1.1.1.3.2" xref="alg1.l3.m1.1.1.1.3.2.cmml">E</mi><mi id="alg1.l3.m1.1.1.1.3.3" xref="alg1.l3.m1.1.1.1.3.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="alg1.l3.m1.1.1.1.2" xref="alg1.l3.m1.1.1.1.2.cmml">​</mo><mrow id="alg1.l3.m1.1.1.1.1.1" xref="alg1.l3.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l3.m1.1.1.1.1.1.2" xref="alg1.l3.m1.1.1.1.1.1.1.cmml">(</mo><msup id="alg1.l3.m1.1.1.1.1.1.1" xref="alg1.l3.m1.1.1.1.1.1.1.cmml"><mi id="alg1.l3.m1.1.1.1.1.1.1.2" xref="alg1.l3.m1.1.1.1.1.1.1.2.cmml">x</mi><mi id="alg1.l3.m1.1.1.1.1.1.1.3" xref="alg1.l3.m1.1.1.1.1.1.1.3.cmml">i</mi></msup><mo stretchy="false" id="alg1.l3.m1.1.1.1.1.1.3" xref="alg1.l3.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.1b"><apply id="alg1.l3.m1.1.1.cmml" xref="alg1.l3.m1.1.1"><eq id="alg1.l3.m1.1.1.2.cmml" xref="alg1.l3.m1.1.1.2"></eq><apply id="alg1.l3.m1.1.1.3.cmml" xref="alg1.l3.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l3.m1.1.1.3.1.cmml" xref="alg1.l3.m1.1.1.3">superscript</csymbol><ci id="alg1.l3.m1.1.1.3.2.cmml" xref="alg1.l3.m1.1.1.3.2">𝑠</ci><ci id="alg1.l3.m1.1.1.3.3.cmml" xref="alg1.l3.m1.1.1.3.3">𝑖</ci></apply><apply id="alg1.l3.m1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1"><times id="alg1.l3.m1.1.1.1.2.cmml" xref="alg1.l3.m1.1.1.1.2"></times><apply id="alg1.l3.m1.1.1.1.3.cmml" xref="alg1.l3.m1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l3.m1.1.1.1.3.1.cmml" xref="alg1.l3.m1.1.1.1.3">subscript</csymbol><ci id="alg1.l3.m1.1.1.1.3.2.cmml" xref="alg1.l3.m1.1.1.1.3.2">𝐸</ci><ci id="alg1.l3.m1.1.1.1.3.3.cmml" xref="alg1.l3.m1.1.1.1.3.3">𝜃</ci></apply><apply id="alg1.l3.m1.1.1.1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l3.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1.1.1">superscript</csymbol><ci id="alg1.l3.m1.1.1.1.1.1.1.2.cmml" xref="alg1.l3.m1.1.1.1.1.1.1.2">𝑥</ci><ci id="alg1.l3.m1.1.1.1.1.1.1.3.cmml" xref="alg1.l3.m1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.1c">s^{i}=E_{\theta}(x^{i})</annotation></semantics></math> for each <math id="alg1.l3.m2.3" class="ltx_Math" alttext="x^{i}\in\{x^{1},\cdots,x^{n}\}" display="inline"><semantics id="alg1.l3.m2.3a"><mrow id="alg1.l3.m2.3.3" xref="alg1.l3.m2.3.3.cmml"><msup id="alg1.l3.m2.3.3.4" xref="alg1.l3.m2.3.3.4.cmml"><mi id="alg1.l3.m2.3.3.4.2" xref="alg1.l3.m2.3.3.4.2.cmml">x</mi><mi id="alg1.l3.m2.3.3.4.3" xref="alg1.l3.m2.3.3.4.3.cmml">i</mi></msup><mo id="alg1.l3.m2.3.3.3" xref="alg1.l3.m2.3.3.3.cmml">∈</mo><mrow id="alg1.l3.m2.3.3.2.2" xref="alg1.l3.m2.3.3.2.3.cmml"><mo stretchy="false" id="alg1.l3.m2.3.3.2.2.3" xref="alg1.l3.m2.3.3.2.3.cmml">{</mo><msup id="alg1.l3.m2.2.2.1.1.1" xref="alg1.l3.m2.2.2.1.1.1.cmml"><mi id="alg1.l3.m2.2.2.1.1.1.2" xref="alg1.l3.m2.2.2.1.1.1.2.cmml">x</mi><mn id="alg1.l3.m2.2.2.1.1.1.3" xref="alg1.l3.m2.2.2.1.1.1.3.cmml">1</mn></msup><mo id="alg1.l3.m2.3.3.2.2.4" xref="alg1.l3.m2.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="alg1.l3.m2.1.1" xref="alg1.l3.m2.1.1.cmml">⋯</mi><mo id="alg1.l3.m2.3.3.2.2.5" xref="alg1.l3.m2.3.3.2.3.cmml">,</mo><msup id="alg1.l3.m2.3.3.2.2.2" xref="alg1.l3.m2.3.3.2.2.2.cmml"><mi id="alg1.l3.m2.3.3.2.2.2.2" xref="alg1.l3.m2.3.3.2.2.2.2.cmml">x</mi><mi id="alg1.l3.m2.3.3.2.2.2.3" xref="alg1.l3.m2.3.3.2.2.2.3.cmml">n</mi></msup><mo stretchy="false" id="alg1.l3.m2.3.3.2.2.6" xref="alg1.l3.m2.3.3.2.3.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l3.m2.3b"><apply id="alg1.l3.m2.3.3.cmml" xref="alg1.l3.m2.3.3"><in id="alg1.l3.m2.3.3.3.cmml" xref="alg1.l3.m2.3.3.3"></in><apply id="alg1.l3.m2.3.3.4.cmml" xref="alg1.l3.m2.3.3.4"><csymbol cd="ambiguous" id="alg1.l3.m2.3.3.4.1.cmml" xref="alg1.l3.m2.3.3.4">superscript</csymbol><ci id="alg1.l3.m2.3.3.4.2.cmml" xref="alg1.l3.m2.3.3.4.2">𝑥</ci><ci id="alg1.l3.m2.3.3.4.3.cmml" xref="alg1.l3.m2.3.3.4.3">𝑖</ci></apply><set id="alg1.l3.m2.3.3.2.3.cmml" xref="alg1.l3.m2.3.3.2.2"><apply id="alg1.l3.m2.2.2.1.1.1.cmml" xref="alg1.l3.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="alg1.l3.m2.2.2.1.1.1.1.cmml" xref="alg1.l3.m2.2.2.1.1.1">superscript</csymbol><ci id="alg1.l3.m2.2.2.1.1.1.2.cmml" xref="alg1.l3.m2.2.2.1.1.1.2">𝑥</ci><cn type="integer" id="alg1.l3.m2.2.2.1.1.1.3.cmml" xref="alg1.l3.m2.2.2.1.1.1.3">1</cn></apply><ci id="alg1.l3.m2.1.1.cmml" xref="alg1.l3.m2.1.1">⋯</ci><apply id="alg1.l3.m2.3.3.2.2.2.cmml" xref="alg1.l3.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="alg1.l3.m2.3.3.2.2.2.1.cmml" xref="alg1.l3.m2.3.3.2.2.2">superscript</csymbol><ci id="alg1.l3.m2.3.3.2.2.2.2.cmml" xref="alg1.l3.m2.3.3.2.2.2.2">𝑥</ci><ci id="alg1.l3.m2.3.3.2.2.2.3.cmml" xref="alg1.l3.m2.3.3.2.2.2.3">𝑛</ci></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m2.3c">x^{i}\in\{x^{1},\cdots,x^{n}\}</annotation></semantics></math>

</div>
<div id="alg1.1.1.1.3" class="ltx_listingline">
<span id="alg1.1.1.1.3.1" class="ltx_text ltx_font_typewriter">// </span><span id="alg1.1.1.1.3.2" class="ltx_text ltx_font_typewriter">Resample from the set of LM samples<span id="alg1.1.1.1.3.2.1" class="ltx_text" style="float:right;"></span></span>
</div>
<div id="alg1.1.1.1.4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span>

</div>
<div id="alg1.1.1.1.5" class="ltx_listingline">
</div>
<div id="alg1.l4" class="ltx_listingline">  sample <math id="alg1.l4.m1.1" class="ltx_Math" alttext="x=x^{i}" display="inline"><semantics id="alg1.l4.m1.1a"><mrow id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml"><mi id="alg1.l4.m1.1.1.2" xref="alg1.l4.m1.1.1.2.cmml">x</mi><mo id="alg1.l4.m1.1.1.1" xref="alg1.l4.m1.1.1.1.cmml">=</mo><msup id="alg1.l4.m1.1.1.3" xref="alg1.l4.m1.1.1.3.cmml"><mi id="alg1.l4.m1.1.1.3.2" xref="alg1.l4.m1.1.1.3.2.cmml">x</mi><mi id="alg1.l4.m1.1.1.3.3" xref="alg1.l4.m1.1.1.3.3.cmml">i</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.1b"><apply id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1"><eq id="alg1.l4.m1.1.1.1.cmml" xref="alg1.l4.m1.1.1.1"></eq><ci id="alg1.l4.m1.1.1.2.cmml" xref="alg1.l4.m1.1.1.2">𝑥</ci><apply id="alg1.l4.m1.1.1.3.cmml" xref="alg1.l4.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l4.m1.1.1.3.1.cmml" xref="alg1.l4.m1.1.1.3">superscript</csymbol><ci id="alg1.l4.m1.1.1.3.2.cmml" xref="alg1.l4.m1.1.1.3.2">𝑥</ci><ci id="alg1.l4.m1.1.1.3.3.cmml" xref="alg1.l4.m1.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.1c">x=x^{i}</annotation></semantics></math> with probability <math id="alg1.l4.m2.4" class="ltx_Math" alttext="\frac{\exp(-s^{i})}{\sum_{j=1}^{n}\exp(-s^{j})}" display="inline"><semantics id="alg1.l4.m2.4a"><mfrac id="alg1.l4.m2.4.4" xref="alg1.l4.m2.4.4.cmml"><mrow id="alg1.l4.m2.2.2.2.2" xref="alg1.l4.m2.2.2.2.3.cmml"><mi id="alg1.l4.m2.1.1.1.1" xref="alg1.l4.m2.1.1.1.1.cmml">exp</mi><mo id="alg1.l4.m2.2.2.2.2a" xref="alg1.l4.m2.2.2.2.3.cmml">⁡</mo><mrow id="alg1.l4.m2.2.2.2.2.1" xref="alg1.l4.m2.2.2.2.3.cmml"><mo stretchy="false" id="alg1.l4.m2.2.2.2.2.1.2" xref="alg1.l4.m2.2.2.2.3.cmml">(</mo><mrow id="alg1.l4.m2.2.2.2.2.1.1" xref="alg1.l4.m2.2.2.2.2.1.1.cmml"><mo id="alg1.l4.m2.2.2.2.2.1.1a" xref="alg1.l4.m2.2.2.2.2.1.1.cmml">−</mo><msup id="alg1.l4.m2.2.2.2.2.1.1.2" xref="alg1.l4.m2.2.2.2.2.1.1.2.cmml"><mi id="alg1.l4.m2.2.2.2.2.1.1.2.2" xref="alg1.l4.m2.2.2.2.2.1.1.2.2.cmml">s</mi><mi id="alg1.l4.m2.2.2.2.2.1.1.2.3" xref="alg1.l4.m2.2.2.2.2.1.1.2.3.cmml">i</mi></msup></mrow><mo stretchy="false" id="alg1.l4.m2.2.2.2.2.1.3" xref="alg1.l4.m2.2.2.2.3.cmml">)</mo></mrow></mrow><mrow id="alg1.l4.m2.4.4.4" xref="alg1.l4.m2.4.4.4.cmml"><mstyle displaystyle="false" id="alg1.l4.m2.4.4.4.3" xref="alg1.l4.m2.4.4.4.3.cmml"><msubsup id="alg1.l4.m2.4.4.4.3a" xref="alg1.l4.m2.4.4.4.3.cmml"><mo id="alg1.l4.m2.4.4.4.3.2.2" xref="alg1.l4.m2.4.4.4.3.2.2.cmml">∑</mo><mrow id="alg1.l4.m2.4.4.4.3.2.3" xref="alg1.l4.m2.4.4.4.3.2.3.cmml"><mi id="alg1.l4.m2.4.4.4.3.2.3.2" xref="alg1.l4.m2.4.4.4.3.2.3.2.cmml">j</mi><mo id="alg1.l4.m2.4.4.4.3.2.3.1" xref="alg1.l4.m2.4.4.4.3.2.3.1.cmml">=</mo><mn id="alg1.l4.m2.4.4.4.3.2.3.3" xref="alg1.l4.m2.4.4.4.3.2.3.3.cmml">1</mn></mrow><mi id="alg1.l4.m2.4.4.4.3.3" xref="alg1.l4.m2.4.4.4.3.3.cmml">n</mi></msubsup></mstyle><mrow id="alg1.l4.m2.4.4.4.2.1" xref="alg1.l4.m2.4.4.4.2.2.cmml"><mi id="alg1.l4.m2.3.3.3.1" xref="alg1.l4.m2.3.3.3.1.cmml">exp</mi><mo id="alg1.l4.m2.4.4.4.2.1a" xref="alg1.l4.m2.4.4.4.2.2.cmml">⁡</mo><mrow id="alg1.l4.m2.4.4.4.2.1.1" xref="alg1.l4.m2.4.4.4.2.2.cmml"><mo stretchy="false" id="alg1.l4.m2.4.4.4.2.1.1.2" xref="alg1.l4.m2.4.4.4.2.2.cmml">(</mo><mrow id="alg1.l4.m2.4.4.4.2.1.1.1" xref="alg1.l4.m2.4.4.4.2.1.1.1.cmml"><mo id="alg1.l4.m2.4.4.4.2.1.1.1a" xref="alg1.l4.m2.4.4.4.2.1.1.1.cmml">−</mo><msup id="alg1.l4.m2.4.4.4.2.1.1.1.2" xref="alg1.l4.m2.4.4.4.2.1.1.1.2.cmml"><mi id="alg1.l4.m2.4.4.4.2.1.1.1.2.2" xref="alg1.l4.m2.4.4.4.2.1.1.1.2.2.cmml">s</mi><mi id="alg1.l4.m2.4.4.4.2.1.1.1.2.3" xref="alg1.l4.m2.4.4.4.2.1.1.1.2.3.cmml">j</mi></msup></mrow><mo stretchy="false" id="alg1.l4.m2.4.4.4.2.1.1.3" xref="alg1.l4.m2.4.4.4.2.2.cmml">)</mo></mrow></mrow></mrow></mfrac><annotation-xml encoding="MathML-Content" id="alg1.l4.m2.4b"><apply id="alg1.l4.m2.4.4.cmml" xref="alg1.l4.m2.4.4"><divide id="alg1.l4.m2.4.4.5.cmml" xref="alg1.l4.m2.4.4"></divide><apply id="alg1.l4.m2.2.2.2.3.cmml" xref="alg1.l4.m2.2.2.2.2"><exp id="alg1.l4.m2.1.1.1.1.cmml" xref="alg1.l4.m2.1.1.1.1"></exp><apply id="alg1.l4.m2.2.2.2.2.1.1.cmml" xref="alg1.l4.m2.2.2.2.2.1.1"><minus id="alg1.l4.m2.2.2.2.2.1.1.1.cmml" xref="alg1.l4.m2.2.2.2.2.1.1"></minus><apply id="alg1.l4.m2.2.2.2.2.1.1.2.cmml" xref="alg1.l4.m2.2.2.2.2.1.1.2"><csymbol cd="ambiguous" id="alg1.l4.m2.2.2.2.2.1.1.2.1.cmml" xref="alg1.l4.m2.2.2.2.2.1.1.2">superscript</csymbol><ci id="alg1.l4.m2.2.2.2.2.1.1.2.2.cmml" xref="alg1.l4.m2.2.2.2.2.1.1.2.2">𝑠</ci><ci id="alg1.l4.m2.2.2.2.2.1.1.2.3.cmml" xref="alg1.l4.m2.2.2.2.2.1.1.2.3">𝑖</ci></apply></apply></apply><apply id="alg1.l4.m2.4.4.4.cmml" xref="alg1.l4.m2.4.4.4"><apply id="alg1.l4.m2.4.4.4.3.cmml" xref="alg1.l4.m2.4.4.4.3"><csymbol cd="ambiguous" id="alg1.l4.m2.4.4.4.3.1.cmml" xref="alg1.l4.m2.4.4.4.3">superscript</csymbol><apply id="alg1.l4.m2.4.4.4.3.2.cmml" xref="alg1.l4.m2.4.4.4.3"><csymbol cd="ambiguous" id="alg1.l4.m2.4.4.4.3.2.1.cmml" xref="alg1.l4.m2.4.4.4.3">subscript</csymbol><sum id="alg1.l4.m2.4.4.4.3.2.2.cmml" xref="alg1.l4.m2.4.4.4.3.2.2"></sum><apply id="alg1.l4.m2.4.4.4.3.2.3.cmml" xref="alg1.l4.m2.4.4.4.3.2.3"><eq id="alg1.l4.m2.4.4.4.3.2.3.1.cmml" xref="alg1.l4.m2.4.4.4.3.2.3.1"></eq><ci id="alg1.l4.m2.4.4.4.3.2.3.2.cmml" xref="alg1.l4.m2.4.4.4.3.2.3.2">𝑗</ci><cn type="integer" id="alg1.l4.m2.4.4.4.3.2.3.3.cmml" xref="alg1.l4.m2.4.4.4.3.2.3.3">1</cn></apply></apply><ci id="alg1.l4.m2.4.4.4.3.3.cmml" xref="alg1.l4.m2.4.4.4.3.3">𝑛</ci></apply><apply id="alg1.l4.m2.4.4.4.2.2.cmml" xref="alg1.l4.m2.4.4.4.2.1"><exp id="alg1.l4.m2.3.3.3.1.cmml" xref="alg1.l4.m2.3.3.3.1"></exp><apply id="alg1.l4.m2.4.4.4.2.1.1.1.cmml" xref="alg1.l4.m2.4.4.4.2.1.1.1"><minus id="alg1.l4.m2.4.4.4.2.1.1.1.1.cmml" xref="alg1.l4.m2.4.4.4.2.1.1.1"></minus><apply id="alg1.l4.m2.4.4.4.2.1.1.1.2.cmml" xref="alg1.l4.m2.4.4.4.2.1.1.1.2"><csymbol cd="ambiguous" id="alg1.l4.m2.4.4.4.2.1.1.1.2.1.cmml" xref="alg1.l4.m2.4.4.4.2.1.1.1.2">superscript</csymbol><ci id="alg1.l4.m2.4.4.4.2.1.1.1.2.2.cmml" xref="alg1.l4.m2.4.4.4.2.1.1.1.2.2">𝑠</ci><ci id="alg1.l4.m2.4.4.4.2.1.1.1.2.3.cmml" xref="alg1.l4.m2.4.4.4.2.1.1.1.2.3">𝑗</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m2.4c">\frac{\exp(-s^{i})}{\sum_{j=1}^{n}\exp(-s^{j})}</annotation></semantics></math>

</div>
<div id="alg1.l5" class="ltx_listingline">  <span id="alg1.l5.1" class="ltx_text ltx_font_bold">return</span> <math id="alg1.l5.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="alg1.l5.m1.1a"><mi id="alg1.l5.m1.1.1" xref="alg1.l5.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.1b"><ci id="alg1.l5.m1.1.1.cmml" xref="alg1.l5.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.1c">x</annotation></semantics></math>

</div>
</div>

</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.3.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span>Top-k Joint Sampling</figcaption>
</figure>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Generation</h3>

<div id="S3.SS3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.p1.1" class="ltx_p">Generating from the joint model is a non-trivial task. A naive way is to generate from the joint model auto-regressively, by marginalizing the future as in Eq. <a href="#S3.E5" title="In 3.2 Evaluation ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, which we term <span id="S3.SS3.p1.1.1" class="ltx_text ltx_font_bold">Top-k auto-regressive sampling</span>.
However, doing so is computationally expensive and impractical, and we only use this method for a qualitative analysis of the joint model in Appendix <a href="#A1.SS1" title="A.1 Top-k auto-regressive sampling ‣ Appendix A Appendix ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.1</span></a>.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para ltx_noindent">
<p id="S3.SS3.p2.1" class="ltx_p">In order to generate efficiently, we use self-normalizing importance sampling <cite class="ltx_cite ltx_citemacro_citep">(Owen, <a href="#bib.bib24" title="" class="ltx_ref">2013</a>; Grover et al., <a href="#bib.bib10" title="" class="ltx_ref">2019</a>)</cite>.
Under the assumptions
that the model from which we wish to draw samples is the joint model, which is the product of the auto-regressive
model and the energy function, and that the proposal distribution is the auto-regressive model itself,
sampling proceeds simply by: a) sampling from the auto-regressive language model, followed by b) resampling according
to the energy function.
The algorithm is shown in Algorithm <a href="#alg1" title="In 3.2 Evaluation ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, where we introduce an optional top-k constraint on the pretrained language
model to improve the quality of samples in the set<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>Adapting to other types of local constraints
such as nucleus sampling <cite class="ltx_cite ltx_citemacro_citep">(Holtzman et al., <a href="#bib.bib14" title="" class="ltx_ref">2019</a>)</cite> is straightforward.</span></span></span>.
Without the top-k constraint, as the number of samples goes to infinity, we would recover exact samples
from the joint model distribution.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<div id="S4.p1" class="ltx_para ltx_noindent">
<p id="S4.p1.1" class="ltx_p">In this section, we describe the experimental set up and the results we obtained by using the residual EBM for text generation,
both in terms of perplexity and generation quality.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Experimental Setup</h3>

<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Datasets</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p">We consider two datasets: the Toronto Book Corpus <cite class="ltx_cite ltx_citemacro_citep">(Zhu et al., <a href="#bib.bib46" title="" class="ltx_ref">2015</a>; Kiros et al., <a href="#bib.bib17" title="" class="ltx_ref">2015</a>)</cite> and CC-News <cite class="ltx_cite ltx_citemacro_citep">(Bakhtin et al., <a href="#bib.bib2" title="" class="ltx_ref">2019</a>)</cite>.
The former dataset consists of fiction books in 16 different genres, totaling about half a billion words.
The latter is a de-duplicated subset of the English portion of the CommonCrawl news dataset <cite class="ltx_cite ltx_citemacro_citep">(Nagel, <a href="#bib.bib21" title="" class="ltx_ref">2016</a>)</cite>,
which totals around 16 Billion words. The book corpus is more challenging because the range of style and topics is
more diverse than CC-News. Also, the book corpus is 30 times smaller than CC-News and may pose generalization challenges because
of its smaller size.</p>
</div>
<div id="S4.SS1.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p2.2" class="ltx_p">In all our experiments we use a prefix of size 120 tokens and we generate the following 40 tokens; with the notation
of Eq. <a href="#S3.E2" title="In 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, <math id="S4.SS1.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="p=120" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.1.m1.1a"><mrow id="S4.SS1.SSS0.Px1.p2.1.m1.1.1" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.cmml"><mi id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.cmml">p</mi><mo id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.1" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.cmml">120</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.1.m1.1b"><apply id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1"><eq id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.1"></eq><ci id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2">𝑝</ci><cn type="integer" id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3">120</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.1.m1.1c">p=120</annotation></semantics></math> and <math id="S4.SS1.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="T=160" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.2.m2.1a"><mrow id="S4.SS1.SSS0.Px1.p2.2.m2.1.1" xref="S4.SS1.SSS0.Px1.p2.2.m2.1.1.cmml"><mi id="S4.SS1.SSS0.Px1.p2.2.m2.1.1.2" xref="S4.SS1.SSS0.Px1.p2.2.m2.1.1.2.cmml">T</mi><mo id="S4.SS1.SSS0.Px1.p2.2.m2.1.1.1" xref="S4.SS1.SSS0.Px1.p2.2.m2.1.1.1.cmml">=</mo><mn id="S4.SS1.SSS0.Px1.p2.2.m2.1.1.3" xref="S4.SS1.SSS0.Px1.p2.2.m2.1.1.3.cmml">160</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.2.m2.1b"><apply id="S4.SS1.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.2.m2.1.1"><eq id="S4.SS1.SSS0.Px1.p2.2.m2.1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.2.m2.1.1.1"></eq><ci id="S4.SS1.SSS0.Px1.p2.2.m2.1.1.2.cmml" xref="S4.SS1.SSS0.Px1.p2.2.m2.1.1.2">𝑇</ci><cn type="integer" id="S4.SS1.SSS0.Px1.p2.2.m2.1.1.3.cmml" xref="S4.SS1.SSS0.Px1.p2.2.m2.1.1.3">160</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.2.m2.1c">T=160</annotation></semantics></math>. For training the joint models, for efficiency we generated 16/128 samples per prefix for CC-News/Book Corpus offline, and sample uniformly from those samples at training time.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Baselines</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p1.3" class="ltx_p">We consider as base language model (<span id="S4.SS1.SSS0.Px2.p1.3.1" class="ltx_text ltx_font_smallcaps">Base LM</span>) used to generate negatives for the residual EBM, a transformer language
model with 12 layers, <math id="S4.SS1.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="h=16" display="inline"><semantics id="S4.SS1.SSS0.Px2.p1.1.m1.1a"><mrow id="S4.SS1.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS1.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="S4.SS1.SSS0.Px2.p1.1.m1.1.1.2" xref="S4.SS1.SSS0.Px2.p1.1.m1.1.1.2.cmml">h</mi><mo id="S4.SS1.SSS0.Px2.p1.1.m1.1.1.1" xref="S4.SS1.SSS0.Px2.p1.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS1.SSS0.Px2.p1.1.m1.1.1.3" xref="S4.SS1.SSS0.Px2.p1.1.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p1.1.m1.1b"><apply id="S4.SS1.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p1.1.m1.1.1"><eq id="S4.SS1.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p1.1.m1.1.1.1"></eq><ci id="S4.SS1.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p1.1.m1.1.1.2">ℎ</ci><cn type="integer" id="S4.SS1.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p1.1.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p1.1.m1.1c">h=16</annotation></semantics></math>, <math id="S4.SS1.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="d_{model}=1024" display="inline"><semantics id="S4.SS1.SSS0.Px2.p1.2.m2.1a"><mrow id="S4.SS1.SSS0.Px2.p1.2.m2.1.1" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.cmml"><msub id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.cmml"><mi id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.2" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.2.cmml">d</mi><mrow id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.cmml"><mi id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.2" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.1" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.3" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.1a" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.4" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.1b" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.5" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.1c" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.6" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.6.cmml">l</mi></mrow></msub><mo id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.1" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.1.cmml">=</mo><mn id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.3" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.3.cmml">1024</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p1.2.m2.1b"><apply id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1"><eq id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.1"></eq><apply id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.1.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.2.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.2">𝑑</ci><apply id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3"><times id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.1.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.1"></times><ci id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.2.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.2">𝑚</ci><ci id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.3.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.3">𝑜</ci><ci id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.4.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.4">𝑑</ci><ci id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.5.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.5">𝑒</ci><ci id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.6.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.2.3.6">𝑙</ci></apply></apply><cn type="integer" id="S4.SS1.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p1.2.m2.1.1.3">1024</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p1.2.m2.1c">d_{model}=1024</annotation></semantics></math>, <math id="S4.SS1.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="d_{ff}=4096" display="inline"><semantics id="S4.SS1.SSS0.Px2.p1.3.m3.1a"><mrow id="S4.SS1.SSS0.Px2.p1.3.m3.1.1" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.cmml"><msub id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.cmml"><mi id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.2" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.2.cmml">d</mi><mrow id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.cmml"><mi id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.2" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.1" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.3" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.3.cmml">f</mi></mrow></msub><mo id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.1" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.1.cmml">=</mo><mn id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.3" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.3.cmml">4096</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p1.3.m3.1b"><apply id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1"><eq id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.1"></eq><apply id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.1.cmml" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.2.cmml" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.2">𝑑</ci><apply id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.cmml" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3"><times id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.1.cmml" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.1"></times><ci id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.2.cmml" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.2">𝑓</ci><ci id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.3.cmml" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.2.3.3">𝑓</ci></apply></apply><cn type="integer" id="S4.SS1.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p1.3.m3.1.1.3">4096</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p1.3.m3.1c">d_{ff}=4096</annotation></semantics></math> (we refer to <cite class="ltx_cite ltx_citemacro_citet">Vaswani et al. (<a href="#bib.bib34" title="" class="ltx_ref">2017</a>)</cite> for notations).
This is also our first baseline model.</p>
</div>
<div id="S4.SS1.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p2.1" class="ltx_p">The <span id="S4.SS1.SSS0.Px2.p2.1.1" class="ltx_text ltx_font_italic">joint</span> model has as many parameters as the sum of the number of parameters in the base LM
and the number of parameters in the energy network.
To make a fair comparison, we consider two additional baselines that have the same number of parameters
as our joint model.</p>
</div>
<div id="S4.SS1.SSS0.Px2.p3" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p3.4" class="ltx_p">The first baseline is a Residual Auto-regressive Language Model baseline (<span id="S4.SS1.SSS0.Px2.p3.4.1" class="ltx_text ltx_font_smallcaps">RALM</span>):</p>
<table id="S4.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E6.m1.3" class="ltx_Math" alttext="\log P_{RALM}(x_{t}|x_{&lt;t})=\log P_{LM}(x_{t}|x_{&lt;t})+\log P_{\phi}(x_{t}|x_{&lt;t})+const" display="block"><semantics id="S4.E6.m1.3a"><mrow id="S4.E6.m1.3.3" xref="S4.E6.m1.3.3.cmml"><mrow id="S4.E6.m1.1.1.1" xref="S4.E6.m1.1.1.1.cmml"><mrow id="S4.E6.m1.1.1.1.3" xref="S4.E6.m1.1.1.1.3.cmml"><mi id="S4.E6.m1.1.1.1.3.1" xref="S4.E6.m1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S4.E6.m1.1.1.1.3a" xref="S4.E6.m1.1.1.1.3.cmml">⁡</mo><msub id="S4.E6.m1.1.1.1.3.2" xref="S4.E6.m1.1.1.1.3.2.cmml"><mi id="S4.E6.m1.1.1.1.3.2.2" xref="S4.E6.m1.1.1.1.3.2.2.cmml">P</mi><mrow id="S4.E6.m1.1.1.1.3.2.3" xref="S4.E6.m1.1.1.1.3.2.3.cmml"><mi id="S4.E6.m1.1.1.1.3.2.3.2" xref="S4.E6.m1.1.1.1.3.2.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="S4.E6.m1.1.1.1.3.2.3.1" xref="S4.E6.m1.1.1.1.3.2.3.1.cmml">​</mo><mi id="S4.E6.m1.1.1.1.3.2.3.3" xref="S4.E6.m1.1.1.1.3.2.3.3.cmml">A</mi><mo lspace="0em" rspace="0em" id="S4.E6.m1.1.1.1.3.2.3.1a" xref="S4.E6.m1.1.1.1.3.2.3.1.cmml">​</mo><mi id="S4.E6.m1.1.1.1.3.2.3.4" xref="S4.E6.m1.1.1.1.3.2.3.4.cmml">L</mi><mo lspace="0em" rspace="0em" id="S4.E6.m1.1.1.1.3.2.3.1b" xref="S4.E6.m1.1.1.1.3.2.3.1.cmml">​</mo><mi id="S4.E6.m1.1.1.1.3.2.3.5" xref="S4.E6.m1.1.1.1.3.2.3.5.cmml">M</mi></mrow></msub></mrow><mo lspace="0em" rspace="0em" id="S4.E6.m1.1.1.1.2" xref="S4.E6.m1.1.1.1.2.cmml">​</mo><mrow id="S4.E6.m1.1.1.1.1.1" xref="S4.E6.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E6.m1.1.1.1.1.1.2" xref="S4.E6.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E6.m1.1.1.1.1.1.1" xref="S4.E6.m1.1.1.1.1.1.1.cmml"><msub id="S4.E6.m1.1.1.1.1.1.1.2" xref="S4.E6.m1.1.1.1.1.1.1.2.cmml"><mi id="S4.E6.m1.1.1.1.1.1.1.2.2" xref="S4.E6.m1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S4.E6.m1.1.1.1.1.1.1.2.3" xref="S4.E6.m1.1.1.1.1.1.1.2.3.cmml">t</mi></msub><mo fence="false" id="S4.E6.m1.1.1.1.1.1.1.1" xref="S4.E6.m1.1.1.1.1.1.1.1.cmml">|</mo><msub id="S4.E6.m1.1.1.1.1.1.1.3" xref="S4.E6.m1.1.1.1.1.1.1.3.cmml"><mi id="S4.E6.m1.1.1.1.1.1.1.3.2" xref="S4.E6.m1.1.1.1.1.1.1.3.2.cmml">x</mi><mrow id="S4.E6.m1.1.1.1.1.1.1.3.3" xref="S4.E6.m1.1.1.1.1.1.1.3.3.cmml"><mi id="S4.E6.m1.1.1.1.1.1.1.3.3.2" xref="S4.E6.m1.1.1.1.1.1.1.3.3.2.cmml"></mi><mo id="S4.E6.m1.1.1.1.1.1.1.3.3.1" xref="S4.E6.m1.1.1.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="S4.E6.m1.1.1.1.1.1.1.3.3.3" xref="S4.E6.m1.1.1.1.1.1.1.3.3.3.cmml">t</mi></mrow></msub></mrow><mo stretchy="false" id="S4.E6.m1.1.1.1.1.1.3" xref="S4.E6.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.E6.m1.3.3.4" xref="S4.E6.m1.3.3.4.cmml">=</mo><mrow id="S4.E6.m1.3.3.3" xref="S4.E6.m1.3.3.3.cmml"><mrow id="S4.E6.m1.2.2.2.1" xref="S4.E6.m1.2.2.2.1.cmml"><mrow id="S4.E6.m1.2.2.2.1.3" xref="S4.E6.m1.2.2.2.1.3.cmml"><mi id="S4.E6.m1.2.2.2.1.3.1" xref="S4.E6.m1.2.2.2.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S4.E6.m1.2.2.2.1.3a" xref="S4.E6.m1.2.2.2.1.3.cmml">⁡</mo><msub id="S4.E6.m1.2.2.2.1.3.2" xref="S4.E6.m1.2.2.2.1.3.2.cmml"><mi id="S4.E6.m1.2.2.2.1.3.2.2" xref="S4.E6.m1.2.2.2.1.3.2.2.cmml">P</mi><mrow id="S4.E6.m1.2.2.2.1.3.2.3" xref="S4.E6.m1.2.2.2.1.3.2.3.cmml"><mi id="S4.E6.m1.2.2.2.1.3.2.3.2" xref="S4.E6.m1.2.2.2.1.3.2.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S4.E6.m1.2.2.2.1.3.2.3.1" xref="S4.E6.m1.2.2.2.1.3.2.3.1.cmml">​</mo><mi id="S4.E6.m1.2.2.2.1.3.2.3.3" xref="S4.E6.m1.2.2.2.1.3.2.3.3.cmml">M</mi></mrow></msub></mrow><mo lspace="0em" rspace="0em" id="S4.E6.m1.2.2.2.1.2" xref="S4.E6.m1.2.2.2.1.2.cmml">​</mo><mrow id="S4.E6.m1.2.2.2.1.1.1" xref="S4.E6.m1.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S4.E6.m1.2.2.2.1.1.1.2" xref="S4.E6.m1.2.2.2.1.1.1.1.cmml">(</mo><mrow id="S4.E6.m1.2.2.2.1.1.1.1" xref="S4.E6.m1.2.2.2.1.1.1.1.cmml"><msub id="S4.E6.m1.2.2.2.1.1.1.1.2" xref="S4.E6.m1.2.2.2.1.1.1.1.2.cmml"><mi id="S4.E6.m1.2.2.2.1.1.1.1.2.2" xref="S4.E6.m1.2.2.2.1.1.1.1.2.2.cmml">x</mi><mi id="S4.E6.m1.2.2.2.1.1.1.1.2.3" xref="S4.E6.m1.2.2.2.1.1.1.1.2.3.cmml">t</mi></msub><mo fence="false" id="S4.E6.m1.2.2.2.1.1.1.1.1" xref="S4.E6.m1.2.2.2.1.1.1.1.1.cmml">|</mo><msub id="S4.E6.m1.2.2.2.1.1.1.1.3" xref="S4.E6.m1.2.2.2.1.1.1.1.3.cmml"><mi id="S4.E6.m1.2.2.2.1.1.1.1.3.2" xref="S4.E6.m1.2.2.2.1.1.1.1.3.2.cmml">x</mi><mrow id="S4.E6.m1.2.2.2.1.1.1.1.3.3" xref="S4.E6.m1.2.2.2.1.1.1.1.3.3.cmml"><mi id="S4.E6.m1.2.2.2.1.1.1.1.3.3.2" xref="S4.E6.m1.2.2.2.1.1.1.1.3.3.2.cmml"></mi><mo id="S4.E6.m1.2.2.2.1.1.1.1.3.3.1" xref="S4.E6.m1.2.2.2.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="S4.E6.m1.2.2.2.1.1.1.1.3.3.3" xref="S4.E6.m1.2.2.2.1.1.1.1.3.3.3.cmml">t</mi></mrow></msub></mrow><mo stretchy="false" id="S4.E6.m1.2.2.2.1.1.1.3" xref="S4.E6.m1.2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.E6.m1.3.3.3.3" xref="S4.E6.m1.3.3.3.3.cmml">+</mo><mrow id="S4.E6.m1.3.3.3.2" xref="S4.E6.m1.3.3.3.2.cmml"><mrow id="S4.E6.m1.3.3.3.2.3" xref="S4.E6.m1.3.3.3.2.3.cmml"><mi id="S4.E6.m1.3.3.3.2.3.1" xref="S4.E6.m1.3.3.3.2.3.1.cmml">log</mi><mo lspace="0.167em" id="S4.E6.m1.3.3.3.2.3a" xref="S4.E6.m1.3.3.3.2.3.cmml">⁡</mo><msub id="S4.E6.m1.3.3.3.2.3.2" xref="S4.E6.m1.3.3.3.2.3.2.cmml"><mi id="S4.E6.m1.3.3.3.2.3.2.2" xref="S4.E6.m1.3.3.3.2.3.2.2.cmml">P</mi><mi id="S4.E6.m1.3.3.3.2.3.2.3" xref="S4.E6.m1.3.3.3.2.3.2.3.cmml">ϕ</mi></msub></mrow><mo lspace="0em" rspace="0em" id="S4.E6.m1.3.3.3.2.2" xref="S4.E6.m1.3.3.3.2.2.cmml">​</mo><mrow id="S4.E6.m1.3.3.3.2.1.1" xref="S4.E6.m1.3.3.3.2.1.1.1.cmml"><mo stretchy="false" id="S4.E6.m1.3.3.3.2.1.1.2" xref="S4.E6.m1.3.3.3.2.1.1.1.cmml">(</mo><mrow id="S4.E6.m1.3.3.3.2.1.1.1" xref="S4.E6.m1.3.3.3.2.1.1.1.cmml"><msub id="S4.E6.m1.3.3.3.2.1.1.1.2" xref="S4.E6.m1.3.3.3.2.1.1.1.2.cmml"><mi id="S4.E6.m1.3.3.3.2.1.1.1.2.2" xref="S4.E6.m1.3.3.3.2.1.1.1.2.2.cmml">x</mi><mi id="S4.E6.m1.3.3.3.2.1.1.1.2.3" xref="S4.E6.m1.3.3.3.2.1.1.1.2.3.cmml">t</mi></msub><mo fence="false" id="S4.E6.m1.3.3.3.2.1.1.1.1" xref="S4.E6.m1.3.3.3.2.1.1.1.1.cmml">|</mo><msub id="S4.E6.m1.3.3.3.2.1.1.1.3" xref="S4.E6.m1.3.3.3.2.1.1.1.3.cmml"><mi id="S4.E6.m1.3.3.3.2.1.1.1.3.2" xref="S4.E6.m1.3.3.3.2.1.1.1.3.2.cmml">x</mi><mrow id="S4.E6.m1.3.3.3.2.1.1.1.3.3" xref="S4.E6.m1.3.3.3.2.1.1.1.3.3.cmml"><mi id="S4.E6.m1.3.3.3.2.1.1.1.3.3.2" xref="S4.E6.m1.3.3.3.2.1.1.1.3.3.2.cmml"></mi><mo id="S4.E6.m1.3.3.3.2.1.1.1.3.3.1" xref="S4.E6.m1.3.3.3.2.1.1.1.3.3.1.cmml">&lt;</mo><mi id="S4.E6.m1.3.3.3.2.1.1.1.3.3.3" xref="S4.E6.m1.3.3.3.2.1.1.1.3.3.3.cmml">t</mi></mrow></msub></mrow><mo stretchy="false" id="S4.E6.m1.3.3.3.2.1.1.3" xref="S4.E6.m1.3.3.3.2.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.E6.m1.3.3.3.3a" xref="S4.E6.m1.3.3.3.3.cmml">+</mo><mrow id="S4.E6.m1.3.3.3.4" xref="S4.E6.m1.3.3.3.4.cmml"><mi id="S4.E6.m1.3.3.3.4.2" xref="S4.E6.m1.3.3.3.4.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S4.E6.m1.3.3.3.4.1" xref="S4.E6.m1.3.3.3.4.1.cmml">​</mo><mi id="S4.E6.m1.3.3.3.4.3" xref="S4.E6.m1.3.3.3.4.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S4.E6.m1.3.3.3.4.1a" xref="S4.E6.m1.3.3.3.4.1.cmml">​</mo><mi id="S4.E6.m1.3.3.3.4.4" xref="S4.E6.m1.3.3.3.4.4.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.E6.m1.3.3.3.4.1b" xref="S4.E6.m1.3.3.3.4.1.cmml">​</mo><mi id="S4.E6.m1.3.3.3.4.5" xref="S4.E6.m1.3.3.3.4.5.cmml">s</mi><mo lspace="0em" rspace="0em" id="S4.E6.m1.3.3.3.4.1c" xref="S4.E6.m1.3.3.3.4.1.cmml">​</mo><mi id="S4.E6.m1.3.3.3.4.6" xref="S4.E6.m1.3.3.3.4.6.cmml">t</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E6.m1.3b"><apply id="S4.E6.m1.3.3.cmml" xref="S4.E6.m1.3.3"><eq id="S4.E6.m1.3.3.4.cmml" xref="S4.E6.m1.3.3.4"></eq><apply id="S4.E6.m1.1.1.1.cmml" xref="S4.E6.m1.1.1.1"><times id="S4.E6.m1.1.1.1.2.cmml" xref="S4.E6.m1.1.1.1.2"></times><apply id="S4.E6.m1.1.1.1.3.cmml" xref="S4.E6.m1.1.1.1.3"><log id="S4.E6.m1.1.1.1.3.1.cmml" xref="S4.E6.m1.1.1.1.3.1"></log><apply id="S4.E6.m1.1.1.1.3.2.cmml" xref="S4.E6.m1.1.1.1.3.2"><csymbol cd="ambiguous" id="S4.E6.m1.1.1.1.3.2.1.cmml" xref="S4.E6.m1.1.1.1.3.2">subscript</csymbol><ci id="S4.E6.m1.1.1.1.3.2.2.cmml" xref="S4.E6.m1.1.1.1.3.2.2">𝑃</ci><apply id="S4.E6.m1.1.1.1.3.2.3.cmml" xref="S4.E6.m1.1.1.1.3.2.3"><times id="S4.E6.m1.1.1.1.3.2.3.1.cmml" xref="S4.E6.m1.1.1.1.3.2.3.1"></times><ci id="S4.E6.m1.1.1.1.3.2.3.2.cmml" xref="S4.E6.m1.1.1.1.3.2.3.2">𝑅</ci><ci id="S4.E6.m1.1.1.1.3.2.3.3.cmml" xref="S4.E6.m1.1.1.1.3.2.3.3">𝐴</ci><ci id="S4.E6.m1.1.1.1.3.2.3.4.cmml" xref="S4.E6.m1.1.1.1.3.2.3.4">𝐿</ci><ci id="S4.E6.m1.1.1.1.3.2.3.5.cmml" xref="S4.E6.m1.1.1.1.3.2.3.5">𝑀</ci></apply></apply></apply><apply id="S4.E6.m1.1.1.1.1.1.1.cmml" xref="S4.E6.m1.1.1.1.1.1"><csymbol cd="latexml" id="S4.E6.m1.1.1.1.1.1.1.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1">conditional</csymbol><apply id="S4.E6.m1.1.1.1.1.1.1.2.cmml" xref="S4.E6.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E6.m1.1.1.1.1.1.1.2.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S4.E6.m1.1.1.1.1.1.1.2.2.cmml" xref="S4.E6.m1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S4.E6.m1.1.1.1.1.1.1.2.3.cmml" xref="S4.E6.m1.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S4.E6.m1.1.1.1.1.1.1.3.cmml" xref="S4.E6.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E6.m1.1.1.1.1.1.1.3.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S4.E6.m1.1.1.1.1.1.1.3.2.cmml" xref="S4.E6.m1.1.1.1.1.1.1.3.2">𝑥</ci><apply id="S4.E6.m1.1.1.1.1.1.1.3.3.cmml" xref="S4.E6.m1.1.1.1.1.1.1.3.3"><lt id="S4.E6.m1.1.1.1.1.1.1.3.3.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="S4.E6.m1.1.1.1.1.1.1.3.3.2.cmml" xref="S4.E6.m1.1.1.1.1.1.1.3.3.2">absent</csymbol><ci id="S4.E6.m1.1.1.1.1.1.1.3.3.3.cmml" xref="S4.E6.m1.1.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply><apply id="S4.E6.m1.3.3.3.cmml" xref="S4.E6.m1.3.3.3"><plus id="S4.E6.m1.3.3.3.3.cmml" xref="S4.E6.m1.3.3.3.3"></plus><apply id="S4.E6.m1.2.2.2.1.cmml" xref="S4.E6.m1.2.2.2.1"><times id="S4.E6.m1.2.2.2.1.2.cmml" xref="S4.E6.m1.2.2.2.1.2"></times><apply id="S4.E6.m1.2.2.2.1.3.cmml" xref="S4.E6.m1.2.2.2.1.3"><log id="S4.E6.m1.2.2.2.1.3.1.cmml" xref="S4.E6.m1.2.2.2.1.3.1"></log><apply id="S4.E6.m1.2.2.2.1.3.2.cmml" xref="S4.E6.m1.2.2.2.1.3.2"><csymbol cd="ambiguous" id="S4.E6.m1.2.2.2.1.3.2.1.cmml" xref="S4.E6.m1.2.2.2.1.3.2">subscript</csymbol><ci id="S4.E6.m1.2.2.2.1.3.2.2.cmml" xref="S4.E6.m1.2.2.2.1.3.2.2">𝑃</ci><apply id="S4.E6.m1.2.2.2.1.3.2.3.cmml" xref="S4.E6.m1.2.2.2.1.3.2.3"><times id="S4.E6.m1.2.2.2.1.3.2.3.1.cmml" xref="S4.E6.m1.2.2.2.1.3.2.3.1"></times><ci id="S4.E6.m1.2.2.2.1.3.2.3.2.cmml" xref="S4.E6.m1.2.2.2.1.3.2.3.2">𝐿</ci><ci id="S4.E6.m1.2.2.2.1.3.2.3.3.cmml" xref="S4.E6.m1.2.2.2.1.3.2.3.3">𝑀</ci></apply></apply></apply><apply id="S4.E6.m1.2.2.2.1.1.1.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1"><csymbol cd="latexml" id="S4.E6.m1.2.2.2.1.1.1.1.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1">conditional</csymbol><apply id="S4.E6.m1.2.2.2.1.1.1.1.2.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E6.m1.2.2.2.1.1.1.1.2.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.2">subscript</csymbol><ci id="S4.E6.m1.2.2.2.1.1.1.1.2.2.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.2.2">𝑥</ci><ci id="S4.E6.m1.2.2.2.1.1.1.1.2.3.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.2.3">𝑡</ci></apply><apply id="S4.E6.m1.2.2.2.1.1.1.1.3.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E6.m1.2.2.2.1.1.1.1.3.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.3">subscript</csymbol><ci id="S4.E6.m1.2.2.2.1.1.1.1.3.2.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.3.2">𝑥</ci><apply id="S4.E6.m1.2.2.2.1.1.1.1.3.3.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.3.3"><lt id="S4.E6.m1.2.2.2.1.1.1.1.3.3.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="S4.E6.m1.2.2.2.1.1.1.1.3.3.2.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.3.3.2">absent</csymbol><ci id="S4.E6.m1.2.2.2.1.1.1.1.3.3.3.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply><apply id="S4.E6.m1.3.3.3.2.cmml" xref="S4.E6.m1.3.3.3.2"><times id="S4.E6.m1.3.3.3.2.2.cmml" xref="S4.E6.m1.3.3.3.2.2"></times><apply id="S4.E6.m1.3.3.3.2.3.cmml" xref="S4.E6.m1.3.3.3.2.3"><log id="S4.E6.m1.3.3.3.2.3.1.cmml" xref="S4.E6.m1.3.3.3.2.3.1"></log><apply id="S4.E6.m1.3.3.3.2.3.2.cmml" xref="S4.E6.m1.3.3.3.2.3.2"><csymbol cd="ambiguous" id="S4.E6.m1.3.3.3.2.3.2.1.cmml" xref="S4.E6.m1.3.3.3.2.3.2">subscript</csymbol><ci id="S4.E6.m1.3.3.3.2.3.2.2.cmml" xref="S4.E6.m1.3.3.3.2.3.2.2">𝑃</ci><ci id="S4.E6.m1.3.3.3.2.3.2.3.cmml" xref="S4.E6.m1.3.3.3.2.3.2.3">italic-ϕ</ci></apply></apply><apply id="S4.E6.m1.3.3.3.2.1.1.1.cmml" xref="S4.E6.m1.3.3.3.2.1.1"><csymbol cd="latexml" id="S4.E6.m1.3.3.3.2.1.1.1.1.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.1">conditional</csymbol><apply id="S4.E6.m1.3.3.3.2.1.1.1.2.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.2"><csymbol cd="ambiguous" id="S4.E6.m1.3.3.3.2.1.1.1.2.1.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.2">subscript</csymbol><ci id="S4.E6.m1.3.3.3.2.1.1.1.2.2.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.2.2">𝑥</ci><ci id="S4.E6.m1.3.3.3.2.1.1.1.2.3.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.2.3">𝑡</ci></apply><apply id="S4.E6.m1.3.3.3.2.1.1.1.3.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.3"><csymbol cd="ambiguous" id="S4.E6.m1.3.3.3.2.1.1.1.3.1.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.3">subscript</csymbol><ci id="S4.E6.m1.3.3.3.2.1.1.1.3.2.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.3.2">𝑥</ci><apply id="S4.E6.m1.3.3.3.2.1.1.1.3.3.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.3.3"><lt id="S4.E6.m1.3.3.3.2.1.1.1.3.3.1.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="S4.E6.m1.3.3.3.2.1.1.1.3.3.2.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.3.3.2">absent</csymbol><ci id="S4.E6.m1.3.3.3.2.1.1.1.3.3.3.cmml" xref="S4.E6.m1.3.3.3.2.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply><apply id="S4.E6.m1.3.3.3.4.cmml" xref="S4.E6.m1.3.3.3.4"><times id="S4.E6.m1.3.3.3.4.1.cmml" xref="S4.E6.m1.3.3.3.4.1"></times><ci id="S4.E6.m1.3.3.3.4.2.cmml" xref="S4.E6.m1.3.3.3.4.2">𝑐</ci><ci id="S4.E6.m1.3.3.3.4.3.cmml" xref="S4.E6.m1.3.3.3.4.3">𝑜</ci><ci id="S4.E6.m1.3.3.3.4.4.cmml" xref="S4.E6.m1.3.3.3.4.4">𝑛</ci><ci id="S4.E6.m1.3.3.3.4.5.cmml" xref="S4.E6.m1.3.3.3.4.5">𝑠</ci><ci id="S4.E6.m1.3.3.3.4.6.cmml" xref="S4.E6.m1.3.3.3.4.6">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E6.m1.3c">\log P_{RALM}(x_{t}|x_{&lt;t})=\log P_{LM}(x_{t}|x_{&lt;t})+\log P_{\phi}(x_{t}|x_{&lt;t})+const</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="S4.SS1.SSS0.Px2.p3.3" class="ltx_p">where <math id="S4.SS1.SSS0.Px2.p3.1.m1.1" class="ltx_Math" alttext="P_{\phi}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.1.m1.1a"><msub id="S4.SS1.SSS0.Px2.p3.1.m1.1.1" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.cmml"><mi id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.2" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.2.cmml">P</mi><mi id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.3" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.3.cmml">ϕ</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.1.m1.1b"><apply id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.2">𝑃</ci><ci id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.3">italic-ϕ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.1.m1.1c">P_{\phi}</annotation></semantics></math> takes the form of another auto-regressive language model. The parameters of <math id="S4.SS1.SSS0.Px2.p3.2.m2.1" class="ltx_Math" alttext="P_{\phi}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.2.m2.1a"><msub id="S4.SS1.SSS0.Px2.p3.2.m2.1.1" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.cmml"><mi id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.cmml">P</mi><mi id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.cmml">ϕ</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.2.m2.1b"><apply id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2">𝑃</ci><ci id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3">italic-ϕ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.2.m2.1c">P_{\phi}</annotation></semantics></math> are trained by
exact maximum likelihood training of <math id="S4.SS1.SSS0.Px2.p3.3.m3.1" class="ltx_Math" alttext="P_{RALM}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.3.m3.1a"><msub id="S4.SS1.SSS0.Px2.p3.3.m3.1.1" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.cmml"><mi id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.2" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.2.cmml">P</mi><mrow id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.cmml"><mi id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.2" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.1" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.3" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.3.cmml">A</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.1a" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.4" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.4.cmml">L</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.1b" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.5" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.5.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.3.m3.1b"><apply id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.2">𝑃</ci><apply id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3"><times id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.1.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.1"></times><ci id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.2.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.2">𝑅</ci><ci id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.3.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.3">𝐴</ci><ci id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.4.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.4">𝐿</ci><ci id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.5.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.5">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.3.m3.1c">P_{RALM}</annotation></semantics></math>.</p>
</div>
<div id="S4.SS1.SSS0.Px2.p4" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p4.3" class="ltx_p">The second baseline is an auto-regressive language model of the same size of our joint model (sum of the base LM
and energy function parameters), we dub this model Big
Auto-regressive Language Model (<span id="S4.SS1.SSS0.Px2.p4.3.1" class="ltx_text ltx_font_smallcaps">BALM</span>). <span id="S4.SS1.SSS0.Px2.p4.3.2" class="ltx_text ltx_font_smallcaps">BALM</span> has 12 layers, <math id="S4.SS1.SSS0.Px2.p4.1.m1.1" class="ltx_Math" alttext="h=16" display="inline"><semantics id="S4.SS1.SSS0.Px2.p4.1.m1.1a"><mrow id="S4.SS1.SSS0.Px2.p4.1.m1.1.1" xref="S4.SS1.SSS0.Px2.p4.1.m1.1.1.cmml"><mi id="S4.SS1.SSS0.Px2.p4.1.m1.1.1.2" xref="S4.SS1.SSS0.Px2.p4.1.m1.1.1.2.cmml">h</mi><mo id="S4.SS1.SSS0.Px2.p4.1.m1.1.1.1" xref="S4.SS1.SSS0.Px2.p4.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS1.SSS0.Px2.p4.1.m1.1.1.3" xref="S4.SS1.SSS0.Px2.p4.1.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p4.1.m1.1b"><apply id="S4.SS1.SSS0.Px2.p4.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p4.1.m1.1.1"><eq id="S4.SS1.SSS0.Px2.p4.1.m1.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p4.1.m1.1.1.1"></eq><ci id="S4.SS1.SSS0.Px2.p4.1.m1.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p4.1.m1.1.1.2">ℎ</ci><cn type="integer" id="S4.SS1.SSS0.Px2.p4.1.m1.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p4.1.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p4.1.m1.1c">h=16</annotation></semantics></math>, <math id="S4.SS1.SSS0.Px2.p4.2.m2.1" class="ltx_Math" alttext="d_{model}=1568" display="inline"><semantics id="S4.SS1.SSS0.Px2.p4.2.m2.1a"><mrow id="S4.SS1.SSS0.Px2.p4.2.m2.1.1" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.cmml"><msub id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.cmml"><mi id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.2" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.2.cmml">d</mi><mrow id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.cmml"><mi id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.2" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.1" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.3" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.1a" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.4" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.1b" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.5" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.1c" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.6" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.6.cmml">l</mi></mrow></msub><mo id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.1" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.1.cmml">=</mo><mn id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.3" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.3.cmml">1568</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p4.2.m2.1b"><apply id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1"><eq id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.1"></eq><apply id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.1.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.2.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.2">𝑑</ci><apply id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3"><times id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.1.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.1"></times><ci id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.2.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.2">𝑚</ci><ci id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.3.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.3">𝑜</ci><ci id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.4.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.4">𝑑</ci><ci id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.5.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.5">𝑒</ci><ci id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.6.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.3.6">𝑙</ci></apply></apply><cn type="integer" id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.3">1568</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p4.2.m2.1c">d_{model}=1568</annotation></semantics></math>, <math id="S4.SS1.SSS0.Px2.p4.3.m3.1" class="ltx_Math" alttext="d_{ff}=6272" display="inline"><semantics id="S4.SS1.SSS0.Px2.p4.3.m3.1a"><mrow id="S4.SS1.SSS0.Px2.p4.3.m3.1.1" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.cmml"><msub id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.cmml"><mi id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.2" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.2.cmml">d</mi><mrow id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.cmml"><mi id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.2" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.1" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.1.cmml">​</mo><mi id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.3" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.3.cmml">f</mi></mrow></msub><mo id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.1" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.1.cmml">=</mo><mn id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.3" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.3.cmml">6272</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p4.3.m3.1b"><apply id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1"><eq id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.1"></eq><apply id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.1.cmml" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.2.cmml" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.2">𝑑</ci><apply id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.cmml" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3"><times id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.1.cmml" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.1"></times><ci id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.2.cmml" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.2">𝑓</ci><ci id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.3.cmml" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.2.3.3">𝑓</ci></apply></apply><cn type="integer" id="S4.SS1.SSS0.Px2.p4.3.m3.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p4.3.m3.1.1.3">6272</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p4.3.m3.1c">d_{ff}=6272</annotation></semantics></math>, and is trained by standard token level cross-entropy loss.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Residual EBM Architecture</h4>

<div id="S4.SS1.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px3.p1.2" class="ltx_p">We consider two architectures for our residual EBM, both of them are based on
transformers <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al., <a href="#bib.bib34" title="" class="ltx_ref">2017</a>; Devlin et al., <a href="#bib.bib5" title="" class="ltx_ref">2018</a>)</cite>. The first version uses causal self-attention
and is derived from the base LM, a unidirectional transformer (<span id="S4.SS1.SSS0.Px3.p1.2.1" class="ltx_text ltx_markedasmath ltx_font_smallcaps">UniT</span>). It is of the same architecture as <span id="S4.SS1.SSS0.Px3.p1.2.2" class="ltx_text ltx_markedasmath ltx_font_smallcaps">Base LM</span>, except that in the final layer we project the mean-pooled hidden states to a scalar energy value. We initialize its parameters with a language model trained on the same dataset.</p>
</div>
<div id="S4.SS1.SSS0.Px3.p2" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px3.p2.8" class="ltx_p">The second version is instead bi-directional (<span id="S4.SS1.SSS0.Px3.p2.8.1" class="ltx_text ltx_markedasmath ltx_font_smallcaps">BiT</span>), and the energy function is computed by projecting the mean-pooled top hidden states
down to a single scalar value. We consider three variants, a <span id="S4.SS1.SSS0.Px3.p2.8.2" class="ltx_text ltx_markedasmath ltx_font_smallcaps">BiT-Base</span> following the architecture of RoBERTa-Base, and a <math id="S4.SS1.SSS0.Px3.p2.3.m3.1" class="ltx_math_unparsed" alttext="\textsc{BiT-Large}*" display="inline"><semantics id="S4.SS1.SSS0.Px3.p2.3.m3.1a"><mrow id="S4.SS1.SSS0.Px3.p2.3.m3.1b"><mtext class="ltx_font_smallcaps" id="S4.SS1.SSS0.Px3.p2.3.m3.1.1">BiT-Large</mtext><mo lspace="0.222em" id="S4.SS1.SSS0.Px3.p2.3.m3.1.2">∗</mo></mrow><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p2.3.m3.1c">\textsc{BiT-Large}*</annotation></semantics></math> following RoBERTa-Large <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib19" title="" class="ltx_ref">2019</a>)</cite>, and a <span id="S4.SS1.SSS0.Px3.p2.8.3" class="ltx_text ltx_markedasmath ltx_font_smallcaps">BiT-Med</span> with the same number of parameters as <span id="S4.SS1.SSS0.Px3.p2.8.4" class="ltx_text ltx_markedasmath ltx_font_smallcaps">UniT</span> (such that <span id="S4.SS1.SSS0.Px3.p2.8.5" class="ltx_text ltx_markedasmath ltx_font_smallcaps">Joint BiT-Med</span> has roughly the same number of parameters as <span id="S4.SS1.SSS0.Px3.p2.8.6" class="ltx_text ltx_markedasmath ltx_font_smallcaps">BALM</span>)<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>We use models from the HuggingFace repository at <a target="_blank" href="https://github.com/huggingface/transformers" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/huggingface/transformers</a></span></span></span>. We initialize the parameters with a trained BERT, and we use <math id="S4.SS1.SSS0.Px3.p2.8.m8.1" class="ltx_Math" alttext="*" display="inline"><semantics id="S4.SS1.SSS0.Px3.p2.8.m8.1a"><mo id="S4.SS1.SSS0.Px3.p2.8.m8.1.1" xref="S4.SS1.SSS0.Px3.p2.8.m8.1.1.cmml">∗</mo><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p2.8.m8.1b"><times id="S4.SS1.SSS0.Px3.p2.8.m8.1.1.cmml" xref="S4.SS1.SSS0.Px3.p2.8.m8.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p2.8.m8.1c">*</annotation></semantics></math> to mark usage of external data, otherwise it means that BERT was trained on our training set.
Notice how our model can be interpreted as a natural way to finetune large bidirectional pretrained models for the
text generation task.</p>
</div>
<div id="S4.SS1.SSS0.Px3.p3" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px3.p3.2" class="ltx_p">While we expect <span id="S4.SS1.SSS0.Px3.p3.2.1" class="ltx_text ltx_markedasmath ltx_font_smallcaps">BiT</span> to yield better results because it can fully leverage context also for intermediate tokens,
we also consider <span id="S4.SS1.SSS0.Px3.p3.2.2" class="ltx_text ltx_markedasmath ltx_font_smallcaps">UniT</span> to compare to the RALM baseline, which uses the same architecture and only differs in the way
parameters are trained and in the presence of local normalization.</p>
</div>
<div id="S4.SS1.SSS0.Px3.p4" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px3.p4.1" class="ltx_p">We train our models on 8 DGX nodes, each with 8 Nvidia V100s. To improve training speed, we use mixed precision training<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span><a target="_blank" href="https://github.com/NVIDIA/apex" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/NVIDIA/apex</a></span></span></span>. We use the Adam optimizer, with cosine learning rate decay and learning rate warmup. To stabilize training we used gradient norm clipping <cite class="ltx_cite ltx_citemacro_citep">(Pascanu et al., <a href="#bib.bib26" title="" class="ltx_ref">2013</a>)</cite>. Detailed hyper-parameter settings can be found in Appendix <a href="#A1.SS3" title="A.3 Optimization Settings ‣ Appendix A Appendix ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.3</span></a>.</p>
</div>
<div id="S4.SS1.SSS0.Px3.p5" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px3.p5.2" class="ltx_p">For generation, we use top-k sampling with <math id="S4.SS1.SSS0.Px3.p5.1.m1.1" class="ltx_Math" alttext="k=10" display="inline"><semantics id="S4.SS1.SSS0.Px3.p5.1.m1.1a"><mrow id="S4.SS1.SSS0.Px3.p5.1.m1.1.1" xref="S4.SS1.SSS0.Px3.p5.1.m1.1.1.cmml"><mi id="S4.SS1.SSS0.Px3.p5.1.m1.1.1.2" xref="S4.SS1.SSS0.Px3.p5.1.m1.1.1.2.cmml">k</mi><mo id="S4.SS1.SSS0.Px3.p5.1.m1.1.1.1" xref="S4.SS1.SSS0.Px3.p5.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS1.SSS0.Px3.p5.1.m1.1.1.3" xref="S4.SS1.SSS0.Px3.p5.1.m1.1.1.3.cmml">10</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p5.1.m1.1b"><apply id="S4.SS1.SSS0.Px3.p5.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px3.p5.1.m1.1.1"><eq id="S4.SS1.SSS0.Px3.p5.1.m1.1.1.1.cmml" xref="S4.SS1.SSS0.Px3.p5.1.m1.1.1.1"></eq><ci id="S4.SS1.SSS0.Px3.p5.1.m1.1.1.2.cmml" xref="S4.SS1.SSS0.Px3.p5.1.m1.1.1.2">𝑘</ci><cn type="integer" id="S4.SS1.SSS0.Px3.p5.1.m1.1.1.3.cmml" xref="S4.SS1.SSS0.Px3.p5.1.m1.1.1.3">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p5.1.m1.1c">k=10</annotation></semantics></math> for all human evaluations. We take 10,000 samples from <span id="S4.SS1.SSS0.Px3.p5.2.1" class="ltx_text ltx_markedasmath ltx_font_smallcaps">Base LM</span> for our joint sampling.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Results</h3>

<section id="S4.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Automatic Evaluation</h4>

<figure id="S4.T1" class="ltx_table">
<table id="S4.T1.67" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T1.67.68.1" class="ltx_tr">
<th id="S4.T1.67.68.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt" rowspan="2"><span id="S4.T1.67.68.1.1.1" class="ltx_text">Model (#parameters)</span></th>
<td id="S4.T1.67.68.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">CC-News</td>
<td id="S4.T1.67.68.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">Toronto Book Corpus</td>
</tr>
<tr id="S4.T1.67.69.2" class="ltx_tr">
<td id="S4.T1.67.69.2.1" class="ltx_td ltx_align_center">Val</td>
<td id="S4.T1.67.69.2.2" class="ltx_td ltx_align_center">Test</td>
<td id="S4.T1.67.69.2.3" class="ltx_td ltx_align_center">Val</td>
<td id="S4.T1.67.69.2.4" class="ltx_td ltx_align_center">Test</td>
</tr>
<tr id="S4.T1.67.70.3" class="ltx_tr">
<th id="S4.T1.67.70.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">
<span id="S4.T1.67.70.3.1.1" class="ltx_text ltx_font_smallcaps">base LM</span> (203M)</th>
<td id="S4.T1.67.70.3.2" class="ltx_td ltx_align_center ltx_border_t">18.41</td>
<td id="S4.T1.67.70.3.3" class="ltx_td ltx_align_center ltx_border_t">17.57</td>
<td id="S4.T1.67.70.3.4" class="ltx_td ltx_align_center ltx_border_t">16.16</td>
<td id="S4.T1.67.70.3.5" class="ltx_td ltx_align_center ltx_border_t">18.29</td>
</tr>
<tr id="S4.T1.67.71.4" class="ltx_tr">
<th id="S4.T1.67.71.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.67.71.4.1.1" class="ltx_text ltx_font_smallcaps">RALM</span> (LM+203M)</th>
<td id="S4.T1.67.71.4.2" class="ltx_td ltx_align_center">17.01</td>
<td id="S4.T1.67.71.4.3" class="ltx_td ltx_align_center">16.17</td>
<td id="S4.T1.67.71.4.4" class="ltx_td ltx_align_center">15.71</td>
<td id="S4.T1.67.71.4.5" class="ltx_td ltx_align_center">17.85</td>
</tr>
<tr id="S4.T1.4.4" class="ltx_tr">
<th id="S4.T1.4.4.5" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.4.4.5.1" class="ltx_text ltx_font_smallcaps">BALM</span> (408M)</th>
<td id="S4.T1.1.1.1" class="ltx_td ltx_align_center"><math id="S4.T1.1.1.1.m1.1" class="ltx_Math" alttext="16.50" display="inline"><semantics id="S4.T1.1.1.1.m1.1a"><mn id="S4.T1.1.1.1.m1.1.1" xref="S4.T1.1.1.1.m1.1.1.cmml">16.50</mn><annotation-xml encoding="MathML-Content" id="S4.T1.1.1.1.m1.1b"><cn type="float" id="S4.T1.1.1.1.m1.1.1.cmml" xref="S4.T1.1.1.1.m1.1.1">16.50</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.1.1.1.m1.1c">16.50</annotation></semantics></math></td>
<td id="S4.T1.2.2.2" class="ltx_td ltx_align_center"><math id="S4.T1.2.2.2.m1.1" class="ltx_Math" alttext="15.74" display="inline"><semantics id="S4.T1.2.2.2.m1.1a"><mn id="S4.T1.2.2.2.m1.1.1" xref="S4.T1.2.2.2.m1.1.1.cmml">15.74</mn><annotation-xml encoding="MathML-Content" id="S4.T1.2.2.2.m1.1b"><cn type="float" id="S4.T1.2.2.2.m1.1.1.cmml" xref="S4.T1.2.2.2.m1.1.1">15.74</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.2.2.m1.1c">15.74</annotation></semantics></math></td>
<td id="S4.T1.3.3.3" class="ltx_td ltx_align_center"><math id="S4.T1.3.3.3.m1.1" class="ltx_Math" alttext="15.00" display="inline"><semantics id="S4.T1.3.3.3.m1.1a"><mn id="S4.T1.3.3.3.m1.1.1" xref="S4.T1.3.3.3.m1.1.1.cmml">15.00</mn><annotation-xml encoding="MathML-Content" id="S4.T1.3.3.3.m1.1b"><cn type="float" id="S4.T1.3.3.3.m1.1.1.cmml" xref="S4.T1.3.3.3.m1.1.1">15.00</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.3.3.3.m1.1c">15.00</annotation></semantics></math></td>
<td id="S4.T1.4.4.4" class="ltx_td ltx_align_center"><math id="S4.T1.4.4.4.m1.1" class="ltx_Math" alttext="16.99" display="inline"><semantics id="S4.T1.4.4.4.m1.1a"><mn id="S4.T1.4.4.4.m1.1.1" xref="S4.T1.4.4.4.m1.1.1.cmml">16.99</mn><annotation-xml encoding="MathML-Content" id="S4.T1.4.4.4.m1.1b"><cn type="float" id="S4.T1.4.4.4.m1.1.1.cmml" xref="S4.T1.4.4.4.m1.1.1">16.99</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.4.4.4.m1.1c">16.99</annotation></semantics></math></td>
</tr>
<tr id="S4.T1.67.72.5" class="ltx_tr">
<th id="S4.T1.67.72.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.67.72.5.1.1" class="ltx_text ltx_font_smallcaps">joint UniT</span> (LM+203M)</th>
<td id="S4.T1.67.72.5.2" class="ltx_td ltx_align_center">16.42-16.44</td>
<td id="S4.T1.67.72.5.3" class="ltx_td ltx_align_center">15.57-15.58</td>
<td id="S4.T1.67.72.5.4" class="ltx_td ltx_align_center">15.12-15.13</td>
<td id="S4.T1.67.72.5.5" class="ltx_td ltx_align_center">16.98-17.00</td>
</tr>
<tr id="S4.T1.8.8" class="ltx_tr">
<th id="S4.T1.8.8.5" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.8.8.5.1" class="ltx_text ltx_font_smallcaps">joint BiT-Base</span> (LM+125M)</th>
<td id="S4.T1.6.6.2" class="ltx_td ltx_align_center">
<math id="S4.T1.5.5.1.m1.1" class="ltx_Math" alttext="15.32" display="inline"><semantics id="S4.T1.5.5.1.m1.1a"><mn id="S4.T1.5.5.1.m1.1.1" xref="S4.T1.5.5.1.m1.1.1.cmml">15.32</mn><annotation-xml encoding="MathML-Content" id="S4.T1.5.5.1.m1.1b"><cn type="float" id="S4.T1.5.5.1.m1.1.1.cmml" xref="S4.T1.5.5.1.m1.1.1">15.32</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.5.5.1.m1.1c">15.32</annotation></semantics></math>-<math id="S4.T1.6.6.2.m2.1" class="ltx_Math" alttext="15.35" display="inline"><semantics id="S4.T1.6.6.2.m2.1a"><mn id="S4.T1.6.6.2.m2.1.1" xref="S4.T1.6.6.2.m2.1.1.cmml">15.35</mn><annotation-xml encoding="MathML-Content" id="S4.T1.6.6.2.m2.1b"><cn type="float" id="S4.T1.6.6.2.m2.1.1.cmml" xref="S4.T1.6.6.2.m2.1.1">15.35</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.6.6.2.m2.1c">15.35</annotation></semantics></math>
</td>
<td id="S4.T1.8.8.4" class="ltx_td ltx_align_center">
<math id="S4.T1.7.7.3.m1.1" class="ltx_Math" alttext="14.61" display="inline"><semantics id="S4.T1.7.7.3.m1.1a"><mn id="S4.T1.7.7.3.m1.1.1" xref="S4.T1.7.7.3.m1.1.1.cmml">14.61</mn><annotation-xml encoding="MathML-Content" id="S4.T1.7.7.3.m1.1b"><cn type="float" id="S4.T1.7.7.3.m1.1.1.cmml" xref="S4.T1.7.7.3.m1.1.1">14.61</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.7.7.3.m1.1c">14.61</annotation></semantics></math>-<math id="S4.T1.8.8.4.m2.1" class="ltx_Math" alttext="14.64" display="inline"><semantics id="S4.T1.8.8.4.m2.1a"><mn id="S4.T1.8.8.4.m2.1.1" xref="S4.T1.8.8.4.m2.1.1.cmml">14.64</mn><annotation-xml encoding="MathML-Content" id="S4.T1.8.8.4.m2.1b"><cn type="float" id="S4.T1.8.8.4.m2.1.1.cmml" xref="S4.T1.8.8.4.m2.1.1">14.64</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.8.8.4.m2.1c">14.64</annotation></semantics></math>
</td>
<td id="S4.T1.8.8.6" class="ltx_td ltx_align_center">-</td>
<td id="S4.T1.8.8.7" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="S4.T1.16.16" class="ltx_tr">
<th id="S4.T1.16.16.9" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.16.16.9.1" class="ltx_text ltx_font_smallcaps">joint BiT-Base*</span> (LM+125M)</th>
<td id="S4.T1.10.10.2" class="ltx_td ltx_align_center">
<math id="S4.T1.9.9.1.m1.1" class="ltx_Math" alttext="15.11" display="inline"><semantics id="S4.T1.9.9.1.m1.1a"><mn id="S4.T1.9.9.1.m1.1.1" xref="S4.T1.9.9.1.m1.1.1.cmml">15.11</mn><annotation-xml encoding="MathML-Content" id="S4.T1.9.9.1.m1.1b"><cn type="float" id="S4.T1.9.9.1.m1.1.1.cmml" xref="S4.T1.9.9.1.m1.1.1">15.11</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.9.9.1.m1.1c">15.11</annotation></semantics></math>-<math id="S4.T1.10.10.2.m2.1" class="ltx_Math" alttext="15.17" display="inline"><semantics id="S4.T1.10.10.2.m2.1a"><mn id="S4.T1.10.10.2.m2.1.1" xref="S4.T1.10.10.2.m2.1.1.cmml">15.17</mn><annotation-xml encoding="MathML-Content" id="S4.T1.10.10.2.m2.1b"><cn type="float" id="S4.T1.10.10.2.m2.1.1.cmml" xref="S4.T1.10.10.2.m2.1.1">15.17</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.10.10.2.m2.1c">15.17</annotation></semantics></math>
</td>
<td id="S4.T1.12.12.4" class="ltx_td ltx_align_center">
<math id="S4.T1.11.11.3.m1.1" class="ltx_Math" alttext="14.37" display="inline"><semantics id="S4.T1.11.11.3.m1.1a"><mn id="S4.T1.11.11.3.m1.1.1" xref="S4.T1.11.11.3.m1.1.1.cmml">14.37</mn><annotation-xml encoding="MathML-Content" id="S4.T1.11.11.3.m1.1b"><cn type="float" id="S4.T1.11.11.3.m1.1.1.cmml" xref="S4.T1.11.11.3.m1.1.1">14.37</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.11.11.3.m1.1c">14.37</annotation></semantics></math>-<math id="S4.T1.12.12.4.m2.1" class="ltx_Math" alttext="14.42" display="inline"><semantics id="S4.T1.12.12.4.m2.1a"><mn id="S4.T1.12.12.4.m2.1.1" xref="S4.T1.12.12.4.m2.1.1.cmml">14.42</mn><annotation-xml encoding="MathML-Content" id="S4.T1.12.12.4.m2.1b"><cn type="float" id="S4.T1.12.12.4.m2.1.1.cmml" xref="S4.T1.12.12.4.m2.1.1">14.42</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.12.12.4.m2.1c">14.42</annotation></semantics></math>
</td>
<td id="S4.T1.14.14.6" class="ltx_td ltx_align_center">
<math id="S4.T1.13.13.5.m1.1" class="ltx_Math" alttext="14.14" display="inline"><semantics id="S4.T1.13.13.5.m1.1a"><mn id="S4.T1.13.13.5.m1.1.1" xref="S4.T1.13.13.5.m1.1.1.cmml">14.14</mn><annotation-xml encoding="MathML-Content" id="S4.T1.13.13.5.m1.1b"><cn type="float" id="S4.T1.13.13.5.m1.1.1.cmml" xref="S4.T1.13.13.5.m1.1.1">14.14</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.13.13.5.m1.1c">14.14</annotation></semantics></math>-<math id="S4.T1.14.14.6.m2.1" class="ltx_Math" alttext="14.16" display="inline"><semantics id="S4.T1.14.14.6.m2.1a"><mn id="S4.T1.14.14.6.m2.1.1" xref="S4.T1.14.14.6.m2.1.1.cmml">14.16</mn><annotation-xml encoding="MathML-Content" id="S4.T1.14.14.6.m2.1b"><cn type="float" id="S4.T1.14.14.6.m2.1.1.cmml" xref="S4.T1.14.14.6.m2.1.1">14.16</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.14.14.6.m2.1c">14.16</annotation></semantics></math>
</td>
<td id="S4.T1.16.16.8" class="ltx_td ltx_align_center">
<math id="S4.T1.15.15.7.m1.1" class="ltx_Math" alttext="15.72" display="inline"><semantics id="S4.T1.15.15.7.m1.1a"><mn id="S4.T1.15.15.7.m1.1.1" xref="S4.T1.15.15.7.m1.1.1.cmml">15.72</mn><annotation-xml encoding="MathML-Content" id="S4.T1.15.15.7.m1.1b"><cn type="float" id="S4.T1.15.15.7.m1.1.1.cmml" xref="S4.T1.15.15.7.m1.1.1">15.72</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.15.15.7.m1.1c">15.72</annotation></semantics></math>-<math id="S4.T1.16.16.8.m2.1" class="ltx_Math" alttext="15.74" display="inline"><semantics id="S4.T1.16.16.8.m2.1a"><mn id="S4.T1.16.16.8.m2.1.1" xref="S4.T1.16.16.8.m2.1.1.cmml">15.74</mn><annotation-xml encoding="MathML-Content" id="S4.T1.16.16.8.m2.1b"><cn type="float" id="S4.T1.16.16.8.m2.1.1.cmml" xref="S4.T1.16.16.8.m2.1.1">15.74</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.16.16.8.m2.1c">15.74</annotation></semantics></math>
</td>
</tr>
<tr id="S4.T1.24.24" class="ltx_tr">
<th id="S4.T1.24.24.9" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.24.24.9.1" class="ltx_text ltx_font_smallcaps">joint BiT-Large*</span> (LM+355M)</th>
<td id="S4.T1.18.18.2" class="ltx_td ltx_align_center">
<math id="S4.T1.17.17.1.m1.1" class="ltx_Math" alttext="\mathbf{14.59}" display="inline"><semantics id="S4.T1.17.17.1.m1.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.17.17.1.m1.1.1" xref="S4.T1.17.17.1.m1.1.1.cmml">14.59</mn><annotation-xml encoding="MathML-Content" id="S4.T1.17.17.1.m1.1b"><cn type="float" id="S4.T1.17.17.1.m1.1.1.cmml" xref="S4.T1.17.17.1.m1.1.1">14.59</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.17.17.1.m1.1c">\mathbf{14.59}</annotation></semantics></math>-<math id="S4.T1.18.18.2.m2.1" class="ltx_Math" alttext="\mathbf{14.61}" display="inline"><semantics id="S4.T1.18.18.2.m2.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.18.18.2.m2.1.1" xref="S4.T1.18.18.2.m2.1.1.cmml">14.61</mn><annotation-xml encoding="MathML-Content" id="S4.T1.18.18.2.m2.1b"><cn type="float" id="S4.T1.18.18.2.m2.1.1.cmml" xref="S4.T1.18.18.2.m2.1.1">14.61</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.18.18.2.m2.1c">\mathbf{14.61}</annotation></semantics></math>
</td>
<td id="S4.T1.20.20.4" class="ltx_td ltx_align_center">
<math id="S4.T1.19.19.3.m1.1" class="ltx_Math" alttext="\mathbf{13.97}" display="inline"><semantics id="S4.T1.19.19.3.m1.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.19.19.3.m1.1.1" xref="S4.T1.19.19.3.m1.1.1.cmml">13.97</mn><annotation-xml encoding="MathML-Content" id="S4.T1.19.19.3.m1.1b"><cn type="float" id="S4.T1.19.19.3.m1.1.1.cmml" xref="S4.T1.19.19.3.m1.1.1">13.97</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.19.19.3.m1.1c">\mathbf{13.97}</annotation></semantics></math>-<math id="S4.T1.20.20.4.m2.1" class="ltx_Math" alttext="\mathbf{14.00}" display="inline"><semantics id="S4.T1.20.20.4.m2.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.20.20.4.m2.1.1" xref="S4.T1.20.20.4.m2.1.1.cmml">14.00</mn><annotation-xml encoding="MathML-Content" id="S4.T1.20.20.4.m2.1b"><cn type="float" id="S4.T1.20.20.4.m2.1.1.cmml" xref="S4.T1.20.20.4.m2.1.1">14.00</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.20.20.4.m2.1c">\mathbf{14.00}</annotation></semantics></math>
</td>
<td id="S4.T1.22.22.6" class="ltx_td ltx_align_center">
<math id="S4.T1.21.21.5.m1.1" class="ltx_Math" alttext="\mathbf{13.80}" display="inline"><semantics id="S4.T1.21.21.5.m1.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.21.21.5.m1.1.1" xref="S4.T1.21.21.5.m1.1.1.cmml">13.80</mn><annotation-xml encoding="MathML-Content" id="S4.T1.21.21.5.m1.1b"><cn type="float" id="S4.T1.21.21.5.m1.1.1.cmml" xref="S4.T1.21.21.5.m1.1.1">13.80</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.21.21.5.m1.1c">\mathbf{13.80}</annotation></semantics></math>-<math id="S4.T1.22.22.6.m2.1" class="ltx_Math" alttext="\mathbf{13.83}" display="inline"><semantics id="S4.T1.22.22.6.m2.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.22.22.6.m2.1.1" xref="S4.T1.22.22.6.m2.1.1.cmml">13.83</mn><annotation-xml encoding="MathML-Content" id="S4.T1.22.22.6.m2.1b"><cn type="float" id="S4.T1.22.22.6.m2.1.1.cmml" xref="S4.T1.22.22.6.m2.1.1">13.83</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.22.22.6.m2.1c">\mathbf{13.83}</annotation></semantics></math>
</td>
<td id="S4.T1.24.24.8" class="ltx_td ltx_align_center">
<math id="S4.T1.23.23.7.m1.1" class="ltx_Math" alttext="\mathbf{15.33}" display="inline"><semantics id="S4.T1.23.23.7.m1.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.23.23.7.m1.1.1" xref="S4.T1.23.23.7.m1.1.1.cmml">15.33</mn><annotation-xml encoding="MathML-Content" id="S4.T1.23.23.7.m1.1b"><cn type="float" id="S4.T1.23.23.7.m1.1.1.cmml" xref="S4.T1.23.23.7.m1.1.1">15.33</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.23.23.7.m1.1c">\mathbf{15.33}</annotation></semantics></math>-<math id="S4.T1.24.24.8.m2.1" class="ltx_Math" alttext="\mathbf{15.36}" display="inline"><semantics id="S4.T1.24.24.8.m2.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.24.24.8.m2.1.1" xref="S4.T1.24.24.8.m2.1.1.cmml">15.36</mn><annotation-xml encoding="MathML-Content" id="S4.T1.24.24.8.m2.1b"><cn type="float" id="S4.T1.24.24.8.m2.1.1.cmml" xref="S4.T1.24.24.8.m2.1.1">15.36</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.24.24.8.m2.1c">\mathbf{15.36}</annotation></semantics></math>
</td>
</tr>
<tr id="S4.T1.28.28" class="ltx_tr">
<th id="S4.T1.28.28.5" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">
<span id="S4.T1.28.28.5.1" class="ltx_text ltx_font_smallcaps">Base LM-24L</span> (203M)</th>
<td id="S4.T1.25.25.1" class="ltx_td ltx_align_center ltx_border_t"><math id="S4.T1.25.25.1.m1.1" class="ltx_Math" alttext="15.71" display="inline"><semantics id="S4.T1.25.25.1.m1.1a"><mn id="S4.T1.25.25.1.m1.1.1" xref="S4.T1.25.25.1.m1.1.1.cmml">15.71</mn><annotation-xml encoding="MathML-Content" id="S4.T1.25.25.1.m1.1b"><cn type="float" id="S4.T1.25.25.1.m1.1.1.cmml" xref="S4.T1.25.25.1.m1.1.1">15.71</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.25.25.1.m1.1c">15.71</annotation></semantics></math></td>
<td id="S4.T1.26.26.2" class="ltx_td ltx_align_center ltx_border_t"><math id="S4.T1.26.26.2.m1.1" class="ltx_Math" alttext="14.89" display="inline"><semantics id="S4.T1.26.26.2.m1.1a"><mn id="S4.T1.26.26.2.m1.1.1" xref="S4.T1.26.26.2.m1.1.1.cmml">14.89</mn><annotation-xml encoding="MathML-Content" id="S4.T1.26.26.2.m1.1b"><cn type="float" id="S4.T1.26.26.2.m1.1.1.cmml" xref="S4.T1.26.26.2.m1.1.1">14.89</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.26.26.2.m1.1c">14.89</annotation></semantics></math></td>
<td id="S4.T1.27.27.3" class="ltx_td ltx_align_center ltx_border_t"><math id="S4.T1.27.27.3.m1.1" class="ltx_Math" alttext="15.61" display="inline"><semantics id="S4.T1.27.27.3.m1.1a"><mn id="S4.T1.27.27.3.m1.1.1" xref="S4.T1.27.27.3.m1.1.1.cmml">15.61</mn><annotation-xml encoding="MathML-Content" id="S4.T1.27.27.3.m1.1b"><cn type="float" id="S4.T1.27.27.3.m1.1.1.cmml" xref="S4.T1.27.27.3.m1.1.1">15.61</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.27.27.3.m1.1c">15.61</annotation></semantics></math></td>
<td id="S4.T1.28.28.4" class="ltx_td ltx_align_center ltx_border_t"><math id="S4.T1.28.28.4.m1.1" class="ltx_Math" alttext="18.14" display="inline"><semantics id="S4.T1.28.28.4.m1.1a"><mn id="S4.T1.28.28.4.m1.1.1" xref="S4.T1.28.28.4.m1.1.1.cmml">18.14</mn><annotation-xml encoding="MathML-Content" id="S4.T1.28.28.4.m1.1b"><cn type="float" id="S4.T1.28.28.4.m1.1.1.cmml" xref="S4.T1.28.28.4.m1.1.1">18.14</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.28.28.4.m1.1c">18.14</annotation></semantics></math></td>
</tr>
<tr id="S4.T1.32.32" class="ltx_tr">
<th id="S4.T1.32.32.5" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.32.32.5.1" class="ltx_text ltx_font_smallcaps">RALM-24L</span> (LM-24L+203M)</th>
<td id="S4.T1.29.29.1" class="ltx_td ltx_align_center"><math id="S4.T1.29.29.1.m1.1" class="ltx_Math" alttext="15.70" display="inline"><semantics id="S4.T1.29.29.1.m1.1a"><mn id="S4.T1.29.29.1.m1.1.1" xref="S4.T1.29.29.1.m1.1.1.cmml">15.70</mn><annotation-xml encoding="MathML-Content" id="S4.T1.29.29.1.m1.1b"><cn type="float" id="S4.T1.29.29.1.m1.1.1.cmml" xref="S4.T1.29.29.1.m1.1.1">15.70</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.29.29.1.m1.1c">15.70</annotation></semantics></math></td>
<td id="S4.T1.30.30.2" class="ltx_td ltx_align_center"><math id="S4.T1.30.30.2.m1.1" class="ltx_Math" alttext="14.89" display="inline"><semantics id="S4.T1.30.30.2.m1.1a"><mn id="S4.T1.30.30.2.m1.1.1" xref="S4.T1.30.30.2.m1.1.1.cmml">14.89</mn><annotation-xml encoding="MathML-Content" id="S4.T1.30.30.2.m1.1b"><cn type="float" id="S4.T1.30.30.2.m1.1.1.cmml" xref="S4.T1.30.30.2.m1.1.1">14.89</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.30.30.2.m1.1c">14.89</annotation></semantics></math></td>
<td id="S4.T1.31.31.3" class="ltx_td ltx_align_center"><math id="S4.T1.31.31.3.m1.1" class="ltx_Math" alttext="15.63" display="inline"><semantics id="S4.T1.31.31.3.m1.1a"><mn id="S4.T1.31.31.3.m1.1.1" xref="S4.T1.31.31.3.m1.1.1.cmml">15.63</mn><annotation-xml encoding="MathML-Content" id="S4.T1.31.31.3.m1.1b"><cn type="float" id="S4.T1.31.31.3.m1.1.1.cmml" xref="S4.T1.31.31.3.m1.1.1">15.63</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.31.31.3.m1.1c">15.63</annotation></semantics></math></td>
<td id="S4.T1.32.32.4" class="ltx_td ltx_align_center"><math id="S4.T1.32.32.4.m1.1" class="ltx_Math" alttext="18.17" display="inline"><semantics id="S4.T1.32.32.4.m1.1a"><mn id="S4.T1.32.32.4.m1.1.1" xref="S4.T1.32.32.4.m1.1.1.cmml">18.17</mn><annotation-xml encoding="MathML-Content" id="S4.T1.32.32.4.m1.1b"><cn type="float" id="S4.T1.32.32.4.m1.1.1.cmml" xref="S4.T1.32.32.4.m1.1.1">18.17</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.32.32.4.m1.1c">18.17</annotation></semantics></math></td>
</tr>
<tr id="S4.T1.36.36" class="ltx_tr">
<th id="S4.T1.36.36.5" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.36.36.5.1" class="ltx_text ltx_font_smallcaps">BALM-24L</span> (408M)</th>
<td id="S4.T1.33.33.1" class="ltx_td ltx_align_center"><math id="S4.T1.33.33.1.m1.1" class="ltx_Math" alttext="14.58" display="inline"><semantics id="S4.T1.33.33.1.m1.1a"><mn id="S4.T1.33.33.1.m1.1.1" xref="S4.T1.33.33.1.m1.1.1.cmml">14.58</mn><annotation-xml encoding="MathML-Content" id="S4.T1.33.33.1.m1.1b"><cn type="float" id="S4.T1.33.33.1.m1.1.1.cmml" xref="S4.T1.33.33.1.m1.1.1">14.58</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.33.33.1.m1.1c">14.58</annotation></semantics></math></td>
<td id="S4.T1.34.34.2" class="ltx_td ltx_align_center"><math id="S4.T1.34.34.2.m1.1" class="ltx_Math" alttext="13.92" display="inline"><semantics id="S4.T1.34.34.2.m1.1a"><mn id="S4.T1.34.34.2.m1.1.1" xref="S4.T1.34.34.2.m1.1.1.cmml">13.92</mn><annotation-xml encoding="MathML-Content" id="S4.T1.34.34.2.m1.1b"><cn type="float" id="S4.T1.34.34.2.m1.1.1.cmml" xref="S4.T1.34.34.2.m1.1.1">13.92</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.34.34.2.m1.1c">13.92</annotation></semantics></math></td>
<td id="S4.T1.35.35.3" class="ltx_td ltx_align_center"><math id="S4.T1.35.35.3.m1.1" class="ltx_Math" alttext="15.20" display="inline"><semantics id="S4.T1.35.35.3.m1.1a"><mn id="S4.T1.35.35.3.m1.1.1" xref="S4.T1.35.35.3.m1.1.1.cmml">15.20</mn><annotation-xml encoding="MathML-Content" id="S4.T1.35.35.3.m1.1b"><cn type="float" id="S4.T1.35.35.3.m1.1.1.cmml" xref="S4.T1.35.35.3.m1.1.1">15.20</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.35.35.3.m1.1c">15.20</annotation></semantics></math></td>
<td id="S4.T1.36.36.4" class="ltx_td ltx_align_center"><math id="S4.T1.36.36.4.m1.1" class="ltx_Math" alttext="18.24" display="inline"><semantics id="S4.T1.36.36.4.m1.1a"><mn id="S4.T1.36.36.4.m1.1.1" xref="S4.T1.36.36.4.m1.1.1.cmml">18.24</mn><annotation-xml encoding="MathML-Content" id="S4.T1.36.36.4.m1.1b"><cn type="float" id="S4.T1.36.36.4.m1.1.1.cmml" xref="S4.T1.36.36.4.m1.1.1">18.24</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.36.36.4.m1.1c">18.24</annotation></semantics></math></td>
</tr>
<tr id="S4.T1.43.43" class="ltx_tr">
<th id="S4.T1.43.43.8" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.43.43.8.1" class="ltx_text ltx_font_smallcaps">joint UniT</span> (LM-24L+203M)</th>
<td id="S4.T1.38.38.2" class="ltx_td ltx_align_center">
<math id="S4.T1.37.37.1.m1.1" class="ltx_Math" alttext="14.59" display="inline"><semantics id="S4.T1.37.37.1.m1.1a"><mn id="S4.T1.37.37.1.m1.1.1" xref="S4.T1.37.37.1.m1.1.1.cmml">14.59</mn><annotation-xml encoding="MathML-Content" id="S4.T1.37.37.1.m1.1b"><cn type="float" id="S4.T1.37.37.1.m1.1.1.cmml" xref="S4.T1.37.37.1.m1.1.1">14.59</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.37.37.1.m1.1c">14.59</annotation></semantics></math>-<math id="S4.T1.38.38.2.m2.1" class="ltx_Math" alttext="14.61" display="inline"><semantics id="S4.T1.38.38.2.m2.1a"><mn id="S4.T1.38.38.2.m2.1.1" xref="S4.T1.38.38.2.m2.1.1.cmml">14.61</mn><annotation-xml encoding="MathML-Content" id="S4.T1.38.38.2.m2.1b"><cn type="float" id="S4.T1.38.38.2.m2.1.1.cmml" xref="S4.T1.38.38.2.m2.1.1">14.61</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.38.38.2.m2.1c">14.61</annotation></semantics></math>
</td>
<td id="S4.T1.40.40.4" class="ltx_td ltx_align_center">
<math id="S4.T1.39.39.3.m1.1" class="ltx_Math" alttext="13.81" display="inline"><semantics id="S4.T1.39.39.3.m1.1a"><mn id="S4.T1.39.39.3.m1.1.1" xref="S4.T1.39.39.3.m1.1.1.cmml">13.81</mn><annotation-xml encoding="MathML-Content" id="S4.T1.39.39.3.m1.1b"><cn type="float" id="S4.T1.39.39.3.m1.1.1.cmml" xref="S4.T1.39.39.3.m1.1.1">13.81</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.39.39.3.m1.1c">13.81</annotation></semantics></math>-<math id="S4.T1.40.40.4.m2.1" class="ltx_Math" alttext="13.82" display="inline"><semantics id="S4.T1.40.40.4.m2.1a"><mn id="S4.T1.40.40.4.m2.1.1" xref="S4.T1.40.40.4.m2.1.1.cmml">13.82</mn><annotation-xml encoding="MathML-Content" id="S4.T1.40.40.4.m2.1b"><cn type="float" id="S4.T1.40.40.4.m2.1.1.cmml" xref="S4.T1.40.40.4.m2.1.1">13.82</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.40.40.4.m2.1c">13.82</annotation></semantics></math>
</td>
<td id="S4.T1.41.41.5" class="ltx_td ltx_align_center"><math id="S4.T1.41.41.5.m1.1" class="ltx_Math" alttext="15.12-15.16" display="inline"><semantics id="S4.T1.41.41.5.m1.1a"><mrow id="S4.T1.41.41.5.m1.1.1" xref="S4.T1.41.41.5.m1.1.1.cmml"><mn id="S4.T1.41.41.5.m1.1.1.2" xref="S4.T1.41.41.5.m1.1.1.2.cmml">15.12</mn><mo id="S4.T1.41.41.5.m1.1.1.1" xref="S4.T1.41.41.5.m1.1.1.1.cmml">−</mo><mn id="S4.T1.41.41.5.m1.1.1.3" xref="S4.T1.41.41.5.m1.1.1.3.cmml">15.16</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.41.41.5.m1.1b"><apply id="S4.T1.41.41.5.m1.1.1.cmml" xref="S4.T1.41.41.5.m1.1.1"><minus id="S4.T1.41.41.5.m1.1.1.1.cmml" xref="S4.T1.41.41.5.m1.1.1.1"></minus><cn type="float" id="S4.T1.41.41.5.m1.1.1.2.cmml" xref="S4.T1.41.41.5.m1.1.1.2">15.12</cn><cn type="float" id="S4.T1.41.41.5.m1.1.1.3.cmml" xref="S4.T1.41.41.5.m1.1.1.3">15.16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.41.41.5.m1.1c">15.12-15.16</annotation></semantics></math></td>
<td id="S4.T1.43.43.7" class="ltx_td ltx_align_center">
<math id="S4.T1.42.42.6.m1.1" class="ltx_Math" alttext="17.46" display="inline"><semantics id="S4.T1.42.42.6.m1.1a"><mn id="S4.T1.42.42.6.m1.1.1" xref="S4.T1.42.42.6.m1.1.1.cmml">17.46</mn><annotation-xml encoding="MathML-Content" id="S4.T1.42.42.6.m1.1b"><cn type="float" id="S4.T1.42.42.6.m1.1.1.cmml" xref="S4.T1.42.42.6.m1.1.1">17.46</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.42.42.6.m1.1c">17.46</annotation></semantics></math>-<math id="S4.T1.43.43.7.m2.1" class="ltx_Math" alttext="17.48" display="inline"><semantics id="S4.T1.43.43.7.m2.1a"><mn id="S4.T1.43.43.7.m2.1.1" xref="S4.T1.43.43.7.m2.1.1.cmml">17.48</mn><annotation-xml encoding="MathML-Content" id="S4.T1.43.43.7.m2.1b"><cn type="float" id="S4.T1.43.43.7.m2.1.1.cmml" xref="S4.T1.43.43.7.m2.1.1">17.48</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.43.43.7.m2.1c">17.48</annotation></semantics></math>
</td>
</tr>
<tr id="S4.T1.47.47" class="ltx_tr">
<th id="S4.T1.47.47.5" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.47.47.5.1" class="ltx_text ltx_font_smallcaps">joint BiT-Base</span> (LM-24L+125M)</th>
<td id="S4.T1.45.45.2" class="ltx_td ltx_align_center">
<math id="S4.T1.44.44.1.m1.1" class="ltx_Math" alttext="{13.68}" display="inline"><semantics id="S4.T1.44.44.1.m1.1a"><mn id="S4.T1.44.44.1.m1.1.1" xref="S4.T1.44.44.1.m1.1.1.cmml">13.68</mn><annotation-xml encoding="MathML-Content" id="S4.T1.44.44.1.m1.1b"><cn type="float" id="S4.T1.44.44.1.m1.1.1.cmml" xref="S4.T1.44.44.1.m1.1.1">13.68</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.44.44.1.m1.1c">{13.68}</annotation></semantics></math>-<math id="S4.T1.45.45.2.m2.1" class="ltx_Math" alttext="{13.69}" display="inline"><semantics id="S4.T1.45.45.2.m2.1a"><mn id="S4.T1.45.45.2.m2.1.1" xref="S4.T1.45.45.2.m2.1.1.cmml">13.69</mn><annotation-xml encoding="MathML-Content" id="S4.T1.45.45.2.m2.1b"><cn type="float" id="S4.T1.45.45.2.m2.1.1.cmml" xref="S4.T1.45.45.2.m2.1.1">13.69</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.45.45.2.m2.1c">{13.69}</annotation></semantics></math>
</td>
<td id="S4.T1.47.47.4" class="ltx_td ltx_align_center">
<math id="S4.T1.46.46.3.m1.1" class="ltx_Math" alttext="{13.01}" display="inline"><semantics id="S4.T1.46.46.3.m1.1a"><mn id="S4.T1.46.46.3.m1.1.1" xref="S4.T1.46.46.3.m1.1.1.cmml">13.01</mn><annotation-xml encoding="MathML-Content" id="S4.T1.46.46.3.m1.1b"><cn type="float" id="S4.T1.46.46.3.m1.1.1.cmml" xref="S4.T1.46.46.3.m1.1.1">13.01</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.46.46.3.m1.1c">{13.01}</annotation></semantics></math>-<math id="S4.T1.47.47.4.m2.1" class="ltx_Math" alttext="{13.03}" display="inline"><semantics id="S4.T1.47.47.4.m2.1a"><mn id="S4.T1.47.47.4.m2.1.1" xref="S4.T1.47.47.4.m2.1.1.cmml">13.03</mn><annotation-xml encoding="MathML-Content" id="S4.T1.47.47.4.m2.1b"><cn type="float" id="S4.T1.47.47.4.m2.1.1.cmml" xref="S4.T1.47.47.4.m2.1.1">13.03</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.47.47.4.m2.1c">{13.03}</annotation></semantics></math>
</td>
<td id="S4.T1.47.47.6" class="ltx_td ltx_align_center">-</td>
<td id="S4.T1.47.47.7" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="S4.T1.55.55" class="ltx_tr">
<th id="S4.T1.55.55.9" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.55.55.9.1" class="ltx_text ltx_font_smallcaps">joint BiT-Base*</span> (LM-24L+125M)</th>
<td id="S4.T1.49.49.2" class="ltx_td ltx_align_center">
<math id="S4.T1.48.48.1.m1.1" class="ltx_Math" alttext="13.60" display="inline"><semantics id="S4.T1.48.48.1.m1.1a"><mn id="S4.T1.48.48.1.m1.1.1" xref="S4.T1.48.48.1.m1.1.1.cmml">13.60</mn><annotation-xml encoding="MathML-Content" id="S4.T1.48.48.1.m1.1b"><cn type="float" id="S4.T1.48.48.1.m1.1.1.cmml" xref="S4.T1.48.48.1.m1.1.1">13.60</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.48.48.1.m1.1c">13.60</annotation></semantics></math>-<math id="S4.T1.49.49.2.m2.1" class="ltx_Math" alttext="13.62" display="inline"><semantics id="S4.T1.49.49.2.m2.1a"><mn id="S4.T1.49.49.2.m2.1.1" xref="S4.T1.49.49.2.m2.1.1.cmml">13.62</mn><annotation-xml encoding="MathML-Content" id="S4.T1.49.49.2.m2.1b"><cn type="float" id="S4.T1.49.49.2.m2.1.1.cmml" xref="S4.T1.49.49.2.m2.1.1">13.62</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.49.49.2.m2.1c">13.62</annotation></semantics></math>
</td>
<td id="S4.T1.51.51.4" class="ltx_td ltx_align_center">
<math id="S4.T1.50.50.3.m1.1" class="ltx_Math" alttext="12.93" display="inline"><semantics id="S4.T1.50.50.3.m1.1a"><mn id="S4.T1.50.50.3.m1.1.1" xref="S4.T1.50.50.3.m1.1.1.cmml">12.93</mn><annotation-xml encoding="MathML-Content" id="S4.T1.50.50.3.m1.1b"><cn type="float" id="S4.T1.50.50.3.m1.1.1.cmml" xref="S4.T1.50.50.3.m1.1.1">12.93</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.50.50.3.m1.1c">12.93</annotation></semantics></math>-<math id="S4.T1.51.51.4.m2.1" class="ltx_Math" alttext="12.95" display="inline"><semantics id="S4.T1.51.51.4.m2.1a"><mn id="S4.T1.51.51.4.m2.1.1" xref="S4.T1.51.51.4.m2.1.1.cmml">12.95</mn><annotation-xml encoding="MathML-Content" id="S4.T1.51.51.4.m2.1b"><cn type="float" id="S4.T1.51.51.4.m2.1.1.cmml" xref="S4.T1.51.51.4.m2.1.1">12.95</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.51.51.4.m2.1c">12.95</annotation></semantics></math>
</td>
<td id="S4.T1.53.53.6" class="ltx_td ltx_align_center">
<math id="S4.T1.52.52.5.m1.1" class="ltx_Math" alttext="14.11" display="inline"><semantics id="S4.T1.52.52.5.m1.1a"><mn id="S4.T1.52.52.5.m1.1.1" xref="S4.T1.52.52.5.m1.1.1.cmml">14.11</mn><annotation-xml encoding="MathML-Content" id="S4.T1.52.52.5.m1.1b"><cn type="float" id="S4.T1.52.52.5.m1.1.1.cmml" xref="S4.T1.52.52.5.m1.1.1">14.11</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.52.52.5.m1.1c">14.11</annotation></semantics></math>-<math id="S4.T1.53.53.6.m2.1" class="ltx_Math" alttext="14.12" display="inline"><semantics id="S4.T1.53.53.6.m2.1a"><mn id="S4.T1.53.53.6.m2.1.1" xref="S4.T1.53.53.6.m2.1.1.cmml">14.12</mn><annotation-xml encoding="MathML-Content" id="S4.T1.53.53.6.m2.1b"><cn type="float" id="S4.T1.53.53.6.m2.1.1.cmml" xref="S4.T1.53.53.6.m2.1.1">14.12</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.53.53.6.m2.1c">14.12</annotation></semantics></math>
</td>
<td id="S4.T1.55.55.8" class="ltx_td ltx_align_center">
<math id="S4.T1.54.54.7.m1.1" class="ltx_Math" alttext="16.17" display="inline"><semantics id="S4.T1.54.54.7.m1.1a"><mn id="S4.T1.54.54.7.m1.1.1" xref="S4.T1.54.54.7.m1.1.1.cmml">16.17</mn><annotation-xml encoding="MathML-Content" id="S4.T1.54.54.7.m1.1b"><cn type="float" id="S4.T1.54.54.7.m1.1.1.cmml" xref="S4.T1.54.54.7.m1.1.1">16.17</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.54.54.7.m1.1c">16.17</annotation></semantics></math>-<math id="S4.T1.55.55.8.m2.1" class="ltx_Math" alttext="16.18" display="inline"><semantics id="S4.T1.55.55.8.m2.1a"><mn id="S4.T1.55.55.8.m2.1.1" xref="S4.T1.55.55.8.m2.1.1.cmml">16.18</mn><annotation-xml encoding="MathML-Content" id="S4.T1.55.55.8.m2.1b"><cn type="float" id="S4.T1.55.55.8.m2.1.1.cmml" xref="S4.T1.55.55.8.m2.1.1">16.18</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.55.55.8.m2.1c">16.18</annotation></semantics></math>
</td>
</tr>
<tr id="S4.T1.59.59" class="ltx_tr">
<th id="S4.T1.59.59.5" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T1.59.59.5.1" class="ltx_text ltx_font_smallcaps">joint BiT-Med</span> (LM-24L+203M)</th>
<td id="S4.T1.57.57.2" class="ltx_td ltx_align_center">
<math id="S4.T1.56.56.1.m1.1" class="ltx_Math" alttext="12.97" display="inline"><semantics id="S4.T1.56.56.1.m1.1a"><mn id="S4.T1.56.56.1.m1.1.1" xref="S4.T1.56.56.1.m1.1.1.cmml">12.97</mn><annotation-xml encoding="MathML-Content" id="S4.T1.56.56.1.m1.1b"><cn type="float" id="S4.T1.56.56.1.m1.1.1.cmml" xref="S4.T1.56.56.1.m1.1.1">12.97</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.56.56.1.m1.1c">12.97</annotation></semantics></math>-<math id="S4.T1.57.57.2.m2.1" class="ltx_Math" alttext="13.01" display="inline"><semantics id="S4.T1.57.57.2.m2.1a"><mn id="S4.T1.57.57.2.m2.1.1" xref="S4.T1.57.57.2.m2.1.1.cmml">13.01</mn><annotation-xml encoding="MathML-Content" id="S4.T1.57.57.2.m2.1b"><cn type="float" id="S4.T1.57.57.2.m2.1.1.cmml" xref="S4.T1.57.57.2.m2.1.1">13.01</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.57.57.2.m2.1c">13.01</annotation></semantics></math>
</td>
<td id="S4.T1.59.59.4" class="ltx_td ltx_align_center">
<math id="S4.T1.58.58.3.m1.1" class="ltx_Math" alttext="12.38" display="inline"><semantics id="S4.T1.58.58.3.m1.1a"><mn id="S4.T1.58.58.3.m1.1.1" xref="S4.T1.58.58.3.m1.1.1.cmml">12.38</mn><annotation-xml encoding="MathML-Content" id="S4.T1.58.58.3.m1.1b"><cn type="float" id="S4.T1.58.58.3.m1.1.1.cmml" xref="S4.T1.58.58.3.m1.1.1">12.38</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.58.58.3.m1.1c">12.38</annotation></semantics></math>-<math id="S4.T1.59.59.4.m2.1" class="ltx_Math" alttext="12.42" display="inline"><semantics id="S4.T1.59.59.4.m2.1a"><mn id="S4.T1.59.59.4.m2.1.1" xref="S4.T1.59.59.4.m2.1.1.cmml">12.42</mn><annotation-xml encoding="MathML-Content" id="S4.T1.59.59.4.m2.1b"><cn type="float" id="S4.T1.59.59.4.m2.1.1.cmml" xref="S4.T1.59.59.4.m2.1.1">12.42</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.59.59.4.m2.1c">12.42</annotation></semantics></math>
</td>
<td id="S4.T1.59.59.6" class="ltx_td ltx_align_center">-</td>
<td id="S4.T1.59.59.7" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="S4.T1.67.67" class="ltx_tr">
<th id="S4.T1.67.67.9" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">
<span id="S4.T1.67.67.9.1" class="ltx_text ltx_font_smallcaps">joint BiT-Large*</span> (LM-24L+355M)</th>
<td id="S4.T1.61.61.2" class="ltx_td ltx_align_center ltx_border_bb">
<math id="S4.T1.60.60.1.m1.1" class="ltx_Math" alttext="\mathbf{12.71}" display="inline"><semantics id="S4.T1.60.60.1.m1.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.60.60.1.m1.1.1" xref="S4.T1.60.60.1.m1.1.1.cmml">12.71</mn><annotation-xml encoding="MathML-Content" id="S4.T1.60.60.1.m1.1b"><cn type="float" id="S4.T1.60.60.1.m1.1.1.cmml" xref="S4.T1.60.60.1.m1.1.1">12.71</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.60.60.1.m1.1c">\mathbf{12.71}</annotation></semantics></math>-<math id="S4.T1.61.61.2.m2.1" class="ltx_Math" alttext="\mathbf{12.77}" display="inline"><semantics id="S4.T1.61.61.2.m2.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.61.61.2.m2.1.1" xref="S4.T1.61.61.2.m2.1.1.cmml">12.77</mn><annotation-xml encoding="MathML-Content" id="S4.T1.61.61.2.m2.1b"><cn type="float" id="S4.T1.61.61.2.m2.1.1.cmml" xref="S4.T1.61.61.2.m2.1.1">12.77</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.61.61.2.m2.1c">\mathbf{12.77}</annotation></semantics></math>
</td>
<td id="S4.T1.63.63.4" class="ltx_td ltx_align_center ltx_border_bb">
<math id="S4.T1.62.62.3.m1.1" class="ltx_Math" alttext="\mathbf{12.10}" display="inline"><semantics id="S4.T1.62.62.3.m1.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.62.62.3.m1.1.1" xref="S4.T1.62.62.3.m1.1.1.cmml">12.10</mn><annotation-xml encoding="MathML-Content" id="S4.T1.62.62.3.m1.1b"><cn type="float" id="S4.T1.62.62.3.m1.1.1.cmml" xref="S4.T1.62.62.3.m1.1.1">12.10</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.62.62.3.m1.1c">\mathbf{12.10}</annotation></semantics></math>-<math id="S4.T1.63.63.4.m2.1" class="ltx_Math" alttext="\mathbf{12.16}" display="inline"><semantics id="S4.T1.63.63.4.m2.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.63.63.4.m2.1.1" xref="S4.T1.63.63.4.m2.1.1.cmml">12.16</mn><annotation-xml encoding="MathML-Content" id="S4.T1.63.63.4.m2.1b"><cn type="float" id="S4.T1.63.63.4.m2.1.1.cmml" xref="S4.T1.63.63.4.m2.1.1">12.16</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.63.63.4.m2.1c">\mathbf{12.16}</annotation></semantics></math>
</td>
<td id="S4.T1.65.65.6" class="ltx_td ltx_align_center ltx_border_bb">
<math id="S4.T1.64.64.5.m1.1" class="ltx_Math" alttext="\mathbf{13.30}" display="inline"><semantics id="S4.T1.64.64.5.m1.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.64.64.5.m1.1.1" xref="S4.T1.64.64.5.m1.1.1.cmml">13.30</mn><annotation-xml encoding="MathML-Content" id="S4.T1.64.64.5.m1.1b"><cn type="float" id="S4.T1.64.64.5.m1.1.1.cmml" xref="S4.T1.64.64.5.m1.1.1">13.30</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.64.64.5.m1.1c">\mathbf{13.30}</annotation></semantics></math>-<math id="S4.T1.65.65.6.m2.1" class="ltx_Math" alttext="\mathbf{13.34}" display="inline"><semantics id="S4.T1.65.65.6.m2.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.65.65.6.m2.1.1" xref="S4.T1.65.65.6.m2.1.1.cmml">13.34</mn><annotation-xml encoding="MathML-Content" id="S4.T1.65.65.6.m2.1b"><cn type="float" id="S4.T1.65.65.6.m2.1.1.cmml" xref="S4.T1.65.65.6.m2.1.1">13.34</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.65.65.6.m2.1c">\mathbf{13.34}</annotation></semantics></math>
</td>
<td id="S4.T1.67.67.8" class="ltx_td ltx_align_center ltx_border_bb">
<math id="S4.T1.66.66.7.m1.1" class="ltx_Math" alttext="\mathbf{15.17}" display="inline"><semantics id="S4.T1.66.66.7.m1.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.66.66.7.m1.1.1" xref="S4.T1.66.66.7.m1.1.1.cmml">15.17</mn><annotation-xml encoding="MathML-Content" id="S4.T1.66.66.7.m1.1b"><cn type="float" id="S4.T1.66.66.7.m1.1.1.cmml" xref="S4.T1.66.66.7.m1.1.1">15.17</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.66.66.7.m1.1c">\mathbf{15.17}</annotation></semantics></math>-<math id="S4.T1.67.67.8.m2.1" class="ltx_Math" alttext="\mathbf{15.22}" display="inline"><semantics id="S4.T1.67.67.8.m2.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S4.T1.67.67.8.m2.1.1" xref="S4.T1.67.67.8.m2.1.1.cmml">15.22</mn><annotation-xml encoding="MathML-Content" id="S4.T1.67.67.8.m2.1b"><cn type="float" id="S4.T1.67.67.8.m2.1.1.cmml" xref="S4.T1.67.67.8.m2.1.1">15.22</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.67.67.8.m2.1c">\mathbf{15.22}</annotation></semantics></math>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Validation and test perplexity on CC-News and Toronto Book Corpus. * denotes models initialized with RoBERTa
trained on additional data. The joint model perplexity ranges are <span id="S4.T1.69.1" class="ltx_text ltx_font_bold">estimated</span> using 100,000 samples, see
Eq. <a href="#S3.E4" title="In Theorem 2. ‣ 3.2 Evaluation ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. The number of parameters of each model is shown in parentheses.</figcaption>
</figure>
<figure id="S4.F1" class="ltx_figure"><img src="/html/2004.11714/assets/x1.png" id="S4.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="664" height="341" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Perplexity gain of <span id="S4.F1.12.1" class="ltx_text ltx_markedasmath ltx_font_smallcaps">Joint BiT-Med</span> and <math id="S4.F1.6.m2.1" class="ltx_math_unparsed" alttext="\textsc{Joint BiT-Large}*" display="inline"><semantics id="S4.F1.6.m2.1b"><mrow id="S4.F1.6.m2.1c"><mtext class="ltx_font_smallcaps" id="S4.F1.6.m2.1.1">Joint BiT-Large</mtext><mo lspace="0.222em" id="S4.F1.6.m2.1.2">∗</mo></mrow><annotation encoding="application/x-tex" id="S4.F1.6.m2.1d">\textsc{Joint BiT-Large}*</annotation></semantics></math> (using <span id="S4.F1.13.2" class="ltx_text ltx_markedasmath ltx_font_smallcaps">Base LM-24L</span>) at each position relative to <span id="S4.F1.14.3" class="ltx_text ltx_markedasmath ltx_font_smallcaps">Base LM-24L</span>
on the test set of CC-News. At each position the lower and upper bounds (Eq. <a href="#S3.E5" title="In 3.2 Evaluation ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> estimated using the method in Eq. <a href="#S3.E4" title="In Theorem 2. ‣ 3.2 Evaluation ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, see §<a href="#S3.SS2" title="3.2 Evaluation ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a> for more details) are estimated
using 20,000 samples. The shorter the horizon (moving to the right), the tighter the estimation is but also the more
limited the gains compared to base LM as un-normalized models are most useful on longer generations.</figcaption>
</figure>
<figure id="S4.T2" class="ltx_table">
<table id="S4.T2.2" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T2.2.3.1" class="ltx_tr">
<th id="S4.T2.2.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Model1 (baseline)</th>
<th id="S4.T2.2.3.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="S4.T2.2.3.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Model2 (compared model)</th>
<th id="S4.T2.2.3.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Rate</th>
<th id="S4.T2.2.3.1.5" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">p-value</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T2.1.1" class="ltx_tr">
<td id="S4.T2.1.1.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T2.1.1.2.1" class="ltx_text ltx_font_smallcaps">base LM</span></td>
<td id="S4.T2.1.1.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="9"><span id="S4.T2.1.1.1.1" class="ltx_text"><math id="S4.T2.1.1.1.1.m1.1" class="ltx_Math" alttext="&lt;" display="inline"><semantics id="S4.T2.1.1.1.1.m1.1a"><mo id="S4.T2.1.1.1.1.m1.1.1" xref="S4.T2.1.1.1.1.m1.1.1.cmml">&lt;</mo><annotation-xml encoding="MathML-Content" id="S4.T2.1.1.1.1.m1.1b"><lt id="S4.T2.1.1.1.1.m1.1.1.cmml" xref="S4.T2.1.1.1.1.m1.1.1"></lt></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.1.1.1.1.m1.1c">&lt;</annotation></semantics></math></span></td>
<td id="S4.T2.1.1.3" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T2.1.1.3.1" class="ltx_text ltx_font_smallcaps">joint uniT</span></td>
<td id="S4.T2.1.1.4" class="ltx_td ltx_align_center ltx_border_t">52.85%</td>
<td id="S4.T2.1.1.5" class="ltx_td ltx_align_right ltx_border_t">0.16</td>
</tr>
<tr id="S4.T2.2.4.1" class="ltx_tr">
<td id="S4.T2.2.4.1.1" class="ltx_td ltx_align_left"><span id="S4.T2.2.4.1.1.1" class="ltx_text ltx_font_smallcaps">base LM</span></td>
<td id="S4.T2.2.4.1.2" class="ltx_td ltx_align_left"><span id="S4.T2.2.4.1.2.1" class="ltx_text ltx_font_smallcaps">joint BiT-Base</span></td>
<td id="S4.T2.2.4.1.3" class="ltx_td ltx_align_center">56.25%</td>
<td id="S4.T2.2.4.1.4" class="ltx_td ltx_align_right">0.015</td>
</tr>
<tr id="S4.T2.2.5.2" class="ltx_tr">
<td id="S4.T2.2.5.2.1" class="ltx_td ltx_align_left"><span id="S4.T2.2.5.2.1.1" class="ltx_text ltx_font_smallcaps">base LM</span></td>
<td id="S4.T2.2.5.2.2" class="ltx_td ltx_align_left"><span id="S4.T2.2.5.2.2.1" class="ltx_text ltx_font_smallcaps">joint BiT-Large*</span></td>
<td id="S4.T2.2.5.2.3" class="ltx_td ltx_align_center">58.93%</td>
<td id="S4.T2.2.5.2.4" class="ltx_td ltx_align_right">0.00084</td>
</tr>
<tr id="S4.T2.2.6.3" class="ltx_tr">
<td id="S4.T2.2.6.3.1" class="ltx_td ltx_align_left"><span id="S4.T2.2.6.3.1.1" class="ltx_text ltx_font_smallcaps">base LM</span></td>
<td id="S4.T2.2.6.3.2" class="ltx_td ltx_align_left"><span id="S4.T2.2.6.3.2.1" class="ltx_text ltx_font_smallcaps">BALM</span></td>
<td id="S4.T2.2.6.3.3" class="ltx_td ltx_align_center">46.77%</td>
<td id="S4.T2.2.6.3.4" class="ltx_td ltx_align_right">0.88</td>
</tr>
<tr id="S4.T2.2.7.4" class="ltx_tr">
<td id="S4.T2.2.7.4.1" class="ltx_td ltx_align_left"><span id="S4.T2.2.7.4.1.1" class="ltx_text ltx_font_smallcaps">BALM</span></td>
<td id="S4.T2.2.7.4.2" class="ltx_td ltx_align_left"><span id="S4.T2.2.7.4.2.1" class="ltx_text ltx_font_smallcaps">joint UniT</span></td>
<td id="S4.T2.2.7.4.3" class="ltx_td ltx_align_center">50.00%</td>
<td id="S4.T2.2.7.4.4" class="ltx_td ltx_align_right">0.52</td>
</tr>
<tr id="S4.T2.2.8.5" class="ltx_tr">
<td id="S4.T2.2.8.5.1" class="ltx_td ltx_align_left"><span id="S4.T2.2.8.5.1.1" class="ltx_text ltx_font_smallcaps">BALM</span></td>
<td id="S4.T2.2.8.5.2" class="ltx_td ltx_align_left"><span id="S4.T2.2.8.5.2.1" class="ltx_text ltx_font_smallcaps">joint BiT-Base</span></td>
<td id="S4.T2.2.8.5.3" class="ltx_td ltx_align_center">57.89%</td>
<td id="S4.T2.2.8.5.4" class="ltx_td ltx_align_right">0.0027</td>
</tr>
<tr id="S4.T2.2.9.6" class="ltx_tr">
<td id="S4.T2.2.9.6.1" class="ltx_td ltx_align_left"><span id="S4.T2.2.9.6.1.1" class="ltx_text ltx_font_smallcaps">BALM</span></td>
<td id="S4.T2.2.9.6.2" class="ltx_td ltx_align_left"><span id="S4.T2.2.9.6.2.1" class="ltx_text ltx_font_smallcaps">joint BiT-Large*</span></td>
<td id="S4.T2.2.9.6.3" class="ltx_td ltx_align_center">59.89%</td>
<td id="S4.T2.2.9.6.4" class="ltx_td ltx_align_right">0.00020</td>
</tr>
<tr id="S4.T2.2.10.7" class="ltx_tr">
<td id="S4.T2.2.10.7.1" class="ltx_td ltx_align_left"><span id="S4.T2.2.10.7.1.1" class="ltx_text ltx_font_smallcaps">BALM-24L</span></td>
<td id="S4.T2.2.10.7.2" class="ltx_td ltx_align_left">
<span id="S4.T2.2.10.7.2.1" class="ltx_text ltx_font_smallcaps">joint BiT-Med</span> (24L)</td>
<td id="S4.T2.2.10.7.3" class="ltx_td ltx_align_center">56.23%</td>
<td id="S4.T2.2.10.7.4" class="ltx_td ltx_align_right">0.015</td>
</tr>
<tr id="S4.T2.2.11.8" class="ltx_tr">
<td id="S4.T2.2.11.8.1" class="ltx_td ltx_align_left">
<span id="S4.T2.2.11.8.1.1" class="ltx_text ltx_font_smallcaps">joint BiT-Large*</span> (24L)</td>
<td id="S4.T2.2.11.8.2" class="ltx_td ltx_align_left"><span id="S4.T2.2.11.8.2.1" class="ltx_text ltx_font_smallcaps">Human</span></td>
<td id="S4.T2.2.11.8.3" class="ltx_td ltx_align_center">55.21%</td>
<td id="S4.T2.2.11.8.4" class="ltx_td ltx_align_right">0.036</td>
</tr>
<tr id="S4.T2.2.2" class="ltx_tr">
<td id="S4.T2.2.2.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="S4.T2.2.2.2.1" class="ltx_text ltx_font_smallcaps">base LM</span></td>
<td id="S4.T2.2.2.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><math id="S4.T2.2.2.1.m1.1" class="ltx_Math" alttext="\leq" display="inline"><semantics id="S4.T2.2.2.1.m1.1a"><mo id="S4.T2.2.2.1.m1.1.1" xref="S4.T2.2.2.1.m1.1.1.cmml">≤</mo><annotation-xml encoding="MathML-Content" id="S4.T2.2.2.1.m1.1b"><leq id="S4.T2.2.2.1.m1.1.1.cmml" xref="S4.T2.2.2.1.m1.1.1"></leq></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.2.2.1.m1.1c">\leq</annotation></semantics></math></td>
<td id="S4.T2.2.2.3" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="S4.T2.2.2.3.1" class="ltx_text ltx_font_smallcaps">BALM</span></td>
<td id="S4.T2.2.2.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">54.85%</td>
<td id="S4.T2.2.2.5" class="ltx_td ltx_align_right ltx_border_bb ltx_border_t">0.050</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Human evaluation results on a subset of 333 sentences on the CC-News test set. The rate is computed as the percentage of sentences where the number of turkers preferring Model1 is strictly less than (denoted with <math id="S4.T2.5.m1.1" class="ltx_Math" alttext="&lt;" display="inline"><semantics id="S4.T2.5.m1.1b"><mo id="S4.T2.5.m1.1.1" xref="S4.T2.5.m1.1.1.cmml">&lt;</mo><annotation-xml encoding="MathML-Content" id="S4.T2.5.m1.1c"><lt id="S4.T2.5.m1.1.1.cmml" xref="S4.T2.5.m1.1.1"></lt></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.5.m1.1d">&lt;</annotation></semantics></math>) or not greater than (denoted with <math id="S4.T2.6.m2.1" class="ltx_Math" alttext="\leq" display="inline"><semantics id="S4.T2.6.m2.1b"><mo id="S4.T2.6.m2.1.1" xref="S4.T2.6.m2.1.1.cmml">≤</mo><annotation-xml encoding="MathML-Content" id="S4.T2.6.m2.1c"><leq id="S4.T2.6.m2.1.1.cmml" xref="S4.T2.6.m2.1.1"></leq></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.6.m2.1d">\leq</annotation></semantics></math>) those preferring Model2. Attention check is used to drop some votes, so there might exist ties. p-value is based on single-sided binomial test. </figcaption>
</figure>
<div id="S4.SS2.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px1.p1.3" class="ltx_p">Our main result is reported in Table <a href="#S4.T1" title="Table 1 ‣ Automatic Evaluation ‣ 4.2 Results ‣ 4 Experiments ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> where we compare models in terms of their perplexity.
We can see that on both datasets, residual EBMs with causal attention <span id="S4.SS2.SSS0.Px1.p1.3.1" class="ltx_text ltx_font_smallcaps">joint UniT</span> outperforms
the baseline <span id="S4.SS2.SSS0.Px1.p1.3.2" class="ltx_text ltx_font_smallcaps">RALM</span> with approximately the same number of parameters. The non-residual baseline <span id="S4.SS2.SSS0.Px1.p1.3.3" class="ltx_text ltx_markedasmath ltx_font_smallcaps">BALM</span> performs similarly to <span id="S4.SS2.SSS0.Px1.p1.3.4" class="ltx_text ltx_font_smallcaps">joint UniT</span>, which might be due to the limitation that <math id="S4.SS2.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="P_{LM}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.2.m2.1a"><msub id="S4.SS2.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.cmml"><mi id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.2" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.2.cmml">P</mi><mrow id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.cmml"><mi id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.2" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.1" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.3" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.3.cmml">M</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.2.m2.1b"><apply id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.2">𝑃</ci><apply id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3"><times id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.1.cmml" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.1"></times><ci id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.2.cmml" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.2">𝐿</ci><ci id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.3.cmml" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.2.m2.1c">P_{LM}</annotation></semantics></math> is not trained jointly with the residual model in both <span id="S4.SS2.SSS0.Px1.p1.3.5" class="ltx_text ltx_font_smallcaps">joint UniT</span> and <span id="S4.SS2.SSS0.Px1.p1.3.6" class="ltx_text ltx_markedasmath ltx_font_smallcaps">RALM</span>. However, by using our EBM approach, we can remove the causal attention mask and use bi-directional models, which achieves better performance than baselines and <span id="S4.SS2.SSS0.Px1.p1.3.7" class="ltx_text ltx_font_smallcaps">joint UniT</span>:
without external data, <span id="S4.SS2.SSS0.Px1.p1.3.8" class="ltx_text ltx_font_smallcaps">joint BiT-Base</span> reaches a higher performance than <span id="S4.SS2.SSS0.Px1.p1.3.9" class="ltx_text ltx_font_smallcaps">joint UniT</span>
with fewer parameters. By initializing from the state-of-the-art pretrained bi-directional transformers
RoBERTa-Base and RoBERTa-Large, <span id="S4.SS2.SSS0.Px1.p1.3.10" class="ltx_text ltx_font_smallcaps">joint BiT-Base*</span> and <span id="S4.SS2.SSS0.Px1.p1.3.11" class="ltx_text ltx_font_smallcaps">Joint BiT-Large*</span> reach even
better performance than <span id="S4.SS2.SSS0.Px1.p1.3.12" class="ltx_text ltx_font_smallcaps">joint BiT-Base</span>.</p>
</div>
<div id="S4.SS2.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px1.p2.1" class="ltx_p">In the lower part of the table, we show that if we make the big language model baseline <span id="S4.SS2.SSS0.Px1.p2.1.1" class="ltx_text ltx_font_smallcaps">BALM</span>
deeper (<span id="S4.SS2.SSS0.Px1.p2.1.2" class="ltx_text ltx_font_smallcaps">BALM-24L</span>) (24 layers instead of 12, for the same number of parameters) we attain lower perplexity.
However, training the joint model <span id="S4.SS2.SSS0.Px1.p2.1.3" class="ltx_text ltx_font_smallcaps">Joint BiT-Base</span> on the residual of a deeper language model <span id="S4.SS2.SSS0.Px1.p2.1.4" class="ltx_text ltx_font_smallcaps">BASE LM-24L</span>
yields even lower perplexity, despite having fewer parameters. By using the same number of parameters as <span id="S4.SS2.SSS0.Px1.p2.1.5" class="ltx_text ltx_font_smallcaps">BALM-24L</span>, <span id="S4.SS2.SSS0.Px1.p2.1.6" class="ltx_text ltx_font_smallcaps">Joint Bit-Med</span> further decreases perplexity. Finally, by initializing from RoBERTa-Large, <span id="S4.SS2.SSS0.Px1.p2.1.7" class="ltx_text ltx_font_smallcaps">joint BiT-Base*</span> obtains the best results.</p>
</div>
<div id="S4.SS2.SSS0.Px1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px1.p3.1" class="ltx_p">One caveat of our evaluation protocol is that the perplexity bounds are only estimates,
which might not reflect the true value, particularly since the number of possible sequences grows exponentially
with the number of words that are generated.
We therefore break down perplexity per position in the generated sequences as in Eq. <a href="#S3.E5" title="In 3.2 Evaluation ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>,
and compare the estimated PPLs to the true enumerated
PPLs at the last position, as shown in Figure <a href="#S4.F1" title="Figure 1 ‣ Automatic Evaluation ‣ 4.2 Results ‣ 4 Experiments ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. We find that at the final generation step, the estimated
bounds agree remarkably well with the exact values, proving that our method at least gets a reasonable PPL estimate at the last generation step, and that <span id="S4.SS2.SSS0.Px1.p3.1.1" class="ltx_text ltx_markedasmath ltx_font_smallcaps">Joint BiT-Med</span> outperforms baselines at the last generation step for sure.</p>
</div>
</section>
<section id="S4.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Human Evaluation</h4>

<div id="S4.SS2.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px2.p1.1" class="ltx_p">Better perplexity results do not necessarily imply better generations.
Besides, since generation from the residual EBM
requires approximations as in Algorithm <a href="#alg1" title="In 3.2 Evaluation ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, the limited sample size might induce approximation errors
compared to truly sampling from the joint distribution. Therefore, we conducted human evaluations to compare generations
from the residual EBM model to generations from the baseline language models.</p>
</div>
<div id="S4.SS2.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px2.p2.4" class="ltx_p">For each prefix, we present one completion from each model, and ask humans to select the one that is a better continuation.
More details about human evaluation can be found in the Appendix <a href="#A1.SS4" title="A.4 Human Evaluation ‣ Appendix A Appendix ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.4</span></a>.
The preference rates reported in Table <a href="#S4.T2" title="Table 2 ‣ Automatic Evaluation ‣ 4.2 Results ‣ 4 Experiments ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> confirm that indeed the generation quality of
<span id="S4.SS2.SSS0.Px2.p2.4.1" class="ltx_text ltx_markedasmath ltx_font_smallcaps">Joint Bit-Base</span> and <math id="S4.SS2.SSS0.Px2.p2.2.m2.1" class="ltx_math_unparsed" alttext="\textsc{Joint Bit-Large}*" display="inline"><semantics id="S4.SS2.SSS0.Px2.p2.2.m2.1a"><mrow id="S4.SS2.SSS0.Px2.p2.2.m2.1b"><mtext class="ltx_font_smallcaps" id="S4.SS2.SSS0.Px2.p2.2.m2.1.1">Joint Bit-Large</mtext><mo lspace="0.222em" id="S4.SS2.SSS0.Px2.p2.2.m2.1.2">∗</mo></mrow><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p2.2.m2.1c">\textsc{Joint Bit-Large}*</annotation></semantics></math> is better than both language model baselines.
Depending on the model variant, our joint model (with bidirectional EBM) is preferred between 56% and almost 60% of the times;
interestingly, the preference rate does not change much as we compare against base LM as opposed to BALM.
In fact, humans do not seem to have a strong preference for BALM over base LM, despite the former scores two perplexity points lower.
Similarly, <span id="S4.SS2.SSS0.Px2.p2.4.2" class="ltx_text ltx_markedasmath ltx_font_smallcaps">Joint Unit</span> is not strongly preferred over <span id="S4.SS2.SSS0.Px2.p2.4.3" class="ltx_text ltx_markedasmath ltx_font_smallcaps">Base LM</span> despite its lower perplexity score.
We surmise that unidirectional scoring functions and auto-regressive models exhibit generation artifacts which are easily
detected by humans, and these may overshadow the improvements brought by perplexity gains.</p>
</div>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Analyses</h3>

<figure id="S4.F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F2.1" class="ltx_figure ltx_figure_panel"><img src="/html/2004.11714/assets/images/vary_sample_size.png" id="S4.F2.1.g1" class="ltx_graphics ltx_img_landscape" width="538" height="404" alt="Refer to caption">
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F2.2" class="ltx_figure ltx_figure_panel"><img src="/html/2004.11714/assets/images/ngram_ccnews_roberta_base_newsonly.png" id="S4.F2.2.g1" class="ltx_graphics ltx_img_landscape" width="538" height="404" alt="Refer to caption">
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Left: PPL estimation for joint <span id="S4.F2.7.1" class="ltx_text ltx_font_smallcaps">BiT-Base</span> on CC-News validation set as we vary
the number of samples. Right: Percentage of Unique n-grams found in real data,
samples from the joint model <span id="S4.F2.8.2" class="ltx_text ltx_markedasmath ltx_font_smallcaps">BiT-Base</span> and samples from the base language model.
The joint sampling is done with 10,000 samples.</figcaption>
</figure>
<div id="S4.SS3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.p1.1" class="ltx_p">In this section, we analyze some of the results we obtained. First, we check whether we used a sufficient number of
samples in our perplexity estimates. Second, we assess whether the joint model produces fewer repetitions compared
to the base language model, and finally we check how well some statistics of the model and data distributions match.</p>
</div>
<section id="S4.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Number of samples.</h4>

<div id="S4.SS3.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px1.p1.1" class="ltx_p">In Figure <a href="#S4.F2" title="Figure 2 ‣ 4.3 Analyses ‣ 4 Experiments ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, we vary the number of samples we take in order to
estimate PPL upper and lower bounds. Beyond 20,000 samples the upper estimate becomes very stable,
although we have to emphasize that these estimates might be biased even though the gap between lower and upper bound
closes as we take more samples.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Repetitions.</h4>

<div id="S4.SS3.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px2.p1.1" class="ltx_p">A typical artifact of auto-regressive language models is their tendency to repeat phrases.
It is then interesting to check whether the joint model is able to alleviate this artifact. Fig. <a href="#S4.F2" title="Figure 2 ‣ 4.3 Analyses ‣ 4 Experiments ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>
shows that indeed the joint model has a slightly higher percentage of unique n-grams compared to the baseline language model with <math id="S4.SS3.SSS0.Px2.p1.1.m1.3" class="ltx_Math" alttext="n=2,3,4" display="inline"><semantics id="S4.SS3.SSS0.Px2.p1.1.m1.3a"><mrow id="S4.SS3.SSS0.Px2.p1.1.m1.3.4" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.4.cmml"><mi id="S4.SS3.SSS0.Px2.p1.1.m1.3.4.2" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.4.2.cmml">n</mi><mo id="S4.SS3.SSS0.Px2.p1.1.m1.3.4.1" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.4.1.cmml">=</mo><mrow id="S4.SS3.SSS0.Px2.p1.1.m1.3.4.3.2" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.4.3.1.cmml"><mn id="S4.SS3.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.cmml">2</mn><mo id="S4.SS3.SSS0.Px2.p1.1.m1.3.4.3.2.1" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.4.3.1.cmml">,</mo><mn id="S4.SS3.SSS0.Px2.p1.1.m1.2.2" xref="S4.SS3.SSS0.Px2.p1.1.m1.2.2.cmml">3</mn><mo id="S4.SS3.SSS0.Px2.p1.1.m1.3.4.3.2.2" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.4.3.1.cmml">,</mo><mn id="S4.SS3.SSS0.Px2.p1.1.m1.3.3" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.3.cmml">4</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p1.1.m1.3b"><apply id="S4.SS3.SSS0.Px2.p1.1.m1.3.4.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.4"><eq id="S4.SS3.SSS0.Px2.p1.1.m1.3.4.1.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.4.1"></eq><ci id="S4.SS3.SSS0.Px2.p1.1.m1.3.4.2.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.4.2">𝑛</ci><list id="S4.SS3.SSS0.Px2.p1.1.m1.3.4.3.1.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.4.3.2"><cn type="integer" id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1">2</cn><cn type="integer" id="S4.SS3.SSS0.Px2.p1.1.m1.2.2.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.2.2">3</cn><cn type="integer" id="S4.SS3.SSS0.Px2.p1.1.m1.3.3.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.3.3">4</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p1.1.m1.3c">n=2,3,4</annotation></semantics></math>,
although still not as high as the original human generated text.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">A necessary condition for the model to match the data distribution.</h4>

<div id="S4.SS3.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px3.p1.3" class="ltx_p">If the joint model <math id="S4.SS3.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="p_{\theta}" display="inline"><semantics id="S4.SS3.SSS0.Px3.p1.1.m1.1a"><msub id="S4.SS3.SSS0.Px3.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1.cmml"><mi id="S4.SS3.SSS0.Px3.p1.1.m1.1.1.2" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1.2.cmml">p</mi><mi id="S4.SS3.SSS0.Px3.p1.1.m1.1.1.3" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px3.p1.1.m1.1b"><apply id="S4.SS3.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS3.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1">subscript</csymbol><ci id="S4.SS3.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1.2">𝑝</ci><ci id="S4.SS3.SSS0.Px3.p1.1.m1.1.1.3.cmml" xref="S4.SS3.SSS0.Px3.p1.1.m1.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px3.p1.1.m1.1c">p_{\theta}</annotation></semantics></math> matches the data distribution <math id="S4.SS3.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="p_{d}" display="inline"><semantics id="S4.SS3.SSS0.Px3.p1.2.m2.1a"><msub id="S4.SS3.SSS0.Px3.p1.2.m2.1.1" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1.cmml"><mi id="S4.SS3.SSS0.Px3.p1.2.m2.1.1.2" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1.2.cmml">p</mi><mi id="S4.SS3.SSS0.Px3.p1.2.m2.1.1.3" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1.3.cmml">d</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px3.p1.2.m2.1b"><apply id="S4.SS3.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS3.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1">subscript</csymbol><ci id="S4.SS3.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1.2">𝑝</ci><ci id="S4.SS3.SSS0.Px3.p1.2.m2.1.1.3.cmml" xref="S4.SS3.SSS0.Px3.p1.2.m2.1.1.3">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px3.p1.2.m2.1c">p_{d}</annotation></semantics></math>, then statistics computed on a large population of samples
from the two distributions should also match. In particular, Fig. <a href="#S4.F3" title="Figure 3 ‣ A necessary condition for the model to match the data distribution. ‣ 4.3 Analyses ‣ 4 Experiments ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> show the density plots of log-likelihood scores
of the baseline language model (left) and joint model (right) when fed with their own samples versus samples from the test set.
We observe that the histogram of samples from the joint model matches the real data distribution more closely: The difference of means in the <span id="S4.SS3.SSS0.Px3.p1.3.1" class="ltx_text ltx_markedasmath ltx_font_smallcaps">LM Base</span> case is 21.64 whereas the difference is 6.20 in the joint approach.</p>
</div>
<figure id="S4.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F3.1" class="ltx_figure ltx_figure_panel"><img src="/html/2004.11714/assets/images/energy_orig_ccnews.png" id="S4.F3.1.g1" class="ltx_graphics ltx_img_landscape" width="538" height="404" alt="Refer to caption">
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F3.2" class="ltx_figure ltx_figure_panel"><img src="/html/2004.11714/assets/images/energy_orig_ccnews_newsonly_joint.png" id="S4.F3.2.g1" class="ltx_graphics ltx_img_landscape" width="538" height="404" alt="Refer to caption">
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Density plot of log-probability scores using the base
language model (left) or the joint model (right). The red curve corresponds to real samples, the black curve to samples from <span id="S4.F3.5.1" class="ltx_text ltx_font_smallcaps">Base LM</span> and the green curve to samples from <span id="S4.F3.6.2" class="ltx_text ltx_font_smallcaps">BiT-Base</span>. The joint model provides a much better fit than the base language model.</figcaption>
</figure>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Limitations</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p">In the previous sections we highlighted the strengths of residual EBMs, namely their simplicity, efficiency both at training
and test time, and their improved perplexity scores against strong auto-regressive language model baselines. In this section, we comment on their limitations to caution the reader about when these methods are more likely to succeed
and to inform other researchers about what future avenues of research may naturally derive from this work.</p>
</div>
<div id="S5.p2" class="ltx_para ltx_noindent">
<p id="S5.p2.1" class="ltx_p">In order to make training efficient and side step costly negative mining using the energy function itself, the current approach
uses negatives generated from a pretrained auto-regressive language model. Therefore, our model works as long as the base
language model from which we draw samples is strong enough, and as long as the ground truth and other plausible sequences are
<span id="S5.p2.1.1" class="ltx_text ltx_font_italic">reachable</span> by the baseline language model.</p>
</div>
<div id="S5.p3" class="ltx_para ltx_noindent">
<p id="S5.p3.1" class="ltx_p">If the base language model has poor quality, then generation from our
joint model is going to be poor as well, as the joint model merely resamples generations from the original language model.
Moreover, training is going to be trivial if the base language model is poor, because the residual energy function
merely needs to detect trivial generation artifacts from the base language model. In fact, observe that the role of
positive and negative samples is symmetric in the loss of Eq. <a href="#S3.E3" title="In 3.1 Training ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. This means that the energy function can choose
to minimize the loss by either modeling the true data or the negative samples; since the latter have much simpler structure,
it is going to model the negative samples. Therefore, importance sampling amounts to mostly down-weighing the worst samples
from the base language model.
The consequence of this is that search with a poor base language model is going to be catastrophically inefficient, as
we would need to sample an impractically large number of negatives in order to find samples that are reasonably close
to the true data manifold.</p>
</div>
<div id="S5.p4" class="ltx_para ltx_noindent">
<p id="S5.p4.1" class="ltx_p">To summarize, this work makes a rather strong implicit assumption on the quality of the base language model, and it
is expected to work well only when this is rather strong. In our application, this assumption is met quite well in practice
as large auto-regressive language models trained on large datasets have improved significantly in recent years <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib27" title="" class="ltx_ref">2019</a>)</cite>.
In general however, residual learning always carries liability to its
base model.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusions and Future work</h2>

<div id="S6.p1" class="ltx_para ltx_noindent">
<p id="S6.p1.1" class="ltx_p">We investigated an EBM trained on the residual of a pretrained autoregressive language model <cite class="ltx_cite ltx_citemacro_citep">(Wang &amp; Ou, <a href="#bib.bib37" title="" class="ltx_ref">2018b</a>; Parshakova et al., <a href="#bib.bib25" title="" class="ltx_ref">2019</a>)</cite>.
The resulting joint model scores sequences holistically, thanks to the energy function. Training is very efficient
and consists of a binary classification task between positives from the training set and pregenerated negatives from
the fixed language model. Generation is also very efficient as it amounts to resampling from the large set of negatives
produced by the base language model. Our estimates show that the resulting model has lower perplexity than the base
language model.
Finally, this approach may be interpreted
as a natural way to finetune a large bidirectional transformer like BERT for text generation applications.
</p>
</div>
<div id="S6.p2" class="ltx_para ltx_noindent">
<p id="S6.p2.1" class="ltx_p">In the future, we plan to investigate other ways to generate negatives that may strike a better trade-off
between the amount of compute each negative requires and their closeness to the joint model distribution.
It would also be interesting to explore other loss functions and the generation of longer pieces of text by using this
model auto-regressively at the chunk level, as opposed to the token level.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Azadi et al. (2018)</span>
<span class="ltx_bibblock">
Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus
Odena.

</span>
<span class="ltx_bibblock">Discriminator rejection sampling.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1810.06758</em>, 2018.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bakhtin et al. (2019)</span>
<span class="ltx_bibblock">
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and
Arthur Szlam.

</span>
<span class="ltx_bibblock">Real or fake? learning to discriminate machine from human generated
text.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1906.03351</em>, 2019.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bottou (1991)</span>
<span class="ltx_bibblock">
Léon Bottou.

</span>
<span class="ltx_bibblock">Une approche théorique de l’apprentissage connexionniste:
Applications à la reconnaissance de la parole, 1991.

</span>
<span class="ltx_bibblock">Ph.D. thesis, Université de Paris XI, Orsay, France.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bowman et al. (2016)</span>
<span class="ltx_bibblock">
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz,
and Samy Bengio.

</span>
<span class="ltx_bibblock">Generating sentences from a continuous space.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">SIGNLL Conference on Computational Natural Language
Learning</em>, 2016.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al. (2018)</span>
<span class="ltx_bibblock">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.

</span>
<span class="ltx_bibblock">BERT: pre-training of deep bidirectional transformers for language
understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1810.04805, 2018.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1810.04805" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1810.04805</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Du &amp; Mordatch (2019)</span>
<span class="ltx_bibblock">
Yilun Du and Igor Mordatch.

</span>
<span class="ltx_bibblock">Implicit generation and generalization in energy-based models.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1903.08689, 2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1903.08689" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1903.08689</a>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Edunov et al. (2018)</span>
<span class="ltx_bibblock">
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio
Ranzato.

</span>
<span class="ltx_bibblock">Classical structured prediction losses for sequence to sequence
learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational
Linguistics</em>, 2018.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2018)</span>
<span class="ltx_bibblock">
Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu.

</span>
<span class="ltx_bibblock">Learning generative convnets via multi-grid modeling and sampling.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition</em>, pp.  9155–9164, 2018.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Goodfellow et al. (2014)</span>
<span class="ltx_bibblock">
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David
Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.

</span>
<span class="ltx_bibblock">Generative adversarial nets.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">NIPS</em>, 2014.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Grover et al. (2019)</span>
<span class="ltx_bibblock">
Aditya Grover, Jiaming Song, Alekh Agarwal, Kenneth Tran, Ashish Kapoor, Eric
Horvitz, and Stefano Ermon.

</span>
<span class="ltx_bibblock">Bias correction of learned generative models using likelihood-free
importance weighting.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1906.09531</em>, 2019.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gutmann &amp; Hyvärinen (2010)</span>
<span class="ltx_bibblock">
Michael Gutmann and Aapo Hyvärinen.

</span>
<span class="ltx_bibblock">Noise-contrastive estimation: A new estimation principle for
unnormalized statistical models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Thirteenth International Conference on
Artificial Intelligence and Statistics</em>, pp.  297–304, 2010.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton (2002)</span>
<span class="ltx_bibblock">
Geoffrey E. Hinton.

</span>
<span class="ltx_bibblock">Training products of experts by minimizing contrastive divergence.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Neural Computation</em>, 14:1771–1800, 2002.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton (2012)</span>
<span class="ltx_bibblock">
Geoffrey E Hinton.

</span>
<span class="ltx_bibblock">A practical guide to training restricted boltzmann machines.

</span>
<span class="ltx_bibblock">In <em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Neural networks: Tricks of the trade</em>, pp.  599–619.
Springer, 2012.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Holtzman et al. (2019)</span>
<span class="ltx_bibblock">
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi.

</span>
<span class="ltx_bibblock">The curious case of neural text degeneration.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1904.09751</em>, 2019.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hopfield (1982)</span>
<span class="ltx_bibblock">
John Hopfield.

</span>
<span class="ltx_bibblock">Neural networks and physical systems with emergent collective
computational abilities.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">National Academy of Sciences of the USA</em>, volume 79, pp. 2554––2558, 1982.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Horvitz &amp; Thompson (1952)</span>
<span class="ltx_bibblock">
Daniel G. Horvitz and Donovan J. Thompson.

</span>
<span class="ltx_bibblock">A generalization of sampling without replacement from a finite
universe.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Journal of the American Statistical Association</em>, 1952.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kiros et al. (2015)</span>
<span class="ltx_bibblock">
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba,
and Sanja Urtasun, Raquel an d Fidler.

</span>
<span class="ltx_bibblock">Skip-thought vectors.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1506.06726</em>, 2015.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">LeCun et al. (2006)</span>
<span class="ltx_bibblock">
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc’Aurelio Ranzato, and Fu-Jie Huang.

</span>
<span class="ltx_bibblock">A tutorial on energy-based learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Predicting Structured Outputs</em>, 2006.

</span>
<span class="ltx_bibblock">MIT Press.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2019)</span>
<span class="ltx_bibblock">
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.

</span>
<span class="ltx_bibblock">Roberta: A robustly optimized bert pretraining approach.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1907.11692</em>, 2019.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ma &amp; Collins (2018)</span>
<span class="ltx_bibblock">
Zhuang Ma and Michael Collins.

</span>
<span class="ltx_bibblock">Noise contrastive estimation and negative sampling for conditional
models: Consistency and statistical efficiency.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Empirical Methods for Natural Language Processing</em>, 2018.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nagel (2016)</span>
<span class="ltx_bibblock">
Sebastian Nagel.

</span>
<span class="ltx_bibblock">Cc-news.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://web.archive.org/save/http://commoncrawl.org/2016/10/news-dataset-available/" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://web.archive.org/save/http://commoncrawl.org/2016/10/news-dataset-available/</a>,
2016.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nijkamp et al. (2019)</span>
<span class="ltx_bibblock">
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu.

</span>
<span class="ltx_bibblock">Learning non-convergent non-persistent short-run mcmc toward
energy-based model.

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, pp. 5233–5243, 2019.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nowozin (2018)</span>
<span class="ltx_bibblock">
Sebastian Nowozin.

</span>
<span class="ltx_bibblock">Debiasing evidence approximations: On importance-weighted
autoencoders and jackknife variational inference.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2018.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Owen (2013)</span>
<span class="ltx_bibblock">
Art B. Owen.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Monte Carlo theory, methods and examples</em>.

</span>
<span class="ltx_bibblock">2013.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://statweb.stanford.edu/~owen/mc/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://statweb.stanford.edu/~owen/mc/</a>.

</span>
<span class="ltx_bibblock">chapter 9.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Parshakova et al. (2019)</span>
<span class="ltx_bibblock">
Tetiana Parshakova, Jean-Marc Andreoli, and Marc Dymetman.

</span>
<span class="ltx_bibblock">Global autoregressive models for data-efficient sequence learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Conference on Computational Natural Language Learning</em>,
2019.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pascanu et al. (2013)</span>
<span class="ltx_bibblock">
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.

</span>
<span class="ltx_bibblock">On the difficulty of training recurrent neural networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pp. 1310–1318, 2013.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2019)</span>
<span class="ltx_bibblock">
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever.

</span>
<span class="ltx_bibblock">Language models are unsupervised multitask learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">OpenAI Blog</em>, 1(8), 2019.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ranzato et al. (2007)</span>
<span class="ltx_bibblock">
Marc’Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun.

</span>
<span class="ltx_bibblock">A unified energy-based framework for unsupervised learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">11-th International Workshop on Artificial Intelligence and
Statistics (AISTATS)</em>, 2007.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ranzato et al. (2016)</span>
<span class="ltx_bibblock">
Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba.

</span>
<span class="ltx_bibblock">Sequence level training with recurrent neural networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representation</em>, 2016.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rosenfeld et al. (2001)</span>
<span class="ltx_bibblock">
Ronald Rosenfeld, Stanley F Chen, and Xiaojin Zhu.

</span>
<span class="ltx_bibblock">Whole-sentence exponential language models: a vehicle for
linguistic-statistical integration.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Computer Speech &amp; Language</em>, 15(1):55–73,
2001.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sennrich et al. (2015)</span>
<span class="ltx_bibblock">
Rico Sennrich, Barry Haddow, and Alexandra Birch.

</span>
<span class="ltx_bibblock">Neural machine translation of rare words with subword units.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1508.07909</em>, 2015.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shen et al. (2004)</span>
<span class="ltx_bibblock">
Libin Shen, Anoop Sarkar, and Franz Josef Och.

</span>
<span class="ltx_bibblock">Discriminative reranking for machine translation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Human Language Technology Conference of the North American
Chapter of the Association for Computational Linguistics: HLT-NAACL 2004</em>,
pp.  177–184, 2004.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Teh et al. (2003)</span>
<span class="ltx_bibblock">
Y. W. Teh, M. Welling, S. Osindero, and Hinton G. E.

</span>
<span class="ltx_bibblock">Energy-based models for sparse overcomplete representations.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Journal of Machine Learning Research</em>, 4:1235–1260,
2003.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. (2017)</span>
<span class="ltx_bibblock">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, pp. 5998–6008, 2017.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang &amp; Ou (2017)</span>
<span class="ltx_bibblock">
Bin Wang and Zhijian Ou.

</span>
<span class="ltx_bibblock">Language modeling with neural trans-dimensional random fields.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">2017 IEEE Automatic Speech Recognition and Understanding
Workshop (ASRU)</em>, pp.  294–300. IEEE, 2017.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang &amp; Ou (2018a)</span>
<span class="ltx_bibblock">
Bin Wang and Zhijian Ou.

</span>
<span class="ltx_bibblock">Improved training of neural trans-dimensional random field language
models with dynamic noise-contrastive estimation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">2018 IEEE Spoken Language Technology Workshop (SLT)</em>, pp. 70–76. IEEE, 2018a.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang &amp; Ou (2018b)</span>
<span class="ltx_bibblock">
Bin Wang and Zhijian Ou.

</span>
<span class="ltx_bibblock">Learning neural trans-dimensional random field language models with
noise-contrastive estimation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">2018 IEEE International Conference on Acoustics, Speech and
Signal Processing (ICASSP)</em>, pp.  6134–6138. IEEE, 2018b.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2015)</span>
<span class="ltx_bibblock">
Bin Wang, Zhijian Ou, and Zhiqiang Tan.

</span>
<span class="ltx_bibblock">Trans-dimensional random fields for language modeling.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)</em>, pp.  785–794, 2015.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2017)</span>
<span class="ltx_bibblock">
Bin Wang, Zhijian Ou, and Zhiqiang Tan.

</span>
<span class="ltx_bibblock">Learning trans-dimensional random fields with applications to
language modeling.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">IEEE transactions on pattern analysis and machine
intelligence</em>, 40(4):876–890, 2017.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Welling et al. (2005)</span>
<span class="ltx_bibblock">
Max Welling, Michal Rosen-Zvi, and Geoffrey E. Hinton.

</span>
<span class="ltx_bibblock">Exponential family harmoniums with an application to information
retrieval.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">Neural Information Processing Systems</em>, 2005.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xie et al. (2016)</span>
<span class="ltx_bibblock">
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu.

</span>
<span class="ltx_bibblock">A theory of generative convnet.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp. 2635–2644, 2016.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xie et al. (2017)</span>
<span class="ltx_bibblock">
Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu.

</span>
<span class="ltx_bibblock">Synthesizing dynamic patterns by spatial-temporal generative convnet.

</span>
<span class="ltx_bibblock">In <em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">Proceedings of the ieee conference on computer vision and
pattern recognition</em>, pp.  7093–7101, 2017.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xie et al. (2018)</span>
<span class="ltx_bibblock">
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying
Nian Wu.

</span>
<span class="ltx_bibblock">Learning descriptor networks for 3d shape synthesis and analysis.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE conference on computer vision and
pattern recognition</em>, pp.  8629–8638, 2018.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xie et al. (2019)</span>
<span class="ltx_bibblock">
Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu.

</span>
<span class="ltx_bibblock">Learning energy-based spatial-temporal generative convnets for
dynamic patterns.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">IEEE transactions on pattern analysis and machine
intelligence</em>, 2019.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhao et al. (2018)</span>
<span class="ltx_bibblock">
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, and Yann LeCun.

</span>
<span class="ltx_bibblock">Adversarially regularized autoencoders.

</span>
<span class="ltx_bibblock">In <em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">International Conference in Machine Learning</em>, 2018.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhu et al. (2015)</span>
<span class="ltx_bibblock">
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun,
Antonio Torralba, and Sanja Fidler.

</span>
<span class="ltx_bibblock">Aligning books and movies: Towards story-like visual explanations by
watching movies and reading books.

</span>
<span class="ltx_bibblock">In <em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">The IEEE International Conference on Computer Vision
(ICCV)</em>, December 2015.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Appendix</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Top-k auto-regressive sampling</h3>

<div id="A1.SS1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS1.p1.1" class="ltx_p">In this subsection, we factorize the joint model <span id="A1.SS1.p1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base</span> auto-regressively, and compare its differences with <span id="A1.SS1.p1.1.2" class="ltx_text ltx_font_smallcaps">Base LM</span>. Since even estimating the per step probabilities according to Eq. <a href="#S3.E5" title="In 3.2 Evaluation ‣ 3 Residual Energy-Based Models ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> is too computationally expensive, we further approximate it by only considering the top 128 words predicted by <span id="A1.SS1.p1.1.3" class="ltx_text ltx_font_smallcaps">Base LM</span>, where we sample 10,000 completions for each of them to estimate <math id="A1.SS1.p1.1.m1.1" class="ltx_Math" alttext="P(x_{t}|x_{&lt;t})" display="inline"><semantics id="A1.SS1.p1.1.m1.1a"><mrow id="A1.SS1.p1.1.m1.1.1" xref="A1.SS1.p1.1.m1.1.1.cmml"><mi id="A1.SS1.p1.1.m1.1.1.3" xref="A1.SS1.p1.1.m1.1.1.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="A1.SS1.p1.1.m1.1.1.2" xref="A1.SS1.p1.1.m1.1.1.2.cmml">​</mo><mrow id="A1.SS1.p1.1.m1.1.1.1.1" xref="A1.SS1.p1.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS1.p1.1.m1.1.1.1.1.2" xref="A1.SS1.p1.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS1.p1.1.m1.1.1.1.1.1" xref="A1.SS1.p1.1.m1.1.1.1.1.1.cmml"><msub id="A1.SS1.p1.1.m1.1.1.1.1.1.2" xref="A1.SS1.p1.1.m1.1.1.1.1.1.2.cmml"><mi id="A1.SS1.p1.1.m1.1.1.1.1.1.2.2" xref="A1.SS1.p1.1.m1.1.1.1.1.1.2.2.cmml">x</mi><mi id="A1.SS1.p1.1.m1.1.1.1.1.1.2.3" xref="A1.SS1.p1.1.m1.1.1.1.1.1.2.3.cmml">t</mi></msub><mo fence="false" id="A1.SS1.p1.1.m1.1.1.1.1.1.1" xref="A1.SS1.p1.1.m1.1.1.1.1.1.1.cmml">|</mo><msub id="A1.SS1.p1.1.m1.1.1.1.1.1.3" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.cmml"><mi id="A1.SS1.p1.1.m1.1.1.1.1.1.3.2" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.2.cmml">x</mi><mrow id="A1.SS1.p1.1.m1.1.1.1.1.1.3.3" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.cmml"><mi id="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.2" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.2.cmml"></mi><mo id="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.1" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.3" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.3.cmml">t</mi></mrow></msub></mrow><mo stretchy="false" id="A1.SS1.p1.1.m1.1.1.1.1.3" xref="A1.SS1.p1.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.1.m1.1b"><apply id="A1.SS1.p1.1.m1.1.1.cmml" xref="A1.SS1.p1.1.m1.1.1"><times id="A1.SS1.p1.1.m1.1.1.2.cmml" xref="A1.SS1.p1.1.m1.1.1.2"></times><ci id="A1.SS1.p1.1.m1.1.1.3.cmml" xref="A1.SS1.p1.1.m1.1.1.3">𝑃</ci><apply id="A1.SS1.p1.1.m1.1.1.1.1.1.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1"><csymbol cd="latexml" id="A1.SS1.p1.1.m1.1.1.1.1.1.1.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.1">conditional</csymbol><apply id="A1.SS1.p1.1.m1.1.1.1.1.1.2.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.SS1.p1.1.m1.1.1.1.1.1.2.1.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.2">subscript</csymbol><ci id="A1.SS1.p1.1.m1.1.1.1.1.1.2.2.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.2.2">𝑥</ci><ci id="A1.SS1.p1.1.m1.1.1.1.1.1.2.3.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="A1.SS1.p1.1.m1.1.1.1.1.1.3.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.SS1.p1.1.m1.1.1.1.1.1.3.1.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.SS1.p1.1.m1.1.1.1.1.1.3.2.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.2">𝑥</ci><apply id="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.3"><lt id="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.1.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.2.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.2">absent</csymbol><ci id="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.3.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.1.m1.1c">P(x_{t}|x_{&lt;t})</annotation></semantics></math>. Then we take the top 10 entries and re-normalize, and compare it to the top 10 probabilities of <span id="A1.SS1.p1.1.4" class="ltx_text ltx_font_smallcaps">Base LM</span>.</p>
</div>
<div id="A1.SS1.p2" class="ltx_para ltx_noindent">
<p id="A1.SS1.p2.4" class="ltx_p">Our initial explorations suggested that the joint model tends to generate fewer repetitions. Therefore we picked a few LM samples where there are repetitions at <math id="A1.SS1.p2.1.m1.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="A1.SS1.p2.1.m1.1a"><msub id="A1.SS1.p2.1.m1.1.1" xref="A1.SS1.p2.1.m1.1.1.cmml"><mi id="A1.SS1.p2.1.m1.1.1.2" xref="A1.SS1.p2.1.m1.1.1.2.cmml">x</mi><mi id="A1.SS1.p2.1.m1.1.1.3" xref="A1.SS1.p2.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.1.m1.1b"><apply id="A1.SS1.p2.1.m1.1.1.cmml" xref="A1.SS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p2.1.m1.1.1.1.cmml" xref="A1.SS1.p2.1.m1.1.1">subscript</csymbol><ci id="A1.SS1.p2.1.m1.1.1.2.cmml" xref="A1.SS1.p2.1.m1.1.1.2">𝑥</ci><ci id="A1.SS1.p2.1.m1.1.1.3.cmml" xref="A1.SS1.p2.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.1.m1.1c">x_{t}</annotation></semantics></math>, and use the same context <math id="A1.SS1.p2.2.m2.1" class="ltx_Math" alttext="x_{&lt;t}" display="inline"><semantics id="A1.SS1.p2.2.m2.1a"><msub id="A1.SS1.p2.2.m2.1.1" xref="A1.SS1.p2.2.m2.1.1.cmml"><mi id="A1.SS1.p2.2.m2.1.1.2" xref="A1.SS1.p2.2.m2.1.1.2.cmml">x</mi><mrow id="A1.SS1.p2.2.m2.1.1.3" xref="A1.SS1.p2.2.m2.1.1.3.cmml"><mi id="A1.SS1.p2.2.m2.1.1.3.2" xref="A1.SS1.p2.2.m2.1.1.3.2.cmml"></mi><mo id="A1.SS1.p2.2.m2.1.1.3.1" xref="A1.SS1.p2.2.m2.1.1.3.1.cmml">&lt;</mo><mi id="A1.SS1.p2.2.m2.1.1.3.3" xref="A1.SS1.p2.2.m2.1.1.3.3.cmml">t</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.2.m2.1b"><apply id="A1.SS1.p2.2.m2.1.1.cmml" xref="A1.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A1.SS1.p2.2.m2.1.1.1.cmml" xref="A1.SS1.p2.2.m2.1.1">subscript</csymbol><ci id="A1.SS1.p2.2.m2.1.1.2.cmml" xref="A1.SS1.p2.2.m2.1.1.2">𝑥</ci><apply id="A1.SS1.p2.2.m2.1.1.3.cmml" xref="A1.SS1.p2.2.m2.1.1.3"><lt id="A1.SS1.p2.2.m2.1.1.3.1.cmml" xref="A1.SS1.p2.2.m2.1.1.3.1"></lt><csymbol cd="latexml" id="A1.SS1.p2.2.m2.1.1.3.2.cmml" xref="A1.SS1.p2.2.m2.1.1.3.2">absent</csymbol><ci id="A1.SS1.p2.2.m2.1.1.3.3.cmml" xref="A1.SS1.p2.2.m2.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.2.m2.1c">x_{&lt;t}</annotation></semantics></math> to estimate <math id="A1.SS1.p2.3.m3.1" class="ltx_Math" alttext="P(x_{t}|x_{&lt;t})" display="inline"><semantics id="A1.SS1.p2.3.m3.1a"><mrow id="A1.SS1.p2.3.m3.1.1" xref="A1.SS1.p2.3.m3.1.1.cmml"><mi id="A1.SS1.p2.3.m3.1.1.3" xref="A1.SS1.p2.3.m3.1.1.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="A1.SS1.p2.3.m3.1.1.2" xref="A1.SS1.p2.3.m3.1.1.2.cmml">​</mo><mrow id="A1.SS1.p2.3.m3.1.1.1.1" xref="A1.SS1.p2.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS1.p2.3.m3.1.1.1.1.2" xref="A1.SS1.p2.3.m3.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS1.p2.3.m3.1.1.1.1.1" xref="A1.SS1.p2.3.m3.1.1.1.1.1.cmml"><msub id="A1.SS1.p2.3.m3.1.1.1.1.1.2" xref="A1.SS1.p2.3.m3.1.1.1.1.1.2.cmml"><mi id="A1.SS1.p2.3.m3.1.1.1.1.1.2.2" xref="A1.SS1.p2.3.m3.1.1.1.1.1.2.2.cmml">x</mi><mi id="A1.SS1.p2.3.m3.1.1.1.1.1.2.3" xref="A1.SS1.p2.3.m3.1.1.1.1.1.2.3.cmml">t</mi></msub><mo fence="false" id="A1.SS1.p2.3.m3.1.1.1.1.1.1" xref="A1.SS1.p2.3.m3.1.1.1.1.1.1.cmml">|</mo><msub id="A1.SS1.p2.3.m3.1.1.1.1.1.3" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3.cmml"><mi id="A1.SS1.p2.3.m3.1.1.1.1.1.3.2" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3.2.cmml">x</mi><mrow id="A1.SS1.p2.3.m3.1.1.1.1.1.3.3" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.cmml"><mi id="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.2" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.2.cmml"></mi><mo id="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.1" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.3" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.3.cmml">t</mi></mrow></msub></mrow><mo stretchy="false" id="A1.SS1.p2.3.m3.1.1.1.1.3" xref="A1.SS1.p2.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.3.m3.1b"><apply id="A1.SS1.p2.3.m3.1.1.cmml" xref="A1.SS1.p2.3.m3.1.1"><times id="A1.SS1.p2.3.m3.1.1.2.cmml" xref="A1.SS1.p2.3.m3.1.1.2"></times><ci id="A1.SS1.p2.3.m3.1.1.3.cmml" xref="A1.SS1.p2.3.m3.1.1.3">𝑃</ci><apply id="A1.SS1.p2.3.m3.1.1.1.1.1.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1"><csymbol cd="latexml" id="A1.SS1.p2.3.m3.1.1.1.1.1.1.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.1">conditional</csymbol><apply id="A1.SS1.p2.3.m3.1.1.1.1.1.2.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.SS1.p2.3.m3.1.1.1.1.1.2.1.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.2">subscript</csymbol><ci id="A1.SS1.p2.3.m3.1.1.1.1.1.2.2.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.2.2">𝑥</ci><ci id="A1.SS1.p2.3.m3.1.1.1.1.1.2.3.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="A1.SS1.p2.3.m3.1.1.1.1.1.3.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.SS1.p2.3.m3.1.1.1.1.1.3.1.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3">subscript</csymbol><ci id="A1.SS1.p2.3.m3.1.1.1.1.1.3.2.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3.2">𝑥</ci><apply id="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3.3"><lt id="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.1.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.2.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.2">absent</csymbol><ci id="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.3.cmml" xref="A1.SS1.p2.3.m3.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.3.m3.1c">P(x_{t}|x_{&lt;t})</annotation></semantics></math> for the joint model. Some examples of <math id="A1.SS1.p2.4.m4.1" class="ltx_Math" alttext="P(x_{t}|x_{&lt;t})" display="inline"><semantics id="A1.SS1.p2.4.m4.1a"><mrow id="A1.SS1.p2.4.m4.1.1" xref="A1.SS1.p2.4.m4.1.1.cmml"><mi id="A1.SS1.p2.4.m4.1.1.3" xref="A1.SS1.p2.4.m4.1.1.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="A1.SS1.p2.4.m4.1.1.2" xref="A1.SS1.p2.4.m4.1.1.2.cmml">​</mo><mrow id="A1.SS1.p2.4.m4.1.1.1.1" xref="A1.SS1.p2.4.m4.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS1.p2.4.m4.1.1.1.1.2" xref="A1.SS1.p2.4.m4.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS1.p2.4.m4.1.1.1.1.1" xref="A1.SS1.p2.4.m4.1.1.1.1.1.cmml"><msub id="A1.SS1.p2.4.m4.1.1.1.1.1.2" xref="A1.SS1.p2.4.m4.1.1.1.1.1.2.cmml"><mi id="A1.SS1.p2.4.m4.1.1.1.1.1.2.2" xref="A1.SS1.p2.4.m4.1.1.1.1.1.2.2.cmml">x</mi><mi id="A1.SS1.p2.4.m4.1.1.1.1.1.2.3" xref="A1.SS1.p2.4.m4.1.1.1.1.1.2.3.cmml">t</mi></msub><mo fence="false" id="A1.SS1.p2.4.m4.1.1.1.1.1.1" xref="A1.SS1.p2.4.m4.1.1.1.1.1.1.cmml">|</mo><msub id="A1.SS1.p2.4.m4.1.1.1.1.1.3" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3.cmml"><mi id="A1.SS1.p2.4.m4.1.1.1.1.1.3.2" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3.2.cmml">x</mi><mrow id="A1.SS1.p2.4.m4.1.1.1.1.1.3.3" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.cmml"><mi id="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.2" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.2.cmml"></mi><mo id="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.1" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.3" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.3.cmml">t</mi></mrow></msub></mrow><mo stretchy="false" id="A1.SS1.p2.4.m4.1.1.1.1.3" xref="A1.SS1.p2.4.m4.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.4.m4.1b"><apply id="A1.SS1.p2.4.m4.1.1.cmml" xref="A1.SS1.p2.4.m4.1.1"><times id="A1.SS1.p2.4.m4.1.1.2.cmml" xref="A1.SS1.p2.4.m4.1.1.2"></times><ci id="A1.SS1.p2.4.m4.1.1.3.cmml" xref="A1.SS1.p2.4.m4.1.1.3">𝑃</ci><apply id="A1.SS1.p2.4.m4.1.1.1.1.1.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1"><csymbol cd="latexml" id="A1.SS1.p2.4.m4.1.1.1.1.1.1.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.1">conditional</csymbol><apply id="A1.SS1.p2.4.m4.1.1.1.1.1.2.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.SS1.p2.4.m4.1.1.1.1.1.2.1.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.2">subscript</csymbol><ci id="A1.SS1.p2.4.m4.1.1.1.1.1.2.2.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.2.2">𝑥</ci><ci id="A1.SS1.p2.4.m4.1.1.1.1.1.2.3.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="A1.SS1.p2.4.m4.1.1.1.1.1.3.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.SS1.p2.4.m4.1.1.1.1.1.3.1.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3">subscript</csymbol><ci id="A1.SS1.p2.4.m4.1.1.1.1.1.3.2.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3.2">𝑥</ci><apply id="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3.3"><lt id="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.1.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.2.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.2">absent</csymbol><ci id="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.3.cmml" xref="A1.SS1.p2.4.m4.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.4.m4.1c">P(x_{t}|x_{&lt;t})</annotation></semantics></math> of <span id="A1.SS1.p2.4.1" class="ltx_text ltx_font_smallcaps">Base LM</span> and <span id="A1.SS1.p2.4.2" class="ltx_text ltx_font_smallcaps">BiT-Base</span> are presented in Table <a href="#A1.T3" title="Table 3 ‣ A.1 Top-k auto-regressive sampling ‣ Appendix A Appendix ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Indeed <span id="A1.SS1.p2.4.3" class="ltx_text ltx_font_smallcaps">BiT-Base</span> usually assigns lower probabilities to repetitions even though the top k words remain the same, which is not surprising given that the existence of repetition is a strong indicator of coming from the LM, which would lead to a higher energy value hence lower joint probability.</p>
</div>
<figure id="A1.T3" class="ltx_table">
<table id="A1.T3.3" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T3.3.3" class="ltx_tr">
<th id="A1.T3.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_tt">
<span id="A1.T3.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T3.1.1.1.1.1" class="ltx_p" style="width:170.7pt;">Context <math id="A1.T3.1.1.1.1.1.m1.1" class="ltx_Math" alttext="x_{&lt;t}" display="inline"><semantics id="A1.T3.1.1.1.1.1.m1.1a"><msub id="A1.T3.1.1.1.1.1.m1.1.1" xref="A1.T3.1.1.1.1.1.m1.1.1.cmml"><mi id="A1.T3.1.1.1.1.1.m1.1.1.2" xref="A1.T3.1.1.1.1.1.m1.1.1.2.cmml">x</mi><mrow id="A1.T3.1.1.1.1.1.m1.1.1.3" xref="A1.T3.1.1.1.1.1.m1.1.1.3.cmml"><mi id="A1.T3.1.1.1.1.1.m1.1.1.3.2" xref="A1.T3.1.1.1.1.1.m1.1.1.3.2.cmml"></mi><mo id="A1.T3.1.1.1.1.1.m1.1.1.3.1" xref="A1.T3.1.1.1.1.1.m1.1.1.3.1.cmml">&lt;</mo><mi id="A1.T3.1.1.1.1.1.m1.1.1.3.3" xref="A1.T3.1.1.1.1.1.m1.1.1.3.3.cmml">t</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A1.T3.1.1.1.1.1.m1.1b"><apply id="A1.T3.1.1.1.1.1.m1.1.1.cmml" xref="A1.T3.1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.T3.1.1.1.1.1.m1.1.1.1.cmml" xref="A1.T3.1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="A1.T3.1.1.1.1.1.m1.1.1.2.cmml" xref="A1.T3.1.1.1.1.1.m1.1.1.2">𝑥</ci><apply id="A1.T3.1.1.1.1.1.m1.1.1.3.cmml" xref="A1.T3.1.1.1.1.1.m1.1.1.3"><lt id="A1.T3.1.1.1.1.1.m1.1.1.3.1.cmml" xref="A1.T3.1.1.1.1.1.m1.1.1.3.1"></lt><csymbol cd="latexml" id="A1.T3.1.1.1.1.1.m1.1.1.3.2.cmml" xref="A1.T3.1.1.1.1.1.m1.1.1.3.2">absent</csymbol><ci id="A1.T3.1.1.1.1.1.m1.1.1.3.3.cmml" xref="A1.T3.1.1.1.1.1.m1.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T3.1.1.1.1.1.m1.1c">x_{&lt;t}</annotation></semantics></math></span>
</span>
</th>
<th id="A1.T3.3.3.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Model</th>
<th id="A1.T3.3.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Rank</th>
<th id="A1.T3.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><math id="A1.T3.2.2.2.m1.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="A1.T3.2.2.2.m1.1a"><msub id="A1.T3.2.2.2.m1.1.1" xref="A1.T3.2.2.2.m1.1.1.cmml"><mi id="A1.T3.2.2.2.m1.1.1.2" xref="A1.T3.2.2.2.m1.1.1.2.cmml">x</mi><mi id="A1.T3.2.2.2.m1.1.1.3" xref="A1.T3.2.2.2.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="A1.T3.2.2.2.m1.1b"><apply id="A1.T3.2.2.2.m1.1.1.cmml" xref="A1.T3.2.2.2.m1.1.1"><csymbol cd="ambiguous" id="A1.T3.2.2.2.m1.1.1.1.cmml" xref="A1.T3.2.2.2.m1.1.1">subscript</csymbol><ci id="A1.T3.2.2.2.m1.1.1.2.cmml" xref="A1.T3.2.2.2.m1.1.1.2">𝑥</ci><ci id="A1.T3.2.2.2.m1.1.1.3.cmml" xref="A1.T3.2.2.2.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T3.2.2.2.m1.1c">x_{t}</annotation></semantics></math></th>
<th id="A1.T3.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><math id="A1.T3.3.3.3.m1.1" class="ltx_Math" alttext="P(x_{t}|x_{&lt;t})" display="inline"><semantics id="A1.T3.3.3.3.m1.1a"><mrow id="A1.T3.3.3.3.m1.1.1" xref="A1.T3.3.3.3.m1.1.1.cmml"><mi id="A1.T3.3.3.3.m1.1.1.3" xref="A1.T3.3.3.3.m1.1.1.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="A1.T3.3.3.3.m1.1.1.2" xref="A1.T3.3.3.3.m1.1.1.2.cmml">​</mo><mrow id="A1.T3.3.3.3.m1.1.1.1.1" xref="A1.T3.3.3.3.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.T3.3.3.3.m1.1.1.1.1.2" xref="A1.T3.3.3.3.m1.1.1.1.1.1.cmml">(</mo><mrow id="A1.T3.3.3.3.m1.1.1.1.1.1" xref="A1.T3.3.3.3.m1.1.1.1.1.1.cmml"><msub id="A1.T3.3.3.3.m1.1.1.1.1.1.2" xref="A1.T3.3.3.3.m1.1.1.1.1.1.2.cmml"><mi id="A1.T3.3.3.3.m1.1.1.1.1.1.2.2" xref="A1.T3.3.3.3.m1.1.1.1.1.1.2.2.cmml">x</mi><mi id="A1.T3.3.3.3.m1.1.1.1.1.1.2.3" xref="A1.T3.3.3.3.m1.1.1.1.1.1.2.3.cmml">t</mi></msub><mo fence="false" id="A1.T3.3.3.3.m1.1.1.1.1.1.1" xref="A1.T3.3.3.3.m1.1.1.1.1.1.1.cmml">|</mo><msub id="A1.T3.3.3.3.m1.1.1.1.1.1.3" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3.cmml"><mi id="A1.T3.3.3.3.m1.1.1.1.1.1.3.2" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3.2.cmml">x</mi><mrow id="A1.T3.3.3.3.m1.1.1.1.1.1.3.3" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.cmml"><mi id="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.2" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.2.cmml"></mi><mo id="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.1" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.3" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.3.cmml">t</mi></mrow></msub></mrow><mo stretchy="false" id="A1.T3.3.3.3.m1.1.1.1.1.3" xref="A1.T3.3.3.3.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.T3.3.3.3.m1.1b"><apply id="A1.T3.3.3.3.m1.1.1.cmml" xref="A1.T3.3.3.3.m1.1.1"><times id="A1.T3.3.3.3.m1.1.1.2.cmml" xref="A1.T3.3.3.3.m1.1.1.2"></times><ci id="A1.T3.3.3.3.m1.1.1.3.cmml" xref="A1.T3.3.3.3.m1.1.1.3">𝑃</ci><apply id="A1.T3.3.3.3.m1.1.1.1.1.1.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1"><csymbol cd="latexml" id="A1.T3.3.3.3.m1.1.1.1.1.1.1.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.1">conditional</csymbol><apply id="A1.T3.3.3.3.m1.1.1.1.1.1.2.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.T3.3.3.3.m1.1.1.1.1.1.2.1.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.2">subscript</csymbol><ci id="A1.T3.3.3.3.m1.1.1.1.1.1.2.2.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.2.2">𝑥</ci><ci id="A1.T3.3.3.3.m1.1.1.1.1.1.2.3.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="A1.T3.3.3.3.m1.1.1.1.1.1.3.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.T3.3.3.3.m1.1.1.1.1.1.3.1.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.T3.3.3.3.m1.1.1.1.1.1.3.2.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3.2">𝑥</ci><apply id="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3.3"><lt id="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.1.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.2.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.2">absent</csymbol><ci id="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.3.cmml" xref="A1.T3.3.3.3.m1.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T3.3.3.3.m1.1c">P(x_{t}|x_{&lt;t})</annotation></semantics></math></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T3.3.4.1" class="ltx_tr">
<td id="A1.T3.3.4.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" rowspan="10">
<span id="A1.T3.3.4.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T3.3.4.1.1.1.1" class="ltx_p" style="width:170.7pt;"><span id="A1.T3.3.4.1.1.1.1.1" class="ltx_text"><span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span>Excerpt from <a target="_blank" href="https://www.swissinfo.ch/eng/multinational-principles_swiss-government-gives-green-light-for-un-migration-accord/44464186" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.swissinfo.ch/eng/multinational-principles_swiss-government-gives-green-light-for-un-migration-accord/44464186</a>.</span></span></span>… is aimed at setting common benchmarks for orderly migration practices, thereby reducing irregular flows. The Global Compact contains ten guiding principles, including that migrants cannot be settled by countries with better integration policies and a fair and sustainable development. ”For the first time in our history, a <span id="A1.T3.3.4.1.1.1.1.1.1" class="ltx_text" style="color:#FF0000;">legally binding</span> and</span></span>
</span>
</td>
<td id="A1.T3.3.4.1.2" class="ltx_td ltx_align_center ltx_border_t" rowspan="5"><span id="A1.T3.3.4.1.2.1" class="ltx_text ltx_font_smallcaps">Base LM</span></td>
<td id="A1.T3.3.4.1.3" class="ltx_td ltx_align_center ltx_border_t">0</td>
<td id="A1.T3.3.4.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T3.3.4.1.4.1" class="ltx_text" style="color:#FF0000;">binding</span></td>
<td id="A1.T3.3.4.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T3.3.4.1.5.1" class="ltx_text" style="color:#FF0000;">0.39</span></td>
</tr>
<tr id="A1.T3.3.5.2" class="ltx_tr">
<td id="A1.T3.3.5.2.1" class="ltx_td ltx_align_center">1</td>
<td id="A1.T3.3.5.2.2" class="ltx_td ltx_align_center"><span id="A1.T3.3.5.2.2.1" class="ltx_text" style="color:#FF0000;">legally</span></td>
<td id="A1.T3.3.5.2.3" class="ltx_td ltx_align_center"><span id="A1.T3.3.5.2.3.1" class="ltx_text" style="color:#FF0000;">0.33</span></td>
</tr>
<tr id="A1.T3.3.6.3" class="ltx_tr">
<td id="A1.T3.3.6.3.1" class="ltx_td ltx_align_center">2</td>
<td id="A1.T3.3.6.3.2" class="ltx_td ltx_align_center">internationally</td>
<td id="A1.T3.3.6.3.3" class="ltx_td ltx_align_center">0.06</td>
</tr>
<tr id="A1.T3.3.7.4" class="ltx_tr">
<td id="A1.T3.3.7.4.1" class="ltx_td ltx_align_center">3</td>
<td id="A1.T3.3.7.4.2" class="ltx_td ltx_align_center">comprehensive</td>
<td id="A1.T3.3.7.4.3" class="ltx_td ltx_align_center">0.05</td>
</tr>
<tr id="A1.T3.3.8.5" class="ltx_tr">
<td id="A1.T3.3.8.5.1" class="ltx_td ltx_align_center">4</td>
<td id="A1.T3.3.8.5.2" class="ltx_td ltx_align_center">transparent</td>
<td id="A1.T3.3.8.5.3" class="ltx_td ltx_align_center">0.04</td>
</tr>
<tr id="A1.T3.3.9.6" class="ltx_tr">
<td id="A1.T3.3.9.6.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="5"><span id="A1.T3.3.9.6.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base</span></td>
<td id="A1.T3.3.9.6.2" class="ltx_td ltx_align_center ltx_border_t">0</td>
<td id="A1.T3.3.9.6.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T3.3.9.6.3.1" class="ltx_text" style="color:#FF0000;">binding</span></td>
<td id="A1.T3.3.9.6.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T3.3.9.6.4.1" class="ltx_text" style="color:#FF0000;">0.18</span></td>
</tr>
<tr id="A1.T3.3.10.7" class="ltx_tr">
<td id="A1.T3.3.10.7.1" class="ltx_td ltx_align_center">1</td>
<td id="A1.T3.3.10.7.2" class="ltx_td ltx_align_center"><span id="A1.T3.3.10.7.2.1" class="ltx_text" style="color:#FF0000;">legally</span></td>
<td id="A1.T3.3.10.7.3" class="ltx_td ltx_align_center"><span id="A1.T3.3.10.7.3.1" class="ltx_text" style="color:#FF0000;">0.17</span></td>
</tr>
<tr id="A1.T3.3.11.8" class="ltx_tr">
<td id="A1.T3.3.11.8.1" class="ltx_td ltx_align_center">2</td>
<td id="A1.T3.3.11.8.2" class="ltx_td ltx_align_center">internationally</td>
<td id="A1.T3.3.11.8.3" class="ltx_td ltx_align_center">0.12</td>
</tr>
<tr id="A1.T3.3.12.9" class="ltx_tr">
<td id="A1.T3.3.12.9.1" class="ltx_td ltx_align_center">3</td>
<td id="A1.T3.3.12.9.2" class="ltx_td ltx_align_center">comprehensive</td>
<td id="A1.T3.3.12.9.3" class="ltx_td ltx_align_center">0.09</td>
</tr>
<tr id="A1.T3.3.13.10" class="ltx_tr">
<td id="A1.T3.3.13.10.1" class="ltx_td ltx_align_center">4</td>
<td id="A1.T3.3.13.10.2" class="ltx_td ltx_align_center">transparent</td>
<td id="A1.T3.3.13.10.3" class="ltx_td ltx_align_center">0.08</td>
</tr>
<tr id="A1.T3.3.14.11" class="ltx_tr">
<td id="A1.T3.3.14.11.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" rowspan="10">
<span id="A1.T3.3.14.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T3.3.14.11.1.1.1" class="ltx_p" style="width:170.7pt;"><span id="A1.T3.3.14.11.1.1.1.1" class="ltx_text">
<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span>Excerpt from <a target="_blank" href="https://www.forbes.com/sites/markmurphy/2018/05/11/this-is-the-one-piece-of-data-that-85-of-recruiters-are-missing/#25917c765dad" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.forbes.com/sites/markmurphy/2018/05/11/this-is-the-one-piece-of-data-that-85-of-recruiters-are-missing/#25917c765dad</a>.</span></span></span>
… companies that land their first-choice candidates 90-100% of the time, 24% of them have ”thoroughly defined” their high performer attitudes. By contrast, only 1% of companies that struggle to land their first-choice candidates ”thoroughly defined” their high performer attitudes. So it seems pretty clear that companies that land their top-choice candidates are not always as <span id="A1.T3.3.14.11.1.1.1.1.1" class="ltx_text" style="color:#FF0000;">willing</span> and</span></span>
</span>
</td>
<td id="A1.T3.3.14.11.2" class="ltx_td ltx_align_center ltx_border_t" rowspan="5"><span id="A1.T3.3.14.11.2.1" class="ltx_text ltx_font_smallcaps">Base LM</span></td>
<td id="A1.T3.3.14.11.3" class="ltx_td ltx_align_center ltx_border_t">0</td>
<td id="A1.T3.3.14.11.4" class="ltx_td ltx_align_center ltx_border_t">able</td>
<td id="A1.T3.3.14.11.5" class="ltx_td ltx_align_center ltx_border_t">0.66</td>
</tr>
<tr id="A1.T3.3.15.12" class="ltx_tr">
<td id="A1.T3.3.15.12.1" class="ltx_td ltx_align_center">1</td>
<td id="A1.T3.3.15.12.2" class="ltx_td ltx_align_center"><span id="A1.T3.3.15.12.2.1" class="ltx_text" style="color:#FF0000;">willing</span></td>
<td id="A1.T3.3.15.12.3" class="ltx_td ltx_align_center"><span id="A1.T3.3.15.12.3.1" class="ltx_text" style="color:#FF0000;">0.09</span></td>
</tr>
<tr id="A1.T3.3.16.13" class="ltx_tr">
<td id="A1.T3.3.16.13.1" class="ltx_td ltx_align_center">2</td>
<td id="A1.T3.3.16.13.2" class="ltx_td ltx_align_center">eager</td>
<td id="A1.T3.3.16.13.3" class="ltx_td ltx_align_center">0.07</td>
</tr>
<tr id="A1.T3.3.17.14" class="ltx_tr">
<td id="A1.T3.3.17.14.1" class="ltx_td ltx_align_center">3</td>
<td id="A1.T3.3.17.14.2" class="ltx_td ltx_align_center">ready</td>
<td id="A1.T3.3.17.14.3" class="ltx_td ltx_align_center">0.05</td>
</tr>
<tr id="A1.T3.3.18.15" class="ltx_tr">
<td id="A1.T3.3.18.15.1" class="ltx_td ltx_align_center">4</td>
<td id="A1.T3.3.18.15.2" class="ltx_td ltx_align_center">well</td>
<td id="A1.T3.3.18.15.3" class="ltx_td ltx_align_center">0.04</td>
</tr>
<tr id="A1.T3.3.19.16" class="ltx_tr">
<td id="A1.T3.3.19.16.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="5"><span id="A1.T3.3.19.16.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base</span></td>
<td id="A1.T3.3.19.16.2" class="ltx_td ltx_align_center ltx_border_t">0</td>
<td id="A1.T3.3.19.16.3" class="ltx_td ltx_align_center ltx_border_t">able</td>
<td id="A1.T3.3.19.16.4" class="ltx_td ltx_align_center ltx_border_t">0.75</td>
</tr>
<tr id="A1.T3.3.20.17" class="ltx_tr">
<td id="A1.T3.3.20.17.1" class="ltx_td ltx_align_center">1</td>
<td id="A1.T3.3.20.17.2" class="ltx_td ltx_align_center"><span id="A1.T3.3.20.17.2.1" class="ltx_text" style="color:#FF0000;">willing</span></td>
<td id="A1.T3.3.20.17.3" class="ltx_td ltx_align_center"><span id="A1.T3.3.20.17.3.1" class="ltx_text" style="color:#FF0000;">0.05</span></td>
</tr>
<tr id="A1.T3.3.21.18" class="ltx_tr">
<td id="A1.T3.3.21.18.1" class="ltx_td ltx_align_center">2</td>
<td id="A1.T3.3.21.18.2" class="ltx_td ltx_align_center">eager</td>
<td id="A1.T3.3.21.18.3" class="ltx_td ltx_align_center">0.05</td>
</tr>
<tr id="A1.T3.3.22.19" class="ltx_tr">
<td id="A1.T3.3.22.19.1" class="ltx_td ltx_align_center">3</td>
<td id="A1.T3.3.22.19.2" class="ltx_td ltx_align_center">ready</td>
<td id="A1.T3.3.22.19.3" class="ltx_td ltx_align_center">0.04</td>
</tr>
<tr id="A1.T3.3.23.20" class="ltx_tr">
<td id="A1.T3.3.23.20.1" class="ltx_td ltx_align_center">4</td>
<td id="A1.T3.3.23.20.2" class="ltx_td ltx_align_center">well</td>
<td id="A1.T3.3.23.20.3" class="ltx_td ltx_align_center">0.03</td>
</tr>
<tr id="A1.T3.3.24.21" class="ltx_tr">
<td id="A1.T3.3.24.21.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t" rowspan="10">
<span id="A1.T3.3.24.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T3.3.24.21.1.1.1" class="ltx_p" style="width:170.7pt;"><span id="A1.T3.3.24.21.1.1.1.1" class="ltx_text"><span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span>Excerpt from <a target="_blank" href="https://www.reuters.com/article/us-usa-fed-powell/fed-nominee-powell-once-hawkish-now-champions-yellens-focus-on-jobs-idUSKBN1DS0FG" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.reuters.com/article/us-usa-fed-powell/fed-nominee-powell-once-hawkish-now-champions-yellens-focus-on-jobs-idUSKBN1DS0FG</a></span></span></span>… it reveals a key skill needed to lead the Fed. ”You need to know what you don’t know. And you need to be willing to listen when you don’t know something,” said Karen Dynan, who as an assistant Treasury Secretary in Barack Obama’s second administration would regularly meet Fed governors. ¡EOS¿ New Delhi Dec 5 The following are mergers under review by India’s <span id="A1.T3.3.24.21.1.1.1.1.1" class="ltx_text" style="color:#FF0000;">financial</span> services and</span></span>
</span>
</td>
<td id="A1.T3.3.24.21.2" class="ltx_td ltx_align_center ltx_border_t" rowspan="5"><span id="A1.T3.3.24.21.2.1" class="ltx_text ltx_font_smallcaps">Base LM</span></td>
<td id="A1.T3.3.24.21.3" class="ltx_td ltx_align_center ltx_border_t">0</td>
<td id="A1.T3.3.24.21.4" class="ltx_td ltx_align_center ltx_border_t">banking</td>
<td id="A1.T3.3.24.21.5" class="ltx_td ltx_align_center ltx_border_t">0.64</td>
</tr>
<tr id="A1.T3.3.25.22" class="ltx_tr">
<td id="A1.T3.3.25.22.1" class="ltx_td ltx_align_center">1</td>
<td id="A1.T3.3.25.22.2" class="ltx_td ltx_align_center"><span id="A1.T3.3.25.22.2.1" class="ltx_text" style="color:#FF0000;">financial</span></td>
<td id="A1.T3.3.25.22.3" class="ltx_td ltx_align_center"><span id="A1.T3.3.25.22.3.1" class="ltx_text" style="color:#FF0000;">0.10</span></td>
</tr>
<tr id="A1.T3.3.26.23" class="ltx_tr">
<td id="A1.T3.3.26.23.1" class="ltx_td ltx_align_center">2</td>
<td id="A1.T3.3.26.23.2" class="ltx_td ltx_align_center">insurance</td>
<td id="A1.T3.3.26.23.3" class="ltx_td ltx_align_center">0.09</td>
</tr>
<tr id="A1.T3.3.27.24" class="ltx_tr">
<td id="A1.T3.3.27.24.1" class="ltx_td ltx_align_center">3</td>
<td id="A1.T3.3.27.24.2" class="ltx_td ltx_align_center">technology</td>
<td id="A1.T3.3.27.24.3" class="ltx_td ltx_align_center">0.05</td>
</tr>
<tr id="A1.T3.3.28.25" class="ltx_tr">
<td id="A1.T3.3.28.25.1" class="ltx_td ltx_align_center">4</td>
<td id="A1.T3.3.28.25.2" class="ltx_td ltx_align_center">IT</td>
<td id="A1.T3.3.28.25.3" class="ltx_td ltx_align_center">0.04</td>
</tr>
<tr id="A1.T3.3.29.26" class="ltx_tr">
<td id="A1.T3.3.29.26.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" rowspan="5"><span id="A1.T3.3.29.26.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base</span></td>
<td id="A1.T3.3.29.26.2" class="ltx_td ltx_align_center ltx_border_t">0</td>
<td id="A1.T3.3.29.26.3" class="ltx_td ltx_align_center ltx_border_t">banking</td>
<td id="A1.T3.3.29.26.4" class="ltx_td ltx_align_center ltx_border_t">0.92</td>
</tr>
<tr id="A1.T3.3.30.27" class="ltx_tr">
<td id="A1.T3.3.30.27.1" class="ltx_td ltx_align_center">1</td>
<td id="A1.T3.3.30.27.2" class="ltx_td ltx_align_center"><span id="A1.T3.3.30.27.2.1" class="ltx_text" style="color:#FF0000;">financial</span></td>
<td id="A1.T3.3.30.27.3" class="ltx_td ltx_align_center"><span id="A1.T3.3.30.27.3.1" class="ltx_text" style="color:#FF0000;">0.06</span></td>
</tr>
<tr id="A1.T3.3.31.28" class="ltx_tr">
<td id="A1.T3.3.31.28.1" class="ltx_td ltx_align_center">2</td>
<td id="A1.T3.3.31.28.2" class="ltx_td ltx_align_center">insurance</td>
<td id="A1.T3.3.31.28.3" class="ltx_td ltx_align_center">0.01</td>
</tr>
<tr id="A1.T3.3.32.29" class="ltx_tr">
<td id="A1.T3.3.32.29.1" class="ltx_td ltx_align_center">3</td>
<td id="A1.T3.3.32.29.2" class="ltx_td ltx_align_center">technology</td>
<td id="A1.T3.3.32.29.3" class="ltx_td ltx_align_center">0.00</td>
</tr>
<tr id="A1.T3.3.33.30" class="ltx_tr">
<td id="A1.T3.3.33.30.1" class="ltx_td ltx_align_center ltx_border_bb">4</td>
<td id="A1.T3.3.33.30.2" class="ltx_td ltx_align_center ltx_border_bb">IT</td>
<td id="A1.T3.3.33.30.3" class="ltx_td ltx_align_center ltx_border_bb">0.00</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Comparison of <math id="A1.T3.5.m1.1" class="ltx_Math" alttext="P(x_{t}|x_{&lt;t})" display="inline"><semantics id="A1.T3.5.m1.1b"><mrow id="A1.T3.5.m1.1.1" xref="A1.T3.5.m1.1.1.cmml"><mi id="A1.T3.5.m1.1.1.3" xref="A1.T3.5.m1.1.1.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="A1.T3.5.m1.1.1.2" xref="A1.T3.5.m1.1.1.2.cmml">​</mo><mrow id="A1.T3.5.m1.1.1.1.1" xref="A1.T3.5.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.T3.5.m1.1.1.1.1.2" xref="A1.T3.5.m1.1.1.1.1.1.cmml">(</mo><mrow id="A1.T3.5.m1.1.1.1.1.1" xref="A1.T3.5.m1.1.1.1.1.1.cmml"><msub id="A1.T3.5.m1.1.1.1.1.1.2" xref="A1.T3.5.m1.1.1.1.1.1.2.cmml"><mi id="A1.T3.5.m1.1.1.1.1.1.2.2" xref="A1.T3.5.m1.1.1.1.1.1.2.2.cmml">x</mi><mi id="A1.T3.5.m1.1.1.1.1.1.2.3" xref="A1.T3.5.m1.1.1.1.1.1.2.3.cmml">t</mi></msub><mo fence="false" id="A1.T3.5.m1.1.1.1.1.1.1" xref="A1.T3.5.m1.1.1.1.1.1.1.cmml">|</mo><msub id="A1.T3.5.m1.1.1.1.1.1.3" xref="A1.T3.5.m1.1.1.1.1.1.3.cmml"><mi id="A1.T3.5.m1.1.1.1.1.1.3.2" xref="A1.T3.5.m1.1.1.1.1.1.3.2.cmml">x</mi><mrow id="A1.T3.5.m1.1.1.1.1.1.3.3" xref="A1.T3.5.m1.1.1.1.1.1.3.3.cmml"><mi id="A1.T3.5.m1.1.1.1.1.1.3.3.2" xref="A1.T3.5.m1.1.1.1.1.1.3.3.2.cmml"></mi><mo id="A1.T3.5.m1.1.1.1.1.1.3.3.1" xref="A1.T3.5.m1.1.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="A1.T3.5.m1.1.1.1.1.1.3.3.3" xref="A1.T3.5.m1.1.1.1.1.1.3.3.3.cmml">t</mi></mrow></msub></mrow><mo stretchy="false" id="A1.T3.5.m1.1.1.1.1.3" xref="A1.T3.5.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.T3.5.m1.1c"><apply id="A1.T3.5.m1.1.1.cmml" xref="A1.T3.5.m1.1.1"><times id="A1.T3.5.m1.1.1.2.cmml" xref="A1.T3.5.m1.1.1.2"></times><ci id="A1.T3.5.m1.1.1.3.cmml" xref="A1.T3.5.m1.1.1.3">𝑃</ci><apply id="A1.T3.5.m1.1.1.1.1.1.cmml" xref="A1.T3.5.m1.1.1.1.1"><csymbol cd="latexml" id="A1.T3.5.m1.1.1.1.1.1.1.cmml" xref="A1.T3.5.m1.1.1.1.1.1.1">conditional</csymbol><apply id="A1.T3.5.m1.1.1.1.1.1.2.cmml" xref="A1.T3.5.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.T3.5.m1.1.1.1.1.1.2.1.cmml" xref="A1.T3.5.m1.1.1.1.1.1.2">subscript</csymbol><ci id="A1.T3.5.m1.1.1.1.1.1.2.2.cmml" xref="A1.T3.5.m1.1.1.1.1.1.2.2">𝑥</ci><ci id="A1.T3.5.m1.1.1.1.1.1.2.3.cmml" xref="A1.T3.5.m1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="A1.T3.5.m1.1.1.1.1.1.3.cmml" xref="A1.T3.5.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.T3.5.m1.1.1.1.1.1.3.1.cmml" xref="A1.T3.5.m1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.T3.5.m1.1.1.1.1.1.3.2.cmml" xref="A1.T3.5.m1.1.1.1.1.1.3.2">𝑥</ci><apply id="A1.T3.5.m1.1.1.1.1.1.3.3.cmml" xref="A1.T3.5.m1.1.1.1.1.1.3.3"><lt id="A1.T3.5.m1.1.1.1.1.1.3.3.1.cmml" xref="A1.T3.5.m1.1.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="A1.T3.5.m1.1.1.1.1.1.3.3.2.cmml" xref="A1.T3.5.m1.1.1.1.1.1.3.3.2">absent</csymbol><ci id="A1.T3.5.m1.1.1.1.1.1.3.3.3.cmml" xref="A1.T3.5.m1.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T3.5.m1.1d">P(x_{t}|x_{&lt;t})</annotation></semantics></math> between <span id="A1.T3.8.1" class="ltx_text ltx_font_smallcaps">Base LM</span> and <span id="A1.T3.9.2" class="ltx_text ltx_font_smallcaps">BiT-Base</span> on a few examples. Repetitions are marked with red. Only the top 5 probabilities are shown.</figcaption>
</figure>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Proof of Theorem 2</h3>

<div id="Thmtheorem2a" class="ltx_theorem ltx_theorem_theorem">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmtheorem2a.1.1.1" class="ltx_text ltx_font_bold">Theorem 2</span></span><span id="Thmtheorem2a.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmtheorem2a.p1" class="ltx_para">
<p id="Thmtheorem2a.p1.8" class="ltx_p"><span id="Thmtheorem2a.p1.8.8" class="ltx_text ltx_font_italic">Denote <math id="Thmtheorem2a.p1.1.1.m1.1" class="ltx_Math" alttext="T_{n}" display="inline"><semantics id="Thmtheorem2a.p1.1.1.m1.1a"><msub id="Thmtheorem2a.p1.1.1.m1.1.1" xref="Thmtheorem2a.p1.1.1.m1.1.1.cmml"><mi id="Thmtheorem2a.p1.1.1.m1.1.1.2" xref="Thmtheorem2a.p1.1.1.m1.1.1.2.cmml">T</mi><mi id="Thmtheorem2a.p1.1.1.m1.1.1.3" xref="Thmtheorem2a.p1.1.1.m1.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="Thmtheorem2a.p1.1.1.m1.1b"><apply id="Thmtheorem2a.p1.1.1.m1.1.1.cmml" xref="Thmtheorem2a.p1.1.1.m1.1.1"><csymbol cd="ambiguous" id="Thmtheorem2a.p1.1.1.m1.1.1.1.cmml" xref="Thmtheorem2a.p1.1.1.m1.1.1">subscript</csymbol><ci id="Thmtheorem2a.p1.1.1.m1.1.1.2.cmml" xref="Thmtheorem2a.p1.1.1.m1.1.1.2">𝑇</ci><ci id="Thmtheorem2a.p1.1.1.m1.1.1.3.cmml" xref="Thmtheorem2a.p1.1.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2a.p1.1.1.m1.1c">T_{n}</annotation></semantics></math> as the empirical estimate of <math id="Thmtheorem2a.p1.2.2.m2.3" class="ltx_Math" alttext="\log\mathbb{E}_{x\sim P_{LM}}\exp(-E(x))" display="inline"><semantics id="Thmtheorem2a.p1.2.2.m2.3a"><mrow id="Thmtheorem2a.p1.2.2.m2.3.3" xref="Thmtheorem2a.p1.2.2.m2.3.3.cmml"><mi id="Thmtheorem2a.p1.2.2.m2.3.3.2" xref="Thmtheorem2a.p1.2.2.m2.3.3.2.cmml">log</mi><mo lspace="0.167em" id="Thmtheorem2a.p1.2.2.m2.3.3a" xref="Thmtheorem2a.p1.2.2.m2.3.3.cmml">⁡</mo><mrow id="Thmtheorem2a.p1.2.2.m2.3.3.1" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.cmml"><msub id="Thmtheorem2a.p1.2.2.m2.3.3.1.3" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.cmml"><mi id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.2" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.2.cmml">𝔼</mi><mrow id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.cmml"><mi id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.2" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.2.cmml">x</mi><mo id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.1" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.1.cmml">∼</mo><msub id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.cmml"><mi id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.2" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.2.cmml">P</mi><mrow id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.cmml"><mi id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.2" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.1" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.1.cmml">​</mo><mi id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.3" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.3.cmml">M</mi></mrow></msub></mrow></msub><mo lspace="0.167em" rspace="0em" id="Thmtheorem2a.p1.2.2.m2.3.3.1.2" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.2.cmml">​</mo><mrow id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.2.cmml"><mi id="Thmtheorem2a.p1.2.2.m2.2.2" xref="Thmtheorem2a.p1.2.2.m2.2.2.cmml">exp</mi><mo id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1a" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.2.cmml">⁡</mo><mrow id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.2.cmml"><mo stretchy="false" id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.2" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.2.cmml">(</mo><mrow id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.cmml"><mo id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1a" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.cmml">−</mo><mrow id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.cmml"><mi id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.2" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.2.cmml">E</mi><mo lspace="0em" rspace="0em" id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.1" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.1.cmml">​</mo><mrow id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.3.2" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.cmml"><mo stretchy="false" id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.3.2.1" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.cmml">(</mo><mi id="Thmtheorem2a.p1.2.2.m2.1.1" xref="Thmtheorem2a.p1.2.2.m2.1.1.cmml">x</mi><mo stretchy="false" id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.3.2.2" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.3" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2a.p1.2.2.m2.3b"><apply id="Thmtheorem2a.p1.2.2.m2.3.3.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3"><log id="Thmtheorem2a.p1.2.2.m2.3.3.2.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.2"></log><apply id="Thmtheorem2a.p1.2.2.m2.3.3.1.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1"><times id="Thmtheorem2a.p1.2.2.m2.3.3.1.2.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.2"></times><apply id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3"><csymbol cd="ambiguous" id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.1.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3">subscript</csymbol><ci id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.2.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.2">𝔼</ci><apply id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3"><csymbol cd="latexml" id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.1.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.1">similar-to</csymbol><ci id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.2.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.2">𝑥</ci><apply id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3"><csymbol cd="ambiguous" id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.1.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3">subscript</csymbol><ci id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.2.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.2">𝑃</ci><apply id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3"><times id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.1.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.1"></times><ci id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.2.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.2">𝐿</ci><ci id="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.3.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.3.3.3.3.3">𝑀</ci></apply></apply></apply></apply><apply id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.2.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1"><exp id="Thmtheorem2a.p1.2.2.m2.2.2.cmml" xref="Thmtheorem2a.p1.2.2.m2.2.2"></exp><apply id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1"><minus id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.1.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1"></minus><apply id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2"><times id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.1.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.1"></times><ci id="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.2.cmml" xref="Thmtheorem2a.p1.2.2.m2.3.3.1.1.1.1.1.2.2">𝐸</ci><ci id="Thmtheorem2a.p1.2.2.m2.1.1.cmml" xref="Thmtheorem2a.p1.2.2.m2.1.1">𝑥</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2a.p1.2.2.m2.3c">\log\mathbb{E}_{x\sim P_{LM}}\exp(-E(x))</annotation></semantics></math> with <math id="Thmtheorem2a.p1.3.3.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="Thmtheorem2a.p1.3.3.m3.1a"><mi id="Thmtheorem2a.p1.3.3.m3.1.1" xref="Thmtheorem2a.p1.3.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="Thmtheorem2a.p1.3.3.m3.1b"><ci id="Thmtheorem2a.p1.3.3.m3.1.1.cmml" xref="Thmtheorem2a.p1.3.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2a.p1.3.3.m3.1c">n</annotation></semantics></math> samples <math id="Thmtheorem2a.p1.4.4.m4.4" class="ltx_Math" alttext="x_{i}\sim P_{LM}(i=1,\cdots,n)" display="inline"><semantics id="Thmtheorem2a.p1.4.4.m4.4a"><mrow id="Thmtheorem2a.p1.4.4.m4.4.4" xref="Thmtheorem2a.p1.4.4.m4.4.4.cmml"><msub id="Thmtheorem2a.p1.4.4.m4.4.4.3" xref="Thmtheorem2a.p1.4.4.m4.4.4.3.cmml"><mi id="Thmtheorem2a.p1.4.4.m4.4.4.3.2" xref="Thmtheorem2a.p1.4.4.m4.4.4.3.2.cmml">x</mi><mi id="Thmtheorem2a.p1.4.4.m4.4.4.3.3" xref="Thmtheorem2a.p1.4.4.m4.4.4.3.3.cmml">i</mi></msub><mo id="Thmtheorem2a.p1.4.4.m4.4.4.2" xref="Thmtheorem2a.p1.4.4.m4.4.4.2.cmml">∼</mo><mrow id="Thmtheorem2a.p1.4.4.m4.4.4.1" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.cmml"><msub id="Thmtheorem2a.p1.4.4.m4.4.4.1.3" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3.cmml"><mi id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.2" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3.2.cmml">P</mi><mrow id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.cmml"><mi id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.2" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.1" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.1.cmml">​</mo><mi id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.3" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.3.cmml">M</mi></mrow></msub><mo lspace="0em" rspace="0em" id="Thmtheorem2a.p1.4.4.m4.4.4.1.2" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.2.cmml">​</mo><mrow id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.cmml"><mo stretchy="false" id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.2" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.cmml">(</mo><mrow id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.cmml"><mi id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.2" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.2.cmml">i</mi><mo id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.1" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.1.cmml">=</mo><mrow id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.3.2" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.3.1.cmml"><mn id="Thmtheorem2a.p1.4.4.m4.1.1" xref="Thmtheorem2a.p1.4.4.m4.1.1.cmml">1</mn><mo id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.3.2.1" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.3.1.cmml">,</mo><mi mathvariant="normal" id="Thmtheorem2a.p1.4.4.m4.2.2" xref="Thmtheorem2a.p1.4.4.m4.2.2.cmml">⋯</mi><mo id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.3.2.2" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.3.1.cmml">,</mo><mi id="Thmtheorem2a.p1.4.4.m4.3.3" xref="Thmtheorem2a.p1.4.4.m4.3.3.cmml">n</mi></mrow></mrow><mo stretchy="false" id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.3" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2a.p1.4.4.m4.4b"><apply id="Thmtheorem2a.p1.4.4.m4.4.4.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4"><csymbol cd="latexml" id="Thmtheorem2a.p1.4.4.m4.4.4.2.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.2">similar-to</csymbol><apply id="Thmtheorem2a.p1.4.4.m4.4.4.3.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.3"><csymbol cd="ambiguous" id="Thmtheorem2a.p1.4.4.m4.4.4.3.1.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.3">subscript</csymbol><ci id="Thmtheorem2a.p1.4.4.m4.4.4.3.2.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.3.2">𝑥</ci><ci id="Thmtheorem2a.p1.4.4.m4.4.4.3.3.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.3.3">𝑖</ci></apply><apply id="Thmtheorem2a.p1.4.4.m4.4.4.1.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1"><times id="Thmtheorem2a.p1.4.4.m4.4.4.1.2.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.2"></times><apply id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3"><csymbol cd="ambiguous" id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.1.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3">subscript</csymbol><ci id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.2.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3.2">𝑃</ci><apply id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3"><times id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.1.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.1"></times><ci id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.2.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.2">𝐿</ci><ci id="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.3.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.3.3.3">𝑀</ci></apply></apply><apply id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1"><eq id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.1.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.1"></eq><ci id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.2.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.2">𝑖</ci><list id="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.3.1.cmml" xref="Thmtheorem2a.p1.4.4.m4.4.4.1.1.1.1.3.2"><cn type="integer" id="Thmtheorem2a.p1.4.4.m4.1.1.cmml" xref="Thmtheorem2a.p1.4.4.m4.1.1">1</cn><ci id="Thmtheorem2a.p1.4.4.m4.2.2.cmml" xref="Thmtheorem2a.p1.4.4.m4.2.2">⋯</ci><ci id="Thmtheorem2a.p1.4.4.m4.3.3.cmml" xref="Thmtheorem2a.p1.4.4.m4.3.3">𝑛</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2a.p1.4.4.m4.4c">x_{i}\sim P_{LM}(i=1,\cdots,n)</annotation></semantics></math>, and let <math id="Thmtheorem2a.p1.5.5.m5.2" class="ltx_Math" alttext="T_{n}=\log\frac{1}{n}\sum_{i=1}^{n}\exp(-E(x_{i}))" display="inline"><semantics id="Thmtheorem2a.p1.5.5.m5.2a"><mrow id="Thmtheorem2a.p1.5.5.m5.2.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.cmml"><msub id="Thmtheorem2a.p1.5.5.m5.2.2.3" xref="Thmtheorem2a.p1.5.5.m5.2.2.3.cmml"><mi id="Thmtheorem2a.p1.5.5.m5.2.2.3.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.3.2.cmml">T</mi><mi id="Thmtheorem2a.p1.5.5.m5.2.2.3.3" xref="Thmtheorem2a.p1.5.5.m5.2.2.3.3.cmml">n</mi></msub><mo id="Thmtheorem2a.p1.5.5.m5.2.2.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.2.cmml">=</mo><mrow id="Thmtheorem2a.p1.5.5.m5.2.2.1" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.cmml"><mrow id="Thmtheorem2a.p1.5.5.m5.2.2.1.3" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3.cmml"><mi id="Thmtheorem2a.p1.5.5.m5.2.2.1.3.1" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3.1.cmml">log</mi><mo lspace="0.167em" id="Thmtheorem2a.p1.5.5.m5.2.2.1.3a" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3.cmml">⁡</mo><mfrac id="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2.cmml"><mn id="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2.2.cmml">1</mn><mi id="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2.3" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2.3.cmml">n</mi></mfrac></mrow><mo lspace="0em" rspace="0em" id="Thmtheorem2a.p1.5.5.m5.2.2.1.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.2.cmml">​</mo><mrow id="Thmtheorem2a.p1.5.5.m5.2.2.1.1" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.cmml"><msubsup id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.cmml"><mo id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.2.cmml">∑</mo><mrow id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.cmml"><mi id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.2.cmml">i</mi><mo id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.1" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.1.cmml">=</mo><mn id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.3" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.3" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.3.cmml">n</mi></msubsup><mrow id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.2.cmml"><mi id="Thmtheorem2a.p1.5.5.m5.1.1" xref="Thmtheorem2a.p1.5.5.m5.1.1.cmml">exp</mi><mo id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1a" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.2.cmml">⁡</mo><mrow id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.2.cmml"><mo stretchy="false" id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.2.cmml">(</mo><mrow id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.cmml"><mo id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1a" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.cmml">−</mo><mrow id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.cmml"><mi id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.3" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.2" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.3" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.3" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.3" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2a.p1.5.5.m5.2b"><apply id="Thmtheorem2a.p1.5.5.m5.2.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2"><eq id="Thmtheorem2a.p1.5.5.m5.2.2.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.2"></eq><apply id="Thmtheorem2a.p1.5.5.m5.2.2.3.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.3"><csymbol cd="ambiguous" id="Thmtheorem2a.p1.5.5.m5.2.2.3.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.3">subscript</csymbol><ci id="Thmtheorem2a.p1.5.5.m5.2.2.3.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.3.2">𝑇</ci><ci id="Thmtheorem2a.p1.5.5.m5.2.2.3.3.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.3.3">𝑛</ci></apply><apply id="Thmtheorem2a.p1.5.5.m5.2.2.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1"><times id="Thmtheorem2a.p1.5.5.m5.2.2.1.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.2"></times><apply id="Thmtheorem2a.p1.5.5.m5.2.2.1.3.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3"><log id="Thmtheorem2a.p1.5.5.m5.2.2.1.3.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3.1"></log><apply id="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2"><divide id="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2"></divide><cn type="integer" id="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2.2">1</cn><ci id="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2.3.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.3.2.3">𝑛</ci></apply></apply><apply id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1"><apply id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2"><csymbol cd="ambiguous" id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2">superscript</csymbol><apply id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2"><csymbol cd="ambiguous" id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2">subscript</csymbol><sum id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.2"></sum><apply id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3"><eq id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.1"></eq><ci id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.3.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.2.3.3">1</cn></apply></apply><ci id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.3.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.2.3">𝑛</ci></apply><apply id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1"><exp id="Thmtheorem2a.p1.5.5.m5.1.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.1.1"></exp><apply id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1"><minus id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1"></minus><apply id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1"><times id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.2"></times><ci id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.3.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.3">𝐸</ci><apply id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Thmtheorem2a.p1.5.5.m5.2.2.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2a.p1.5.5.m5.2c">T_{n}=\log\frac{1}{n}\sum_{i=1}^{n}\exp(-E(x_{i}))</annotation></semantics></math>, then <math id="Thmtheorem2a.p1.6.6.m6.1" class="ltx_Math" alttext="\forall\epsilon&gt;0" display="inline"><semantics id="Thmtheorem2a.p1.6.6.m6.1a"><mrow id="Thmtheorem2a.p1.6.6.m6.1.1" xref="Thmtheorem2a.p1.6.6.m6.1.1.cmml"><mrow id="Thmtheorem2a.p1.6.6.m6.1.1.2" xref="Thmtheorem2a.p1.6.6.m6.1.1.2.cmml"><mo rspace="0.167em" id="Thmtheorem2a.p1.6.6.m6.1.1.2.1" xref="Thmtheorem2a.p1.6.6.m6.1.1.2.1.cmml">∀</mo><mi id="Thmtheorem2a.p1.6.6.m6.1.1.2.2" xref="Thmtheorem2a.p1.6.6.m6.1.1.2.2.cmml">ϵ</mi></mrow><mo id="Thmtheorem2a.p1.6.6.m6.1.1.1" xref="Thmtheorem2a.p1.6.6.m6.1.1.1.cmml">&gt;</mo><mn id="Thmtheorem2a.p1.6.6.m6.1.1.3" xref="Thmtheorem2a.p1.6.6.m6.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2a.p1.6.6.m6.1b"><apply id="Thmtheorem2a.p1.6.6.m6.1.1.cmml" xref="Thmtheorem2a.p1.6.6.m6.1.1"><gt id="Thmtheorem2a.p1.6.6.m6.1.1.1.cmml" xref="Thmtheorem2a.p1.6.6.m6.1.1.1"></gt><apply id="Thmtheorem2a.p1.6.6.m6.1.1.2.cmml" xref="Thmtheorem2a.p1.6.6.m6.1.1.2"><csymbol cd="latexml" id="Thmtheorem2a.p1.6.6.m6.1.1.2.1.cmml" xref="Thmtheorem2a.p1.6.6.m6.1.1.2.1">for-all</csymbol><ci id="Thmtheorem2a.p1.6.6.m6.1.1.2.2.cmml" xref="Thmtheorem2a.p1.6.6.m6.1.1.2.2">italic-ϵ</ci></apply><cn type="integer" id="Thmtheorem2a.p1.6.6.m6.1.1.3.cmml" xref="Thmtheorem2a.p1.6.6.m6.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2a.p1.6.6.m6.1c">\forall\epsilon&gt;0</annotation></semantics></math>, <math id="Thmtheorem2a.p1.7.7.m7.1" class="ltx_Math" alttext="\exists N&gt;0" display="inline"><semantics id="Thmtheorem2a.p1.7.7.m7.1a"><mrow id="Thmtheorem2a.p1.7.7.m7.1.1" xref="Thmtheorem2a.p1.7.7.m7.1.1.cmml"><mrow id="Thmtheorem2a.p1.7.7.m7.1.1.2" xref="Thmtheorem2a.p1.7.7.m7.1.1.2.cmml"><mo rspace="0.167em" id="Thmtheorem2a.p1.7.7.m7.1.1.2.1" xref="Thmtheorem2a.p1.7.7.m7.1.1.2.1.cmml">∃</mo><mi id="Thmtheorem2a.p1.7.7.m7.1.1.2.2" xref="Thmtheorem2a.p1.7.7.m7.1.1.2.2.cmml">N</mi></mrow><mo id="Thmtheorem2a.p1.7.7.m7.1.1.1" xref="Thmtheorem2a.p1.7.7.m7.1.1.1.cmml">&gt;</mo><mn id="Thmtheorem2a.p1.7.7.m7.1.1.3" xref="Thmtheorem2a.p1.7.7.m7.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2a.p1.7.7.m7.1b"><apply id="Thmtheorem2a.p1.7.7.m7.1.1.cmml" xref="Thmtheorem2a.p1.7.7.m7.1.1"><gt id="Thmtheorem2a.p1.7.7.m7.1.1.1.cmml" xref="Thmtheorem2a.p1.7.7.m7.1.1.1"></gt><apply id="Thmtheorem2a.p1.7.7.m7.1.1.2.cmml" xref="Thmtheorem2a.p1.7.7.m7.1.1.2"><exists id="Thmtheorem2a.p1.7.7.m7.1.1.2.1.cmml" xref="Thmtheorem2a.p1.7.7.m7.1.1.2.1"></exists><ci id="Thmtheorem2a.p1.7.7.m7.1.1.2.2.cmml" xref="Thmtheorem2a.p1.7.7.m7.1.1.2.2">𝑁</ci></apply><cn type="integer" id="Thmtheorem2a.p1.7.7.m7.1.1.3.cmml" xref="Thmtheorem2a.p1.7.7.m7.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2a.p1.7.7.m7.1c">\exists N&gt;0</annotation></semantics></math> such that <math id="Thmtheorem2a.p1.8.8.m8.1" class="ltx_Math" alttext="\forall n&gt;N" display="inline"><semantics id="Thmtheorem2a.p1.8.8.m8.1a"><mrow id="Thmtheorem2a.p1.8.8.m8.1.1" xref="Thmtheorem2a.p1.8.8.m8.1.1.cmml"><mrow id="Thmtheorem2a.p1.8.8.m8.1.1.2" xref="Thmtheorem2a.p1.8.8.m8.1.1.2.cmml"><mo rspace="0.167em" id="Thmtheorem2a.p1.8.8.m8.1.1.2.1" xref="Thmtheorem2a.p1.8.8.m8.1.1.2.1.cmml">∀</mo><mi id="Thmtheorem2a.p1.8.8.m8.1.1.2.2" xref="Thmtheorem2a.p1.8.8.m8.1.1.2.2.cmml">n</mi></mrow><mo id="Thmtheorem2a.p1.8.8.m8.1.1.1" xref="Thmtheorem2a.p1.8.8.m8.1.1.1.cmml">&gt;</mo><mi id="Thmtheorem2a.p1.8.8.m8.1.1.3" xref="Thmtheorem2a.p1.8.8.m8.1.1.3.cmml">N</mi></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2a.p1.8.8.m8.1b"><apply id="Thmtheorem2a.p1.8.8.m8.1.1.cmml" xref="Thmtheorem2a.p1.8.8.m8.1.1"><gt id="Thmtheorem2a.p1.8.8.m8.1.1.1.cmml" xref="Thmtheorem2a.p1.8.8.m8.1.1.1"></gt><apply id="Thmtheorem2a.p1.8.8.m8.1.1.2.cmml" xref="Thmtheorem2a.p1.8.8.m8.1.1.2"><csymbol cd="latexml" id="Thmtheorem2a.p1.8.8.m8.1.1.2.1.cmml" xref="Thmtheorem2a.p1.8.8.m8.1.1.2.1">for-all</csymbol><ci id="Thmtheorem2a.p1.8.8.m8.1.1.2.2.cmml" xref="Thmtheorem2a.p1.8.8.m8.1.1.2.2">𝑛</ci></apply><ci id="Thmtheorem2a.p1.8.8.m8.1.1.3.cmml" xref="Thmtheorem2a.p1.8.8.m8.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2a.p1.8.8.m8.1c">\forall n&gt;N</annotation></semantics></math> we have</span></p>
<table id="A1.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E7.m1.2" class="ltx_Math" alttext="Z_{\theta}-\epsilon&lt;\mathbb{E}[T_{n}]&lt;Z_{\theta}&lt;\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]&lt;Z_{\theta}+\epsilon" display="block"><semantics id="A1.E7.m1.2a"><mrow id="A1.E7.m1.2.2" xref="A1.E7.m1.2.2.cmml"><mrow id="A1.E7.m1.2.2.4" xref="A1.E7.m1.2.2.4.cmml"><msub id="A1.E7.m1.2.2.4.2" xref="A1.E7.m1.2.2.4.2.cmml"><mi id="A1.E7.m1.2.2.4.2.2" xref="A1.E7.m1.2.2.4.2.2.cmml">Z</mi><mi id="A1.E7.m1.2.2.4.2.3" xref="A1.E7.m1.2.2.4.2.3.cmml">θ</mi></msub><mo id="A1.E7.m1.2.2.4.1" xref="A1.E7.m1.2.2.4.1.cmml">−</mo><mi id="A1.E7.m1.2.2.4.3" xref="A1.E7.m1.2.2.4.3.cmml">ϵ</mi></mrow><mo id="A1.E7.m1.2.2.5" xref="A1.E7.m1.2.2.5.cmml">&lt;</mo><mrow id="A1.E7.m1.1.1.1" xref="A1.E7.m1.1.1.1.cmml"><mi id="A1.E7.m1.1.1.1.3" xref="A1.E7.m1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.E7.m1.1.1.1.2" xref="A1.E7.m1.1.1.1.2.cmml">​</mo><mrow id="A1.E7.m1.1.1.1.1.1" xref="A1.E7.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E7.m1.1.1.1.1.1.2" xref="A1.E7.m1.1.1.1.1.2.1.cmml">[</mo><msub id="A1.E7.m1.1.1.1.1.1.1" xref="A1.E7.m1.1.1.1.1.1.1.cmml"><mi id="A1.E7.m1.1.1.1.1.1.1.2" xref="A1.E7.m1.1.1.1.1.1.1.2.cmml">T</mi><mi id="A1.E7.m1.1.1.1.1.1.1.3" xref="A1.E7.m1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.E7.m1.1.1.1.1.1.3" xref="A1.E7.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.E7.m1.2.2.6" xref="A1.E7.m1.2.2.6.cmml">&lt;</mo><msub id="A1.E7.m1.2.2.7" xref="A1.E7.m1.2.2.7.cmml"><mi id="A1.E7.m1.2.2.7.2" xref="A1.E7.m1.2.2.7.2.cmml">Z</mi><mi id="A1.E7.m1.2.2.7.3" xref="A1.E7.m1.2.2.7.3.cmml">θ</mi></msub><mo id="A1.E7.m1.2.2.8" xref="A1.E7.m1.2.2.8.cmml">&lt;</mo><mrow id="A1.E7.m1.2.2.2" xref="A1.E7.m1.2.2.2.cmml"><mi id="A1.E7.m1.2.2.2.3" xref="A1.E7.m1.2.2.2.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.E7.m1.2.2.2.2" xref="A1.E7.m1.2.2.2.2.cmml">​</mo><mrow id="A1.E7.m1.2.2.2.1.1" xref="A1.E7.m1.2.2.2.1.2.cmml"><mo stretchy="false" id="A1.E7.m1.2.2.2.1.1.2" xref="A1.E7.m1.2.2.2.1.2.1.cmml">[</mo><mrow id="A1.E7.m1.2.2.2.1.1.1" xref="A1.E7.m1.2.2.2.1.1.1.cmml"><mrow id="A1.E7.m1.2.2.2.1.1.1.1" xref="A1.E7.m1.2.2.2.1.1.1.1.cmml"><mrow id="A1.E7.m1.2.2.2.1.1.1.1.1.1" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E7.m1.2.2.2.1.1.1.1.1.1.2" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.cmml"><mrow id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.cmml"><mn id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.2" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.1" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.3" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.3.cmml">n</mi></mrow><mo id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.1" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.1.cmml">−</mo><mn id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.3" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.E7.m1.2.2.2.1.1.1.1.1.1.3" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.E7.m1.2.2.2.1.1.1.1.2" xref="A1.E7.m1.2.2.2.1.1.1.1.2.cmml">​</mo><msub id="A1.E7.m1.2.2.2.1.1.1.1.3" xref="A1.E7.m1.2.2.2.1.1.1.1.3.cmml"><mi id="A1.E7.m1.2.2.2.1.1.1.1.3.2" xref="A1.E7.m1.2.2.2.1.1.1.1.3.2.cmml">T</mi><mi id="A1.E7.m1.2.2.2.1.1.1.1.3.3" xref="A1.E7.m1.2.2.2.1.1.1.1.3.3.cmml">n</mi></msub></mrow><mo id="A1.E7.m1.2.2.2.1.1.1.3" xref="A1.E7.m1.2.2.2.1.1.1.3.cmml">−</mo><mrow id="A1.E7.m1.2.2.2.1.1.1.2" xref="A1.E7.m1.2.2.2.1.1.1.2.cmml"><mn id="A1.E7.m1.2.2.2.1.1.1.2.3" xref="A1.E7.m1.2.2.2.1.1.1.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.E7.m1.2.2.2.1.1.1.2.2" xref="A1.E7.m1.2.2.2.1.1.1.2.2.cmml">​</mo><mrow id="A1.E7.m1.2.2.2.1.1.1.2.1.1" xref="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="A1.E7.m1.2.2.2.1.1.1.2.1.1.2" xref="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.cmml">(</mo><mrow id="A1.E7.m1.2.2.2.1.1.1.2.1.1.1" xref="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.cmml"><mi id="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.2" xref="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.2.cmml">n</mi><mo id="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.1" xref="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.1.cmml">−</mo><mn id="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.3" xref="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.E7.m1.2.2.2.1.1.1.2.1.1.3" xref="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.E7.m1.2.2.2.1.1.1.2.2a" xref="A1.E7.m1.2.2.2.1.1.1.2.2.cmml">​</mo><msub id="A1.E7.m1.2.2.2.1.1.1.2.4" xref="A1.E7.m1.2.2.2.1.1.1.2.4.cmml"><mi id="A1.E7.m1.2.2.2.1.1.1.2.4.2" xref="A1.E7.m1.2.2.2.1.1.1.2.4.2.cmml">T</mi><mrow id="A1.E7.m1.2.2.2.1.1.1.2.4.3" xref="A1.E7.m1.2.2.2.1.1.1.2.4.3.cmml"><mi id="A1.E7.m1.2.2.2.1.1.1.2.4.3.2" xref="A1.E7.m1.2.2.2.1.1.1.2.4.3.2.cmml">n</mi><mo id="A1.E7.m1.2.2.2.1.1.1.2.4.3.1" xref="A1.E7.m1.2.2.2.1.1.1.2.4.3.1.cmml">−</mo><mn id="A1.E7.m1.2.2.2.1.1.1.2.4.3.3" xref="A1.E7.m1.2.2.2.1.1.1.2.4.3.3.cmml">1</mn></mrow></msub></mrow></mrow><mo stretchy="false" id="A1.E7.m1.2.2.2.1.1.3" xref="A1.E7.m1.2.2.2.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.E7.m1.2.2.9" xref="A1.E7.m1.2.2.9.cmml">&lt;</mo><mrow id="A1.E7.m1.2.2.10" xref="A1.E7.m1.2.2.10.cmml"><msub id="A1.E7.m1.2.2.10.2" xref="A1.E7.m1.2.2.10.2.cmml"><mi id="A1.E7.m1.2.2.10.2.2" xref="A1.E7.m1.2.2.10.2.2.cmml">Z</mi><mi id="A1.E7.m1.2.2.10.2.3" xref="A1.E7.m1.2.2.10.2.3.cmml">θ</mi></msub><mo id="A1.E7.m1.2.2.10.1" xref="A1.E7.m1.2.2.10.1.cmml">+</mo><mi id="A1.E7.m1.2.2.10.3" xref="A1.E7.m1.2.2.10.3.cmml">ϵ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E7.m1.2b"><apply id="A1.E7.m1.2.2.cmml" xref="A1.E7.m1.2.2"><and id="A1.E7.m1.2.2a.cmml" xref="A1.E7.m1.2.2"></and><apply id="A1.E7.m1.2.2b.cmml" xref="A1.E7.m1.2.2"><lt id="A1.E7.m1.2.2.5.cmml" xref="A1.E7.m1.2.2.5"></lt><apply id="A1.E7.m1.2.2.4.cmml" xref="A1.E7.m1.2.2.4"><minus id="A1.E7.m1.2.2.4.1.cmml" xref="A1.E7.m1.2.2.4.1"></minus><apply id="A1.E7.m1.2.2.4.2.cmml" xref="A1.E7.m1.2.2.4.2"><csymbol cd="ambiguous" id="A1.E7.m1.2.2.4.2.1.cmml" xref="A1.E7.m1.2.2.4.2">subscript</csymbol><ci id="A1.E7.m1.2.2.4.2.2.cmml" xref="A1.E7.m1.2.2.4.2.2">𝑍</ci><ci id="A1.E7.m1.2.2.4.2.3.cmml" xref="A1.E7.m1.2.2.4.2.3">𝜃</ci></apply><ci id="A1.E7.m1.2.2.4.3.cmml" xref="A1.E7.m1.2.2.4.3">italic-ϵ</ci></apply><apply id="A1.E7.m1.1.1.1.cmml" xref="A1.E7.m1.1.1.1"><times id="A1.E7.m1.1.1.1.2.cmml" xref="A1.E7.m1.1.1.1.2"></times><ci id="A1.E7.m1.1.1.1.3.cmml" xref="A1.E7.m1.1.1.1.3">𝔼</ci><apply id="A1.E7.m1.1.1.1.1.2.cmml" xref="A1.E7.m1.1.1.1.1.1"><csymbol cd="latexml" id="A1.E7.m1.1.1.1.1.2.1.cmml" xref="A1.E7.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.E7.m1.1.1.1.1.1.1.cmml" xref="A1.E7.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.E7.m1.1.1.1.1.1.1.1.cmml" xref="A1.E7.m1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.E7.m1.1.1.1.1.1.1.2.cmml" xref="A1.E7.m1.1.1.1.1.1.1.2">𝑇</ci><ci id="A1.E7.m1.1.1.1.1.1.1.3.cmml" xref="A1.E7.m1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></apply><apply id="A1.E7.m1.2.2c.cmml" xref="A1.E7.m1.2.2"><lt id="A1.E7.m1.2.2.6.cmml" xref="A1.E7.m1.2.2.6"></lt><share href="#A1.E7.m1.1.1.1.cmml" id="A1.E7.m1.2.2d.cmml" xref="A1.E7.m1.2.2"></share><apply id="A1.E7.m1.2.2.7.cmml" xref="A1.E7.m1.2.2.7"><csymbol cd="ambiguous" id="A1.E7.m1.2.2.7.1.cmml" xref="A1.E7.m1.2.2.7">subscript</csymbol><ci id="A1.E7.m1.2.2.7.2.cmml" xref="A1.E7.m1.2.2.7.2">𝑍</ci><ci id="A1.E7.m1.2.2.7.3.cmml" xref="A1.E7.m1.2.2.7.3">𝜃</ci></apply></apply><apply id="A1.E7.m1.2.2e.cmml" xref="A1.E7.m1.2.2"><lt id="A1.E7.m1.2.2.8.cmml" xref="A1.E7.m1.2.2.8"></lt><share href="#A1.E7.m1.2.2.7.cmml" id="A1.E7.m1.2.2f.cmml" xref="A1.E7.m1.2.2"></share><apply id="A1.E7.m1.2.2.2.cmml" xref="A1.E7.m1.2.2.2"><times id="A1.E7.m1.2.2.2.2.cmml" xref="A1.E7.m1.2.2.2.2"></times><ci id="A1.E7.m1.2.2.2.3.cmml" xref="A1.E7.m1.2.2.2.3">𝔼</ci><apply id="A1.E7.m1.2.2.2.1.2.cmml" xref="A1.E7.m1.2.2.2.1.1"><csymbol cd="latexml" id="A1.E7.m1.2.2.2.1.2.1.cmml" xref="A1.E7.m1.2.2.2.1.1.2">delimited-[]</csymbol><apply id="A1.E7.m1.2.2.2.1.1.1.cmml" xref="A1.E7.m1.2.2.2.1.1.1"><minus id="A1.E7.m1.2.2.2.1.1.1.3.cmml" xref="A1.E7.m1.2.2.2.1.1.1.3"></minus><apply id="A1.E7.m1.2.2.2.1.1.1.1.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1"><times id="A1.E7.m1.2.2.2.1.1.1.1.2.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.2"></times><apply id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1"><minus id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.1.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.1"></minus><apply id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2"><times id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.1.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.2.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.2">2</cn><ci id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.3.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.2.3">𝑛</ci></apply><cn type="integer" id="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.3.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.1.1.1.3">1</cn></apply><apply id="A1.E7.m1.2.2.2.1.1.1.1.3.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E7.m1.2.2.2.1.1.1.1.3.1.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.3">subscript</csymbol><ci id="A1.E7.m1.2.2.2.1.1.1.1.3.2.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.3.2">𝑇</ci><ci id="A1.E7.m1.2.2.2.1.1.1.1.3.3.cmml" xref="A1.E7.m1.2.2.2.1.1.1.1.3.3">𝑛</ci></apply></apply><apply id="A1.E7.m1.2.2.2.1.1.1.2.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2"><times id="A1.E7.m1.2.2.2.1.1.1.2.2.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.2"></times><cn type="integer" id="A1.E7.m1.2.2.2.1.1.1.2.3.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.3">2</cn><apply id="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.1.1"><minus id="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.1.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.1"></minus><ci id="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.2.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.2">𝑛</ci><cn type="integer" id="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.3.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.1.1.1.3">1</cn></apply><apply id="A1.E7.m1.2.2.2.1.1.1.2.4.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.4"><csymbol cd="ambiguous" id="A1.E7.m1.2.2.2.1.1.1.2.4.1.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.4">subscript</csymbol><ci id="A1.E7.m1.2.2.2.1.1.1.2.4.2.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.4.2">𝑇</ci><apply id="A1.E7.m1.2.2.2.1.1.1.2.4.3.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.4.3"><minus id="A1.E7.m1.2.2.2.1.1.1.2.4.3.1.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.4.3.1"></minus><ci id="A1.E7.m1.2.2.2.1.1.1.2.4.3.2.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.4.3.2">𝑛</ci><cn type="integer" id="A1.E7.m1.2.2.2.1.1.1.2.4.3.3.cmml" xref="A1.E7.m1.2.2.2.1.1.1.2.4.3.3">1</cn></apply></apply></apply></apply></apply></apply></apply><apply id="A1.E7.m1.2.2g.cmml" xref="A1.E7.m1.2.2"><lt id="A1.E7.m1.2.2.9.cmml" xref="A1.E7.m1.2.2.9"></lt><share href="#A1.E7.m1.2.2.2.cmml" id="A1.E7.m1.2.2h.cmml" xref="A1.E7.m1.2.2"></share><apply id="A1.E7.m1.2.2.10.cmml" xref="A1.E7.m1.2.2.10"><plus id="A1.E7.m1.2.2.10.1.cmml" xref="A1.E7.m1.2.2.10.1"></plus><apply id="A1.E7.m1.2.2.10.2.cmml" xref="A1.E7.m1.2.2.10.2"><csymbol cd="ambiguous" id="A1.E7.m1.2.2.10.2.1.cmml" xref="A1.E7.m1.2.2.10.2">subscript</csymbol><ci id="A1.E7.m1.2.2.10.2.2.cmml" xref="A1.E7.m1.2.2.10.2.2">𝑍</ci><ci id="A1.E7.m1.2.2.10.2.3.cmml" xref="A1.E7.m1.2.2.10.2.3">𝜃</ci></apply><ci id="A1.E7.m1.2.2.10.3.cmml" xref="A1.E7.m1.2.2.10.3">italic-ϵ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E7.m1.2c">Z_{\theta}-\epsilon&lt;\mathbb{E}[T_{n}]&lt;Z_{\theta}&lt;\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]&lt;Z_{\theta}+\epsilon</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
</div>
</div>
<div id="A1.SS2.3" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="A1.SS2.1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS2.1.p1.1" class="ltx_p">From <cite class="ltx_cite ltx_citemacro_citet">Nowozin (<a href="#bib.bib23" title="" class="ltx_ref">2018</a>)</cite> Eq. 35, we can write <math id="A1.SS2.1.p1.1.m1.1" class="ltx_Math" alttext="\mathbb{E}[T_{n}]" display="inline"><semantics id="A1.SS2.1.p1.1.m1.1a"><mrow id="A1.SS2.1.p1.1.m1.1.1" xref="A1.SS2.1.p1.1.m1.1.1.cmml"><mi id="A1.SS2.1.p1.1.m1.1.1.3" xref="A1.SS2.1.p1.1.m1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.1.m1.1.1.2" xref="A1.SS2.1.p1.1.m1.1.1.2.cmml">​</mo><mrow id="A1.SS2.1.p1.1.m1.1.1.1.1" xref="A1.SS2.1.p1.1.m1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.1.p1.1.m1.1.1.1.1.2" xref="A1.SS2.1.p1.1.m1.1.1.1.2.1.cmml">[</mo><msub id="A1.SS2.1.p1.1.m1.1.1.1.1.1" xref="A1.SS2.1.p1.1.m1.1.1.1.1.1.cmml"><mi id="A1.SS2.1.p1.1.m1.1.1.1.1.1.2" xref="A1.SS2.1.p1.1.m1.1.1.1.1.1.2.cmml">T</mi><mi id="A1.SS2.1.p1.1.m1.1.1.1.1.1.3" xref="A1.SS2.1.p1.1.m1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.SS2.1.p1.1.m1.1.1.1.1.3" xref="A1.SS2.1.p1.1.m1.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.1.m1.1b"><apply id="A1.SS2.1.p1.1.m1.1.1.cmml" xref="A1.SS2.1.p1.1.m1.1.1"><times id="A1.SS2.1.p1.1.m1.1.1.2.cmml" xref="A1.SS2.1.p1.1.m1.1.1.2"></times><ci id="A1.SS2.1.p1.1.m1.1.1.3.cmml" xref="A1.SS2.1.p1.1.m1.1.1.3">𝔼</ci><apply id="A1.SS2.1.p1.1.m1.1.1.1.2.cmml" xref="A1.SS2.1.p1.1.m1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.1.p1.1.m1.1.1.1.2.1.cmml" xref="A1.SS2.1.p1.1.m1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.1.p1.1.m1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.1.p1.1.m1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.1.m1.1.1.1.1.1">subscript</csymbol><ci id="A1.SS2.1.p1.1.m1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.1.m1.1.1.1.1.1.2">𝑇</ci><ci id="A1.SS2.1.p1.1.m1.1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.1.m1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.1.m1.1c">\mathbb{E}[T_{n}]</annotation></semantics></math> as</p>
<table id="A1.E8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E8.m1.68" class="ltx_Math" alttext="\mathbb{E}[T_{n}]=Z_{\theta}-\frac{\mu_{2}}{2\mu^{2}}\frac{1}{n}+\frac{1}{3\mu^{3}}\frac{\mu_{3}}{n^{2}}-\frac{1}{4\mu^{4}}(\frac{3}{n^{2}}\mu_{2}^{2}+\frac{1}{n^{3}}(\mu_{4}-3\mu_{2}^{2}))\\
+\frac{1}{5\mu^{5}}(\frac{10}{n^{3}}\mu_{3}\mu_{2}+\frac{1}{n^{4}}(\mu_{5}-10\mu_{3}\mu_{2}))+o(n^{-3})" display="block"><semantics id="A1.E8.m1.68a"><mtable displaystyle="true" rowspacing="0pt" id="A1.E8.m1.68.68.8" xref="A1.E8.m1.64.64.4.cmml"><mtr id="A1.E8.m1.68.68.8a" xref="A1.E8.m1.64.64.4.cmml"><mtd class="ltx_align_left" columnalign="left" id="A1.E8.m1.68.68.8b" xref="A1.E8.m1.64.64.4.cmml"><mrow id="A1.E8.m1.66.66.6.62.35.35" xref="A1.E8.m1.64.64.4.cmml"><mrow id="A1.E8.m1.65.65.5.61.34.34.34" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.1.1.1.1.1.1" xref="A1.E8.m1.1.1.1.1.1.1.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.E8.m1.65.65.5.61.34.34.34.2" xref="A1.E8.m1.64.64.4.cmml">​</mo><mrow id="A1.E8.m1.65.65.5.61.34.34.34.1.1" xref="A1.E8.m1.64.64.4.cmml"><mo stretchy="false" id="A1.E8.m1.2.2.2.2.2.2" xref="A1.E8.m1.64.64.4.cmml">[</mo><msub id="A1.E8.m1.65.65.5.61.34.34.34.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.3.3.3.3.3.3" xref="A1.E8.m1.3.3.3.3.3.3.cmml">T</mi><mi id="A1.E8.m1.4.4.4.4.4.4.1" xref="A1.E8.m1.4.4.4.4.4.4.1.cmml">n</mi></msub><mo stretchy="false" id="A1.E8.m1.5.5.5.5.5.5" xref="A1.E8.m1.64.64.4.cmml">]</mo></mrow></mrow><mo id="A1.E8.m1.6.6.6.6.6.6" xref="A1.E8.m1.6.6.6.6.6.6.cmml">=</mo><mrow id="A1.E8.m1.66.66.6.62.35.35.35" xref="A1.E8.m1.64.64.4.cmml"><mrow id="A1.E8.m1.66.66.6.62.35.35.35.2" xref="A1.E8.m1.64.64.4.cmml"><mrow id="A1.E8.m1.66.66.6.62.35.35.35.2.1" xref="A1.E8.m1.64.64.4.cmml"><msub id="A1.E8.m1.66.66.6.62.35.35.35.2.1.1" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.7.7.7.7.7.7" xref="A1.E8.m1.7.7.7.7.7.7.cmml">Z</mi><mi id="A1.E8.m1.8.8.8.8.8.8.1" xref="A1.E8.m1.8.8.8.8.8.8.1.cmml">θ</mi></msub><mo id="A1.E8.m1.9.9.9.9.9.9" xref="A1.E8.m1.9.9.9.9.9.9.cmml">−</mo><mrow id="A1.E8.m1.66.66.6.62.35.35.35.2.1.2" xref="A1.E8.m1.64.64.4.cmml"><mfrac id="A1.E8.m1.10.10.10.10.10.10" xref="A1.E8.m1.10.10.10.10.10.10.cmml"><msub id="A1.E8.m1.10.10.10.10.10.10.2" xref="A1.E8.m1.10.10.10.10.10.10.2.cmml"><mi id="A1.E8.m1.10.10.10.10.10.10.2.2" xref="A1.E8.m1.10.10.10.10.10.10.2.2.cmml">μ</mi><mn id="A1.E8.m1.10.10.10.10.10.10.2.3" xref="A1.E8.m1.10.10.10.10.10.10.2.3.cmml">2</mn></msub><mrow id="A1.E8.m1.10.10.10.10.10.10.3" xref="A1.E8.m1.10.10.10.10.10.10.3.cmml"><mn id="A1.E8.m1.10.10.10.10.10.10.3.2" xref="A1.E8.m1.10.10.10.10.10.10.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.E8.m1.10.10.10.10.10.10.3.1" xref="A1.E8.m1.10.10.10.10.10.10.3.1.cmml">​</mo><msup id="A1.E8.m1.10.10.10.10.10.10.3.3" xref="A1.E8.m1.10.10.10.10.10.10.3.3.cmml"><mi id="A1.E8.m1.10.10.10.10.10.10.3.3.2" xref="A1.E8.m1.10.10.10.10.10.10.3.3.2.cmml">μ</mi><mn id="A1.E8.m1.10.10.10.10.10.10.3.3.3" xref="A1.E8.m1.10.10.10.10.10.10.3.3.3.cmml">2</mn></msup></mrow></mfrac><mo lspace="0em" rspace="0em" id="A1.E8.m1.66.66.6.62.35.35.35.2.1.2.1" xref="A1.E8.m1.64.64.4.cmml">​</mo><mfrac id="A1.E8.m1.11.11.11.11.11.11" xref="A1.E8.m1.11.11.11.11.11.11.cmml"><mn id="A1.E8.m1.11.11.11.11.11.11.2" xref="A1.E8.m1.11.11.11.11.11.11.2.cmml">1</mn><mi id="A1.E8.m1.11.11.11.11.11.11.3" xref="A1.E8.m1.11.11.11.11.11.11.3.cmml">n</mi></mfrac></mrow></mrow><mo id="A1.E8.m1.12.12.12.12.12.12" xref="A1.E8.m1.12.12.12.12.12.12.cmml">+</mo><mrow id="A1.E8.m1.66.66.6.62.35.35.35.2.2" xref="A1.E8.m1.64.64.4.cmml"><mfrac id="A1.E8.m1.13.13.13.13.13.13" xref="A1.E8.m1.13.13.13.13.13.13.cmml"><mn id="A1.E8.m1.13.13.13.13.13.13.2" xref="A1.E8.m1.13.13.13.13.13.13.2.cmml">1</mn><mrow id="A1.E8.m1.13.13.13.13.13.13.3" xref="A1.E8.m1.13.13.13.13.13.13.3.cmml"><mn id="A1.E8.m1.13.13.13.13.13.13.3.2" xref="A1.E8.m1.13.13.13.13.13.13.3.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="A1.E8.m1.13.13.13.13.13.13.3.1" xref="A1.E8.m1.13.13.13.13.13.13.3.1.cmml">​</mo><msup id="A1.E8.m1.13.13.13.13.13.13.3.3" xref="A1.E8.m1.13.13.13.13.13.13.3.3.cmml"><mi id="A1.E8.m1.13.13.13.13.13.13.3.3.2" xref="A1.E8.m1.13.13.13.13.13.13.3.3.2.cmml">μ</mi><mn id="A1.E8.m1.13.13.13.13.13.13.3.3.3" xref="A1.E8.m1.13.13.13.13.13.13.3.3.3.cmml">3</mn></msup></mrow></mfrac><mo lspace="0em" rspace="0em" id="A1.E8.m1.66.66.6.62.35.35.35.2.2.1" xref="A1.E8.m1.64.64.4.cmml">​</mo><mfrac id="A1.E8.m1.14.14.14.14.14.14" xref="A1.E8.m1.14.14.14.14.14.14.cmml"><msub id="A1.E8.m1.14.14.14.14.14.14.2" xref="A1.E8.m1.14.14.14.14.14.14.2.cmml"><mi id="A1.E8.m1.14.14.14.14.14.14.2.2" xref="A1.E8.m1.14.14.14.14.14.14.2.2.cmml">μ</mi><mn id="A1.E8.m1.14.14.14.14.14.14.2.3" xref="A1.E8.m1.14.14.14.14.14.14.2.3.cmml">3</mn></msub><msup id="A1.E8.m1.14.14.14.14.14.14.3" xref="A1.E8.m1.14.14.14.14.14.14.3.cmml"><mi id="A1.E8.m1.14.14.14.14.14.14.3.2" xref="A1.E8.m1.14.14.14.14.14.14.3.2.cmml">n</mi><mn id="A1.E8.m1.14.14.14.14.14.14.3.3" xref="A1.E8.m1.14.14.14.14.14.14.3.3.cmml">2</mn></msup></mfrac></mrow></mrow><mo id="A1.E8.m1.15.15.15.15.15.15" xref="A1.E8.m1.15.15.15.15.15.15.cmml">−</mo><mrow id="A1.E8.m1.66.66.6.62.35.35.35.1" xref="A1.E8.m1.64.64.4.cmml"><mfrac id="A1.E8.m1.16.16.16.16.16.16" xref="A1.E8.m1.16.16.16.16.16.16.cmml"><mn id="A1.E8.m1.16.16.16.16.16.16.2" xref="A1.E8.m1.16.16.16.16.16.16.2.cmml">1</mn><mrow id="A1.E8.m1.16.16.16.16.16.16.3" xref="A1.E8.m1.16.16.16.16.16.16.3.cmml"><mn id="A1.E8.m1.16.16.16.16.16.16.3.2" xref="A1.E8.m1.16.16.16.16.16.16.3.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="A1.E8.m1.16.16.16.16.16.16.3.1" xref="A1.E8.m1.16.16.16.16.16.16.3.1.cmml">​</mo><msup id="A1.E8.m1.16.16.16.16.16.16.3.3" xref="A1.E8.m1.16.16.16.16.16.16.3.3.cmml"><mi id="A1.E8.m1.16.16.16.16.16.16.3.3.2" xref="A1.E8.m1.16.16.16.16.16.16.3.3.2.cmml">μ</mi><mn id="A1.E8.m1.16.16.16.16.16.16.3.3.3" xref="A1.E8.m1.16.16.16.16.16.16.3.3.3.cmml">4</mn></msup></mrow></mfrac><mo lspace="0em" rspace="0em" id="A1.E8.m1.66.66.6.62.35.35.35.1.2" xref="A1.E8.m1.64.64.4.cmml">​</mo><mrow id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mo stretchy="false" id="A1.E8.m1.17.17.17.17.17.17" xref="A1.E8.m1.64.64.4.cmml">(</mo><mrow id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mrow id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1.2" xref="A1.E8.m1.64.64.4.cmml"><mfrac id="A1.E8.m1.18.18.18.18.18.18" xref="A1.E8.m1.18.18.18.18.18.18.cmml"><mn id="A1.E8.m1.18.18.18.18.18.18.2" xref="A1.E8.m1.18.18.18.18.18.18.2.cmml">3</mn><msup id="A1.E8.m1.18.18.18.18.18.18.3" xref="A1.E8.m1.18.18.18.18.18.18.3.cmml"><mi id="A1.E8.m1.18.18.18.18.18.18.3.2" xref="A1.E8.m1.18.18.18.18.18.18.3.2.cmml">n</mi><mn id="A1.E8.m1.18.18.18.18.18.18.3.3" xref="A1.E8.m1.18.18.18.18.18.18.3.3.cmml">2</mn></msup></mfrac><mo lspace="0em" rspace="0em" id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1.2.1" xref="A1.E8.m1.64.64.4.cmml">​</mo><msubsup id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1.2.2" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.19.19.19.19.19.19" xref="A1.E8.m1.19.19.19.19.19.19.cmml">μ</mi><mn id="A1.E8.m1.20.20.20.20.20.20.1" xref="A1.E8.m1.20.20.20.20.20.20.1.cmml">2</mn><mn id="A1.E8.m1.21.21.21.21.21.21.1" xref="A1.E8.m1.21.21.21.21.21.21.1.cmml">2</mn></msubsup></mrow><mo id="A1.E8.m1.22.22.22.22.22.22" xref="A1.E8.m1.22.22.22.22.22.22.cmml">+</mo><mrow id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mfrac id="A1.E8.m1.23.23.23.23.23.23" xref="A1.E8.m1.23.23.23.23.23.23.cmml"><mn id="A1.E8.m1.23.23.23.23.23.23.2" xref="A1.E8.m1.23.23.23.23.23.23.2.cmml">1</mn><msup id="A1.E8.m1.23.23.23.23.23.23.3" xref="A1.E8.m1.23.23.23.23.23.23.3.cmml"><mi id="A1.E8.m1.23.23.23.23.23.23.3.2" xref="A1.E8.m1.23.23.23.23.23.23.3.2.cmml">n</mi><mn id="A1.E8.m1.23.23.23.23.23.23.3.3" xref="A1.E8.m1.23.23.23.23.23.23.3.3.cmml">3</mn></msup></mfrac><mo lspace="0em" rspace="0em" id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1.1.2" xref="A1.E8.m1.64.64.4.cmml">​</mo><mrow id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mo stretchy="false" id="A1.E8.m1.24.24.24.24.24.24" xref="A1.E8.m1.64.64.4.cmml">(</mo><mrow id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1.1.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><msub id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1.1.1.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.25.25.25.25.25.25" xref="A1.E8.m1.25.25.25.25.25.25.cmml">μ</mi><mn id="A1.E8.m1.26.26.26.26.26.26.1" xref="A1.E8.m1.26.26.26.26.26.26.1.cmml">4</mn></msub><mo id="A1.E8.m1.27.27.27.27.27.27" xref="A1.E8.m1.27.27.27.27.27.27.cmml">−</mo><mrow id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1.1.1.1.1.2" xref="A1.E8.m1.64.64.4.cmml"><mn id="A1.E8.m1.28.28.28.28.28.28" xref="A1.E8.m1.28.28.28.28.28.28.cmml">3</mn><mo lspace="0em" rspace="0em" id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1.1.1.1.1.2.1" xref="A1.E8.m1.64.64.4.cmml">​</mo><msubsup id="A1.E8.m1.66.66.6.62.35.35.35.1.1.1.1.1.1.1.1.2.2" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.29.29.29.29.29.29" xref="A1.E8.m1.29.29.29.29.29.29.cmml">μ</mi><mn id="A1.E8.m1.30.30.30.30.30.30.1" xref="A1.E8.m1.30.30.30.30.30.30.1.cmml">2</mn><mn id="A1.E8.m1.31.31.31.31.31.31.1" xref="A1.E8.m1.31.31.31.31.31.31.1.cmml">2</mn></msubsup></mrow></mrow><mo stretchy="false" id="A1.E8.m1.32.32.32.32.32.32" xref="A1.E8.m1.64.64.4.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="A1.E8.m1.33.33.33.33.33.33" xref="A1.E8.m1.64.64.4.cmml">)</mo></mrow></mrow></mrow></mrow></mtd></mtr><mtr id="A1.E8.m1.68.68.8c" xref="A1.E8.m1.64.64.4.cmml"><mtd class="ltx_align_right" columnalign="right" id="A1.E8.m1.68.68.8d" xref="A1.E8.m1.64.64.4.cmml"><mrow id="A1.E8.m1.68.68.8.64.29.29" xref="A1.E8.m1.64.64.4.cmml"><mrow id="A1.E8.m1.67.67.7.63.28.28.28" xref="A1.E8.m1.64.64.4.cmml"><mo id="A1.E8.m1.67.67.7.63.28.28.28a" xref="A1.E8.m1.64.64.4.cmml">+</mo><mrow id="A1.E8.m1.67.67.7.63.28.28.28.1" xref="A1.E8.m1.64.64.4.cmml"><mfrac id="A1.E8.m1.35.35.35.2.2.2" xref="A1.E8.m1.35.35.35.2.2.2.cmml"><mn id="A1.E8.m1.35.35.35.2.2.2.2" xref="A1.E8.m1.35.35.35.2.2.2.2.cmml">1</mn><mrow id="A1.E8.m1.35.35.35.2.2.2.3" xref="A1.E8.m1.35.35.35.2.2.2.3.cmml"><mn id="A1.E8.m1.35.35.35.2.2.2.3.2" xref="A1.E8.m1.35.35.35.2.2.2.3.2.cmml">5</mn><mo lspace="0em" rspace="0em" id="A1.E8.m1.35.35.35.2.2.2.3.1" xref="A1.E8.m1.35.35.35.2.2.2.3.1.cmml">​</mo><msup id="A1.E8.m1.35.35.35.2.2.2.3.3" xref="A1.E8.m1.35.35.35.2.2.2.3.3.cmml"><mi id="A1.E8.m1.35.35.35.2.2.2.3.3.2" xref="A1.E8.m1.35.35.35.2.2.2.3.3.2.cmml">μ</mi><mn id="A1.E8.m1.35.35.35.2.2.2.3.3.3" xref="A1.E8.m1.35.35.35.2.2.2.3.3.3.cmml">5</mn></msup></mrow></mfrac><mo lspace="0em" rspace="0em" id="A1.E8.m1.67.67.7.63.28.28.28.1.2" xref="A1.E8.m1.64.64.4.cmml">​</mo><mrow id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mo stretchy="false" id="A1.E8.m1.36.36.36.3.3.3" xref="A1.E8.m1.64.64.4.cmml">(</mo><mrow id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mrow id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.2" xref="A1.E8.m1.64.64.4.cmml"><mfrac id="A1.E8.m1.37.37.37.4.4.4" xref="A1.E8.m1.37.37.37.4.4.4.cmml"><mn id="A1.E8.m1.37.37.37.4.4.4.2" xref="A1.E8.m1.37.37.37.4.4.4.2.cmml">10</mn><msup id="A1.E8.m1.37.37.37.4.4.4.3" xref="A1.E8.m1.37.37.37.4.4.4.3.cmml"><mi id="A1.E8.m1.37.37.37.4.4.4.3.2" xref="A1.E8.m1.37.37.37.4.4.4.3.2.cmml">n</mi><mn id="A1.E8.m1.37.37.37.4.4.4.3.3" xref="A1.E8.m1.37.37.37.4.4.4.3.3.cmml">3</mn></msup></mfrac><mo lspace="0em" rspace="0em" id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.2.1" xref="A1.E8.m1.64.64.4.cmml">​</mo><msub id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.2.2" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.38.38.38.5.5.5" xref="A1.E8.m1.38.38.38.5.5.5.cmml">μ</mi><mn id="A1.E8.m1.39.39.39.6.6.6.1" xref="A1.E8.m1.39.39.39.6.6.6.1.cmml">3</mn></msub><mo lspace="0em" rspace="0em" id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.2.1a" xref="A1.E8.m1.64.64.4.cmml">​</mo><msub id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.2.3" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.40.40.40.7.7.7" xref="A1.E8.m1.40.40.40.7.7.7.cmml">μ</mi><mn id="A1.E8.m1.41.41.41.8.8.8.1" xref="A1.E8.m1.41.41.41.8.8.8.1.cmml">2</mn></msub></mrow><mo id="A1.E8.m1.42.42.42.9.9.9" xref="A1.E8.m1.42.42.42.9.9.9.cmml">+</mo><mrow id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mfrac id="A1.E8.m1.43.43.43.10.10.10" xref="A1.E8.m1.43.43.43.10.10.10.cmml"><mn id="A1.E8.m1.43.43.43.10.10.10.2" xref="A1.E8.m1.43.43.43.10.10.10.2.cmml">1</mn><msup id="A1.E8.m1.43.43.43.10.10.10.3" xref="A1.E8.m1.43.43.43.10.10.10.3.cmml"><mi id="A1.E8.m1.43.43.43.10.10.10.3.2" xref="A1.E8.m1.43.43.43.10.10.10.3.2.cmml">n</mi><mn id="A1.E8.m1.43.43.43.10.10.10.3.3" xref="A1.E8.m1.43.43.43.10.10.10.3.3.cmml">4</mn></msup></mfrac><mo lspace="0em" rspace="0em" id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.1.2" xref="A1.E8.m1.64.64.4.cmml">​</mo><mrow id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mo stretchy="false" id="A1.E8.m1.44.44.44.11.11.11" xref="A1.E8.m1.64.64.4.cmml">(</mo><mrow id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.1.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><msub id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.1.1.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.45.45.45.12.12.12" xref="A1.E8.m1.45.45.45.12.12.12.cmml">μ</mi><mn id="A1.E8.m1.46.46.46.13.13.13.1" xref="A1.E8.m1.46.46.46.13.13.13.1.cmml">5</mn></msub><mo id="A1.E8.m1.47.47.47.14.14.14" xref="A1.E8.m1.47.47.47.14.14.14.cmml">−</mo><mrow id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.1.1.1.1.2" xref="A1.E8.m1.64.64.4.cmml"><mn id="A1.E8.m1.48.48.48.15.15.15" xref="A1.E8.m1.48.48.48.15.15.15.cmml">10</mn><mo lspace="0em" rspace="0em" id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.1.1.1.1.2.1" xref="A1.E8.m1.64.64.4.cmml">​</mo><msub id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.1.1.1.1.2.2" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.49.49.49.16.16.16" xref="A1.E8.m1.49.49.49.16.16.16.cmml">μ</mi><mn id="A1.E8.m1.50.50.50.17.17.17.1" xref="A1.E8.m1.50.50.50.17.17.17.1.cmml">3</mn></msub><mo lspace="0em" rspace="0em" id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.1.1.1.1.2.1a" xref="A1.E8.m1.64.64.4.cmml">​</mo><msub id="A1.E8.m1.67.67.7.63.28.28.28.1.1.1.1.1.1.1.1.2.3" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.51.51.51.18.18.18" xref="A1.E8.m1.51.51.51.18.18.18.cmml">μ</mi><mn id="A1.E8.m1.52.52.52.19.19.19.1" xref="A1.E8.m1.52.52.52.19.19.19.1.cmml">2</mn></msub></mrow></mrow><mo stretchy="false" id="A1.E8.m1.53.53.53.20.20.20" xref="A1.E8.m1.64.64.4.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="A1.E8.m1.54.54.54.21.21.21" xref="A1.E8.m1.64.64.4.cmml">)</mo></mrow></mrow></mrow><mo id="A1.E8.m1.55.55.55.22.22.22" xref="A1.E8.m1.64.64.4.cmml">+</mo><mrow id="A1.E8.m1.68.68.8.64.29.29.29" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.56.56.56.23.23.23" xref="A1.E8.m1.56.56.56.23.23.23.cmml">o</mi><mo lspace="0em" rspace="0em" id="A1.E8.m1.68.68.8.64.29.29.29.2" xref="A1.E8.m1.64.64.4.cmml">​</mo><mrow id="A1.E8.m1.68.68.8.64.29.29.29.1.1" xref="A1.E8.m1.64.64.4.cmml"><mo stretchy="false" id="A1.E8.m1.57.57.57.24.24.24" xref="A1.E8.m1.64.64.4.cmml">(</mo><msup id="A1.E8.m1.68.68.8.64.29.29.29.1.1.1" xref="A1.E8.m1.64.64.4.cmml"><mi id="A1.E8.m1.58.58.58.25.25.25" xref="A1.E8.m1.58.58.58.25.25.25.cmml">n</mi><mrow id="A1.E8.m1.59.59.59.26.26.26.1" xref="A1.E8.m1.59.59.59.26.26.26.1.cmml"><mo id="A1.E8.m1.59.59.59.26.26.26.1a" xref="A1.E8.m1.59.59.59.26.26.26.1.cmml">−</mo><mn id="A1.E8.m1.59.59.59.26.26.26.1.2" xref="A1.E8.m1.59.59.59.26.26.26.1.2.cmml">3</mn></mrow></msup><mo stretchy="false" id="A1.E8.m1.60.60.60.27.27.27" xref="A1.E8.m1.64.64.4.cmml">)</mo></mrow></mrow></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="A1.E8.m1.68b"><apply id="A1.E8.m1.64.64.4.cmml" xref="A1.E8.m1.68.68.8"><eq id="A1.E8.m1.6.6.6.6.6.6.cmml" xref="A1.E8.m1.6.6.6.6.6.6"></eq><apply id="A1.E8.m1.61.61.1.1.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.61.61.1.1.2.cmml" xref="A1.E8.m1.68.68.8"></times><ci id="A1.E8.m1.1.1.1.1.1.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1">𝔼</ci><apply id="A1.E8.m1.61.61.1.1.1.2.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="latexml" id="A1.E8.m1.61.61.1.1.1.2.1.cmml" xref="A1.E8.m1.68.68.8">delimited-[]</csymbol><apply id="A1.E8.m1.61.61.1.1.1.1.1.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.61.61.1.1.1.1.1.1.cmml" xref="A1.E8.m1.68.68.8">subscript</csymbol><ci id="A1.E8.m1.3.3.3.3.3.3.cmml" xref="A1.E8.m1.3.3.3.3.3.3">𝑇</ci><ci id="A1.E8.m1.4.4.4.4.4.4.1.cmml" xref="A1.E8.m1.4.4.4.4.4.4.1">𝑛</ci></apply></apply></apply><apply id="A1.E8.m1.64.64.4.4.cmml" xref="A1.E8.m1.68.68.8"><plus id="A1.E8.m1.34.34.34.1.1.1.cmml" xref="A1.E8.m1.68.68.8"></plus><apply id="A1.E8.m1.62.62.2.2.1.cmml" xref="A1.E8.m1.68.68.8"><minus id="A1.E8.m1.15.15.15.15.15.15.cmml" xref="A1.E8.m1.15.15.15.15.15.15"></minus><apply id="A1.E8.m1.62.62.2.2.1.3.cmml" xref="A1.E8.m1.68.68.8"><plus id="A1.E8.m1.12.12.12.12.12.12.cmml" xref="A1.E8.m1.12.12.12.12.12.12"></plus><apply id="A1.E8.m1.62.62.2.2.1.3.2.cmml" xref="A1.E8.m1.68.68.8"><minus id="A1.E8.m1.9.9.9.9.9.9.cmml" xref="A1.E8.m1.9.9.9.9.9.9"></minus><apply id="A1.E8.m1.62.62.2.2.1.3.2.2.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.62.62.2.2.1.3.2.2.1.cmml" xref="A1.E8.m1.68.68.8">subscript</csymbol><ci id="A1.E8.m1.7.7.7.7.7.7.cmml" xref="A1.E8.m1.7.7.7.7.7.7">𝑍</ci><ci id="A1.E8.m1.8.8.8.8.8.8.1.cmml" xref="A1.E8.m1.8.8.8.8.8.8.1">𝜃</ci></apply><apply id="A1.E8.m1.62.62.2.2.1.3.2.3.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.62.62.2.2.1.3.2.3.1.cmml" xref="A1.E8.m1.68.68.8"></times><apply id="A1.E8.m1.10.10.10.10.10.10.cmml" xref="A1.E8.m1.10.10.10.10.10.10"><divide id="A1.E8.m1.10.10.10.10.10.10.1.cmml" xref="A1.E8.m1.10.10.10.10.10.10"></divide><apply id="A1.E8.m1.10.10.10.10.10.10.2.cmml" xref="A1.E8.m1.10.10.10.10.10.10.2"><csymbol cd="ambiguous" id="A1.E8.m1.10.10.10.10.10.10.2.1.cmml" xref="A1.E8.m1.10.10.10.10.10.10.2">subscript</csymbol><ci id="A1.E8.m1.10.10.10.10.10.10.2.2.cmml" xref="A1.E8.m1.10.10.10.10.10.10.2.2">𝜇</ci><cn type="integer" id="A1.E8.m1.10.10.10.10.10.10.2.3.cmml" xref="A1.E8.m1.10.10.10.10.10.10.2.3">2</cn></apply><apply id="A1.E8.m1.10.10.10.10.10.10.3.cmml" xref="A1.E8.m1.10.10.10.10.10.10.3"><times id="A1.E8.m1.10.10.10.10.10.10.3.1.cmml" xref="A1.E8.m1.10.10.10.10.10.10.3.1"></times><cn type="integer" id="A1.E8.m1.10.10.10.10.10.10.3.2.cmml" xref="A1.E8.m1.10.10.10.10.10.10.3.2">2</cn><apply id="A1.E8.m1.10.10.10.10.10.10.3.3.cmml" xref="A1.E8.m1.10.10.10.10.10.10.3.3"><csymbol cd="ambiguous" id="A1.E8.m1.10.10.10.10.10.10.3.3.1.cmml" xref="A1.E8.m1.10.10.10.10.10.10.3.3">superscript</csymbol><ci id="A1.E8.m1.10.10.10.10.10.10.3.3.2.cmml" xref="A1.E8.m1.10.10.10.10.10.10.3.3.2">𝜇</ci><cn type="integer" id="A1.E8.m1.10.10.10.10.10.10.3.3.3.cmml" xref="A1.E8.m1.10.10.10.10.10.10.3.3.3">2</cn></apply></apply></apply><apply id="A1.E8.m1.11.11.11.11.11.11.cmml" xref="A1.E8.m1.11.11.11.11.11.11"><divide id="A1.E8.m1.11.11.11.11.11.11.1.cmml" xref="A1.E8.m1.11.11.11.11.11.11"></divide><cn type="integer" id="A1.E8.m1.11.11.11.11.11.11.2.cmml" xref="A1.E8.m1.11.11.11.11.11.11.2">1</cn><ci id="A1.E8.m1.11.11.11.11.11.11.3.cmml" xref="A1.E8.m1.11.11.11.11.11.11.3">𝑛</ci></apply></apply></apply><apply id="A1.E8.m1.62.62.2.2.1.3.3.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.62.62.2.2.1.3.3.1.cmml" xref="A1.E8.m1.68.68.8"></times><apply id="A1.E8.m1.13.13.13.13.13.13.cmml" xref="A1.E8.m1.13.13.13.13.13.13"><divide id="A1.E8.m1.13.13.13.13.13.13.1.cmml" xref="A1.E8.m1.13.13.13.13.13.13"></divide><cn type="integer" id="A1.E8.m1.13.13.13.13.13.13.2.cmml" xref="A1.E8.m1.13.13.13.13.13.13.2">1</cn><apply id="A1.E8.m1.13.13.13.13.13.13.3.cmml" xref="A1.E8.m1.13.13.13.13.13.13.3"><times id="A1.E8.m1.13.13.13.13.13.13.3.1.cmml" xref="A1.E8.m1.13.13.13.13.13.13.3.1"></times><cn type="integer" id="A1.E8.m1.13.13.13.13.13.13.3.2.cmml" xref="A1.E8.m1.13.13.13.13.13.13.3.2">3</cn><apply id="A1.E8.m1.13.13.13.13.13.13.3.3.cmml" xref="A1.E8.m1.13.13.13.13.13.13.3.3"><csymbol cd="ambiguous" id="A1.E8.m1.13.13.13.13.13.13.3.3.1.cmml" xref="A1.E8.m1.13.13.13.13.13.13.3.3">superscript</csymbol><ci id="A1.E8.m1.13.13.13.13.13.13.3.3.2.cmml" xref="A1.E8.m1.13.13.13.13.13.13.3.3.2">𝜇</ci><cn type="integer" id="A1.E8.m1.13.13.13.13.13.13.3.3.3.cmml" xref="A1.E8.m1.13.13.13.13.13.13.3.3.3">3</cn></apply></apply></apply><apply id="A1.E8.m1.14.14.14.14.14.14.cmml" xref="A1.E8.m1.14.14.14.14.14.14"><divide id="A1.E8.m1.14.14.14.14.14.14.1.cmml" xref="A1.E8.m1.14.14.14.14.14.14"></divide><apply id="A1.E8.m1.14.14.14.14.14.14.2.cmml" xref="A1.E8.m1.14.14.14.14.14.14.2"><csymbol cd="ambiguous" id="A1.E8.m1.14.14.14.14.14.14.2.1.cmml" xref="A1.E8.m1.14.14.14.14.14.14.2">subscript</csymbol><ci id="A1.E8.m1.14.14.14.14.14.14.2.2.cmml" xref="A1.E8.m1.14.14.14.14.14.14.2.2">𝜇</ci><cn type="integer" id="A1.E8.m1.14.14.14.14.14.14.2.3.cmml" xref="A1.E8.m1.14.14.14.14.14.14.2.3">3</cn></apply><apply id="A1.E8.m1.14.14.14.14.14.14.3.cmml" xref="A1.E8.m1.14.14.14.14.14.14.3"><csymbol cd="ambiguous" id="A1.E8.m1.14.14.14.14.14.14.3.1.cmml" xref="A1.E8.m1.14.14.14.14.14.14.3">superscript</csymbol><ci id="A1.E8.m1.14.14.14.14.14.14.3.2.cmml" xref="A1.E8.m1.14.14.14.14.14.14.3.2">𝑛</ci><cn type="integer" id="A1.E8.m1.14.14.14.14.14.14.3.3.cmml" xref="A1.E8.m1.14.14.14.14.14.14.3.3">2</cn></apply></apply></apply></apply><apply id="A1.E8.m1.62.62.2.2.1.1.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.62.62.2.2.1.1.2.cmml" xref="A1.E8.m1.68.68.8"></times><apply id="A1.E8.m1.16.16.16.16.16.16.cmml" xref="A1.E8.m1.16.16.16.16.16.16"><divide id="A1.E8.m1.16.16.16.16.16.16.1.cmml" xref="A1.E8.m1.16.16.16.16.16.16"></divide><cn type="integer" id="A1.E8.m1.16.16.16.16.16.16.2.cmml" xref="A1.E8.m1.16.16.16.16.16.16.2">1</cn><apply id="A1.E8.m1.16.16.16.16.16.16.3.cmml" xref="A1.E8.m1.16.16.16.16.16.16.3"><times id="A1.E8.m1.16.16.16.16.16.16.3.1.cmml" xref="A1.E8.m1.16.16.16.16.16.16.3.1"></times><cn type="integer" id="A1.E8.m1.16.16.16.16.16.16.3.2.cmml" xref="A1.E8.m1.16.16.16.16.16.16.3.2">4</cn><apply id="A1.E8.m1.16.16.16.16.16.16.3.3.cmml" xref="A1.E8.m1.16.16.16.16.16.16.3.3"><csymbol cd="ambiguous" id="A1.E8.m1.16.16.16.16.16.16.3.3.1.cmml" xref="A1.E8.m1.16.16.16.16.16.16.3.3">superscript</csymbol><ci id="A1.E8.m1.16.16.16.16.16.16.3.3.2.cmml" xref="A1.E8.m1.16.16.16.16.16.16.3.3.2">𝜇</ci><cn type="integer" id="A1.E8.m1.16.16.16.16.16.16.3.3.3.cmml" xref="A1.E8.m1.16.16.16.16.16.16.3.3.3">4</cn></apply></apply></apply><apply id="A1.E8.m1.62.62.2.2.1.1.1.1.1.cmml" xref="A1.E8.m1.68.68.8"><plus id="A1.E8.m1.22.22.22.22.22.22.cmml" xref="A1.E8.m1.22.22.22.22.22.22"></plus><apply id="A1.E8.m1.62.62.2.2.1.1.1.1.1.3.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.62.62.2.2.1.1.1.1.1.3.1.cmml" xref="A1.E8.m1.68.68.8"></times><apply id="A1.E8.m1.18.18.18.18.18.18.cmml" xref="A1.E8.m1.18.18.18.18.18.18"><divide id="A1.E8.m1.18.18.18.18.18.18.1.cmml" xref="A1.E8.m1.18.18.18.18.18.18"></divide><cn type="integer" id="A1.E8.m1.18.18.18.18.18.18.2.cmml" xref="A1.E8.m1.18.18.18.18.18.18.2">3</cn><apply id="A1.E8.m1.18.18.18.18.18.18.3.cmml" xref="A1.E8.m1.18.18.18.18.18.18.3"><csymbol cd="ambiguous" id="A1.E8.m1.18.18.18.18.18.18.3.1.cmml" xref="A1.E8.m1.18.18.18.18.18.18.3">superscript</csymbol><ci id="A1.E8.m1.18.18.18.18.18.18.3.2.cmml" xref="A1.E8.m1.18.18.18.18.18.18.3.2">𝑛</ci><cn type="integer" id="A1.E8.m1.18.18.18.18.18.18.3.3.cmml" xref="A1.E8.m1.18.18.18.18.18.18.3.3">2</cn></apply></apply><apply id="A1.E8.m1.62.62.2.2.1.1.1.1.1.3.3.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.62.62.2.2.1.1.1.1.1.3.3.1.cmml" xref="A1.E8.m1.68.68.8">superscript</csymbol><apply id="A1.E8.m1.62.62.2.2.1.1.1.1.1.3.3.2.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.62.62.2.2.1.1.1.1.1.3.3.2.1.cmml" xref="A1.E8.m1.68.68.8">subscript</csymbol><ci id="A1.E8.m1.19.19.19.19.19.19.cmml" xref="A1.E8.m1.19.19.19.19.19.19">𝜇</ci><cn type="integer" id="A1.E8.m1.20.20.20.20.20.20.1.cmml" xref="A1.E8.m1.20.20.20.20.20.20.1">2</cn></apply><cn type="integer" id="A1.E8.m1.21.21.21.21.21.21.1.cmml" xref="A1.E8.m1.21.21.21.21.21.21.1">2</cn></apply></apply><apply id="A1.E8.m1.62.62.2.2.1.1.1.1.1.1.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.62.62.2.2.1.1.1.1.1.1.2.cmml" xref="A1.E8.m1.68.68.8"></times><apply id="A1.E8.m1.23.23.23.23.23.23.cmml" xref="A1.E8.m1.23.23.23.23.23.23"><divide id="A1.E8.m1.23.23.23.23.23.23.1.cmml" xref="A1.E8.m1.23.23.23.23.23.23"></divide><cn type="integer" id="A1.E8.m1.23.23.23.23.23.23.2.cmml" xref="A1.E8.m1.23.23.23.23.23.23.2">1</cn><apply id="A1.E8.m1.23.23.23.23.23.23.3.cmml" xref="A1.E8.m1.23.23.23.23.23.23.3"><csymbol cd="ambiguous" id="A1.E8.m1.23.23.23.23.23.23.3.1.cmml" xref="A1.E8.m1.23.23.23.23.23.23.3">superscript</csymbol><ci id="A1.E8.m1.23.23.23.23.23.23.3.2.cmml" xref="A1.E8.m1.23.23.23.23.23.23.3.2">𝑛</ci><cn type="integer" id="A1.E8.m1.23.23.23.23.23.23.3.3.cmml" xref="A1.E8.m1.23.23.23.23.23.23.3.3">3</cn></apply></apply><apply id="A1.E8.m1.62.62.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E8.m1.68.68.8"><minus id="A1.E8.m1.27.27.27.27.27.27.cmml" xref="A1.E8.m1.27.27.27.27.27.27"></minus><apply id="A1.E8.m1.62.62.2.2.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.62.62.2.2.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.E8.m1.68.68.8">subscript</csymbol><ci id="A1.E8.m1.25.25.25.25.25.25.cmml" xref="A1.E8.m1.25.25.25.25.25.25">𝜇</ci><cn type="integer" id="A1.E8.m1.26.26.26.26.26.26.1.cmml" xref="A1.E8.m1.26.26.26.26.26.26.1">4</cn></apply><apply id="A1.E8.m1.62.62.2.2.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.62.62.2.2.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="A1.E8.m1.68.68.8"></times><cn type="integer" id="A1.E8.m1.28.28.28.28.28.28.cmml" xref="A1.E8.m1.28.28.28.28.28.28">3</cn><apply id="A1.E8.m1.62.62.2.2.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.62.62.2.2.1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="A1.E8.m1.68.68.8">superscript</csymbol><apply id="A1.E8.m1.62.62.2.2.1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.62.62.2.2.1.1.1.1.1.1.1.1.1.3.3.2.1.cmml" xref="A1.E8.m1.68.68.8">subscript</csymbol><ci id="A1.E8.m1.29.29.29.29.29.29.cmml" xref="A1.E8.m1.29.29.29.29.29.29">𝜇</ci><cn type="integer" id="A1.E8.m1.30.30.30.30.30.30.1.cmml" xref="A1.E8.m1.30.30.30.30.30.30.1">2</cn></apply><cn type="integer" id="A1.E8.m1.31.31.31.31.31.31.1.cmml" xref="A1.E8.m1.31.31.31.31.31.31.1">2</cn></apply></apply></apply></apply></apply></apply></apply><apply id="A1.E8.m1.63.63.3.3.2.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.63.63.3.3.2.2.cmml" xref="A1.E8.m1.68.68.8"></times><apply id="A1.E8.m1.35.35.35.2.2.2.cmml" xref="A1.E8.m1.35.35.35.2.2.2"><divide id="A1.E8.m1.35.35.35.2.2.2.1.cmml" xref="A1.E8.m1.35.35.35.2.2.2"></divide><cn type="integer" id="A1.E8.m1.35.35.35.2.2.2.2.cmml" xref="A1.E8.m1.35.35.35.2.2.2.2">1</cn><apply id="A1.E8.m1.35.35.35.2.2.2.3.cmml" xref="A1.E8.m1.35.35.35.2.2.2.3"><times id="A1.E8.m1.35.35.35.2.2.2.3.1.cmml" xref="A1.E8.m1.35.35.35.2.2.2.3.1"></times><cn type="integer" id="A1.E8.m1.35.35.35.2.2.2.3.2.cmml" xref="A1.E8.m1.35.35.35.2.2.2.3.2">5</cn><apply id="A1.E8.m1.35.35.35.2.2.2.3.3.cmml" xref="A1.E8.m1.35.35.35.2.2.2.3.3"><csymbol cd="ambiguous" id="A1.E8.m1.35.35.35.2.2.2.3.3.1.cmml" xref="A1.E8.m1.35.35.35.2.2.2.3.3">superscript</csymbol><ci id="A1.E8.m1.35.35.35.2.2.2.3.3.2.cmml" xref="A1.E8.m1.35.35.35.2.2.2.3.3.2">𝜇</ci><cn type="integer" id="A1.E8.m1.35.35.35.2.2.2.3.3.3.cmml" xref="A1.E8.m1.35.35.35.2.2.2.3.3.3">5</cn></apply></apply></apply><apply id="A1.E8.m1.63.63.3.3.2.1.1.1.cmml" xref="A1.E8.m1.68.68.8"><plus id="A1.E8.m1.42.42.42.9.9.9.cmml" xref="A1.E8.m1.42.42.42.9.9.9"></plus><apply id="A1.E8.m1.63.63.3.3.2.1.1.1.3.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.63.63.3.3.2.1.1.1.3.1.cmml" xref="A1.E8.m1.68.68.8"></times><apply id="A1.E8.m1.37.37.37.4.4.4.cmml" xref="A1.E8.m1.37.37.37.4.4.4"><divide id="A1.E8.m1.37.37.37.4.4.4.1.cmml" xref="A1.E8.m1.37.37.37.4.4.4"></divide><cn type="integer" id="A1.E8.m1.37.37.37.4.4.4.2.cmml" xref="A1.E8.m1.37.37.37.4.4.4.2">10</cn><apply id="A1.E8.m1.37.37.37.4.4.4.3.cmml" xref="A1.E8.m1.37.37.37.4.4.4.3"><csymbol cd="ambiguous" id="A1.E8.m1.37.37.37.4.4.4.3.1.cmml" xref="A1.E8.m1.37.37.37.4.4.4.3">superscript</csymbol><ci id="A1.E8.m1.37.37.37.4.4.4.3.2.cmml" xref="A1.E8.m1.37.37.37.4.4.4.3.2">𝑛</ci><cn type="integer" id="A1.E8.m1.37.37.37.4.4.4.3.3.cmml" xref="A1.E8.m1.37.37.37.4.4.4.3.3">3</cn></apply></apply><apply id="A1.E8.m1.63.63.3.3.2.1.1.1.3.3.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.63.63.3.3.2.1.1.1.3.3.1.cmml" xref="A1.E8.m1.68.68.8">subscript</csymbol><ci id="A1.E8.m1.38.38.38.5.5.5.cmml" xref="A1.E8.m1.38.38.38.5.5.5">𝜇</ci><cn type="integer" id="A1.E8.m1.39.39.39.6.6.6.1.cmml" xref="A1.E8.m1.39.39.39.6.6.6.1">3</cn></apply><apply id="A1.E8.m1.63.63.3.3.2.1.1.1.3.4.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.63.63.3.3.2.1.1.1.3.4.1.cmml" xref="A1.E8.m1.68.68.8">subscript</csymbol><ci id="A1.E8.m1.40.40.40.7.7.7.cmml" xref="A1.E8.m1.40.40.40.7.7.7">𝜇</ci><cn type="integer" id="A1.E8.m1.41.41.41.8.8.8.1.cmml" xref="A1.E8.m1.41.41.41.8.8.8.1">2</cn></apply></apply><apply id="A1.E8.m1.63.63.3.3.2.1.1.1.1.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.63.63.3.3.2.1.1.1.1.2.cmml" xref="A1.E8.m1.68.68.8"></times><apply id="A1.E8.m1.43.43.43.10.10.10.cmml" xref="A1.E8.m1.43.43.43.10.10.10"><divide id="A1.E8.m1.43.43.43.10.10.10.1.cmml" xref="A1.E8.m1.43.43.43.10.10.10"></divide><cn type="integer" id="A1.E8.m1.43.43.43.10.10.10.2.cmml" xref="A1.E8.m1.43.43.43.10.10.10.2">1</cn><apply id="A1.E8.m1.43.43.43.10.10.10.3.cmml" xref="A1.E8.m1.43.43.43.10.10.10.3"><csymbol cd="ambiguous" id="A1.E8.m1.43.43.43.10.10.10.3.1.cmml" xref="A1.E8.m1.43.43.43.10.10.10.3">superscript</csymbol><ci id="A1.E8.m1.43.43.43.10.10.10.3.2.cmml" xref="A1.E8.m1.43.43.43.10.10.10.3.2">𝑛</ci><cn type="integer" id="A1.E8.m1.43.43.43.10.10.10.3.3.cmml" xref="A1.E8.m1.43.43.43.10.10.10.3.3">4</cn></apply></apply><apply id="A1.E8.m1.63.63.3.3.2.1.1.1.1.1.1.1.cmml" xref="A1.E8.m1.68.68.8"><minus id="A1.E8.m1.47.47.47.14.14.14.cmml" xref="A1.E8.m1.47.47.47.14.14.14"></minus><apply id="A1.E8.m1.63.63.3.3.2.1.1.1.1.1.1.1.2.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.63.63.3.3.2.1.1.1.1.1.1.1.2.1.cmml" xref="A1.E8.m1.68.68.8">subscript</csymbol><ci id="A1.E8.m1.45.45.45.12.12.12.cmml" xref="A1.E8.m1.45.45.45.12.12.12">𝜇</ci><cn type="integer" id="A1.E8.m1.46.46.46.13.13.13.1.cmml" xref="A1.E8.m1.46.46.46.13.13.13.1">5</cn></apply><apply id="A1.E8.m1.63.63.3.3.2.1.1.1.1.1.1.1.3.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.63.63.3.3.2.1.1.1.1.1.1.1.3.1.cmml" xref="A1.E8.m1.68.68.8"></times><cn type="integer" id="A1.E8.m1.48.48.48.15.15.15.cmml" xref="A1.E8.m1.48.48.48.15.15.15">10</cn><apply id="A1.E8.m1.63.63.3.3.2.1.1.1.1.1.1.1.3.3.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.63.63.3.3.2.1.1.1.1.1.1.1.3.3.1.cmml" xref="A1.E8.m1.68.68.8">subscript</csymbol><ci id="A1.E8.m1.49.49.49.16.16.16.cmml" xref="A1.E8.m1.49.49.49.16.16.16">𝜇</ci><cn type="integer" id="A1.E8.m1.50.50.50.17.17.17.1.cmml" xref="A1.E8.m1.50.50.50.17.17.17.1">3</cn></apply><apply id="A1.E8.m1.63.63.3.3.2.1.1.1.1.1.1.1.3.4.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.63.63.3.3.2.1.1.1.1.1.1.1.3.4.1.cmml" xref="A1.E8.m1.68.68.8">subscript</csymbol><ci id="A1.E8.m1.51.51.51.18.18.18.cmml" xref="A1.E8.m1.51.51.51.18.18.18">𝜇</ci><cn type="integer" id="A1.E8.m1.52.52.52.19.19.19.1.cmml" xref="A1.E8.m1.52.52.52.19.19.19.1">2</cn></apply></apply></apply></apply></apply></apply><apply id="A1.E8.m1.64.64.4.4.3.cmml" xref="A1.E8.m1.68.68.8"><times id="A1.E8.m1.64.64.4.4.3.2.cmml" xref="A1.E8.m1.68.68.8"></times><ci id="A1.E8.m1.56.56.56.23.23.23.cmml" xref="A1.E8.m1.56.56.56.23.23.23">𝑜</ci><apply id="A1.E8.m1.64.64.4.4.3.1.1.1.cmml" xref="A1.E8.m1.68.68.8"><csymbol cd="ambiguous" id="A1.E8.m1.64.64.4.4.3.1.1.1.1.cmml" xref="A1.E8.m1.68.68.8">superscript</csymbol><ci id="A1.E8.m1.58.58.58.25.25.25.cmml" xref="A1.E8.m1.58.58.58.25.25.25">𝑛</ci><apply id="A1.E8.m1.59.59.59.26.26.26.1.cmml" xref="A1.E8.m1.59.59.59.26.26.26.1"><minus id="A1.E8.m1.59.59.59.26.26.26.1.1.cmml" xref="A1.E8.m1.59.59.59.26.26.26.1"></minus><cn type="integer" id="A1.E8.m1.59.59.59.26.26.26.1.2.cmml" xref="A1.E8.m1.59.59.59.26.26.26.1.2">3</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E8.m1.68c">\mathbb{E}[T_{n}]=Z_{\theta}-\frac{\mu_{2}}{2\mu^{2}}\frac{1}{n}+\frac{1}{3\mu^{3}}\frac{\mu_{3}}{n^{2}}-\frac{1}{4\mu^{4}}(\frac{3}{n^{2}}\mu_{2}^{2}+\frac{1}{n^{3}}(\mu_{4}-3\mu_{2}^{2}))\\
+\frac{1}{5\mu^{5}}(\frac{10}{n^{3}}\mu_{3}\mu_{2}+\frac{1}{n^{4}}(\mu_{5}-10\mu_{3}\mu_{2}))+o(n^{-3})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<p id="A1.SS2.1.p1.3" class="ltx_p">Where <math id="A1.SS2.1.p1.2.m1.1" class="ltx_Math" alttext="\mu=\mathbb{E}[T_{n}]" display="inline"><semantics id="A1.SS2.1.p1.2.m1.1a"><mrow id="A1.SS2.1.p1.2.m1.1.1" xref="A1.SS2.1.p1.2.m1.1.1.cmml"><mi id="A1.SS2.1.p1.2.m1.1.1.3" xref="A1.SS2.1.p1.2.m1.1.1.3.cmml">μ</mi><mo id="A1.SS2.1.p1.2.m1.1.1.2" xref="A1.SS2.1.p1.2.m1.1.1.2.cmml">=</mo><mrow id="A1.SS2.1.p1.2.m1.1.1.1" xref="A1.SS2.1.p1.2.m1.1.1.1.cmml"><mi id="A1.SS2.1.p1.2.m1.1.1.1.3" xref="A1.SS2.1.p1.2.m1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.2.m1.1.1.1.2" xref="A1.SS2.1.p1.2.m1.1.1.1.2.cmml">​</mo><mrow id="A1.SS2.1.p1.2.m1.1.1.1.1.1" xref="A1.SS2.1.p1.2.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.1.p1.2.m1.1.1.1.1.1.2" xref="A1.SS2.1.p1.2.m1.1.1.1.1.2.1.cmml">[</mo><msub id="A1.SS2.1.p1.2.m1.1.1.1.1.1.1" xref="A1.SS2.1.p1.2.m1.1.1.1.1.1.1.cmml"><mi id="A1.SS2.1.p1.2.m1.1.1.1.1.1.1.2" xref="A1.SS2.1.p1.2.m1.1.1.1.1.1.1.2.cmml">T</mi><mi id="A1.SS2.1.p1.2.m1.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.2.m1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.SS2.1.p1.2.m1.1.1.1.1.1.3" xref="A1.SS2.1.p1.2.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.2.m1.1b"><apply id="A1.SS2.1.p1.2.m1.1.1.cmml" xref="A1.SS2.1.p1.2.m1.1.1"><eq id="A1.SS2.1.p1.2.m1.1.1.2.cmml" xref="A1.SS2.1.p1.2.m1.1.1.2"></eq><ci id="A1.SS2.1.p1.2.m1.1.1.3.cmml" xref="A1.SS2.1.p1.2.m1.1.1.3">𝜇</ci><apply id="A1.SS2.1.p1.2.m1.1.1.1.cmml" xref="A1.SS2.1.p1.2.m1.1.1.1"><times id="A1.SS2.1.p1.2.m1.1.1.1.2.cmml" xref="A1.SS2.1.p1.2.m1.1.1.1.2"></times><ci id="A1.SS2.1.p1.2.m1.1.1.1.3.cmml" xref="A1.SS2.1.p1.2.m1.1.1.1.3">𝔼</ci><apply id="A1.SS2.1.p1.2.m1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.2.m1.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.1.p1.2.m1.1.1.1.1.2.1.cmml" xref="A1.SS2.1.p1.2.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.1.p1.2.m1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.2.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.1.p1.2.m1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.2.m1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.SS2.1.p1.2.m1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.2.m1.1.1.1.1.1.1.2">𝑇</ci><ci id="A1.SS2.1.p1.2.m1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.2.m1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.2.m1.1c">\mu=\mathbb{E}[T_{n}]</annotation></semantics></math>, <math id="A1.SS2.1.p1.3.m2.1" class="ltx_Math" alttext="\mu_{k}=\mathbb{E}[(T_{n}-\mu)^{k}]" display="inline"><semantics id="A1.SS2.1.p1.3.m2.1a"><mrow id="A1.SS2.1.p1.3.m2.1.1" xref="A1.SS2.1.p1.3.m2.1.1.cmml"><msub id="A1.SS2.1.p1.3.m2.1.1.3" xref="A1.SS2.1.p1.3.m2.1.1.3.cmml"><mi id="A1.SS2.1.p1.3.m2.1.1.3.2" xref="A1.SS2.1.p1.3.m2.1.1.3.2.cmml">μ</mi><mi id="A1.SS2.1.p1.3.m2.1.1.3.3" xref="A1.SS2.1.p1.3.m2.1.1.3.3.cmml">k</mi></msub><mo id="A1.SS2.1.p1.3.m2.1.1.2" xref="A1.SS2.1.p1.3.m2.1.1.2.cmml">=</mo><mrow id="A1.SS2.1.p1.3.m2.1.1.1" xref="A1.SS2.1.p1.3.m2.1.1.1.cmml"><mi id="A1.SS2.1.p1.3.m2.1.1.1.3" xref="A1.SS2.1.p1.3.m2.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.3.m2.1.1.1.2" xref="A1.SS2.1.p1.3.m2.1.1.1.2.cmml">​</mo><mrow id="A1.SS2.1.p1.3.m2.1.1.1.1.1" xref="A1.SS2.1.p1.3.m2.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.1.p1.3.m2.1.1.1.1.1.2" xref="A1.SS2.1.p1.3.m2.1.1.1.1.2.1.cmml">[</mo><msup id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.2" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.cmml"><msub id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2.2" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2.2.cmml">T</mi><mi id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2.3" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2.3.cmml">n</mi></msub><mo id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mi id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.3.cmml">μ</mi></mrow><mo stretchy="false" id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.3.cmml">k</mi></msup><mo stretchy="false" id="A1.SS2.1.p1.3.m2.1.1.1.1.1.3" xref="A1.SS2.1.p1.3.m2.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.3.m2.1b"><apply id="A1.SS2.1.p1.3.m2.1.1.cmml" xref="A1.SS2.1.p1.3.m2.1.1"><eq id="A1.SS2.1.p1.3.m2.1.1.2.cmml" xref="A1.SS2.1.p1.3.m2.1.1.2"></eq><apply id="A1.SS2.1.p1.3.m2.1.1.3.cmml" xref="A1.SS2.1.p1.3.m2.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.1.p1.3.m2.1.1.3.1.cmml" xref="A1.SS2.1.p1.3.m2.1.1.3">subscript</csymbol><ci id="A1.SS2.1.p1.3.m2.1.1.3.2.cmml" xref="A1.SS2.1.p1.3.m2.1.1.3.2">𝜇</ci><ci id="A1.SS2.1.p1.3.m2.1.1.3.3.cmml" xref="A1.SS2.1.p1.3.m2.1.1.3.3">𝑘</ci></apply><apply id="A1.SS2.1.p1.3.m2.1.1.1.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1"><times id="A1.SS2.1.p1.3.m2.1.1.1.2.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.2"></times><ci id="A1.SS2.1.p1.3.m2.1.1.1.3.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.3">𝔼</ci><apply id="A1.SS2.1.p1.3.m2.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.1.p1.3.m2.1.1.1.1.2.1.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1">superscript</csymbol><apply id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1"><minus id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.1"></minus><apply id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2.2">𝑇</ci><ci id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.2.3">𝑛</ci></apply><ci id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.1.1.1.3">𝜇</ci></apply><ci id="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.3.m2.1.1.1.1.1.1.3">𝑘</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.3.m2.1c">\mu_{k}=\mathbb{E}[(T_{n}-\mu)^{k}]</annotation></semantics></math>. Equivalently,</p>
<table id="A1.E9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E9.m1.2" class="ltx_Math" alttext="\mathbb{E}[T_{n}]=Z_{\theta}-\frac{\mu_{2}}{2\mu^{2}}\frac{1}{n}+o(n^{-1})" display="block"><semantics id="A1.E9.m1.2a"><mrow id="A1.E9.m1.2.2" xref="A1.E9.m1.2.2.cmml"><mrow id="A1.E9.m1.1.1.1" xref="A1.E9.m1.1.1.1.cmml"><mi id="A1.E9.m1.1.1.1.3" xref="A1.E9.m1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.E9.m1.1.1.1.2" xref="A1.E9.m1.1.1.1.2.cmml">​</mo><mrow id="A1.E9.m1.1.1.1.1.1" xref="A1.E9.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E9.m1.1.1.1.1.1.2" xref="A1.E9.m1.1.1.1.1.2.1.cmml">[</mo><msub id="A1.E9.m1.1.1.1.1.1.1" xref="A1.E9.m1.1.1.1.1.1.1.cmml"><mi id="A1.E9.m1.1.1.1.1.1.1.2" xref="A1.E9.m1.1.1.1.1.1.1.2.cmml">T</mi><mi id="A1.E9.m1.1.1.1.1.1.1.3" xref="A1.E9.m1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.E9.m1.1.1.1.1.1.3" xref="A1.E9.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.E9.m1.2.2.3" xref="A1.E9.m1.2.2.3.cmml">=</mo><mrow id="A1.E9.m1.2.2.2" xref="A1.E9.m1.2.2.2.cmml"><mrow id="A1.E9.m1.2.2.2.3" xref="A1.E9.m1.2.2.2.3.cmml"><msub id="A1.E9.m1.2.2.2.3.2" xref="A1.E9.m1.2.2.2.3.2.cmml"><mi id="A1.E9.m1.2.2.2.3.2.2" xref="A1.E9.m1.2.2.2.3.2.2.cmml">Z</mi><mi id="A1.E9.m1.2.2.2.3.2.3" xref="A1.E9.m1.2.2.2.3.2.3.cmml">θ</mi></msub><mo id="A1.E9.m1.2.2.2.3.1" xref="A1.E9.m1.2.2.2.3.1.cmml">−</mo><mrow id="A1.E9.m1.2.2.2.3.3" xref="A1.E9.m1.2.2.2.3.3.cmml"><mfrac id="A1.E9.m1.2.2.2.3.3.2" xref="A1.E9.m1.2.2.2.3.3.2.cmml"><msub id="A1.E9.m1.2.2.2.3.3.2.2" xref="A1.E9.m1.2.2.2.3.3.2.2.cmml"><mi id="A1.E9.m1.2.2.2.3.3.2.2.2" xref="A1.E9.m1.2.2.2.3.3.2.2.2.cmml">μ</mi><mn id="A1.E9.m1.2.2.2.3.3.2.2.3" xref="A1.E9.m1.2.2.2.3.3.2.2.3.cmml">2</mn></msub><mrow id="A1.E9.m1.2.2.2.3.3.2.3" xref="A1.E9.m1.2.2.2.3.3.2.3.cmml"><mn id="A1.E9.m1.2.2.2.3.3.2.3.2" xref="A1.E9.m1.2.2.2.3.3.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.E9.m1.2.2.2.3.3.2.3.1" xref="A1.E9.m1.2.2.2.3.3.2.3.1.cmml">​</mo><msup id="A1.E9.m1.2.2.2.3.3.2.3.3" xref="A1.E9.m1.2.2.2.3.3.2.3.3.cmml"><mi id="A1.E9.m1.2.2.2.3.3.2.3.3.2" xref="A1.E9.m1.2.2.2.3.3.2.3.3.2.cmml">μ</mi><mn id="A1.E9.m1.2.2.2.3.3.2.3.3.3" xref="A1.E9.m1.2.2.2.3.3.2.3.3.3.cmml">2</mn></msup></mrow></mfrac><mo lspace="0em" rspace="0em" id="A1.E9.m1.2.2.2.3.3.1" xref="A1.E9.m1.2.2.2.3.3.1.cmml">​</mo><mfrac id="A1.E9.m1.2.2.2.3.3.3" xref="A1.E9.m1.2.2.2.3.3.3.cmml"><mn id="A1.E9.m1.2.2.2.3.3.3.2" xref="A1.E9.m1.2.2.2.3.3.3.2.cmml">1</mn><mi id="A1.E9.m1.2.2.2.3.3.3.3" xref="A1.E9.m1.2.2.2.3.3.3.3.cmml">n</mi></mfrac></mrow></mrow><mo id="A1.E9.m1.2.2.2.2" xref="A1.E9.m1.2.2.2.2.cmml">+</mo><mrow id="A1.E9.m1.2.2.2.1" xref="A1.E9.m1.2.2.2.1.cmml"><mi id="A1.E9.m1.2.2.2.1.3" xref="A1.E9.m1.2.2.2.1.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A1.E9.m1.2.2.2.1.2" xref="A1.E9.m1.2.2.2.1.2.cmml">​</mo><mrow id="A1.E9.m1.2.2.2.1.1.1" xref="A1.E9.m1.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="A1.E9.m1.2.2.2.1.1.1.2" xref="A1.E9.m1.2.2.2.1.1.1.1.cmml">(</mo><msup id="A1.E9.m1.2.2.2.1.1.1.1" xref="A1.E9.m1.2.2.2.1.1.1.1.cmml"><mi id="A1.E9.m1.2.2.2.1.1.1.1.2" xref="A1.E9.m1.2.2.2.1.1.1.1.2.cmml">n</mi><mrow id="A1.E9.m1.2.2.2.1.1.1.1.3" xref="A1.E9.m1.2.2.2.1.1.1.1.3.cmml"><mo id="A1.E9.m1.2.2.2.1.1.1.1.3a" xref="A1.E9.m1.2.2.2.1.1.1.1.3.cmml">−</mo><mn id="A1.E9.m1.2.2.2.1.1.1.1.3.2" xref="A1.E9.m1.2.2.2.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo stretchy="false" id="A1.E9.m1.2.2.2.1.1.1.3" xref="A1.E9.m1.2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E9.m1.2b"><apply id="A1.E9.m1.2.2.cmml" xref="A1.E9.m1.2.2"><eq id="A1.E9.m1.2.2.3.cmml" xref="A1.E9.m1.2.2.3"></eq><apply id="A1.E9.m1.1.1.1.cmml" xref="A1.E9.m1.1.1.1"><times id="A1.E9.m1.1.1.1.2.cmml" xref="A1.E9.m1.1.1.1.2"></times><ci id="A1.E9.m1.1.1.1.3.cmml" xref="A1.E9.m1.1.1.1.3">𝔼</ci><apply id="A1.E9.m1.1.1.1.1.2.cmml" xref="A1.E9.m1.1.1.1.1.1"><csymbol cd="latexml" id="A1.E9.m1.1.1.1.1.2.1.cmml" xref="A1.E9.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.E9.m1.1.1.1.1.1.1.cmml" xref="A1.E9.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.1.1.1.1.1.cmml" xref="A1.E9.m1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.E9.m1.1.1.1.1.1.1.2.cmml" xref="A1.E9.m1.1.1.1.1.1.1.2">𝑇</ci><ci id="A1.E9.m1.1.1.1.1.1.1.3.cmml" xref="A1.E9.m1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply><apply id="A1.E9.m1.2.2.2.cmml" xref="A1.E9.m1.2.2.2"><plus id="A1.E9.m1.2.2.2.2.cmml" xref="A1.E9.m1.2.2.2.2"></plus><apply id="A1.E9.m1.2.2.2.3.cmml" xref="A1.E9.m1.2.2.2.3"><minus id="A1.E9.m1.2.2.2.3.1.cmml" xref="A1.E9.m1.2.2.2.3.1"></minus><apply id="A1.E9.m1.2.2.2.3.2.cmml" xref="A1.E9.m1.2.2.2.3.2"><csymbol cd="ambiguous" id="A1.E9.m1.2.2.2.3.2.1.cmml" xref="A1.E9.m1.2.2.2.3.2">subscript</csymbol><ci id="A1.E9.m1.2.2.2.3.2.2.cmml" xref="A1.E9.m1.2.2.2.3.2.2">𝑍</ci><ci id="A1.E9.m1.2.2.2.3.2.3.cmml" xref="A1.E9.m1.2.2.2.3.2.3">𝜃</ci></apply><apply id="A1.E9.m1.2.2.2.3.3.cmml" xref="A1.E9.m1.2.2.2.3.3"><times id="A1.E9.m1.2.2.2.3.3.1.cmml" xref="A1.E9.m1.2.2.2.3.3.1"></times><apply id="A1.E9.m1.2.2.2.3.3.2.cmml" xref="A1.E9.m1.2.2.2.3.3.2"><divide id="A1.E9.m1.2.2.2.3.3.2.1.cmml" xref="A1.E9.m1.2.2.2.3.3.2"></divide><apply id="A1.E9.m1.2.2.2.3.3.2.2.cmml" xref="A1.E9.m1.2.2.2.3.3.2.2"><csymbol cd="ambiguous" id="A1.E9.m1.2.2.2.3.3.2.2.1.cmml" xref="A1.E9.m1.2.2.2.3.3.2.2">subscript</csymbol><ci id="A1.E9.m1.2.2.2.3.3.2.2.2.cmml" xref="A1.E9.m1.2.2.2.3.3.2.2.2">𝜇</ci><cn type="integer" id="A1.E9.m1.2.2.2.3.3.2.2.3.cmml" xref="A1.E9.m1.2.2.2.3.3.2.2.3">2</cn></apply><apply id="A1.E9.m1.2.2.2.3.3.2.3.cmml" xref="A1.E9.m1.2.2.2.3.3.2.3"><times id="A1.E9.m1.2.2.2.3.3.2.3.1.cmml" xref="A1.E9.m1.2.2.2.3.3.2.3.1"></times><cn type="integer" id="A1.E9.m1.2.2.2.3.3.2.3.2.cmml" xref="A1.E9.m1.2.2.2.3.3.2.3.2">2</cn><apply id="A1.E9.m1.2.2.2.3.3.2.3.3.cmml" xref="A1.E9.m1.2.2.2.3.3.2.3.3"><csymbol cd="ambiguous" id="A1.E9.m1.2.2.2.3.3.2.3.3.1.cmml" xref="A1.E9.m1.2.2.2.3.3.2.3.3">superscript</csymbol><ci id="A1.E9.m1.2.2.2.3.3.2.3.3.2.cmml" xref="A1.E9.m1.2.2.2.3.3.2.3.3.2">𝜇</ci><cn type="integer" id="A1.E9.m1.2.2.2.3.3.2.3.3.3.cmml" xref="A1.E9.m1.2.2.2.3.3.2.3.3.3">2</cn></apply></apply></apply><apply id="A1.E9.m1.2.2.2.3.3.3.cmml" xref="A1.E9.m1.2.2.2.3.3.3"><divide id="A1.E9.m1.2.2.2.3.3.3.1.cmml" xref="A1.E9.m1.2.2.2.3.3.3"></divide><cn type="integer" id="A1.E9.m1.2.2.2.3.3.3.2.cmml" xref="A1.E9.m1.2.2.2.3.3.3.2">1</cn><ci id="A1.E9.m1.2.2.2.3.3.3.3.cmml" xref="A1.E9.m1.2.2.2.3.3.3.3">𝑛</ci></apply></apply></apply><apply id="A1.E9.m1.2.2.2.1.cmml" xref="A1.E9.m1.2.2.2.1"><times id="A1.E9.m1.2.2.2.1.2.cmml" xref="A1.E9.m1.2.2.2.1.2"></times><ci id="A1.E9.m1.2.2.2.1.3.cmml" xref="A1.E9.m1.2.2.2.1.3">𝑜</ci><apply id="A1.E9.m1.2.2.2.1.1.1.1.cmml" xref="A1.E9.m1.2.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.E9.m1.2.2.2.1.1.1.1.1.cmml" xref="A1.E9.m1.2.2.2.1.1.1">superscript</csymbol><ci id="A1.E9.m1.2.2.2.1.1.1.1.2.cmml" xref="A1.E9.m1.2.2.2.1.1.1.1.2">𝑛</ci><apply id="A1.E9.m1.2.2.2.1.1.1.1.3.cmml" xref="A1.E9.m1.2.2.2.1.1.1.1.3"><minus id="A1.E9.m1.2.2.2.1.1.1.1.3.1.cmml" xref="A1.E9.m1.2.2.2.1.1.1.1.3"></minus><cn type="integer" id="A1.E9.m1.2.2.2.1.1.1.1.3.2.cmml" xref="A1.E9.m1.2.2.2.1.1.1.1.3.2">1</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E9.m1.2c">\mathbb{E}[T_{n}]=Z_{\theta}-\frac{\mu_{2}}{2\mu^{2}}\frac{1}{n}+o(n^{-1})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
</table>
<p id="A1.SS2.1.p1.13" class="ltx_p">Therefore, <math id="A1.SS2.1.p1.4.m1.1" class="ltx_Math" alttext="\lim_{n\to\infty}\mathbb{E}[T_{n}]=Z_{\theta}" display="inline"><semantics id="A1.SS2.1.p1.4.m1.1a"><mrow id="A1.SS2.1.p1.4.m1.1.1" xref="A1.SS2.1.p1.4.m1.1.1.cmml"><mrow id="A1.SS2.1.p1.4.m1.1.1.1" xref="A1.SS2.1.p1.4.m1.1.1.1.cmml"><msub id="A1.SS2.1.p1.4.m1.1.1.1.2" xref="A1.SS2.1.p1.4.m1.1.1.1.2.cmml"><mo id="A1.SS2.1.p1.4.m1.1.1.1.2.2" xref="A1.SS2.1.p1.4.m1.1.1.1.2.2.cmml">lim</mo><mrow id="A1.SS2.1.p1.4.m1.1.1.1.2.3" xref="A1.SS2.1.p1.4.m1.1.1.1.2.3.cmml"><mi id="A1.SS2.1.p1.4.m1.1.1.1.2.3.2" xref="A1.SS2.1.p1.4.m1.1.1.1.2.3.2.cmml">n</mi><mo stretchy="false" id="A1.SS2.1.p1.4.m1.1.1.1.2.3.1" xref="A1.SS2.1.p1.4.m1.1.1.1.2.3.1.cmml">→</mo><mi mathvariant="normal" id="A1.SS2.1.p1.4.m1.1.1.1.2.3.3" xref="A1.SS2.1.p1.4.m1.1.1.1.2.3.3.cmml">∞</mi></mrow></msub><mrow id="A1.SS2.1.p1.4.m1.1.1.1.1" xref="A1.SS2.1.p1.4.m1.1.1.1.1.cmml"><mi id="A1.SS2.1.p1.4.m1.1.1.1.1.3" xref="A1.SS2.1.p1.4.m1.1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.4.m1.1.1.1.1.2" xref="A1.SS2.1.p1.4.m1.1.1.1.1.2.cmml">​</mo><mrow id="A1.SS2.1.p1.4.m1.1.1.1.1.1.1" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.2" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.2.1.cmml">[</mo><msub id="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1.cmml"><mi id="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1.2" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1.2.cmml">T</mi><mi id="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><mo id="A1.SS2.1.p1.4.m1.1.1.2" xref="A1.SS2.1.p1.4.m1.1.1.2.cmml">=</mo><msub id="A1.SS2.1.p1.4.m1.1.1.3" xref="A1.SS2.1.p1.4.m1.1.1.3.cmml"><mi id="A1.SS2.1.p1.4.m1.1.1.3.2" xref="A1.SS2.1.p1.4.m1.1.1.3.2.cmml">Z</mi><mi id="A1.SS2.1.p1.4.m1.1.1.3.3" xref="A1.SS2.1.p1.4.m1.1.1.3.3.cmml">θ</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.4.m1.1b"><apply id="A1.SS2.1.p1.4.m1.1.1.cmml" xref="A1.SS2.1.p1.4.m1.1.1"><eq id="A1.SS2.1.p1.4.m1.1.1.2.cmml" xref="A1.SS2.1.p1.4.m1.1.1.2"></eq><apply id="A1.SS2.1.p1.4.m1.1.1.1.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1"><apply id="A1.SS2.1.p1.4.m1.1.1.1.2.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.2"><csymbol cd="ambiguous" id="A1.SS2.1.p1.4.m1.1.1.1.2.1.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.2">subscript</csymbol><limit id="A1.SS2.1.p1.4.m1.1.1.1.2.2.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.2.2"></limit><apply id="A1.SS2.1.p1.4.m1.1.1.1.2.3.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.2.3"><ci id="A1.SS2.1.p1.4.m1.1.1.1.2.3.1.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.2.3.1">→</ci><ci id="A1.SS2.1.p1.4.m1.1.1.1.2.3.2.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.2.3.2">𝑛</ci><infinity id="A1.SS2.1.p1.4.m1.1.1.1.2.3.3.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.2.3.3"></infinity></apply></apply><apply id="A1.SS2.1.p1.4.m1.1.1.1.1.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.1"><times id="A1.SS2.1.p1.4.m1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.1.2"></times><ci id="A1.SS2.1.p1.4.m1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.1.3">𝔼</ci><apply id="A1.SS2.1.p1.4.m1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.1.p1.4.m1.1.1.1.1.1.2.1.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1.2">𝑇</ci><ci id="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.4.m1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></apply><apply id="A1.SS2.1.p1.4.m1.1.1.3.cmml" xref="A1.SS2.1.p1.4.m1.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.1.p1.4.m1.1.1.3.1.cmml" xref="A1.SS2.1.p1.4.m1.1.1.3">subscript</csymbol><ci id="A1.SS2.1.p1.4.m1.1.1.3.2.cmml" xref="A1.SS2.1.p1.4.m1.1.1.3.2">𝑍</ci><ci id="A1.SS2.1.p1.4.m1.1.1.3.3.cmml" xref="A1.SS2.1.p1.4.m1.1.1.3.3">𝜃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.4.m1.1c">\lim_{n\to\infty}\mathbb{E}[T_{n}]=Z_{\theta}</annotation></semantics></math>. So <math id="A1.SS2.1.p1.5.m2.1" class="ltx_Math" alttext="\forall\epsilon&gt;0" display="inline"><semantics id="A1.SS2.1.p1.5.m2.1a"><mrow id="A1.SS2.1.p1.5.m2.1.1" xref="A1.SS2.1.p1.5.m2.1.1.cmml"><mrow id="A1.SS2.1.p1.5.m2.1.1.2" xref="A1.SS2.1.p1.5.m2.1.1.2.cmml"><mo rspace="0.167em" id="A1.SS2.1.p1.5.m2.1.1.2.1" xref="A1.SS2.1.p1.5.m2.1.1.2.1.cmml">∀</mo><mi id="A1.SS2.1.p1.5.m2.1.1.2.2" xref="A1.SS2.1.p1.5.m2.1.1.2.2.cmml">ϵ</mi></mrow><mo id="A1.SS2.1.p1.5.m2.1.1.1" xref="A1.SS2.1.p1.5.m2.1.1.1.cmml">&gt;</mo><mn id="A1.SS2.1.p1.5.m2.1.1.3" xref="A1.SS2.1.p1.5.m2.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.5.m2.1b"><apply id="A1.SS2.1.p1.5.m2.1.1.cmml" xref="A1.SS2.1.p1.5.m2.1.1"><gt id="A1.SS2.1.p1.5.m2.1.1.1.cmml" xref="A1.SS2.1.p1.5.m2.1.1.1"></gt><apply id="A1.SS2.1.p1.5.m2.1.1.2.cmml" xref="A1.SS2.1.p1.5.m2.1.1.2"><csymbol cd="latexml" id="A1.SS2.1.p1.5.m2.1.1.2.1.cmml" xref="A1.SS2.1.p1.5.m2.1.1.2.1">for-all</csymbol><ci id="A1.SS2.1.p1.5.m2.1.1.2.2.cmml" xref="A1.SS2.1.p1.5.m2.1.1.2.2">italic-ϵ</ci></apply><cn type="integer" id="A1.SS2.1.p1.5.m2.1.1.3.cmml" xref="A1.SS2.1.p1.5.m2.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.5.m2.1c">\forall\epsilon&gt;0</annotation></semantics></math>, <math id="A1.SS2.1.p1.6.m3.1" class="ltx_Math" alttext="\exists N_{1}&gt;0" display="inline"><semantics id="A1.SS2.1.p1.6.m3.1a"><mrow id="A1.SS2.1.p1.6.m3.1.1" xref="A1.SS2.1.p1.6.m3.1.1.cmml"><mrow id="A1.SS2.1.p1.6.m3.1.1.2" xref="A1.SS2.1.p1.6.m3.1.1.2.cmml"><mo rspace="0.167em" id="A1.SS2.1.p1.6.m3.1.1.2.1" xref="A1.SS2.1.p1.6.m3.1.1.2.1.cmml">∃</mo><msub id="A1.SS2.1.p1.6.m3.1.1.2.2" xref="A1.SS2.1.p1.6.m3.1.1.2.2.cmml"><mi id="A1.SS2.1.p1.6.m3.1.1.2.2.2" xref="A1.SS2.1.p1.6.m3.1.1.2.2.2.cmml">N</mi><mn id="A1.SS2.1.p1.6.m3.1.1.2.2.3" xref="A1.SS2.1.p1.6.m3.1.1.2.2.3.cmml">1</mn></msub></mrow><mo id="A1.SS2.1.p1.6.m3.1.1.1" xref="A1.SS2.1.p1.6.m3.1.1.1.cmml">&gt;</mo><mn id="A1.SS2.1.p1.6.m3.1.1.3" xref="A1.SS2.1.p1.6.m3.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.6.m3.1b"><apply id="A1.SS2.1.p1.6.m3.1.1.cmml" xref="A1.SS2.1.p1.6.m3.1.1"><gt id="A1.SS2.1.p1.6.m3.1.1.1.cmml" xref="A1.SS2.1.p1.6.m3.1.1.1"></gt><apply id="A1.SS2.1.p1.6.m3.1.1.2.cmml" xref="A1.SS2.1.p1.6.m3.1.1.2"><exists id="A1.SS2.1.p1.6.m3.1.1.2.1.cmml" xref="A1.SS2.1.p1.6.m3.1.1.2.1"></exists><apply id="A1.SS2.1.p1.6.m3.1.1.2.2.cmml" xref="A1.SS2.1.p1.6.m3.1.1.2.2"><csymbol cd="ambiguous" id="A1.SS2.1.p1.6.m3.1.1.2.2.1.cmml" xref="A1.SS2.1.p1.6.m3.1.1.2.2">subscript</csymbol><ci id="A1.SS2.1.p1.6.m3.1.1.2.2.2.cmml" xref="A1.SS2.1.p1.6.m3.1.1.2.2.2">𝑁</ci><cn type="integer" id="A1.SS2.1.p1.6.m3.1.1.2.2.3.cmml" xref="A1.SS2.1.p1.6.m3.1.1.2.2.3">1</cn></apply></apply><cn type="integer" id="A1.SS2.1.p1.6.m3.1.1.3.cmml" xref="A1.SS2.1.p1.6.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.6.m3.1c">\exists N_{1}&gt;0</annotation></semantics></math> such that when <math id="A1.SS2.1.p1.7.m4.1" class="ltx_Math" alttext="n&gt;N_{1}" display="inline"><semantics id="A1.SS2.1.p1.7.m4.1a"><mrow id="A1.SS2.1.p1.7.m4.1.1" xref="A1.SS2.1.p1.7.m4.1.1.cmml"><mi id="A1.SS2.1.p1.7.m4.1.1.2" xref="A1.SS2.1.p1.7.m4.1.1.2.cmml">n</mi><mo id="A1.SS2.1.p1.7.m4.1.1.1" xref="A1.SS2.1.p1.7.m4.1.1.1.cmml">&gt;</mo><msub id="A1.SS2.1.p1.7.m4.1.1.3" xref="A1.SS2.1.p1.7.m4.1.1.3.cmml"><mi id="A1.SS2.1.p1.7.m4.1.1.3.2" xref="A1.SS2.1.p1.7.m4.1.1.3.2.cmml">N</mi><mn id="A1.SS2.1.p1.7.m4.1.1.3.3" xref="A1.SS2.1.p1.7.m4.1.1.3.3.cmml">1</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.7.m4.1b"><apply id="A1.SS2.1.p1.7.m4.1.1.cmml" xref="A1.SS2.1.p1.7.m4.1.1"><gt id="A1.SS2.1.p1.7.m4.1.1.1.cmml" xref="A1.SS2.1.p1.7.m4.1.1.1"></gt><ci id="A1.SS2.1.p1.7.m4.1.1.2.cmml" xref="A1.SS2.1.p1.7.m4.1.1.2">𝑛</ci><apply id="A1.SS2.1.p1.7.m4.1.1.3.cmml" xref="A1.SS2.1.p1.7.m4.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.1.p1.7.m4.1.1.3.1.cmml" xref="A1.SS2.1.p1.7.m4.1.1.3">subscript</csymbol><ci id="A1.SS2.1.p1.7.m4.1.1.3.2.cmml" xref="A1.SS2.1.p1.7.m4.1.1.3.2">𝑁</ci><cn type="integer" id="A1.SS2.1.p1.7.m4.1.1.3.3.cmml" xref="A1.SS2.1.p1.7.m4.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.7.m4.1c">n&gt;N_{1}</annotation></semantics></math>, <math id="A1.SS2.1.p1.8.m5.1" class="ltx_Math" alttext="\mathbb{E}[T_{n}]&gt;Z_{\theta}-\epsilon" display="inline"><semantics id="A1.SS2.1.p1.8.m5.1a"><mrow id="A1.SS2.1.p1.8.m5.1.1" xref="A1.SS2.1.p1.8.m5.1.1.cmml"><mrow id="A1.SS2.1.p1.8.m5.1.1.1" xref="A1.SS2.1.p1.8.m5.1.1.1.cmml"><mi id="A1.SS2.1.p1.8.m5.1.1.1.3" xref="A1.SS2.1.p1.8.m5.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.8.m5.1.1.1.2" xref="A1.SS2.1.p1.8.m5.1.1.1.2.cmml">​</mo><mrow id="A1.SS2.1.p1.8.m5.1.1.1.1.1" xref="A1.SS2.1.p1.8.m5.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.1.p1.8.m5.1.1.1.1.1.2" xref="A1.SS2.1.p1.8.m5.1.1.1.1.2.1.cmml">[</mo><msub id="A1.SS2.1.p1.8.m5.1.1.1.1.1.1" xref="A1.SS2.1.p1.8.m5.1.1.1.1.1.1.cmml"><mi id="A1.SS2.1.p1.8.m5.1.1.1.1.1.1.2" xref="A1.SS2.1.p1.8.m5.1.1.1.1.1.1.2.cmml">T</mi><mi id="A1.SS2.1.p1.8.m5.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.8.m5.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.SS2.1.p1.8.m5.1.1.1.1.1.3" xref="A1.SS2.1.p1.8.m5.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.SS2.1.p1.8.m5.1.1.2" xref="A1.SS2.1.p1.8.m5.1.1.2.cmml">&gt;</mo><mrow id="A1.SS2.1.p1.8.m5.1.1.3" xref="A1.SS2.1.p1.8.m5.1.1.3.cmml"><msub id="A1.SS2.1.p1.8.m5.1.1.3.2" xref="A1.SS2.1.p1.8.m5.1.1.3.2.cmml"><mi id="A1.SS2.1.p1.8.m5.1.1.3.2.2" xref="A1.SS2.1.p1.8.m5.1.1.3.2.2.cmml">Z</mi><mi id="A1.SS2.1.p1.8.m5.1.1.3.2.3" xref="A1.SS2.1.p1.8.m5.1.1.3.2.3.cmml">θ</mi></msub><mo id="A1.SS2.1.p1.8.m5.1.1.3.1" xref="A1.SS2.1.p1.8.m5.1.1.3.1.cmml">−</mo><mi id="A1.SS2.1.p1.8.m5.1.1.3.3" xref="A1.SS2.1.p1.8.m5.1.1.3.3.cmml">ϵ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.8.m5.1b"><apply id="A1.SS2.1.p1.8.m5.1.1.cmml" xref="A1.SS2.1.p1.8.m5.1.1"><gt id="A1.SS2.1.p1.8.m5.1.1.2.cmml" xref="A1.SS2.1.p1.8.m5.1.1.2"></gt><apply id="A1.SS2.1.p1.8.m5.1.1.1.cmml" xref="A1.SS2.1.p1.8.m5.1.1.1"><times id="A1.SS2.1.p1.8.m5.1.1.1.2.cmml" xref="A1.SS2.1.p1.8.m5.1.1.1.2"></times><ci id="A1.SS2.1.p1.8.m5.1.1.1.3.cmml" xref="A1.SS2.1.p1.8.m5.1.1.1.3">𝔼</ci><apply id="A1.SS2.1.p1.8.m5.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.8.m5.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.1.p1.8.m5.1.1.1.1.2.1.cmml" xref="A1.SS2.1.p1.8.m5.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.1.p1.8.m5.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.8.m5.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.1.p1.8.m5.1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.8.m5.1.1.1.1.1.1">subscript</csymbol><ci id="A1.SS2.1.p1.8.m5.1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.8.m5.1.1.1.1.1.1.2">𝑇</ci><ci id="A1.SS2.1.p1.8.m5.1.1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.8.m5.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply><apply id="A1.SS2.1.p1.8.m5.1.1.3.cmml" xref="A1.SS2.1.p1.8.m5.1.1.3"><minus id="A1.SS2.1.p1.8.m5.1.1.3.1.cmml" xref="A1.SS2.1.p1.8.m5.1.1.3.1"></minus><apply id="A1.SS2.1.p1.8.m5.1.1.3.2.cmml" xref="A1.SS2.1.p1.8.m5.1.1.3.2"><csymbol cd="ambiguous" id="A1.SS2.1.p1.8.m5.1.1.3.2.1.cmml" xref="A1.SS2.1.p1.8.m5.1.1.3.2">subscript</csymbol><ci id="A1.SS2.1.p1.8.m5.1.1.3.2.2.cmml" xref="A1.SS2.1.p1.8.m5.1.1.3.2.2">𝑍</ci><ci id="A1.SS2.1.p1.8.m5.1.1.3.2.3.cmml" xref="A1.SS2.1.p1.8.m5.1.1.3.2.3">𝜃</ci></apply><ci id="A1.SS2.1.p1.8.m5.1.1.3.3.cmml" xref="A1.SS2.1.p1.8.m5.1.1.3.3">italic-ϵ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.8.m5.1c">\mathbb{E}[T_{n}]&gt;Z_{\theta}-\epsilon</annotation></semantics></math>. On the other hand, <math id="A1.SS2.1.p1.9.m6.2" class="ltx_Math" alttext="\lim_{n\to\infty}n(Z_{\theta}-\mathbb{E}[T_{n}])=\lim_{n\to\infty}\frac{\mu_{2}}{2\mu^{2}}+o(1)=\frac{\mu_{2}}{2\mu^{2}}&gt;0" display="inline"><semantics id="A1.SS2.1.p1.9.m6.2a"><mrow id="A1.SS2.1.p1.9.m6.2.2" xref="A1.SS2.1.p1.9.m6.2.2.cmml"><mrow id="A1.SS2.1.p1.9.m6.2.2.1" xref="A1.SS2.1.p1.9.m6.2.2.1.cmml"><msub id="A1.SS2.1.p1.9.m6.2.2.1.2" xref="A1.SS2.1.p1.9.m6.2.2.1.2.cmml"><mo id="A1.SS2.1.p1.9.m6.2.2.1.2.2" xref="A1.SS2.1.p1.9.m6.2.2.1.2.2.cmml">lim</mo><mrow id="A1.SS2.1.p1.9.m6.2.2.1.2.3" xref="A1.SS2.1.p1.9.m6.2.2.1.2.3.cmml"><mi id="A1.SS2.1.p1.9.m6.2.2.1.2.3.2" xref="A1.SS2.1.p1.9.m6.2.2.1.2.3.2.cmml">n</mi><mo stretchy="false" id="A1.SS2.1.p1.9.m6.2.2.1.2.3.1" xref="A1.SS2.1.p1.9.m6.2.2.1.2.3.1.cmml">→</mo><mi mathvariant="normal" id="A1.SS2.1.p1.9.m6.2.2.1.2.3.3" xref="A1.SS2.1.p1.9.m6.2.2.1.2.3.3.cmml">∞</mi></mrow></msub><mrow id="A1.SS2.1.p1.9.m6.2.2.1.1" xref="A1.SS2.1.p1.9.m6.2.2.1.1.cmml"><mi id="A1.SS2.1.p1.9.m6.2.2.1.1.3" xref="A1.SS2.1.p1.9.m6.2.2.1.1.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.9.m6.2.2.1.1.2" xref="A1.SS2.1.p1.9.m6.2.2.1.1.2.cmml">​</mo><mrow id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.2" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.cmml"><msub id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3.cmml"><mi id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3.2" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3.2.cmml">Z</mi><mi id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3.3" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3.3.cmml">θ</mi></msub><mo id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.2" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.2.cmml">−</mo><mrow id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.cmml"><mi id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.2" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.2.cmml">​</mo><mrow id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.2" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.2.1.cmml">[</mo><msub id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1.cmml"><mi id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1.2.cmml">T</mi><mi id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><mo stretchy="false" id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.3" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo rspace="0.1389em" id="A1.SS2.1.p1.9.m6.2.2.3" xref="A1.SS2.1.p1.9.m6.2.2.3.cmml">=</mo><mrow id="A1.SS2.1.p1.9.m6.2.2.4" xref="A1.SS2.1.p1.9.m6.2.2.4.cmml"><mrow id="A1.SS2.1.p1.9.m6.2.2.4.2" xref="A1.SS2.1.p1.9.m6.2.2.4.2.cmml"><msub id="A1.SS2.1.p1.9.m6.2.2.4.2.1" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1.cmml"><mo lspace="0.1389em" rspace="0.167em" id="A1.SS2.1.p1.9.m6.2.2.4.2.1.2" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1.2.cmml">lim</mo><mrow id="A1.SS2.1.p1.9.m6.2.2.4.2.1.3" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.cmml"><mi id="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.2" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.2.cmml">n</mi><mo stretchy="false" id="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.1" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.1.cmml">→</mo><mi mathvariant="normal" id="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.3" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.3.cmml">∞</mi></mrow></msub><mfrac id="A1.SS2.1.p1.9.m6.2.2.4.2.2" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.cmml"><msub id="A1.SS2.1.p1.9.m6.2.2.4.2.2.2" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.2.cmml"><mi id="A1.SS2.1.p1.9.m6.2.2.4.2.2.2.2" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.2.2.cmml">μ</mi><mn id="A1.SS2.1.p1.9.m6.2.2.4.2.2.2.3" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.2.3.cmml">2</mn></msub><mrow id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.cmml"><mn id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.2" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.1" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.1.cmml">​</mo><msup id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3.cmml"><mi id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3.2" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3.2.cmml">μ</mi><mn id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3.3" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3.3.cmml">2</mn></msup></mrow></mfrac></mrow><mo id="A1.SS2.1.p1.9.m6.2.2.4.1" xref="A1.SS2.1.p1.9.m6.2.2.4.1.cmml">+</mo><mrow id="A1.SS2.1.p1.9.m6.2.2.4.3" xref="A1.SS2.1.p1.9.m6.2.2.4.3.cmml"><mi id="A1.SS2.1.p1.9.m6.2.2.4.3.2" xref="A1.SS2.1.p1.9.m6.2.2.4.3.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.9.m6.2.2.4.3.1" xref="A1.SS2.1.p1.9.m6.2.2.4.3.1.cmml">​</mo><mrow id="A1.SS2.1.p1.9.m6.2.2.4.3.3.2" xref="A1.SS2.1.p1.9.m6.2.2.4.3.cmml"><mo stretchy="false" id="A1.SS2.1.p1.9.m6.2.2.4.3.3.2.1" xref="A1.SS2.1.p1.9.m6.2.2.4.3.cmml">(</mo><mn id="A1.SS2.1.p1.9.m6.1.1" xref="A1.SS2.1.p1.9.m6.1.1.cmml">1</mn><mo stretchy="false" id="A1.SS2.1.p1.9.m6.2.2.4.3.3.2.2" xref="A1.SS2.1.p1.9.m6.2.2.4.3.cmml">)</mo></mrow></mrow></mrow><mo id="A1.SS2.1.p1.9.m6.2.2.5" xref="A1.SS2.1.p1.9.m6.2.2.5.cmml">=</mo><mfrac id="A1.SS2.1.p1.9.m6.2.2.6" xref="A1.SS2.1.p1.9.m6.2.2.6.cmml"><msub id="A1.SS2.1.p1.9.m6.2.2.6.2" xref="A1.SS2.1.p1.9.m6.2.2.6.2.cmml"><mi id="A1.SS2.1.p1.9.m6.2.2.6.2.2" xref="A1.SS2.1.p1.9.m6.2.2.6.2.2.cmml">μ</mi><mn id="A1.SS2.1.p1.9.m6.2.2.6.2.3" xref="A1.SS2.1.p1.9.m6.2.2.6.2.3.cmml">2</mn></msub><mrow id="A1.SS2.1.p1.9.m6.2.2.6.3" xref="A1.SS2.1.p1.9.m6.2.2.6.3.cmml"><mn id="A1.SS2.1.p1.9.m6.2.2.6.3.2" xref="A1.SS2.1.p1.9.m6.2.2.6.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.9.m6.2.2.6.3.1" xref="A1.SS2.1.p1.9.m6.2.2.6.3.1.cmml">​</mo><msup id="A1.SS2.1.p1.9.m6.2.2.6.3.3" xref="A1.SS2.1.p1.9.m6.2.2.6.3.3.cmml"><mi id="A1.SS2.1.p1.9.m6.2.2.6.3.3.2" xref="A1.SS2.1.p1.9.m6.2.2.6.3.3.2.cmml">μ</mi><mn id="A1.SS2.1.p1.9.m6.2.2.6.3.3.3" xref="A1.SS2.1.p1.9.m6.2.2.6.3.3.3.cmml">2</mn></msup></mrow></mfrac><mo id="A1.SS2.1.p1.9.m6.2.2.7" xref="A1.SS2.1.p1.9.m6.2.2.7.cmml">&gt;</mo><mn id="A1.SS2.1.p1.9.m6.2.2.8" xref="A1.SS2.1.p1.9.m6.2.2.8.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.9.m6.2b"><apply id="A1.SS2.1.p1.9.m6.2.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2"><and id="A1.SS2.1.p1.9.m6.2.2a.cmml" xref="A1.SS2.1.p1.9.m6.2.2"></and><apply id="A1.SS2.1.p1.9.m6.2.2b.cmml" xref="A1.SS2.1.p1.9.m6.2.2"><eq id="A1.SS2.1.p1.9.m6.2.2.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.3"></eq><apply id="A1.SS2.1.p1.9.m6.2.2.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1"><apply id="A1.SS2.1.p1.9.m6.2.2.1.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.2"><csymbol cd="ambiguous" id="A1.SS2.1.p1.9.m6.2.2.1.2.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.2">subscript</csymbol><limit id="A1.SS2.1.p1.9.m6.2.2.1.2.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.2.2"></limit><apply id="A1.SS2.1.p1.9.m6.2.2.1.2.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.2.3"><ci id="A1.SS2.1.p1.9.m6.2.2.1.2.3.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.2.3.1">→</ci><ci id="A1.SS2.1.p1.9.m6.2.2.1.2.3.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.2.3.2">𝑛</ci><infinity id="A1.SS2.1.p1.9.m6.2.2.1.2.3.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.2.3.3"></infinity></apply></apply><apply id="A1.SS2.1.p1.9.m6.2.2.1.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1"><times id="A1.SS2.1.p1.9.m6.2.2.1.1.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.2"></times><ci id="A1.SS2.1.p1.9.m6.2.2.1.1.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.3">𝑛</ci><apply id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1"><minus id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.2"></minus><apply id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3">subscript</csymbol><ci id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3.2">𝑍</ci><ci id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.3.3">𝜃</ci></apply><apply id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1"><times id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.2"></times><ci id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.3">𝔼</ci><apply id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.2.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1.2">𝑇</ci><ci id="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.1.1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></apply></apply></apply><apply id="A1.SS2.1.p1.9.m6.2.2.4.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4"><plus id="A1.SS2.1.p1.9.m6.2.2.4.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.1"></plus><apply id="A1.SS2.1.p1.9.m6.2.2.4.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2"><apply id="A1.SS2.1.p1.9.m6.2.2.4.2.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1"><csymbol cd="ambiguous" id="A1.SS2.1.p1.9.m6.2.2.4.2.1.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1">subscript</csymbol><limit id="A1.SS2.1.p1.9.m6.2.2.4.2.1.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1.2"></limit><apply id="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1.3"><ci id="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.1">→</ci><ci id="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.2">𝑛</ci><infinity id="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.1.3.3"></infinity></apply></apply><apply id="A1.SS2.1.p1.9.m6.2.2.4.2.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2"><divide id="A1.SS2.1.p1.9.m6.2.2.4.2.2.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2"></divide><apply id="A1.SS2.1.p1.9.m6.2.2.4.2.2.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.2"><csymbol cd="ambiguous" id="A1.SS2.1.p1.9.m6.2.2.4.2.2.2.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.2">subscript</csymbol><ci id="A1.SS2.1.p1.9.m6.2.2.4.2.2.2.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.2.2">𝜇</ci><cn type="integer" id="A1.SS2.1.p1.9.m6.2.2.4.2.2.2.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.2.3">2</cn></apply><apply id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3"><times id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.1"></times><cn type="integer" id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.2">2</cn><apply id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3"><csymbol cd="ambiguous" id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3">superscript</csymbol><ci id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3.2">𝜇</ci><cn type="integer" id="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.2.2.3.3.3">2</cn></apply></apply></apply></apply><apply id="A1.SS2.1.p1.9.m6.2.2.4.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.3"><times id="A1.SS2.1.p1.9.m6.2.2.4.3.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.3.1"></times><ci id="A1.SS2.1.p1.9.m6.2.2.4.3.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.4.3.2">𝑜</ci><cn type="integer" id="A1.SS2.1.p1.9.m6.1.1.cmml" xref="A1.SS2.1.p1.9.m6.1.1">1</cn></apply></apply></apply><apply id="A1.SS2.1.p1.9.m6.2.2c.cmml" xref="A1.SS2.1.p1.9.m6.2.2"><eq id="A1.SS2.1.p1.9.m6.2.2.5.cmml" xref="A1.SS2.1.p1.9.m6.2.2.5"></eq><share href="#A1.SS2.1.p1.9.m6.2.2.4.cmml" id="A1.SS2.1.p1.9.m6.2.2d.cmml" xref="A1.SS2.1.p1.9.m6.2.2"></share><apply id="A1.SS2.1.p1.9.m6.2.2.6.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6"><divide id="A1.SS2.1.p1.9.m6.2.2.6.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6"></divide><apply id="A1.SS2.1.p1.9.m6.2.2.6.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6.2"><csymbol cd="ambiguous" id="A1.SS2.1.p1.9.m6.2.2.6.2.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6.2">subscript</csymbol><ci id="A1.SS2.1.p1.9.m6.2.2.6.2.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6.2.2">𝜇</ci><cn type="integer" id="A1.SS2.1.p1.9.m6.2.2.6.2.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6.2.3">2</cn></apply><apply id="A1.SS2.1.p1.9.m6.2.2.6.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6.3"><times id="A1.SS2.1.p1.9.m6.2.2.6.3.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6.3.1"></times><cn type="integer" id="A1.SS2.1.p1.9.m6.2.2.6.3.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6.3.2">2</cn><apply id="A1.SS2.1.p1.9.m6.2.2.6.3.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6.3.3"><csymbol cd="ambiguous" id="A1.SS2.1.p1.9.m6.2.2.6.3.3.1.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6.3.3">superscript</csymbol><ci id="A1.SS2.1.p1.9.m6.2.2.6.3.3.2.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6.3.3.2">𝜇</ci><cn type="integer" id="A1.SS2.1.p1.9.m6.2.2.6.3.3.3.cmml" xref="A1.SS2.1.p1.9.m6.2.2.6.3.3.3">2</cn></apply></apply></apply></apply><apply id="A1.SS2.1.p1.9.m6.2.2e.cmml" xref="A1.SS2.1.p1.9.m6.2.2"><gt id="A1.SS2.1.p1.9.m6.2.2.7.cmml" xref="A1.SS2.1.p1.9.m6.2.2.7"></gt><share href="#A1.SS2.1.p1.9.m6.2.2.6.cmml" id="A1.SS2.1.p1.9.m6.2.2f.cmml" xref="A1.SS2.1.p1.9.m6.2.2"></share><cn type="integer" id="A1.SS2.1.p1.9.m6.2.2.8.cmml" xref="A1.SS2.1.p1.9.m6.2.2.8">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.9.m6.2c">\lim_{n\to\infty}n(Z_{\theta}-\mathbb{E}[T_{n}])=\lim_{n\to\infty}\frac{\mu_{2}}{2\mu^{2}}+o(1)=\frac{\mu_{2}}{2\mu^{2}}&gt;0</annotation></semantics></math>, so <math id="A1.SS2.1.p1.10.m7.1" class="ltx_Math" alttext="\exists N_{2}&gt;0" display="inline"><semantics id="A1.SS2.1.p1.10.m7.1a"><mrow id="A1.SS2.1.p1.10.m7.1.1" xref="A1.SS2.1.p1.10.m7.1.1.cmml"><mrow id="A1.SS2.1.p1.10.m7.1.1.2" xref="A1.SS2.1.p1.10.m7.1.1.2.cmml"><mo rspace="0.167em" id="A1.SS2.1.p1.10.m7.1.1.2.1" xref="A1.SS2.1.p1.10.m7.1.1.2.1.cmml">∃</mo><msub id="A1.SS2.1.p1.10.m7.1.1.2.2" xref="A1.SS2.1.p1.10.m7.1.1.2.2.cmml"><mi id="A1.SS2.1.p1.10.m7.1.1.2.2.2" xref="A1.SS2.1.p1.10.m7.1.1.2.2.2.cmml">N</mi><mn id="A1.SS2.1.p1.10.m7.1.1.2.2.3" xref="A1.SS2.1.p1.10.m7.1.1.2.2.3.cmml">2</mn></msub></mrow><mo id="A1.SS2.1.p1.10.m7.1.1.1" xref="A1.SS2.1.p1.10.m7.1.1.1.cmml">&gt;</mo><mn id="A1.SS2.1.p1.10.m7.1.1.3" xref="A1.SS2.1.p1.10.m7.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.10.m7.1b"><apply id="A1.SS2.1.p1.10.m7.1.1.cmml" xref="A1.SS2.1.p1.10.m7.1.1"><gt id="A1.SS2.1.p1.10.m7.1.1.1.cmml" xref="A1.SS2.1.p1.10.m7.1.1.1"></gt><apply id="A1.SS2.1.p1.10.m7.1.1.2.cmml" xref="A1.SS2.1.p1.10.m7.1.1.2"><exists id="A1.SS2.1.p1.10.m7.1.1.2.1.cmml" xref="A1.SS2.1.p1.10.m7.1.1.2.1"></exists><apply id="A1.SS2.1.p1.10.m7.1.1.2.2.cmml" xref="A1.SS2.1.p1.10.m7.1.1.2.2"><csymbol cd="ambiguous" id="A1.SS2.1.p1.10.m7.1.1.2.2.1.cmml" xref="A1.SS2.1.p1.10.m7.1.1.2.2">subscript</csymbol><ci id="A1.SS2.1.p1.10.m7.1.1.2.2.2.cmml" xref="A1.SS2.1.p1.10.m7.1.1.2.2.2">𝑁</ci><cn type="integer" id="A1.SS2.1.p1.10.m7.1.1.2.2.3.cmml" xref="A1.SS2.1.p1.10.m7.1.1.2.2.3">2</cn></apply></apply><cn type="integer" id="A1.SS2.1.p1.10.m7.1.1.3.cmml" xref="A1.SS2.1.p1.10.m7.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.10.m7.1c">\exists N_{2}&gt;0</annotation></semantics></math> such that when <math id="A1.SS2.1.p1.11.m8.1" class="ltx_Math" alttext="n&gt;N_{2}" display="inline"><semantics id="A1.SS2.1.p1.11.m8.1a"><mrow id="A1.SS2.1.p1.11.m8.1.1" xref="A1.SS2.1.p1.11.m8.1.1.cmml"><mi id="A1.SS2.1.p1.11.m8.1.1.2" xref="A1.SS2.1.p1.11.m8.1.1.2.cmml">n</mi><mo id="A1.SS2.1.p1.11.m8.1.1.1" xref="A1.SS2.1.p1.11.m8.1.1.1.cmml">&gt;</mo><msub id="A1.SS2.1.p1.11.m8.1.1.3" xref="A1.SS2.1.p1.11.m8.1.1.3.cmml"><mi id="A1.SS2.1.p1.11.m8.1.1.3.2" xref="A1.SS2.1.p1.11.m8.1.1.3.2.cmml">N</mi><mn id="A1.SS2.1.p1.11.m8.1.1.3.3" xref="A1.SS2.1.p1.11.m8.1.1.3.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.11.m8.1b"><apply id="A1.SS2.1.p1.11.m8.1.1.cmml" xref="A1.SS2.1.p1.11.m8.1.1"><gt id="A1.SS2.1.p1.11.m8.1.1.1.cmml" xref="A1.SS2.1.p1.11.m8.1.1.1"></gt><ci id="A1.SS2.1.p1.11.m8.1.1.2.cmml" xref="A1.SS2.1.p1.11.m8.1.1.2">𝑛</ci><apply id="A1.SS2.1.p1.11.m8.1.1.3.cmml" xref="A1.SS2.1.p1.11.m8.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.1.p1.11.m8.1.1.3.1.cmml" xref="A1.SS2.1.p1.11.m8.1.1.3">subscript</csymbol><ci id="A1.SS2.1.p1.11.m8.1.1.3.2.cmml" xref="A1.SS2.1.p1.11.m8.1.1.3.2">𝑁</ci><cn type="integer" id="A1.SS2.1.p1.11.m8.1.1.3.3.cmml" xref="A1.SS2.1.p1.11.m8.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.11.m8.1c">n&gt;N_{2}</annotation></semantics></math> we have <math id="A1.SS2.1.p1.12.m9.1" class="ltx_Math" alttext="Z_{\theta}&gt;\mathbb{E}[T_{n}]" display="inline"><semantics id="A1.SS2.1.p1.12.m9.1a"><mrow id="A1.SS2.1.p1.12.m9.1.1" xref="A1.SS2.1.p1.12.m9.1.1.cmml"><msub id="A1.SS2.1.p1.12.m9.1.1.3" xref="A1.SS2.1.p1.12.m9.1.1.3.cmml"><mi id="A1.SS2.1.p1.12.m9.1.1.3.2" xref="A1.SS2.1.p1.12.m9.1.1.3.2.cmml">Z</mi><mi id="A1.SS2.1.p1.12.m9.1.1.3.3" xref="A1.SS2.1.p1.12.m9.1.1.3.3.cmml">θ</mi></msub><mo id="A1.SS2.1.p1.12.m9.1.1.2" xref="A1.SS2.1.p1.12.m9.1.1.2.cmml">&gt;</mo><mrow id="A1.SS2.1.p1.12.m9.1.1.1" xref="A1.SS2.1.p1.12.m9.1.1.1.cmml"><mi id="A1.SS2.1.p1.12.m9.1.1.1.3" xref="A1.SS2.1.p1.12.m9.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.12.m9.1.1.1.2" xref="A1.SS2.1.p1.12.m9.1.1.1.2.cmml">​</mo><mrow id="A1.SS2.1.p1.12.m9.1.1.1.1.1" xref="A1.SS2.1.p1.12.m9.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.1.p1.12.m9.1.1.1.1.1.2" xref="A1.SS2.1.p1.12.m9.1.1.1.1.2.1.cmml">[</mo><msub id="A1.SS2.1.p1.12.m9.1.1.1.1.1.1" xref="A1.SS2.1.p1.12.m9.1.1.1.1.1.1.cmml"><mi id="A1.SS2.1.p1.12.m9.1.1.1.1.1.1.2" xref="A1.SS2.1.p1.12.m9.1.1.1.1.1.1.2.cmml">T</mi><mi id="A1.SS2.1.p1.12.m9.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.12.m9.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.SS2.1.p1.12.m9.1.1.1.1.1.3" xref="A1.SS2.1.p1.12.m9.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.12.m9.1b"><apply id="A1.SS2.1.p1.12.m9.1.1.cmml" xref="A1.SS2.1.p1.12.m9.1.1"><gt id="A1.SS2.1.p1.12.m9.1.1.2.cmml" xref="A1.SS2.1.p1.12.m9.1.1.2"></gt><apply id="A1.SS2.1.p1.12.m9.1.1.3.cmml" xref="A1.SS2.1.p1.12.m9.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.1.p1.12.m9.1.1.3.1.cmml" xref="A1.SS2.1.p1.12.m9.1.1.3">subscript</csymbol><ci id="A1.SS2.1.p1.12.m9.1.1.3.2.cmml" xref="A1.SS2.1.p1.12.m9.1.1.3.2">𝑍</ci><ci id="A1.SS2.1.p1.12.m9.1.1.3.3.cmml" xref="A1.SS2.1.p1.12.m9.1.1.3.3">𝜃</ci></apply><apply id="A1.SS2.1.p1.12.m9.1.1.1.cmml" xref="A1.SS2.1.p1.12.m9.1.1.1"><times id="A1.SS2.1.p1.12.m9.1.1.1.2.cmml" xref="A1.SS2.1.p1.12.m9.1.1.1.2"></times><ci id="A1.SS2.1.p1.12.m9.1.1.1.3.cmml" xref="A1.SS2.1.p1.12.m9.1.1.1.3">𝔼</ci><apply id="A1.SS2.1.p1.12.m9.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.12.m9.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.1.p1.12.m9.1.1.1.1.2.1.cmml" xref="A1.SS2.1.p1.12.m9.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.1.p1.12.m9.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.12.m9.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.1.p1.12.m9.1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.12.m9.1.1.1.1.1.1">subscript</csymbol><ci id="A1.SS2.1.p1.12.m9.1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.12.m9.1.1.1.1.1.1.2">𝑇</ci><ci id="A1.SS2.1.p1.12.m9.1.1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.12.m9.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.12.m9.1c">Z_{\theta}&gt;\mathbb{E}[T_{n}]</annotation></semantics></math>. Up to this point, we have proved that <math id="A1.SS2.1.p1.13.m10.1" class="ltx_Math" alttext="Z_{\theta}-\epsilon&lt;\mathbb{E}[T_{n}]&lt;Z_{\theta}" display="inline"><semantics id="A1.SS2.1.p1.13.m10.1a"><mrow id="A1.SS2.1.p1.13.m10.1.1" xref="A1.SS2.1.p1.13.m10.1.1.cmml"><mrow id="A1.SS2.1.p1.13.m10.1.1.3" xref="A1.SS2.1.p1.13.m10.1.1.3.cmml"><msub id="A1.SS2.1.p1.13.m10.1.1.3.2" xref="A1.SS2.1.p1.13.m10.1.1.3.2.cmml"><mi id="A1.SS2.1.p1.13.m10.1.1.3.2.2" xref="A1.SS2.1.p1.13.m10.1.1.3.2.2.cmml">Z</mi><mi id="A1.SS2.1.p1.13.m10.1.1.3.2.3" xref="A1.SS2.1.p1.13.m10.1.1.3.2.3.cmml">θ</mi></msub><mo id="A1.SS2.1.p1.13.m10.1.1.3.1" xref="A1.SS2.1.p1.13.m10.1.1.3.1.cmml">−</mo><mi id="A1.SS2.1.p1.13.m10.1.1.3.3" xref="A1.SS2.1.p1.13.m10.1.1.3.3.cmml">ϵ</mi></mrow><mo id="A1.SS2.1.p1.13.m10.1.1.4" xref="A1.SS2.1.p1.13.m10.1.1.4.cmml">&lt;</mo><mrow id="A1.SS2.1.p1.13.m10.1.1.1" xref="A1.SS2.1.p1.13.m10.1.1.1.cmml"><mi id="A1.SS2.1.p1.13.m10.1.1.1.3" xref="A1.SS2.1.p1.13.m10.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.1.p1.13.m10.1.1.1.2" xref="A1.SS2.1.p1.13.m10.1.1.1.2.cmml">​</mo><mrow id="A1.SS2.1.p1.13.m10.1.1.1.1.1" xref="A1.SS2.1.p1.13.m10.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.1.p1.13.m10.1.1.1.1.1.2" xref="A1.SS2.1.p1.13.m10.1.1.1.1.2.1.cmml">[</mo><msub id="A1.SS2.1.p1.13.m10.1.1.1.1.1.1" xref="A1.SS2.1.p1.13.m10.1.1.1.1.1.1.cmml"><mi id="A1.SS2.1.p1.13.m10.1.1.1.1.1.1.2" xref="A1.SS2.1.p1.13.m10.1.1.1.1.1.1.2.cmml">T</mi><mi id="A1.SS2.1.p1.13.m10.1.1.1.1.1.1.3" xref="A1.SS2.1.p1.13.m10.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.SS2.1.p1.13.m10.1.1.1.1.1.3" xref="A1.SS2.1.p1.13.m10.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.SS2.1.p1.13.m10.1.1.5" xref="A1.SS2.1.p1.13.m10.1.1.5.cmml">&lt;</mo><msub id="A1.SS2.1.p1.13.m10.1.1.6" xref="A1.SS2.1.p1.13.m10.1.1.6.cmml"><mi id="A1.SS2.1.p1.13.m10.1.1.6.2" xref="A1.SS2.1.p1.13.m10.1.1.6.2.cmml">Z</mi><mi id="A1.SS2.1.p1.13.m10.1.1.6.3" xref="A1.SS2.1.p1.13.m10.1.1.6.3.cmml">θ</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.1.p1.13.m10.1b"><apply id="A1.SS2.1.p1.13.m10.1.1.cmml" xref="A1.SS2.1.p1.13.m10.1.1"><and id="A1.SS2.1.p1.13.m10.1.1a.cmml" xref="A1.SS2.1.p1.13.m10.1.1"></and><apply id="A1.SS2.1.p1.13.m10.1.1b.cmml" xref="A1.SS2.1.p1.13.m10.1.1"><lt id="A1.SS2.1.p1.13.m10.1.1.4.cmml" xref="A1.SS2.1.p1.13.m10.1.1.4"></lt><apply id="A1.SS2.1.p1.13.m10.1.1.3.cmml" xref="A1.SS2.1.p1.13.m10.1.1.3"><minus id="A1.SS2.1.p1.13.m10.1.1.3.1.cmml" xref="A1.SS2.1.p1.13.m10.1.1.3.1"></minus><apply id="A1.SS2.1.p1.13.m10.1.1.3.2.cmml" xref="A1.SS2.1.p1.13.m10.1.1.3.2"><csymbol cd="ambiguous" id="A1.SS2.1.p1.13.m10.1.1.3.2.1.cmml" xref="A1.SS2.1.p1.13.m10.1.1.3.2">subscript</csymbol><ci id="A1.SS2.1.p1.13.m10.1.1.3.2.2.cmml" xref="A1.SS2.1.p1.13.m10.1.1.3.2.2">𝑍</ci><ci id="A1.SS2.1.p1.13.m10.1.1.3.2.3.cmml" xref="A1.SS2.1.p1.13.m10.1.1.3.2.3">𝜃</ci></apply><ci id="A1.SS2.1.p1.13.m10.1.1.3.3.cmml" xref="A1.SS2.1.p1.13.m10.1.1.3.3">italic-ϵ</ci></apply><apply id="A1.SS2.1.p1.13.m10.1.1.1.cmml" xref="A1.SS2.1.p1.13.m10.1.1.1"><times id="A1.SS2.1.p1.13.m10.1.1.1.2.cmml" xref="A1.SS2.1.p1.13.m10.1.1.1.2"></times><ci id="A1.SS2.1.p1.13.m10.1.1.1.3.cmml" xref="A1.SS2.1.p1.13.m10.1.1.1.3">𝔼</ci><apply id="A1.SS2.1.p1.13.m10.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.13.m10.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.1.p1.13.m10.1.1.1.1.2.1.cmml" xref="A1.SS2.1.p1.13.m10.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.1.p1.13.m10.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.13.m10.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.1.p1.13.m10.1.1.1.1.1.1.1.cmml" xref="A1.SS2.1.p1.13.m10.1.1.1.1.1.1">subscript</csymbol><ci id="A1.SS2.1.p1.13.m10.1.1.1.1.1.1.2.cmml" xref="A1.SS2.1.p1.13.m10.1.1.1.1.1.1.2">𝑇</ci><ci id="A1.SS2.1.p1.13.m10.1.1.1.1.1.1.3.cmml" xref="A1.SS2.1.p1.13.m10.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></apply><apply id="A1.SS2.1.p1.13.m10.1.1c.cmml" xref="A1.SS2.1.p1.13.m10.1.1"><lt id="A1.SS2.1.p1.13.m10.1.1.5.cmml" xref="A1.SS2.1.p1.13.m10.1.1.5"></lt><share href="#A1.SS2.1.p1.13.m10.1.1.1.cmml" id="A1.SS2.1.p1.13.m10.1.1d.cmml" xref="A1.SS2.1.p1.13.m10.1.1"></share><apply id="A1.SS2.1.p1.13.m10.1.1.6.cmml" xref="A1.SS2.1.p1.13.m10.1.1.6"><csymbol cd="ambiguous" id="A1.SS2.1.p1.13.m10.1.1.6.1.cmml" xref="A1.SS2.1.p1.13.m10.1.1.6">subscript</csymbol><ci id="A1.SS2.1.p1.13.m10.1.1.6.2.cmml" xref="A1.SS2.1.p1.13.m10.1.1.6.2">𝑍</ci><ci id="A1.SS2.1.p1.13.m10.1.1.6.3.cmml" xref="A1.SS2.1.p1.13.m10.1.1.6.3">𝜃</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.1.p1.13.m10.1c">Z_{\theta}-\epsilon&lt;\mathbb{E}[T_{n}]&lt;Z_{\theta}</annotation></semantics></math>.</p>
</div>
<div id="A1.SS2.2.p2" class="ltx_para ltx_noindent">
<p id="A1.SS2.2.p2.12" class="ltx_p">For the other half part of the proof, using Eq. <a href="#A1.E8" title="In Proof. ‣ A.2 Proof of Theorem 2 ‣ Appendix A Appendix ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> we have</p>
<table id="A1.E10" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E10.m1.2" class="ltx_Math" alttext="\mathbb{E}[T_{n}]=Z_{\theta}-\frac{\mu_{2}}{2\mu^{2}}\frac{1}{n}+\frac{c}{n^{2}}+o(n^{-2})" display="block"><semantics id="A1.E10.m1.2a"><mrow id="A1.E10.m1.2.2" xref="A1.E10.m1.2.2.cmml"><mrow id="A1.E10.m1.1.1.1" xref="A1.E10.m1.1.1.1.cmml"><mi id="A1.E10.m1.1.1.1.3" xref="A1.E10.m1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.E10.m1.1.1.1.2" xref="A1.E10.m1.1.1.1.2.cmml">​</mo><mrow id="A1.E10.m1.1.1.1.1.1" xref="A1.E10.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E10.m1.1.1.1.1.1.2" xref="A1.E10.m1.1.1.1.1.2.1.cmml">[</mo><msub id="A1.E10.m1.1.1.1.1.1.1" xref="A1.E10.m1.1.1.1.1.1.1.cmml"><mi id="A1.E10.m1.1.1.1.1.1.1.2" xref="A1.E10.m1.1.1.1.1.1.1.2.cmml">T</mi><mi id="A1.E10.m1.1.1.1.1.1.1.3" xref="A1.E10.m1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.E10.m1.1.1.1.1.1.3" xref="A1.E10.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.E10.m1.2.2.3" xref="A1.E10.m1.2.2.3.cmml">=</mo><mrow id="A1.E10.m1.2.2.2" xref="A1.E10.m1.2.2.2.cmml"><mrow id="A1.E10.m1.2.2.2.3" xref="A1.E10.m1.2.2.2.3.cmml"><msub id="A1.E10.m1.2.2.2.3.2" xref="A1.E10.m1.2.2.2.3.2.cmml"><mi id="A1.E10.m1.2.2.2.3.2.2" xref="A1.E10.m1.2.2.2.3.2.2.cmml">Z</mi><mi id="A1.E10.m1.2.2.2.3.2.3" xref="A1.E10.m1.2.2.2.3.2.3.cmml">θ</mi></msub><mo id="A1.E10.m1.2.2.2.3.1" xref="A1.E10.m1.2.2.2.3.1.cmml">−</mo><mrow id="A1.E10.m1.2.2.2.3.3" xref="A1.E10.m1.2.2.2.3.3.cmml"><mfrac id="A1.E10.m1.2.2.2.3.3.2" xref="A1.E10.m1.2.2.2.3.3.2.cmml"><msub id="A1.E10.m1.2.2.2.3.3.2.2" xref="A1.E10.m1.2.2.2.3.3.2.2.cmml"><mi id="A1.E10.m1.2.2.2.3.3.2.2.2" xref="A1.E10.m1.2.2.2.3.3.2.2.2.cmml">μ</mi><mn id="A1.E10.m1.2.2.2.3.3.2.2.3" xref="A1.E10.m1.2.2.2.3.3.2.2.3.cmml">2</mn></msub><mrow id="A1.E10.m1.2.2.2.3.3.2.3" xref="A1.E10.m1.2.2.2.3.3.2.3.cmml"><mn id="A1.E10.m1.2.2.2.3.3.2.3.2" xref="A1.E10.m1.2.2.2.3.3.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.E10.m1.2.2.2.3.3.2.3.1" xref="A1.E10.m1.2.2.2.3.3.2.3.1.cmml">​</mo><msup id="A1.E10.m1.2.2.2.3.3.2.3.3" xref="A1.E10.m1.2.2.2.3.3.2.3.3.cmml"><mi id="A1.E10.m1.2.2.2.3.3.2.3.3.2" xref="A1.E10.m1.2.2.2.3.3.2.3.3.2.cmml">μ</mi><mn id="A1.E10.m1.2.2.2.3.3.2.3.3.3" xref="A1.E10.m1.2.2.2.3.3.2.3.3.3.cmml">2</mn></msup></mrow></mfrac><mo lspace="0em" rspace="0em" id="A1.E10.m1.2.2.2.3.3.1" xref="A1.E10.m1.2.2.2.3.3.1.cmml">​</mo><mfrac id="A1.E10.m1.2.2.2.3.3.3" xref="A1.E10.m1.2.2.2.3.3.3.cmml"><mn id="A1.E10.m1.2.2.2.3.3.3.2" xref="A1.E10.m1.2.2.2.3.3.3.2.cmml">1</mn><mi id="A1.E10.m1.2.2.2.3.3.3.3" xref="A1.E10.m1.2.2.2.3.3.3.3.cmml">n</mi></mfrac></mrow></mrow><mo id="A1.E10.m1.2.2.2.2" xref="A1.E10.m1.2.2.2.2.cmml">+</mo><mfrac id="A1.E10.m1.2.2.2.4" xref="A1.E10.m1.2.2.2.4.cmml"><mi id="A1.E10.m1.2.2.2.4.2" xref="A1.E10.m1.2.2.2.4.2.cmml">c</mi><msup id="A1.E10.m1.2.2.2.4.3" xref="A1.E10.m1.2.2.2.4.3.cmml"><mi id="A1.E10.m1.2.2.2.4.3.2" xref="A1.E10.m1.2.2.2.4.3.2.cmml">n</mi><mn id="A1.E10.m1.2.2.2.4.3.3" xref="A1.E10.m1.2.2.2.4.3.3.cmml">2</mn></msup></mfrac><mo id="A1.E10.m1.2.2.2.2a" xref="A1.E10.m1.2.2.2.2.cmml">+</mo><mrow id="A1.E10.m1.2.2.2.1" xref="A1.E10.m1.2.2.2.1.cmml"><mi id="A1.E10.m1.2.2.2.1.3" xref="A1.E10.m1.2.2.2.1.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A1.E10.m1.2.2.2.1.2" xref="A1.E10.m1.2.2.2.1.2.cmml">​</mo><mrow id="A1.E10.m1.2.2.2.1.1.1" xref="A1.E10.m1.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="A1.E10.m1.2.2.2.1.1.1.2" xref="A1.E10.m1.2.2.2.1.1.1.1.cmml">(</mo><msup id="A1.E10.m1.2.2.2.1.1.1.1" xref="A1.E10.m1.2.2.2.1.1.1.1.cmml"><mi id="A1.E10.m1.2.2.2.1.1.1.1.2" xref="A1.E10.m1.2.2.2.1.1.1.1.2.cmml">n</mi><mrow id="A1.E10.m1.2.2.2.1.1.1.1.3" xref="A1.E10.m1.2.2.2.1.1.1.1.3.cmml"><mo id="A1.E10.m1.2.2.2.1.1.1.1.3a" xref="A1.E10.m1.2.2.2.1.1.1.1.3.cmml">−</mo><mn id="A1.E10.m1.2.2.2.1.1.1.1.3.2" xref="A1.E10.m1.2.2.2.1.1.1.1.3.2.cmml">2</mn></mrow></msup><mo stretchy="false" id="A1.E10.m1.2.2.2.1.1.1.3" xref="A1.E10.m1.2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E10.m1.2b"><apply id="A1.E10.m1.2.2.cmml" xref="A1.E10.m1.2.2"><eq id="A1.E10.m1.2.2.3.cmml" xref="A1.E10.m1.2.2.3"></eq><apply id="A1.E10.m1.1.1.1.cmml" xref="A1.E10.m1.1.1.1"><times id="A1.E10.m1.1.1.1.2.cmml" xref="A1.E10.m1.1.1.1.2"></times><ci id="A1.E10.m1.1.1.1.3.cmml" xref="A1.E10.m1.1.1.1.3">𝔼</ci><apply id="A1.E10.m1.1.1.1.1.2.cmml" xref="A1.E10.m1.1.1.1.1.1"><csymbol cd="latexml" id="A1.E10.m1.1.1.1.1.2.1.cmml" xref="A1.E10.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.E10.m1.1.1.1.1.1.1.cmml" xref="A1.E10.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.E10.m1.1.1.1.1.1.1.1.cmml" xref="A1.E10.m1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.E10.m1.1.1.1.1.1.1.2.cmml" xref="A1.E10.m1.1.1.1.1.1.1.2">𝑇</ci><ci id="A1.E10.m1.1.1.1.1.1.1.3.cmml" xref="A1.E10.m1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply><apply id="A1.E10.m1.2.2.2.cmml" xref="A1.E10.m1.2.2.2"><plus id="A1.E10.m1.2.2.2.2.cmml" xref="A1.E10.m1.2.2.2.2"></plus><apply id="A1.E10.m1.2.2.2.3.cmml" xref="A1.E10.m1.2.2.2.3"><minus id="A1.E10.m1.2.2.2.3.1.cmml" xref="A1.E10.m1.2.2.2.3.1"></minus><apply id="A1.E10.m1.2.2.2.3.2.cmml" xref="A1.E10.m1.2.2.2.3.2"><csymbol cd="ambiguous" id="A1.E10.m1.2.2.2.3.2.1.cmml" xref="A1.E10.m1.2.2.2.3.2">subscript</csymbol><ci id="A1.E10.m1.2.2.2.3.2.2.cmml" xref="A1.E10.m1.2.2.2.3.2.2">𝑍</ci><ci id="A1.E10.m1.2.2.2.3.2.3.cmml" xref="A1.E10.m1.2.2.2.3.2.3">𝜃</ci></apply><apply id="A1.E10.m1.2.2.2.3.3.cmml" xref="A1.E10.m1.2.2.2.3.3"><times id="A1.E10.m1.2.2.2.3.3.1.cmml" xref="A1.E10.m1.2.2.2.3.3.1"></times><apply id="A1.E10.m1.2.2.2.3.3.2.cmml" xref="A1.E10.m1.2.2.2.3.3.2"><divide id="A1.E10.m1.2.2.2.3.3.2.1.cmml" xref="A1.E10.m1.2.2.2.3.3.2"></divide><apply id="A1.E10.m1.2.2.2.3.3.2.2.cmml" xref="A1.E10.m1.2.2.2.3.3.2.2"><csymbol cd="ambiguous" id="A1.E10.m1.2.2.2.3.3.2.2.1.cmml" xref="A1.E10.m1.2.2.2.3.3.2.2">subscript</csymbol><ci id="A1.E10.m1.2.2.2.3.3.2.2.2.cmml" xref="A1.E10.m1.2.2.2.3.3.2.2.2">𝜇</ci><cn type="integer" id="A1.E10.m1.2.2.2.3.3.2.2.3.cmml" xref="A1.E10.m1.2.2.2.3.3.2.2.3">2</cn></apply><apply id="A1.E10.m1.2.2.2.3.3.2.3.cmml" xref="A1.E10.m1.2.2.2.3.3.2.3"><times id="A1.E10.m1.2.2.2.3.3.2.3.1.cmml" xref="A1.E10.m1.2.2.2.3.3.2.3.1"></times><cn type="integer" id="A1.E10.m1.2.2.2.3.3.2.3.2.cmml" xref="A1.E10.m1.2.2.2.3.3.2.3.2">2</cn><apply id="A1.E10.m1.2.2.2.3.3.2.3.3.cmml" xref="A1.E10.m1.2.2.2.3.3.2.3.3"><csymbol cd="ambiguous" id="A1.E10.m1.2.2.2.3.3.2.3.3.1.cmml" xref="A1.E10.m1.2.2.2.3.3.2.3.3">superscript</csymbol><ci id="A1.E10.m1.2.2.2.3.3.2.3.3.2.cmml" xref="A1.E10.m1.2.2.2.3.3.2.3.3.2">𝜇</ci><cn type="integer" id="A1.E10.m1.2.2.2.3.3.2.3.3.3.cmml" xref="A1.E10.m1.2.2.2.3.3.2.3.3.3">2</cn></apply></apply></apply><apply id="A1.E10.m1.2.2.2.3.3.3.cmml" xref="A1.E10.m1.2.2.2.3.3.3"><divide id="A1.E10.m1.2.2.2.3.3.3.1.cmml" xref="A1.E10.m1.2.2.2.3.3.3"></divide><cn type="integer" id="A1.E10.m1.2.2.2.3.3.3.2.cmml" xref="A1.E10.m1.2.2.2.3.3.3.2">1</cn><ci id="A1.E10.m1.2.2.2.3.3.3.3.cmml" xref="A1.E10.m1.2.2.2.3.3.3.3">𝑛</ci></apply></apply></apply><apply id="A1.E10.m1.2.2.2.4.cmml" xref="A1.E10.m1.2.2.2.4"><divide id="A1.E10.m1.2.2.2.4.1.cmml" xref="A1.E10.m1.2.2.2.4"></divide><ci id="A1.E10.m1.2.2.2.4.2.cmml" xref="A1.E10.m1.2.2.2.4.2">𝑐</ci><apply id="A1.E10.m1.2.2.2.4.3.cmml" xref="A1.E10.m1.2.2.2.4.3"><csymbol cd="ambiguous" id="A1.E10.m1.2.2.2.4.3.1.cmml" xref="A1.E10.m1.2.2.2.4.3">superscript</csymbol><ci id="A1.E10.m1.2.2.2.4.3.2.cmml" xref="A1.E10.m1.2.2.2.4.3.2">𝑛</ci><cn type="integer" id="A1.E10.m1.2.2.2.4.3.3.cmml" xref="A1.E10.m1.2.2.2.4.3.3">2</cn></apply></apply><apply id="A1.E10.m1.2.2.2.1.cmml" xref="A1.E10.m1.2.2.2.1"><times id="A1.E10.m1.2.2.2.1.2.cmml" xref="A1.E10.m1.2.2.2.1.2"></times><ci id="A1.E10.m1.2.2.2.1.3.cmml" xref="A1.E10.m1.2.2.2.1.3">𝑜</ci><apply id="A1.E10.m1.2.2.2.1.1.1.1.cmml" xref="A1.E10.m1.2.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.E10.m1.2.2.2.1.1.1.1.1.cmml" xref="A1.E10.m1.2.2.2.1.1.1">superscript</csymbol><ci id="A1.E10.m1.2.2.2.1.1.1.1.2.cmml" xref="A1.E10.m1.2.2.2.1.1.1.1.2">𝑛</ci><apply id="A1.E10.m1.2.2.2.1.1.1.1.3.cmml" xref="A1.E10.m1.2.2.2.1.1.1.1.3"><minus id="A1.E10.m1.2.2.2.1.1.1.1.3.1.cmml" xref="A1.E10.m1.2.2.2.1.1.1.1.3"></minus><cn type="integer" id="A1.E10.m1.2.2.2.1.1.1.1.3.2.cmml" xref="A1.E10.m1.2.2.2.1.1.1.1.3.2">2</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E10.m1.2c">\mathbb{E}[T_{n}]=Z_{\theta}-\frac{\mu_{2}}{2\mu^{2}}\frac{1}{n}+\frac{c}{n^{2}}+o(n^{-2})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(10)</span></td>
</tr></tbody>
</table>
<p id="A1.SS2.2.p2.11" class="ltx_p">where <math id="A1.SS2.2.p2.1.m1.1" class="ltx_Math" alttext="c" display="inline"><semantics id="A1.SS2.2.p2.1.m1.1a"><mi id="A1.SS2.2.p2.1.m1.1.1" xref="A1.SS2.2.p2.1.m1.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.2.p2.1.m1.1b"><ci id="A1.SS2.2.p2.1.m1.1.1.cmml" xref="A1.SS2.2.p2.1.m1.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.2.p2.1.m1.1c">c</annotation></semantics></math> is a constant. Therefore, <math id="A1.SS2.2.p2.2.m2.6" class="ltx_Math" alttext="\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]=(2n-1)\mathbb{E}[T_{n}]-2(n-1)\mathbb{E}[T_{n-1}]=Z_{\theta}+\frac{\mu_{2}}{2\mu^{2}}\frac{1}{n}+o(n^{-1})" display="inline"><semantics id="A1.SS2.2.p2.2.m2.6a"><mrow id="A1.SS2.2.p2.2.m2.6.6" xref="A1.SS2.2.p2.2.m2.6.6.cmml"><mrow id="A1.SS2.2.p2.2.m2.1.1.1" xref="A1.SS2.2.p2.2.m2.1.1.1.cmml"><mi id="A1.SS2.2.p2.2.m2.1.1.1.3" xref="A1.SS2.2.p2.2.m2.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.1.1.1.2" xref="A1.SS2.2.p2.2.m2.1.1.1.2.cmml">​</mo><mrow id="A1.SS2.2.p2.2.m2.1.1.1.1.1" xref="A1.SS2.2.p2.2.m2.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.2.1.cmml">[</mo><mrow id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.cmml"><mn id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.1" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.3.cmml">n</mi></mrow><mo id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mn id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.2.cmml">​</mo><msub id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3.cmml"><mi id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3.2.cmml">T</mi><mi id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3.3.cmml">n</mi></msub></mrow><mo id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.3.cmml">−</mo><mrow id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.cmml"><mn id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.2.cmml">​</mo><mrow id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><mrow id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.cmml"><mi id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.2.cmml">n</mi><mo id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.1" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.1.cmml">−</mo><mn id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.2a" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.2.cmml">​</mo><msub id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.cmml"><mi id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.2.cmml">T</mi><mrow id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.cmml"><mi id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.2" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.2.cmml">n</mi><mo id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.1" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.1.cmml">−</mo><mn id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.3.cmml">1</mn></mrow></msub></mrow></mrow><mo stretchy="false" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.3" xref="A1.SS2.2.p2.2.m2.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.SS2.2.p2.2.m2.6.6.8" xref="A1.SS2.2.p2.2.m2.6.6.8.cmml">=</mo><mrow id="A1.SS2.2.p2.2.m2.5.5.5" xref="A1.SS2.2.p2.2.m2.5.5.5.cmml"><mrow id="A1.SS2.2.p2.2.m2.3.3.3.2" xref="A1.SS2.2.p2.2.m2.3.3.3.2.cmml"><mrow id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.2" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.cmml"><mn id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.2" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.1" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.1.cmml">​</mo><mi id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.3" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.3.cmml">n</mi></mrow><mo id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.1" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.1.cmml">−</mo><mn id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.3" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.3" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.3.3.3.2.3" xref="A1.SS2.2.p2.2.m2.3.3.3.2.3.cmml">​</mo><mi id="A1.SS2.2.p2.2.m2.3.3.3.2.4" xref="A1.SS2.2.p2.2.m2.3.3.3.2.4.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.3.3.3.2.3a" xref="A1.SS2.2.p2.2.m2.3.3.3.2.3.cmml">​</mo><mrow id="A1.SS2.2.p2.2.m2.3.3.3.2.2.1" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.2.cmml"><mo stretchy="false" id="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.2" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.2.1.cmml">[</mo><msub id="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1.cmml"><mi id="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1.2" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1.2.cmml">T</mi><mi id="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1.3" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.3" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.2.1.cmml">]</mo></mrow></mrow><mo id="A1.SS2.2.p2.2.m2.5.5.5.5" xref="A1.SS2.2.p2.2.m2.5.5.5.5.cmml">−</mo><mrow id="A1.SS2.2.p2.2.m2.5.5.5.4" xref="A1.SS2.2.p2.2.m2.5.5.5.4.cmml"><mn id="A1.SS2.2.p2.2.m2.5.5.5.4.4" xref="A1.SS2.2.p2.2.m2.5.5.5.4.4.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.5.5.5.4.3" xref="A1.SS2.2.p2.2.m2.5.5.5.4.3.cmml">​</mo><mrow id="A1.SS2.2.p2.2.m2.4.4.4.3.1.1" xref="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.2" xref="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.cmml">(</mo><mrow id="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1" xref="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.cmml"><mi id="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.2" xref="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.2.cmml">n</mi><mo id="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.1" xref="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.1.cmml">−</mo><mn id="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.3" xref="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.3" xref="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.5.5.5.4.3a" xref="A1.SS2.2.p2.2.m2.5.5.5.4.3.cmml">​</mo><mi id="A1.SS2.2.p2.2.m2.5.5.5.4.5" xref="A1.SS2.2.p2.2.m2.5.5.5.4.5.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.5.5.5.4.3b" xref="A1.SS2.2.p2.2.m2.5.5.5.4.3.cmml">​</mo><mrow id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.2.cmml"><mo stretchy="false" id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.2" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.2.1.cmml">[</mo><msub id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.cmml"><mi id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.2" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.2.cmml">T</mi><mrow id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.cmml"><mi id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.2" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.2.cmml">n</mi><mo id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.1" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.1.cmml">−</mo><mn id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.3" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.3.cmml">1</mn></mrow></msub><mo stretchy="false" id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.3" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.2.1.cmml">]</mo></mrow></mrow></mrow><mo id="A1.SS2.2.p2.2.m2.6.6.9" xref="A1.SS2.2.p2.2.m2.6.6.9.cmml">=</mo><mrow id="A1.SS2.2.p2.2.m2.6.6.6" xref="A1.SS2.2.p2.2.m2.6.6.6.cmml"><msub id="A1.SS2.2.p2.2.m2.6.6.6.3" xref="A1.SS2.2.p2.2.m2.6.6.6.3.cmml"><mi id="A1.SS2.2.p2.2.m2.6.6.6.3.2" xref="A1.SS2.2.p2.2.m2.6.6.6.3.2.cmml">Z</mi><mi id="A1.SS2.2.p2.2.m2.6.6.6.3.3" xref="A1.SS2.2.p2.2.m2.6.6.6.3.3.cmml">θ</mi></msub><mo id="A1.SS2.2.p2.2.m2.6.6.6.2" xref="A1.SS2.2.p2.2.m2.6.6.6.2.cmml">+</mo><mrow id="A1.SS2.2.p2.2.m2.6.6.6.4" xref="A1.SS2.2.p2.2.m2.6.6.6.4.cmml"><mfrac id="A1.SS2.2.p2.2.m2.6.6.6.4.2" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.cmml"><msub id="A1.SS2.2.p2.2.m2.6.6.6.4.2.2" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.2.cmml"><mi id="A1.SS2.2.p2.2.m2.6.6.6.4.2.2.2" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.2.2.cmml">μ</mi><mn id="A1.SS2.2.p2.2.m2.6.6.6.4.2.2.3" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.2.3.cmml">2</mn></msub><mrow id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.cmml"><mn id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.2" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.1" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.1.cmml">​</mo><msup id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3.cmml"><mi id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3.2" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3.2.cmml">μ</mi><mn id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3.3" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3.3.cmml">2</mn></msup></mrow></mfrac><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.6.6.6.4.1" xref="A1.SS2.2.p2.2.m2.6.6.6.4.1.cmml">​</mo><mfrac id="A1.SS2.2.p2.2.m2.6.6.6.4.3" xref="A1.SS2.2.p2.2.m2.6.6.6.4.3.cmml"><mn id="A1.SS2.2.p2.2.m2.6.6.6.4.3.2" xref="A1.SS2.2.p2.2.m2.6.6.6.4.3.2.cmml">1</mn><mi id="A1.SS2.2.p2.2.m2.6.6.6.4.3.3" xref="A1.SS2.2.p2.2.m2.6.6.6.4.3.3.cmml">n</mi></mfrac></mrow><mo id="A1.SS2.2.p2.2.m2.6.6.6.2a" xref="A1.SS2.2.p2.2.m2.6.6.6.2.cmml">+</mo><mrow id="A1.SS2.2.p2.2.m2.6.6.6.1" xref="A1.SS2.2.p2.2.m2.6.6.6.1.cmml"><mi id="A1.SS2.2.p2.2.m2.6.6.6.1.3" xref="A1.SS2.2.p2.2.m2.6.6.6.1.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.2.m2.6.6.6.1.2" xref="A1.SS2.2.p2.2.m2.6.6.6.1.2.cmml">​</mo><mrow id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.2" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.cmml">(</mo><msup id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.cmml"><mi id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.2" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.2.cmml">n</mi><mrow id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3.cmml"><mo id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3a" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3.cmml">−</mo><mn id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3.2" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo stretchy="false" id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.3" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.2.p2.2.m2.6b"><apply id="A1.SS2.2.p2.2.m2.6.6.cmml" xref="A1.SS2.2.p2.2.m2.6.6"><and id="A1.SS2.2.p2.2.m2.6.6a.cmml" xref="A1.SS2.2.p2.2.m2.6.6"></and><apply id="A1.SS2.2.p2.2.m2.6.6b.cmml" xref="A1.SS2.2.p2.2.m2.6.6"><eq id="A1.SS2.2.p2.2.m2.6.6.8.cmml" xref="A1.SS2.2.p2.2.m2.6.6.8"></eq><apply id="A1.SS2.2.p2.2.m2.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1"><times id="A1.SS2.2.p2.2.m2.1.1.1.2.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.2"></times><ci id="A1.SS2.2.p2.2.m2.1.1.1.3.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.3">𝔼</ci><apply id="A1.SS2.2.p2.2.m2.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.2.p2.2.m2.1.1.1.1.2.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1"><minus id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.3"></minus><apply id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1"><times id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.2"></times><apply id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1"><minus id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.1"></minus><apply id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2"><times id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.2">2</cn><ci id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.2.3">𝑛</ci></apply><cn type="integer" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.1.1.1.3">1</cn></apply><apply id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3.2.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3.2">𝑇</ci><ci id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3.3.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.1.3.3">𝑛</ci></apply></apply><apply id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2"><times id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.2.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.2"></times><cn type="integer" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.3.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.3">2</cn><apply id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1"><minus id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.1"></minus><ci id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.2">𝑛</ci><cn type="integer" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.1.1.1.3">1</cn></apply><apply id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4">subscript</csymbol><ci id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.2.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.2">𝑇</ci><apply id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3"><minus id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.1.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.1"></minus><ci id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.2.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.2">𝑛</ci><cn type="integer" id="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.3.cmml" xref="A1.SS2.2.p2.2.m2.1.1.1.1.1.1.2.4.3.3">1</cn></apply></apply></apply></apply></apply></apply><apply id="A1.SS2.2.p2.2.m2.5.5.5.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5"><minus id="A1.SS2.2.p2.2.m2.5.5.5.5.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.5"></minus><apply id="A1.SS2.2.p2.2.m2.3.3.3.2.cmml" xref="A1.SS2.2.p2.2.m2.3.3.3.2"><times id="A1.SS2.2.p2.2.m2.3.3.3.2.3.cmml" xref="A1.SS2.2.p2.2.m2.3.3.3.2.3"></times><apply id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1"><minus id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.1"></minus><apply id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2"><times id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.1.cmml" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.1"></times><cn type="integer" id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.2.cmml" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.2">2</cn><ci id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.3.cmml" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.2.3">𝑛</ci></apply><cn type="integer" id="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.2.m2.2.2.2.1.1.1.1.3">1</cn></apply><ci id="A1.SS2.2.p2.2.m2.3.3.3.2.4.cmml" xref="A1.SS2.2.p2.2.m2.3.3.3.2.4">𝔼</ci><apply id="A1.SS2.2.p2.2.m2.3.3.3.2.2.2.cmml" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.1"><csymbol cd="latexml" id="A1.SS2.2.p2.2.m2.3.3.3.2.2.2.1.cmml" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.2">delimited-[]</csymbol><apply id="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1.cmml" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1"><csymbol cd="ambiguous" id="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1">subscript</csymbol><ci id="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1.2.cmml" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1.2">𝑇</ci><ci id="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1.3.cmml" xref="A1.SS2.2.p2.2.m2.3.3.3.2.2.1.1.3">𝑛</ci></apply></apply></apply><apply id="A1.SS2.2.p2.2.m2.5.5.5.4.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4"><times id="A1.SS2.2.p2.2.m2.5.5.5.4.3.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.3"></times><cn type="integer" id="A1.SS2.2.p2.2.m2.5.5.5.4.4.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.4">2</cn><apply id="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.4.4.4.3.1.1"><minus id="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.1"></minus><ci id="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.2.cmml" xref="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.2">𝑛</ci><cn type="integer" id="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.3.cmml" xref="A1.SS2.2.p2.2.m2.4.4.4.3.1.1.1.3">1</cn></apply><ci id="A1.SS2.2.p2.2.m2.5.5.5.4.5.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.5">𝔼</ci><apply id="A1.SS2.2.p2.2.m2.5.5.5.4.2.2.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1"><csymbol cd="latexml" id="A1.SS2.2.p2.2.m2.5.5.5.4.2.2.1.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.2">delimited-[]</csymbol><apply id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1"><csymbol cd="ambiguous" id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1">subscript</csymbol><ci id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.2.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.2">𝑇</ci><apply id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3"><minus id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.1.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.1"></minus><ci id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.2.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.2">𝑛</ci><cn type="integer" id="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.3.cmml" xref="A1.SS2.2.p2.2.m2.5.5.5.4.2.1.1.3.3">1</cn></apply></apply></apply></apply></apply></apply><apply id="A1.SS2.2.p2.2.m2.6.6c.cmml" xref="A1.SS2.2.p2.2.m2.6.6"><eq id="A1.SS2.2.p2.2.m2.6.6.9.cmml" xref="A1.SS2.2.p2.2.m2.6.6.9"></eq><share href="#A1.SS2.2.p2.2.m2.5.5.5.cmml" id="A1.SS2.2.p2.2.m2.6.6d.cmml" xref="A1.SS2.2.p2.2.m2.6.6"></share><apply id="A1.SS2.2.p2.2.m2.6.6.6.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6"><plus id="A1.SS2.2.p2.2.m2.6.6.6.2.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.2"></plus><apply id="A1.SS2.2.p2.2.m2.6.6.6.3.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.3"><csymbol cd="ambiguous" id="A1.SS2.2.p2.2.m2.6.6.6.3.1.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.3">subscript</csymbol><ci id="A1.SS2.2.p2.2.m2.6.6.6.3.2.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.3.2">𝑍</ci><ci id="A1.SS2.2.p2.2.m2.6.6.6.3.3.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.3.3">𝜃</ci></apply><apply id="A1.SS2.2.p2.2.m2.6.6.6.4.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4"><times id="A1.SS2.2.p2.2.m2.6.6.6.4.1.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.1"></times><apply id="A1.SS2.2.p2.2.m2.6.6.6.4.2.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2"><divide id="A1.SS2.2.p2.2.m2.6.6.6.4.2.1.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2"></divide><apply id="A1.SS2.2.p2.2.m2.6.6.6.4.2.2.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.2"><csymbol cd="ambiguous" id="A1.SS2.2.p2.2.m2.6.6.6.4.2.2.1.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.2">subscript</csymbol><ci id="A1.SS2.2.p2.2.m2.6.6.6.4.2.2.2.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.2.2">𝜇</ci><cn type="integer" id="A1.SS2.2.p2.2.m2.6.6.6.4.2.2.3.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.2.3">2</cn></apply><apply id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3"><times id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.1.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.1"></times><cn type="integer" id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.2.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.2">2</cn><apply id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3"><csymbol cd="ambiguous" id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3.1.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3">superscript</csymbol><ci id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3.2.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3.2">𝜇</ci><cn type="integer" id="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3.3.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.2.3.3.3">2</cn></apply></apply></apply><apply id="A1.SS2.2.p2.2.m2.6.6.6.4.3.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.3"><divide id="A1.SS2.2.p2.2.m2.6.6.6.4.3.1.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.3"></divide><cn type="integer" id="A1.SS2.2.p2.2.m2.6.6.6.4.3.2.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.3.2">1</cn><ci id="A1.SS2.2.p2.2.m2.6.6.6.4.3.3.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.4.3.3">𝑛</ci></apply></apply><apply id="A1.SS2.2.p2.2.m2.6.6.6.1.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.1"><times id="A1.SS2.2.p2.2.m2.6.6.6.1.2.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.1.2"></times><ci id="A1.SS2.2.p2.2.m2.6.6.6.1.3.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.1.3">𝑜</ci><apply id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1">superscript</csymbol><ci id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.2">𝑛</ci><apply id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3"><minus id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3.1.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3"></minus><cn type="integer" id="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3.2.cmml" xref="A1.SS2.2.p2.2.m2.6.6.6.1.1.1.1.3.2">1</cn></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.2.p2.2.m2.6c">\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]=(2n-1)\mathbb{E}[T_{n}]-2(n-1)\mathbb{E}[T_{n-1}]=Z_{\theta}+\frac{\mu_{2}}{2\mu^{2}}\frac{1}{n}+o(n^{-1})</annotation></semantics></math>. Therefore <math id="A1.SS2.2.p2.3.m3.1" class="ltx_Math" alttext="\lim_{n\to\infty}\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]=Z_{\theta}" display="inline"><semantics id="A1.SS2.2.p2.3.m3.1a"><mrow id="A1.SS2.2.p2.3.m3.1.1" xref="A1.SS2.2.p2.3.m3.1.1.cmml"><mrow id="A1.SS2.2.p2.3.m3.1.1.1" xref="A1.SS2.2.p2.3.m3.1.1.1.cmml"><msub id="A1.SS2.2.p2.3.m3.1.1.1.2" xref="A1.SS2.2.p2.3.m3.1.1.1.2.cmml"><mo id="A1.SS2.2.p2.3.m3.1.1.1.2.2" xref="A1.SS2.2.p2.3.m3.1.1.1.2.2.cmml">lim</mo><mrow id="A1.SS2.2.p2.3.m3.1.1.1.2.3" xref="A1.SS2.2.p2.3.m3.1.1.1.2.3.cmml"><mi id="A1.SS2.2.p2.3.m3.1.1.1.2.3.2" xref="A1.SS2.2.p2.3.m3.1.1.1.2.3.2.cmml">n</mi><mo stretchy="false" id="A1.SS2.2.p2.3.m3.1.1.1.2.3.1" xref="A1.SS2.2.p2.3.m3.1.1.1.2.3.1.cmml">→</mo><mi mathvariant="normal" id="A1.SS2.2.p2.3.m3.1.1.1.2.3.3" xref="A1.SS2.2.p2.3.m3.1.1.1.2.3.3.cmml">∞</mi></mrow></msub><mrow id="A1.SS2.2.p2.3.m3.1.1.1.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.cmml"><mi id="A1.SS2.2.p2.3.m3.1.1.1.1.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.3.m3.1.1.1.1.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.2.cmml">​</mo><mrow id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.2.1.cmml">[</mo><mrow id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mn id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">n</mi></mrow><mo id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mn id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.2.cmml">​</mo><msub id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3.cmml"><mi id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3.2.cmml">T</mi><mi id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3.3.cmml">n</mi></msub></mrow><mo id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.3.cmml">−</mo><mrow id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.cmml"><mn id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.2.cmml">​</mo><mrow id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><mrow id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.cmml"><mi id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.2.cmml">n</mi><mo id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.1.cmml">−</mo><mn id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.2a" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.2.cmml">​</mo><msub id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.cmml"><mi id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.2.cmml">T</mi><mrow id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.cmml"><mi id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.2" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.2.cmml">n</mi><mo id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.1" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.1.cmml">−</mo><mn id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.3.cmml">1</mn></mrow></msub></mrow></mrow><mo stretchy="false" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><mo id="A1.SS2.2.p2.3.m3.1.1.2" xref="A1.SS2.2.p2.3.m3.1.1.2.cmml">=</mo><msub id="A1.SS2.2.p2.3.m3.1.1.3" xref="A1.SS2.2.p2.3.m3.1.1.3.cmml"><mi id="A1.SS2.2.p2.3.m3.1.1.3.2" xref="A1.SS2.2.p2.3.m3.1.1.3.2.cmml">Z</mi><mi id="A1.SS2.2.p2.3.m3.1.1.3.3" xref="A1.SS2.2.p2.3.m3.1.1.3.3.cmml">θ</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.2.p2.3.m3.1b"><apply id="A1.SS2.2.p2.3.m3.1.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1"><eq id="A1.SS2.2.p2.3.m3.1.1.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.2"></eq><apply id="A1.SS2.2.p2.3.m3.1.1.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1"><apply id="A1.SS2.2.p2.3.m3.1.1.1.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.2"><csymbol cd="ambiguous" id="A1.SS2.2.p2.3.m3.1.1.1.2.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.2">subscript</csymbol><limit id="A1.SS2.2.p2.3.m3.1.1.1.2.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.2.2"></limit><apply id="A1.SS2.2.p2.3.m3.1.1.1.2.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.2.3"><ci id="A1.SS2.2.p2.3.m3.1.1.1.2.3.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.2.3.1">→</ci><ci id="A1.SS2.2.p2.3.m3.1.1.1.2.3.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.2.3.2">𝑛</ci><infinity id="A1.SS2.2.p2.3.m3.1.1.1.2.3.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.2.3.3"></infinity></apply></apply><apply id="A1.SS2.2.p2.3.m3.1.1.1.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1"><times id="A1.SS2.2.p2.3.m3.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.2"></times><ci id="A1.SS2.2.p2.3.m3.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.3">𝔼</ci><apply id="A1.SS2.2.p2.3.m3.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.2.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1"><minus id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.3"></minus><apply id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1"><times id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.2"></times><apply id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1"><minus id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.1"></minus><apply id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2"><times id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.2">2</cn><ci id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.3">𝑛</ci></apply><cn type="integer" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.1.1.1.3">1</cn></apply><apply id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3.2">𝑇</ci><ci id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.1.3.3">𝑛</ci></apply></apply><apply id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2"><times id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.2"></times><cn type="integer" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.3">2</cn><apply id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1"><minus id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.1"></minus><ci id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.2">𝑛</ci><cn type="integer" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.1.1.1.3">1</cn></apply><apply id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4">subscript</csymbol><ci id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.2">𝑇</ci><apply id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3"><minus id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.1"></minus><ci id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.2">𝑛</ci><cn type="integer" id="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.1.1.1.1.1.2.4.3.3">1</cn></apply></apply></apply></apply></apply></apply></apply><apply id="A1.SS2.2.p2.3.m3.1.1.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.2.p2.3.m3.1.1.3.1.cmml" xref="A1.SS2.2.p2.3.m3.1.1.3">subscript</csymbol><ci id="A1.SS2.2.p2.3.m3.1.1.3.2.cmml" xref="A1.SS2.2.p2.3.m3.1.1.3.2">𝑍</ci><ci id="A1.SS2.2.p2.3.m3.1.1.3.3.cmml" xref="A1.SS2.2.p2.3.m3.1.1.3.3">𝜃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.2.p2.3.m3.1c">\lim_{n\to\infty}\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]=Z_{\theta}</annotation></semantics></math>, hence <math id="A1.SS2.2.p2.4.m4.1" class="ltx_Math" alttext="\forall\epsilon&gt;0" display="inline"><semantics id="A1.SS2.2.p2.4.m4.1a"><mrow id="A1.SS2.2.p2.4.m4.1.1" xref="A1.SS2.2.p2.4.m4.1.1.cmml"><mrow id="A1.SS2.2.p2.4.m4.1.1.2" xref="A1.SS2.2.p2.4.m4.1.1.2.cmml"><mo rspace="0.167em" id="A1.SS2.2.p2.4.m4.1.1.2.1" xref="A1.SS2.2.p2.4.m4.1.1.2.1.cmml">∀</mo><mi id="A1.SS2.2.p2.4.m4.1.1.2.2" xref="A1.SS2.2.p2.4.m4.1.1.2.2.cmml">ϵ</mi></mrow><mo id="A1.SS2.2.p2.4.m4.1.1.1" xref="A1.SS2.2.p2.4.m4.1.1.1.cmml">&gt;</mo><mn id="A1.SS2.2.p2.4.m4.1.1.3" xref="A1.SS2.2.p2.4.m4.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.2.p2.4.m4.1b"><apply id="A1.SS2.2.p2.4.m4.1.1.cmml" xref="A1.SS2.2.p2.4.m4.1.1"><gt id="A1.SS2.2.p2.4.m4.1.1.1.cmml" xref="A1.SS2.2.p2.4.m4.1.1.1"></gt><apply id="A1.SS2.2.p2.4.m4.1.1.2.cmml" xref="A1.SS2.2.p2.4.m4.1.1.2"><csymbol cd="latexml" id="A1.SS2.2.p2.4.m4.1.1.2.1.cmml" xref="A1.SS2.2.p2.4.m4.1.1.2.1">for-all</csymbol><ci id="A1.SS2.2.p2.4.m4.1.1.2.2.cmml" xref="A1.SS2.2.p2.4.m4.1.1.2.2">italic-ϵ</ci></apply><cn type="integer" id="A1.SS2.2.p2.4.m4.1.1.3.cmml" xref="A1.SS2.2.p2.4.m4.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.2.p2.4.m4.1c">\forall\epsilon&gt;0</annotation></semantics></math>, <math id="A1.SS2.2.p2.5.m5.1" class="ltx_Math" alttext="\exists N_{3}&gt;0" display="inline"><semantics id="A1.SS2.2.p2.5.m5.1a"><mrow id="A1.SS2.2.p2.5.m5.1.1" xref="A1.SS2.2.p2.5.m5.1.1.cmml"><mrow id="A1.SS2.2.p2.5.m5.1.1.2" xref="A1.SS2.2.p2.5.m5.1.1.2.cmml"><mo rspace="0.167em" id="A1.SS2.2.p2.5.m5.1.1.2.1" xref="A1.SS2.2.p2.5.m5.1.1.2.1.cmml">∃</mo><msub id="A1.SS2.2.p2.5.m5.1.1.2.2" xref="A1.SS2.2.p2.5.m5.1.1.2.2.cmml"><mi id="A1.SS2.2.p2.5.m5.1.1.2.2.2" xref="A1.SS2.2.p2.5.m5.1.1.2.2.2.cmml">N</mi><mn id="A1.SS2.2.p2.5.m5.1.1.2.2.3" xref="A1.SS2.2.p2.5.m5.1.1.2.2.3.cmml">3</mn></msub></mrow><mo id="A1.SS2.2.p2.5.m5.1.1.1" xref="A1.SS2.2.p2.5.m5.1.1.1.cmml">&gt;</mo><mn id="A1.SS2.2.p2.5.m5.1.1.3" xref="A1.SS2.2.p2.5.m5.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.2.p2.5.m5.1b"><apply id="A1.SS2.2.p2.5.m5.1.1.cmml" xref="A1.SS2.2.p2.5.m5.1.1"><gt id="A1.SS2.2.p2.5.m5.1.1.1.cmml" xref="A1.SS2.2.p2.5.m5.1.1.1"></gt><apply id="A1.SS2.2.p2.5.m5.1.1.2.cmml" xref="A1.SS2.2.p2.5.m5.1.1.2"><exists id="A1.SS2.2.p2.5.m5.1.1.2.1.cmml" xref="A1.SS2.2.p2.5.m5.1.1.2.1"></exists><apply id="A1.SS2.2.p2.5.m5.1.1.2.2.cmml" xref="A1.SS2.2.p2.5.m5.1.1.2.2"><csymbol cd="ambiguous" id="A1.SS2.2.p2.5.m5.1.1.2.2.1.cmml" xref="A1.SS2.2.p2.5.m5.1.1.2.2">subscript</csymbol><ci id="A1.SS2.2.p2.5.m5.1.1.2.2.2.cmml" xref="A1.SS2.2.p2.5.m5.1.1.2.2.2">𝑁</ci><cn type="integer" id="A1.SS2.2.p2.5.m5.1.1.2.2.3.cmml" xref="A1.SS2.2.p2.5.m5.1.1.2.2.3">3</cn></apply></apply><cn type="integer" id="A1.SS2.2.p2.5.m5.1.1.3.cmml" xref="A1.SS2.2.p2.5.m5.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.2.p2.5.m5.1c">\exists N_{3}&gt;0</annotation></semantics></math> such that <math id="A1.SS2.2.p2.6.m6.1" class="ltx_Math" alttext="\forall n&gt;N_{3}" display="inline"><semantics id="A1.SS2.2.p2.6.m6.1a"><mrow id="A1.SS2.2.p2.6.m6.1.1" xref="A1.SS2.2.p2.6.m6.1.1.cmml"><mrow id="A1.SS2.2.p2.6.m6.1.1.2" xref="A1.SS2.2.p2.6.m6.1.1.2.cmml"><mo rspace="0.167em" id="A1.SS2.2.p2.6.m6.1.1.2.1" xref="A1.SS2.2.p2.6.m6.1.1.2.1.cmml">∀</mo><mi id="A1.SS2.2.p2.6.m6.1.1.2.2" xref="A1.SS2.2.p2.6.m6.1.1.2.2.cmml">n</mi></mrow><mo id="A1.SS2.2.p2.6.m6.1.1.1" xref="A1.SS2.2.p2.6.m6.1.1.1.cmml">&gt;</mo><msub id="A1.SS2.2.p2.6.m6.1.1.3" xref="A1.SS2.2.p2.6.m6.1.1.3.cmml"><mi id="A1.SS2.2.p2.6.m6.1.1.3.2" xref="A1.SS2.2.p2.6.m6.1.1.3.2.cmml">N</mi><mn id="A1.SS2.2.p2.6.m6.1.1.3.3" xref="A1.SS2.2.p2.6.m6.1.1.3.3.cmml">3</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.2.p2.6.m6.1b"><apply id="A1.SS2.2.p2.6.m6.1.1.cmml" xref="A1.SS2.2.p2.6.m6.1.1"><gt id="A1.SS2.2.p2.6.m6.1.1.1.cmml" xref="A1.SS2.2.p2.6.m6.1.1.1"></gt><apply id="A1.SS2.2.p2.6.m6.1.1.2.cmml" xref="A1.SS2.2.p2.6.m6.1.1.2"><csymbol cd="latexml" id="A1.SS2.2.p2.6.m6.1.1.2.1.cmml" xref="A1.SS2.2.p2.6.m6.1.1.2.1">for-all</csymbol><ci id="A1.SS2.2.p2.6.m6.1.1.2.2.cmml" xref="A1.SS2.2.p2.6.m6.1.1.2.2">𝑛</ci></apply><apply id="A1.SS2.2.p2.6.m6.1.1.3.cmml" xref="A1.SS2.2.p2.6.m6.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.2.p2.6.m6.1.1.3.1.cmml" xref="A1.SS2.2.p2.6.m6.1.1.3">subscript</csymbol><ci id="A1.SS2.2.p2.6.m6.1.1.3.2.cmml" xref="A1.SS2.2.p2.6.m6.1.1.3.2">𝑁</ci><cn type="integer" id="A1.SS2.2.p2.6.m6.1.1.3.3.cmml" xref="A1.SS2.2.p2.6.m6.1.1.3.3">3</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.2.p2.6.m6.1c">\forall n&gt;N_{3}</annotation></semantics></math> <math id="A1.SS2.2.p2.7.m7.1" class="ltx_Math" alttext="\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]&lt;Z_{\theta}+\epsilon" display="inline"><semantics id="A1.SS2.2.p2.7.m7.1a"><mrow id="A1.SS2.2.p2.7.m7.1.1" xref="A1.SS2.2.p2.7.m7.1.1.cmml"><mrow id="A1.SS2.2.p2.7.m7.1.1.1" xref="A1.SS2.2.p2.7.m7.1.1.1.cmml"><mi id="A1.SS2.2.p2.7.m7.1.1.1.3" xref="A1.SS2.2.p2.7.m7.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.7.m7.1.1.1.2" xref="A1.SS2.2.p2.7.m7.1.1.1.2.cmml">​</mo><mrow id="A1.SS2.2.p2.7.m7.1.1.1.1.1" xref="A1.SS2.2.p2.7.m7.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.2.1.cmml">[</mo><mrow id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.cmml"><mn id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.1" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.3.cmml">n</mi></mrow><mo id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mn id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.2.cmml">​</mo><msub id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3.cmml"><mi id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3.2.cmml">T</mi><mi id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3.3.cmml">n</mi></msub></mrow><mo id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.3.cmml">−</mo><mrow id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.cmml"><mn id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.2.cmml">​</mo><mrow id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><mrow id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.cmml"><mi id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.2.cmml">n</mi><mo id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.1" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.1.cmml">−</mo><mn id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.2a" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.2.cmml">​</mo><msub id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.cmml"><mi id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.2.cmml">T</mi><mrow id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.cmml"><mi id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.2" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.2.cmml">n</mi><mo id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.1" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.1.cmml">−</mo><mn id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.3.cmml">1</mn></mrow></msub></mrow></mrow><mo stretchy="false" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.3" xref="A1.SS2.2.p2.7.m7.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.SS2.2.p2.7.m7.1.1.2" xref="A1.SS2.2.p2.7.m7.1.1.2.cmml">&lt;</mo><mrow id="A1.SS2.2.p2.7.m7.1.1.3" xref="A1.SS2.2.p2.7.m7.1.1.3.cmml"><msub id="A1.SS2.2.p2.7.m7.1.1.3.2" xref="A1.SS2.2.p2.7.m7.1.1.3.2.cmml"><mi id="A1.SS2.2.p2.7.m7.1.1.3.2.2" xref="A1.SS2.2.p2.7.m7.1.1.3.2.2.cmml">Z</mi><mi id="A1.SS2.2.p2.7.m7.1.1.3.2.3" xref="A1.SS2.2.p2.7.m7.1.1.3.2.3.cmml">θ</mi></msub><mo id="A1.SS2.2.p2.7.m7.1.1.3.1" xref="A1.SS2.2.p2.7.m7.1.1.3.1.cmml">+</mo><mi id="A1.SS2.2.p2.7.m7.1.1.3.3" xref="A1.SS2.2.p2.7.m7.1.1.3.3.cmml">ϵ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.2.p2.7.m7.1b"><apply id="A1.SS2.2.p2.7.m7.1.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1"><lt id="A1.SS2.2.p2.7.m7.1.1.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.2"></lt><apply id="A1.SS2.2.p2.7.m7.1.1.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1"><times id="A1.SS2.2.p2.7.m7.1.1.1.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.2"></times><ci id="A1.SS2.2.p2.7.m7.1.1.1.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.3">𝔼</ci><apply id="A1.SS2.2.p2.7.m7.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.2.p2.7.m7.1.1.1.1.2.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1"><minus id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.3"></minus><apply id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1"><times id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.2"></times><apply id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1"><minus id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.1"></minus><apply id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2"><times id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.2">2</cn><ci id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.2.3">𝑛</ci></apply><cn type="integer" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.1.1.1.3">1</cn></apply><apply id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3.2">𝑇</ci><ci id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.1.3.3">𝑛</ci></apply></apply><apply id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2"><times id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.2"></times><cn type="integer" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.3">2</cn><apply id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1"><minus id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.1"></minus><ci id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.2">𝑛</ci><cn type="integer" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.1.1.1.3">1</cn></apply><apply id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4">subscript</csymbol><ci id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.2">𝑇</ci><apply id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3"><minus id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.1"></minus><ci id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.2">𝑛</ci><cn type="integer" id="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.1.1.1.1.2.4.3.3">1</cn></apply></apply></apply></apply></apply></apply><apply id="A1.SS2.2.p2.7.m7.1.1.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.3"><plus id="A1.SS2.2.p2.7.m7.1.1.3.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.3.1"></plus><apply id="A1.SS2.2.p2.7.m7.1.1.3.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.3.2"><csymbol cd="ambiguous" id="A1.SS2.2.p2.7.m7.1.1.3.2.1.cmml" xref="A1.SS2.2.p2.7.m7.1.1.3.2">subscript</csymbol><ci id="A1.SS2.2.p2.7.m7.1.1.3.2.2.cmml" xref="A1.SS2.2.p2.7.m7.1.1.3.2.2">𝑍</ci><ci id="A1.SS2.2.p2.7.m7.1.1.3.2.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.3.2.3">𝜃</ci></apply><ci id="A1.SS2.2.p2.7.m7.1.1.3.3.cmml" xref="A1.SS2.2.p2.7.m7.1.1.3.3">italic-ϵ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.2.p2.7.m7.1c">\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]&lt;Z_{\theta}+\epsilon</annotation></semantics></math>. Furthermore, <math id="A1.SS2.2.p2.8.m8.2" class="ltx_Math" alttext="\lim_{n\to\infty}n(\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]-Z_{\theta})=\lim_{n\to\infty}\frac{\mu_{2}}{2\mu^{2}}+o(1)&gt;0" display="inline"><semantics id="A1.SS2.2.p2.8.m8.2a"><mrow id="A1.SS2.2.p2.8.m8.2.2" xref="A1.SS2.2.p2.8.m8.2.2.cmml"><mrow id="A1.SS2.2.p2.8.m8.2.2.1" xref="A1.SS2.2.p2.8.m8.2.2.1.cmml"><msub id="A1.SS2.2.p2.8.m8.2.2.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.2.cmml"><mo id="A1.SS2.2.p2.8.m8.2.2.1.2.2" xref="A1.SS2.2.p2.8.m8.2.2.1.2.2.cmml">lim</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.1.2.3" xref="A1.SS2.2.p2.8.m8.2.2.1.2.3.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.1.2.3.2" xref="A1.SS2.2.p2.8.m8.2.2.1.2.3.2.cmml">n</mi><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.1.2.3.1" xref="A1.SS2.2.p2.8.m8.2.2.1.2.3.1.cmml">→</mo><mi mathvariant="normal" id="A1.SS2.2.p2.8.m8.2.2.1.2.3.3" xref="A1.SS2.2.p2.8.m8.2.2.1.2.3.3.cmml">∞</mi></mrow></msub><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.1.1.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.8.m8.2.2.1.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.2.cmml">​</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.2.cmml">​</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.2.1.cmml">[</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mn id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">n</mi></mrow><mo id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mn id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.2.cmml">​</mo><msub id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3.2.cmml">T</mi><mi id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3.3.cmml">n</mi></msub></mrow><mo id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.3.cmml">−</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.cmml"><mn id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.2.cmml">​</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.2.cmml">n</mi><mo id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.1.cmml">−</mo><mn id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.2a" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.2.cmml">​</mo><msub id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.2.cmml">T</mi><mrow id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.2.cmml">n</mi><mo id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.1" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.1.cmml">−</mo><mn id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.3.cmml">1</mn></mrow></msub></mrow></mrow><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.2.cmml">−</mo><msub id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3.2" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3.2.cmml">Z</mi><mi id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3.3.cmml">θ</mi></msub></mrow><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.3" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo rspace="0.1389em" id="A1.SS2.2.p2.8.m8.2.2.3" xref="A1.SS2.2.p2.8.m8.2.2.3.cmml">=</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.4" xref="A1.SS2.2.p2.8.m8.2.2.4.cmml"><mrow id="A1.SS2.2.p2.8.m8.2.2.4.2" xref="A1.SS2.2.p2.8.m8.2.2.4.2.cmml"><msub id="A1.SS2.2.p2.8.m8.2.2.4.2.1" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1.cmml"><mo lspace="0.1389em" rspace="0.167em" id="A1.SS2.2.p2.8.m8.2.2.4.2.1.2" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1.2.cmml">lim</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.4.2.1.3" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.2" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.2.cmml">n</mi><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.1" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.1.cmml">→</mo><mi mathvariant="normal" id="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.3" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.3.cmml">∞</mi></mrow></msub><mfrac id="A1.SS2.2.p2.8.m8.2.2.4.2.2" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.cmml"><msub id="A1.SS2.2.p2.8.m8.2.2.4.2.2.2" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.2.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.4.2.2.2.2" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.2.2.cmml">μ</mi><mn id="A1.SS2.2.p2.8.m8.2.2.4.2.2.2.3" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.2.3.cmml">2</mn></msub><mrow id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.cmml"><mn id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.2" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.1" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.1.cmml">​</mo><msup id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3.2" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3.2.cmml">μ</mi><mn id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3.3" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3.3.cmml">2</mn></msup></mrow></mfrac></mrow><mo id="A1.SS2.2.p2.8.m8.2.2.4.1" xref="A1.SS2.2.p2.8.m8.2.2.4.1.cmml">+</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.4.3" xref="A1.SS2.2.p2.8.m8.2.2.4.3.cmml"><mi id="A1.SS2.2.p2.8.m8.2.2.4.3.2" xref="A1.SS2.2.p2.8.m8.2.2.4.3.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="A1.SS2.2.p2.8.m8.2.2.4.3.1" xref="A1.SS2.2.p2.8.m8.2.2.4.3.1.cmml">​</mo><mrow id="A1.SS2.2.p2.8.m8.2.2.4.3.3.2" xref="A1.SS2.2.p2.8.m8.2.2.4.3.cmml"><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.4.3.3.2.1" xref="A1.SS2.2.p2.8.m8.2.2.4.3.cmml">(</mo><mn id="A1.SS2.2.p2.8.m8.1.1" xref="A1.SS2.2.p2.8.m8.1.1.cmml">1</mn><mo stretchy="false" id="A1.SS2.2.p2.8.m8.2.2.4.3.3.2.2" xref="A1.SS2.2.p2.8.m8.2.2.4.3.cmml">)</mo></mrow></mrow></mrow><mo id="A1.SS2.2.p2.8.m8.2.2.5" xref="A1.SS2.2.p2.8.m8.2.2.5.cmml">&gt;</mo><mn id="A1.SS2.2.p2.8.m8.2.2.6" xref="A1.SS2.2.p2.8.m8.2.2.6.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.2.p2.8.m8.2b"><apply id="A1.SS2.2.p2.8.m8.2.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2"><and id="A1.SS2.2.p2.8.m8.2.2a.cmml" xref="A1.SS2.2.p2.8.m8.2.2"></and><apply id="A1.SS2.2.p2.8.m8.2.2b.cmml" xref="A1.SS2.2.p2.8.m8.2.2"><eq id="A1.SS2.2.p2.8.m8.2.2.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.3"></eq><apply id="A1.SS2.2.p2.8.m8.2.2.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1"><apply id="A1.SS2.2.p2.8.m8.2.2.1.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.2"><csymbol cd="ambiguous" id="A1.SS2.2.p2.8.m8.2.2.1.2.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.2">subscript</csymbol><limit id="A1.SS2.2.p2.8.m8.2.2.1.2.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.2.2"></limit><apply id="A1.SS2.2.p2.8.m8.2.2.1.2.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.2.3"><ci id="A1.SS2.2.p2.8.m8.2.2.1.2.3.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.2.3.1">→</ci><ci id="A1.SS2.2.p2.8.m8.2.2.1.2.3.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.2.3.2">𝑛</ci><infinity id="A1.SS2.2.p2.8.m8.2.2.1.2.3.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.2.3.3"></infinity></apply></apply><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1"><times id="A1.SS2.2.p2.8.m8.2.2.1.1.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.2"></times><ci id="A1.SS2.2.p2.8.m8.2.2.1.1.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.3">𝑛</ci><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1"><minus id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.2"></minus><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1"><times id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.2"></times><ci id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.3">𝔼</ci><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.2.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1"><minus id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.3"></minus><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1"><times id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.2"></times><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1"><minus id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.1"></minus><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2"><times id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2">2</cn><ci id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3">𝑛</ci></apply><cn type="integer" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.1.1.1.3">1</cn></apply><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3.2">𝑇</ci><ci id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.1.3.3">𝑛</ci></apply></apply><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2"><times id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.2"></times><cn type="integer" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.3">2</cn><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1"><minus id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.1"></minus><ci id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.2">𝑛</ci><cn type="integer" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.1.1.1.3">1</cn></apply><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4">subscript</csymbol><ci id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.2">𝑇</ci><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3"><minus id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.1"></minus><ci id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.2">𝑛</ci><cn type="integer" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.1.1.1.1.2.4.3.3">1</cn></apply></apply></apply></apply></apply></apply><apply id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3">subscript</csymbol><ci id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3.2">𝑍</ci><ci id="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.1.1.1.1.1.3.3">𝜃</ci></apply></apply></apply></apply><apply id="A1.SS2.2.p2.8.m8.2.2.4.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4"><plus id="A1.SS2.2.p2.8.m8.2.2.4.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.1"></plus><apply id="A1.SS2.2.p2.8.m8.2.2.4.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2"><apply id="A1.SS2.2.p2.8.m8.2.2.4.2.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1"><csymbol cd="ambiguous" id="A1.SS2.2.p2.8.m8.2.2.4.2.1.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1">subscript</csymbol><limit id="A1.SS2.2.p2.8.m8.2.2.4.2.1.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1.2"></limit><apply id="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1.3"><ci id="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.1">→</ci><ci id="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.2">𝑛</ci><infinity id="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.1.3.3"></infinity></apply></apply><apply id="A1.SS2.2.p2.8.m8.2.2.4.2.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2"><divide id="A1.SS2.2.p2.8.m8.2.2.4.2.2.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2"></divide><apply id="A1.SS2.2.p2.8.m8.2.2.4.2.2.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.2"><csymbol cd="ambiguous" id="A1.SS2.2.p2.8.m8.2.2.4.2.2.2.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.2">subscript</csymbol><ci id="A1.SS2.2.p2.8.m8.2.2.4.2.2.2.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.2.2">𝜇</ci><cn type="integer" id="A1.SS2.2.p2.8.m8.2.2.4.2.2.2.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.2.3">2</cn></apply><apply id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3"><times id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.1"></times><cn type="integer" id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.2">2</cn><apply id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3"><csymbol cd="ambiguous" id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3">superscript</csymbol><ci id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3.2">𝜇</ci><cn type="integer" id="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.2.2.3.3.3">2</cn></apply></apply></apply></apply><apply id="A1.SS2.2.p2.8.m8.2.2.4.3.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.3"><times id="A1.SS2.2.p2.8.m8.2.2.4.3.1.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.3.1"></times><ci id="A1.SS2.2.p2.8.m8.2.2.4.3.2.cmml" xref="A1.SS2.2.p2.8.m8.2.2.4.3.2">𝑜</ci><cn type="integer" id="A1.SS2.2.p2.8.m8.1.1.cmml" xref="A1.SS2.2.p2.8.m8.1.1">1</cn></apply></apply></apply><apply id="A1.SS2.2.p2.8.m8.2.2c.cmml" xref="A1.SS2.2.p2.8.m8.2.2"><gt id="A1.SS2.2.p2.8.m8.2.2.5.cmml" xref="A1.SS2.2.p2.8.m8.2.2.5"></gt><share href="#A1.SS2.2.p2.8.m8.2.2.4.cmml" id="A1.SS2.2.p2.8.m8.2.2d.cmml" xref="A1.SS2.2.p2.8.m8.2.2"></share><cn type="integer" id="A1.SS2.2.p2.8.m8.2.2.6.cmml" xref="A1.SS2.2.p2.8.m8.2.2.6">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.2.p2.8.m8.2c">\lim_{n\to\infty}n(\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]-Z_{\theta})=\lim_{n\to\infty}\frac{\mu_{2}}{2\mu^{2}}+o(1)&gt;0</annotation></semantics></math>, so <math id="A1.SS2.2.p2.9.m9.1" class="ltx_Math" alttext="\exists N_{4}&gt;0" display="inline"><semantics id="A1.SS2.2.p2.9.m9.1a"><mrow id="A1.SS2.2.p2.9.m9.1.1" xref="A1.SS2.2.p2.9.m9.1.1.cmml"><mrow id="A1.SS2.2.p2.9.m9.1.1.2" xref="A1.SS2.2.p2.9.m9.1.1.2.cmml"><mo rspace="0.167em" id="A1.SS2.2.p2.9.m9.1.1.2.1" xref="A1.SS2.2.p2.9.m9.1.1.2.1.cmml">∃</mo><msub id="A1.SS2.2.p2.9.m9.1.1.2.2" xref="A1.SS2.2.p2.9.m9.1.1.2.2.cmml"><mi id="A1.SS2.2.p2.9.m9.1.1.2.2.2" xref="A1.SS2.2.p2.9.m9.1.1.2.2.2.cmml">N</mi><mn id="A1.SS2.2.p2.9.m9.1.1.2.2.3" xref="A1.SS2.2.p2.9.m9.1.1.2.2.3.cmml">4</mn></msub></mrow><mo id="A1.SS2.2.p2.9.m9.1.1.1" xref="A1.SS2.2.p2.9.m9.1.1.1.cmml">&gt;</mo><mn id="A1.SS2.2.p2.9.m9.1.1.3" xref="A1.SS2.2.p2.9.m9.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.2.p2.9.m9.1b"><apply id="A1.SS2.2.p2.9.m9.1.1.cmml" xref="A1.SS2.2.p2.9.m9.1.1"><gt id="A1.SS2.2.p2.9.m9.1.1.1.cmml" xref="A1.SS2.2.p2.9.m9.1.1.1"></gt><apply id="A1.SS2.2.p2.9.m9.1.1.2.cmml" xref="A1.SS2.2.p2.9.m9.1.1.2"><exists id="A1.SS2.2.p2.9.m9.1.1.2.1.cmml" xref="A1.SS2.2.p2.9.m9.1.1.2.1"></exists><apply id="A1.SS2.2.p2.9.m9.1.1.2.2.cmml" xref="A1.SS2.2.p2.9.m9.1.1.2.2"><csymbol cd="ambiguous" id="A1.SS2.2.p2.9.m9.1.1.2.2.1.cmml" xref="A1.SS2.2.p2.9.m9.1.1.2.2">subscript</csymbol><ci id="A1.SS2.2.p2.9.m9.1.1.2.2.2.cmml" xref="A1.SS2.2.p2.9.m9.1.1.2.2.2">𝑁</ci><cn type="integer" id="A1.SS2.2.p2.9.m9.1.1.2.2.3.cmml" xref="A1.SS2.2.p2.9.m9.1.1.2.2.3">4</cn></apply></apply><cn type="integer" id="A1.SS2.2.p2.9.m9.1.1.3.cmml" xref="A1.SS2.2.p2.9.m9.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.2.p2.9.m9.1c">\exists N_{4}&gt;0</annotation></semantics></math> such that when <math id="A1.SS2.2.p2.10.m10.1" class="ltx_Math" alttext="n&gt;N_{4}" display="inline"><semantics id="A1.SS2.2.p2.10.m10.1a"><mrow id="A1.SS2.2.p2.10.m10.1.1" xref="A1.SS2.2.p2.10.m10.1.1.cmml"><mi id="A1.SS2.2.p2.10.m10.1.1.2" xref="A1.SS2.2.p2.10.m10.1.1.2.cmml">n</mi><mo id="A1.SS2.2.p2.10.m10.1.1.1" xref="A1.SS2.2.p2.10.m10.1.1.1.cmml">&gt;</mo><msub id="A1.SS2.2.p2.10.m10.1.1.3" xref="A1.SS2.2.p2.10.m10.1.1.3.cmml"><mi id="A1.SS2.2.p2.10.m10.1.1.3.2" xref="A1.SS2.2.p2.10.m10.1.1.3.2.cmml">N</mi><mn id="A1.SS2.2.p2.10.m10.1.1.3.3" xref="A1.SS2.2.p2.10.m10.1.1.3.3.cmml">4</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.2.p2.10.m10.1b"><apply id="A1.SS2.2.p2.10.m10.1.1.cmml" xref="A1.SS2.2.p2.10.m10.1.1"><gt id="A1.SS2.2.p2.10.m10.1.1.1.cmml" xref="A1.SS2.2.p2.10.m10.1.1.1"></gt><ci id="A1.SS2.2.p2.10.m10.1.1.2.cmml" xref="A1.SS2.2.p2.10.m10.1.1.2">𝑛</ci><apply id="A1.SS2.2.p2.10.m10.1.1.3.cmml" xref="A1.SS2.2.p2.10.m10.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.2.p2.10.m10.1.1.3.1.cmml" xref="A1.SS2.2.p2.10.m10.1.1.3">subscript</csymbol><ci id="A1.SS2.2.p2.10.m10.1.1.3.2.cmml" xref="A1.SS2.2.p2.10.m10.1.1.3.2">𝑁</ci><cn type="integer" id="A1.SS2.2.p2.10.m10.1.1.3.3.cmml" xref="A1.SS2.2.p2.10.m10.1.1.3.3">4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.2.p2.10.m10.1c">n&gt;N_{4}</annotation></semantics></math> we have <math id="A1.SS2.2.p2.11.m11.1" class="ltx_math_unparsed" alttext="\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}&gt;Z_{\theta}" display="inline"><semantics id="A1.SS2.2.p2.11.m11.1a"><mrow id="A1.SS2.2.p2.11.m11.1b"><mi id="A1.SS2.2.p2.11.m11.1.1">𝔼</mi><mrow id="A1.SS2.2.p2.11.m11.1.2"><mo stretchy="false" id="A1.SS2.2.p2.11.m11.1.2.1">[</mo><mrow id="A1.SS2.2.p2.11.m11.1.2.2"><mo stretchy="false" id="A1.SS2.2.p2.11.m11.1.2.2.1">(</mo><mn id="A1.SS2.2.p2.11.m11.1.2.2.2">2</mn><mi id="A1.SS2.2.p2.11.m11.1.2.2.3">n</mi><mo id="A1.SS2.2.p2.11.m11.1.2.2.4">−</mo><mn id="A1.SS2.2.p2.11.m11.1.2.2.5">1</mn><mo stretchy="false" id="A1.SS2.2.p2.11.m11.1.2.2.6">)</mo></mrow><msub id="A1.SS2.2.p2.11.m11.1.2.3"><mi id="A1.SS2.2.p2.11.m11.1.2.3.2">T</mi><mi id="A1.SS2.2.p2.11.m11.1.2.3.3">n</mi></msub><mo id="A1.SS2.2.p2.11.m11.1.2.4">−</mo><mn id="A1.SS2.2.p2.11.m11.1.2.5">2</mn><mrow id="A1.SS2.2.p2.11.m11.1.2.6"><mo stretchy="false" id="A1.SS2.2.p2.11.m11.1.2.6.1">(</mo><mi id="A1.SS2.2.p2.11.m11.1.2.6.2">n</mi><mo id="A1.SS2.2.p2.11.m11.1.2.6.3">−</mo><mn id="A1.SS2.2.p2.11.m11.1.2.6.4">1</mn><mo stretchy="false" id="A1.SS2.2.p2.11.m11.1.2.6.5">)</mo></mrow><msub id="A1.SS2.2.p2.11.m11.1.2.7"><mi id="A1.SS2.2.p2.11.m11.1.2.7.2">T</mi><mrow id="A1.SS2.2.p2.11.m11.1.2.7.3"><mi id="A1.SS2.2.p2.11.m11.1.2.7.3.2">n</mi><mo id="A1.SS2.2.p2.11.m11.1.2.7.3.1">−</mo><mn id="A1.SS2.2.p2.11.m11.1.2.7.3.3">1</mn></mrow></msub><mo id="A1.SS2.2.p2.11.m11.1.2.8">&gt;</mo><msub id="A1.SS2.2.p2.11.m11.1.2.9"><mi id="A1.SS2.2.p2.11.m11.1.2.9.2">Z</mi><mi id="A1.SS2.2.p2.11.m11.1.2.9.3">θ</mi></msub></mrow></mrow><annotation encoding="application/x-tex" id="A1.SS2.2.p2.11.m11.1c">\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}&gt;Z_{\theta}</annotation></semantics></math>.</p>
</div>
<div id="A1.SS2.3.p3" class="ltx_para">
<p id="A1.SS2.3.p3.3" class="ltx_p">Putting the above together, <math id="A1.SS2.3.p3.1.m1.1" class="ltx_Math" alttext="\forall\epsilon&gt;0" display="inline"><semantics id="A1.SS2.3.p3.1.m1.1a"><mrow id="A1.SS2.3.p3.1.m1.1.1" xref="A1.SS2.3.p3.1.m1.1.1.cmml"><mrow id="A1.SS2.3.p3.1.m1.1.1.2" xref="A1.SS2.3.p3.1.m1.1.1.2.cmml"><mo rspace="0.167em" id="A1.SS2.3.p3.1.m1.1.1.2.1" xref="A1.SS2.3.p3.1.m1.1.1.2.1.cmml">∀</mo><mi id="A1.SS2.3.p3.1.m1.1.1.2.2" xref="A1.SS2.3.p3.1.m1.1.1.2.2.cmml">ϵ</mi></mrow><mo id="A1.SS2.3.p3.1.m1.1.1.1" xref="A1.SS2.3.p3.1.m1.1.1.1.cmml">&gt;</mo><mn id="A1.SS2.3.p3.1.m1.1.1.3" xref="A1.SS2.3.p3.1.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.3.p3.1.m1.1b"><apply id="A1.SS2.3.p3.1.m1.1.1.cmml" xref="A1.SS2.3.p3.1.m1.1.1"><gt id="A1.SS2.3.p3.1.m1.1.1.1.cmml" xref="A1.SS2.3.p3.1.m1.1.1.1"></gt><apply id="A1.SS2.3.p3.1.m1.1.1.2.cmml" xref="A1.SS2.3.p3.1.m1.1.1.2"><csymbol cd="latexml" id="A1.SS2.3.p3.1.m1.1.1.2.1.cmml" xref="A1.SS2.3.p3.1.m1.1.1.2.1">for-all</csymbol><ci id="A1.SS2.3.p3.1.m1.1.1.2.2.cmml" xref="A1.SS2.3.p3.1.m1.1.1.2.2">italic-ϵ</ci></apply><cn type="integer" id="A1.SS2.3.p3.1.m1.1.1.3.cmml" xref="A1.SS2.3.p3.1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.3.p3.1.m1.1c">\forall\epsilon&gt;0</annotation></semantics></math>, let <math id="A1.SS2.3.p3.2.m2.5" class="ltx_Math" alttext="N=\max\{N_{1},N_{2},N_{3},N_{4}\}" display="inline"><semantics id="A1.SS2.3.p3.2.m2.5a"><mrow id="A1.SS2.3.p3.2.m2.5.5" xref="A1.SS2.3.p3.2.m2.5.5.cmml"><mi id="A1.SS2.3.p3.2.m2.5.5.6" xref="A1.SS2.3.p3.2.m2.5.5.6.cmml">N</mi><mo id="A1.SS2.3.p3.2.m2.5.5.5" xref="A1.SS2.3.p3.2.m2.5.5.5.cmml">=</mo><mrow id="A1.SS2.3.p3.2.m2.5.5.4.4" xref="A1.SS2.3.p3.2.m2.5.5.4.5.cmml"><mi id="A1.SS2.3.p3.2.m2.1.1" xref="A1.SS2.3.p3.2.m2.1.1.cmml">max</mi><mo id="A1.SS2.3.p3.2.m2.5.5.4.4a" xref="A1.SS2.3.p3.2.m2.5.5.4.5.cmml">⁡</mo><mrow id="A1.SS2.3.p3.2.m2.5.5.4.4.4" xref="A1.SS2.3.p3.2.m2.5.5.4.5.cmml"><mo stretchy="false" id="A1.SS2.3.p3.2.m2.5.5.4.4.4.5" xref="A1.SS2.3.p3.2.m2.5.5.4.5.cmml">{</mo><msub id="A1.SS2.3.p3.2.m2.2.2.1.1.1.1" xref="A1.SS2.3.p3.2.m2.2.2.1.1.1.1.cmml"><mi id="A1.SS2.3.p3.2.m2.2.2.1.1.1.1.2" xref="A1.SS2.3.p3.2.m2.2.2.1.1.1.1.2.cmml">N</mi><mn id="A1.SS2.3.p3.2.m2.2.2.1.1.1.1.3" xref="A1.SS2.3.p3.2.m2.2.2.1.1.1.1.3.cmml">1</mn></msub><mo id="A1.SS2.3.p3.2.m2.5.5.4.4.4.6" xref="A1.SS2.3.p3.2.m2.5.5.4.5.cmml">,</mo><msub id="A1.SS2.3.p3.2.m2.3.3.2.2.2.2" xref="A1.SS2.3.p3.2.m2.3.3.2.2.2.2.cmml"><mi id="A1.SS2.3.p3.2.m2.3.3.2.2.2.2.2" xref="A1.SS2.3.p3.2.m2.3.3.2.2.2.2.2.cmml">N</mi><mn id="A1.SS2.3.p3.2.m2.3.3.2.2.2.2.3" xref="A1.SS2.3.p3.2.m2.3.3.2.2.2.2.3.cmml">2</mn></msub><mo id="A1.SS2.3.p3.2.m2.5.5.4.4.4.7" xref="A1.SS2.3.p3.2.m2.5.5.4.5.cmml">,</mo><msub id="A1.SS2.3.p3.2.m2.4.4.3.3.3.3" xref="A1.SS2.3.p3.2.m2.4.4.3.3.3.3.cmml"><mi id="A1.SS2.3.p3.2.m2.4.4.3.3.3.3.2" xref="A1.SS2.3.p3.2.m2.4.4.3.3.3.3.2.cmml">N</mi><mn id="A1.SS2.3.p3.2.m2.4.4.3.3.3.3.3" xref="A1.SS2.3.p3.2.m2.4.4.3.3.3.3.3.cmml">3</mn></msub><mo id="A1.SS2.3.p3.2.m2.5.5.4.4.4.8" xref="A1.SS2.3.p3.2.m2.5.5.4.5.cmml">,</mo><msub id="A1.SS2.3.p3.2.m2.5.5.4.4.4.4" xref="A1.SS2.3.p3.2.m2.5.5.4.4.4.4.cmml"><mi id="A1.SS2.3.p3.2.m2.5.5.4.4.4.4.2" xref="A1.SS2.3.p3.2.m2.5.5.4.4.4.4.2.cmml">N</mi><mn id="A1.SS2.3.p3.2.m2.5.5.4.4.4.4.3" xref="A1.SS2.3.p3.2.m2.5.5.4.4.4.4.3.cmml">4</mn></msub><mo stretchy="false" id="A1.SS2.3.p3.2.m2.5.5.4.4.4.9" xref="A1.SS2.3.p3.2.m2.5.5.4.5.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.3.p3.2.m2.5b"><apply id="A1.SS2.3.p3.2.m2.5.5.cmml" xref="A1.SS2.3.p3.2.m2.5.5"><eq id="A1.SS2.3.p3.2.m2.5.5.5.cmml" xref="A1.SS2.3.p3.2.m2.5.5.5"></eq><ci id="A1.SS2.3.p3.2.m2.5.5.6.cmml" xref="A1.SS2.3.p3.2.m2.5.5.6">𝑁</ci><apply id="A1.SS2.3.p3.2.m2.5.5.4.5.cmml" xref="A1.SS2.3.p3.2.m2.5.5.4.4"><max id="A1.SS2.3.p3.2.m2.1.1.cmml" xref="A1.SS2.3.p3.2.m2.1.1"></max><apply id="A1.SS2.3.p3.2.m2.2.2.1.1.1.1.cmml" xref="A1.SS2.3.p3.2.m2.2.2.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.3.p3.2.m2.2.2.1.1.1.1.1.cmml" xref="A1.SS2.3.p3.2.m2.2.2.1.1.1.1">subscript</csymbol><ci id="A1.SS2.3.p3.2.m2.2.2.1.1.1.1.2.cmml" xref="A1.SS2.3.p3.2.m2.2.2.1.1.1.1.2">𝑁</ci><cn type="integer" id="A1.SS2.3.p3.2.m2.2.2.1.1.1.1.3.cmml" xref="A1.SS2.3.p3.2.m2.2.2.1.1.1.1.3">1</cn></apply><apply id="A1.SS2.3.p3.2.m2.3.3.2.2.2.2.cmml" xref="A1.SS2.3.p3.2.m2.3.3.2.2.2.2"><csymbol cd="ambiguous" id="A1.SS2.3.p3.2.m2.3.3.2.2.2.2.1.cmml" xref="A1.SS2.3.p3.2.m2.3.3.2.2.2.2">subscript</csymbol><ci id="A1.SS2.3.p3.2.m2.3.3.2.2.2.2.2.cmml" xref="A1.SS2.3.p3.2.m2.3.3.2.2.2.2.2">𝑁</ci><cn type="integer" id="A1.SS2.3.p3.2.m2.3.3.2.2.2.2.3.cmml" xref="A1.SS2.3.p3.2.m2.3.3.2.2.2.2.3">2</cn></apply><apply id="A1.SS2.3.p3.2.m2.4.4.3.3.3.3.cmml" xref="A1.SS2.3.p3.2.m2.4.4.3.3.3.3"><csymbol cd="ambiguous" id="A1.SS2.3.p3.2.m2.4.4.3.3.3.3.1.cmml" xref="A1.SS2.3.p3.2.m2.4.4.3.3.3.3">subscript</csymbol><ci id="A1.SS2.3.p3.2.m2.4.4.3.3.3.3.2.cmml" xref="A1.SS2.3.p3.2.m2.4.4.3.3.3.3.2">𝑁</ci><cn type="integer" id="A1.SS2.3.p3.2.m2.4.4.3.3.3.3.3.cmml" xref="A1.SS2.3.p3.2.m2.4.4.3.3.3.3.3">3</cn></apply><apply id="A1.SS2.3.p3.2.m2.5.5.4.4.4.4.cmml" xref="A1.SS2.3.p3.2.m2.5.5.4.4.4.4"><csymbol cd="ambiguous" id="A1.SS2.3.p3.2.m2.5.5.4.4.4.4.1.cmml" xref="A1.SS2.3.p3.2.m2.5.5.4.4.4.4">subscript</csymbol><ci id="A1.SS2.3.p3.2.m2.5.5.4.4.4.4.2.cmml" xref="A1.SS2.3.p3.2.m2.5.5.4.4.4.4.2">𝑁</ci><cn type="integer" id="A1.SS2.3.p3.2.m2.5.5.4.4.4.4.3.cmml" xref="A1.SS2.3.p3.2.m2.5.5.4.4.4.4.3">4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.3.p3.2.m2.5c">N=\max\{N_{1},N_{2},N_{3},N_{4}\}</annotation></semantics></math>, then <math id="A1.SS2.3.p3.3.m3.1" class="ltx_Math" alttext="\forall n&gt;N" display="inline"><semantics id="A1.SS2.3.p3.3.m3.1a"><mrow id="A1.SS2.3.p3.3.m3.1.1" xref="A1.SS2.3.p3.3.m3.1.1.cmml"><mrow id="A1.SS2.3.p3.3.m3.1.1.2" xref="A1.SS2.3.p3.3.m3.1.1.2.cmml"><mo rspace="0.167em" id="A1.SS2.3.p3.3.m3.1.1.2.1" xref="A1.SS2.3.p3.3.m3.1.1.2.1.cmml">∀</mo><mi id="A1.SS2.3.p3.3.m3.1.1.2.2" xref="A1.SS2.3.p3.3.m3.1.1.2.2.cmml">n</mi></mrow><mo id="A1.SS2.3.p3.3.m3.1.1.1" xref="A1.SS2.3.p3.3.m3.1.1.1.cmml">&gt;</mo><mi id="A1.SS2.3.p3.3.m3.1.1.3" xref="A1.SS2.3.p3.3.m3.1.1.3.cmml">N</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.3.p3.3.m3.1b"><apply id="A1.SS2.3.p3.3.m3.1.1.cmml" xref="A1.SS2.3.p3.3.m3.1.1"><gt id="A1.SS2.3.p3.3.m3.1.1.1.cmml" xref="A1.SS2.3.p3.3.m3.1.1.1"></gt><apply id="A1.SS2.3.p3.3.m3.1.1.2.cmml" xref="A1.SS2.3.p3.3.m3.1.1.2"><csymbol cd="latexml" id="A1.SS2.3.p3.3.m3.1.1.2.1.cmml" xref="A1.SS2.3.p3.3.m3.1.1.2.1">for-all</csymbol><ci id="A1.SS2.3.p3.3.m3.1.1.2.2.cmml" xref="A1.SS2.3.p3.3.m3.1.1.2.2">𝑛</ci></apply><ci id="A1.SS2.3.p3.3.m3.1.1.3.cmml" xref="A1.SS2.3.p3.3.m3.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.3.p3.3.m3.1c">\forall n&gt;N</annotation></semantics></math>,</p>
<table id="A1.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex1.m1.2" class="ltx_Math" alttext="Z_{\theta}-\epsilon&lt;\mathbb{E}[T_{n}]&lt;Z_{\theta}&lt;\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]&lt;Z_{\theta}+\epsilon" display="block"><semantics id="A1.Ex1.m1.2a"><mrow id="A1.Ex1.m1.2.2" xref="A1.Ex1.m1.2.2.cmml"><mrow id="A1.Ex1.m1.2.2.4" xref="A1.Ex1.m1.2.2.4.cmml"><msub id="A1.Ex1.m1.2.2.4.2" xref="A1.Ex1.m1.2.2.4.2.cmml"><mi id="A1.Ex1.m1.2.2.4.2.2" xref="A1.Ex1.m1.2.2.4.2.2.cmml">Z</mi><mi id="A1.Ex1.m1.2.2.4.2.3" xref="A1.Ex1.m1.2.2.4.2.3.cmml">θ</mi></msub><mo id="A1.Ex1.m1.2.2.4.1" xref="A1.Ex1.m1.2.2.4.1.cmml">−</mo><mi id="A1.Ex1.m1.2.2.4.3" xref="A1.Ex1.m1.2.2.4.3.cmml">ϵ</mi></mrow><mo id="A1.Ex1.m1.2.2.5" xref="A1.Ex1.m1.2.2.5.cmml">&lt;</mo><mrow id="A1.Ex1.m1.1.1.1" xref="A1.Ex1.m1.1.1.1.cmml"><mi id="A1.Ex1.m1.1.1.1.3" xref="A1.Ex1.m1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.Ex1.m1.1.1.1.2" xref="A1.Ex1.m1.1.1.1.2.cmml">​</mo><mrow id="A1.Ex1.m1.1.1.1.1.1" xref="A1.Ex1.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.Ex1.m1.1.1.1.1.1.2" xref="A1.Ex1.m1.1.1.1.1.2.1.cmml">[</mo><msub id="A1.Ex1.m1.1.1.1.1.1.1" xref="A1.Ex1.m1.1.1.1.1.1.1.cmml"><mi id="A1.Ex1.m1.1.1.1.1.1.1.2" xref="A1.Ex1.m1.1.1.1.1.1.1.2.cmml">T</mi><mi id="A1.Ex1.m1.1.1.1.1.1.1.3" xref="A1.Ex1.m1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.Ex1.m1.1.1.1.1.1.3" xref="A1.Ex1.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.Ex1.m1.2.2.6" xref="A1.Ex1.m1.2.2.6.cmml">&lt;</mo><msub id="A1.Ex1.m1.2.2.7" xref="A1.Ex1.m1.2.2.7.cmml"><mi id="A1.Ex1.m1.2.2.7.2" xref="A1.Ex1.m1.2.2.7.2.cmml">Z</mi><mi id="A1.Ex1.m1.2.2.7.3" xref="A1.Ex1.m1.2.2.7.3.cmml">θ</mi></msub><mo id="A1.Ex1.m1.2.2.8" xref="A1.Ex1.m1.2.2.8.cmml">&lt;</mo><mrow id="A1.Ex1.m1.2.2.2" xref="A1.Ex1.m1.2.2.2.cmml"><mi id="A1.Ex1.m1.2.2.2.3" xref="A1.Ex1.m1.2.2.2.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A1.Ex1.m1.2.2.2.2" xref="A1.Ex1.m1.2.2.2.2.cmml">​</mo><mrow id="A1.Ex1.m1.2.2.2.1.1" xref="A1.Ex1.m1.2.2.2.1.2.cmml"><mo stretchy="false" id="A1.Ex1.m1.2.2.2.1.1.2" xref="A1.Ex1.m1.2.2.2.1.2.1.cmml">[</mo><mrow id="A1.Ex1.m1.2.2.2.1.1.1" xref="A1.Ex1.m1.2.2.2.1.1.1.cmml"><mrow id="A1.Ex1.m1.2.2.2.1.1.1.1" xref="A1.Ex1.m1.2.2.2.1.1.1.1.cmml"><mrow id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.2" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.cmml"><mrow id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.cmml"><mn id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.2" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.1" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.3" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.3.cmml">n</mi></mrow><mo id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.1" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.1.cmml">−</mo><mn id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.3" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.3" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.Ex1.m1.2.2.2.1.1.1.1.2" xref="A1.Ex1.m1.2.2.2.1.1.1.1.2.cmml">​</mo><msub id="A1.Ex1.m1.2.2.2.1.1.1.1.3" xref="A1.Ex1.m1.2.2.2.1.1.1.1.3.cmml"><mi id="A1.Ex1.m1.2.2.2.1.1.1.1.3.2" xref="A1.Ex1.m1.2.2.2.1.1.1.1.3.2.cmml">T</mi><mi id="A1.Ex1.m1.2.2.2.1.1.1.1.3.3" xref="A1.Ex1.m1.2.2.2.1.1.1.1.3.3.cmml">n</mi></msub></mrow><mo id="A1.Ex1.m1.2.2.2.1.1.1.3" xref="A1.Ex1.m1.2.2.2.1.1.1.3.cmml">−</mo><mrow id="A1.Ex1.m1.2.2.2.1.1.1.2" xref="A1.Ex1.m1.2.2.2.1.1.1.2.cmml"><mn id="A1.Ex1.m1.2.2.2.1.1.1.2.3" xref="A1.Ex1.m1.2.2.2.1.1.1.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="A1.Ex1.m1.2.2.2.1.1.1.2.2" xref="A1.Ex1.m1.2.2.2.1.1.1.2.2.cmml">​</mo><mrow id="A1.Ex1.m1.2.2.2.1.1.1.2.1.1" xref="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.2" xref="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.cmml">(</mo><mrow id="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1" xref="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.cmml"><mi id="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.2" xref="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.2.cmml">n</mi><mo id="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.1" xref="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.1.cmml">−</mo><mn id="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.3" xref="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.3" xref="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.Ex1.m1.2.2.2.1.1.1.2.2a" xref="A1.Ex1.m1.2.2.2.1.1.1.2.2.cmml">​</mo><msub id="A1.Ex1.m1.2.2.2.1.1.1.2.4" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4.cmml"><mi id="A1.Ex1.m1.2.2.2.1.1.1.2.4.2" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4.2.cmml">T</mi><mrow id="A1.Ex1.m1.2.2.2.1.1.1.2.4.3" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.cmml"><mi id="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.2" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.2.cmml">n</mi><mo id="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.1" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.1.cmml">−</mo><mn id="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.3" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.3.cmml">1</mn></mrow></msub></mrow></mrow><mo stretchy="false" id="A1.Ex1.m1.2.2.2.1.1.3" xref="A1.Ex1.m1.2.2.2.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.Ex1.m1.2.2.9" xref="A1.Ex1.m1.2.2.9.cmml">&lt;</mo><mrow id="A1.Ex1.m1.2.2.10" xref="A1.Ex1.m1.2.2.10.cmml"><msub id="A1.Ex1.m1.2.2.10.2" xref="A1.Ex1.m1.2.2.10.2.cmml"><mi id="A1.Ex1.m1.2.2.10.2.2" xref="A1.Ex1.m1.2.2.10.2.2.cmml">Z</mi><mi id="A1.Ex1.m1.2.2.10.2.3" xref="A1.Ex1.m1.2.2.10.2.3.cmml">θ</mi></msub><mo id="A1.Ex1.m1.2.2.10.1" xref="A1.Ex1.m1.2.2.10.1.cmml">+</mo><mi id="A1.Ex1.m1.2.2.10.3" xref="A1.Ex1.m1.2.2.10.3.cmml">ϵ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex1.m1.2b"><apply id="A1.Ex1.m1.2.2.cmml" xref="A1.Ex1.m1.2.2"><and id="A1.Ex1.m1.2.2a.cmml" xref="A1.Ex1.m1.2.2"></and><apply id="A1.Ex1.m1.2.2b.cmml" xref="A1.Ex1.m1.2.2"><lt id="A1.Ex1.m1.2.2.5.cmml" xref="A1.Ex1.m1.2.2.5"></lt><apply id="A1.Ex1.m1.2.2.4.cmml" xref="A1.Ex1.m1.2.2.4"><minus id="A1.Ex1.m1.2.2.4.1.cmml" xref="A1.Ex1.m1.2.2.4.1"></minus><apply id="A1.Ex1.m1.2.2.4.2.cmml" xref="A1.Ex1.m1.2.2.4.2"><csymbol cd="ambiguous" id="A1.Ex1.m1.2.2.4.2.1.cmml" xref="A1.Ex1.m1.2.2.4.2">subscript</csymbol><ci id="A1.Ex1.m1.2.2.4.2.2.cmml" xref="A1.Ex1.m1.2.2.4.2.2">𝑍</ci><ci id="A1.Ex1.m1.2.2.4.2.3.cmml" xref="A1.Ex1.m1.2.2.4.2.3">𝜃</ci></apply><ci id="A1.Ex1.m1.2.2.4.3.cmml" xref="A1.Ex1.m1.2.2.4.3">italic-ϵ</ci></apply><apply id="A1.Ex1.m1.1.1.1.cmml" xref="A1.Ex1.m1.1.1.1"><times id="A1.Ex1.m1.1.1.1.2.cmml" xref="A1.Ex1.m1.1.1.1.2"></times><ci id="A1.Ex1.m1.1.1.1.3.cmml" xref="A1.Ex1.m1.1.1.1.3">𝔼</ci><apply id="A1.Ex1.m1.1.1.1.1.2.cmml" xref="A1.Ex1.m1.1.1.1.1.1"><csymbol cd="latexml" id="A1.Ex1.m1.1.1.1.1.2.1.cmml" xref="A1.Ex1.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.Ex1.m1.1.1.1.1.1.1.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.Ex1.m1.1.1.1.1.1.1.1.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.Ex1.m1.1.1.1.1.1.1.2.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.2">𝑇</ci><ci id="A1.Ex1.m1.1.1.1.1.1.1.3.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></apply><apply id="A1.Ex1.m1.2.2c.cmml" xref="A1.Ex1.m1.2.2"><lt id="A1.Ex1.m1.2.2.6.cmml" xref="A1.Ex1.m1.2.2.6"></lt><share href="#A1.Ex1.m1.1.1.1.cmml" id="A1.Ex1.m1.2.2d.cmml" xref="A1.Ex1.m1.2.2"></share><apply id="A1.Ex1.m1.2.2.7.cmml" xref="A1.Ex1.m1.2.2.7"><csymbol cd="ambiguous" id="A1.Ex1.m1.2.2.7.1.cmml" xref="A1.Ex1.m1.2.2.7">subscript</csymbol><ci id="A1.Ex1.m1.2.2.7.2.cmml" xref="A1.Ex1.m1.2.2.7.2">𝑍</ci><ci id="A1.Ex1.m1.2.2.7.3.cmml" xref="A1.Ex1.m1.2.2.7.3">𝜃</ci></apply></apply><apply id="A1.Ex1.m1.2.2e.cmml" xref="A1.Ex1.m1.2.2"><lt id="A1.Ex1.m1.2.2.8.cmml" xref="A1.Ex1.m1.2.2.8"></lt><share href="#A1.Ex1.m1.2.2.7.cmml" id="A1.Ex1.m1.2.2f.cmml" xref="A1.Ex1.m1.2.2"></share><apply id="A1.Ex1.m1.2.2.2.cmml" xref="A1.Ex1.m1.2.2.2"><times id="A1.Ex1.m1.2.2.2.2.cmml" xref="A1.Ex1.m1.2.2.2.2"></times><ci id="A1.Ex1.m1.2.2.2.3.cmml" xref="A1.Ex1.m1.2.2.2.3">𝔼</ci><apply id="A1.Ex1.m1.2.2.2.1.2.cmml" xref="A1.Ex1.m1.2.2.2.1.1"><csymbol cd="latexml" id="A1.Ex1.m1.2.2.2.1.2.1.cmml" xref="A1.Ex1.m1.2.2.2.1.1.2">delimited-[]</csymbol><apply id="A1.Ex1.m1.2.2.2.1.1.1.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1"><minus id="A1.Ex1.m1.2.2.2.1.1.1.3.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.3"></minus><apply id="A1.Ex1.m1.2.2.2.1.1.1.1.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1"><times id="A1.Ex1.m1.2.2.2.1.1.1.1.2.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.2"></times><apply id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1"><minus id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.1.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.1"></minus><apply id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2"><times id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.1.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.2.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.2">2</cn><ci id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.3.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.2.3">𝑛</ci></apply><cn type="integer" id="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.3.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.1.1.1.3">1</cn></apply><apply id="A1.Ex1.m1.2.2.2.1.1.1.1.3.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.Ex1.m1.2.2.2.1.1.1.1.3.1.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.3">subscript</csymbol><ci id="A1.Ex1.m1.2.2.2.1.1.1.1.3.2.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.3.2">𝑇</ci><ci id="A1.Ex1.m1.2.2.2.1.1.1.1.3.3.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.1.3.3">𝑛</ci></apply></apply><apply id="A1.Ex1.m1.2.2.2.1.1.1.2.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2"><times id="A1.Ex1.m1.2.2.2.1.1.1.2.2.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.2"></times><cn type="integer" id="A1.Ex1.m1.2.2.2.1.1.1.2.3.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.3">2</cn><apply id="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.1.1"><minus id="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.1.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.1"></minus><ci id="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.2.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.2">𝑛</ci><cn type="integer" id="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.3.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.1.1.1.3">1</cn></apply><apply id="A1.Ex1.m1.2.2.2.1.1.1.2.4.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4"><csymbol cd="ambiguous" id="A1.Ex1.m1.2.2.2.1.1.1.2.4.1.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4">subscript</csymbol><ci id="A1.Ex1.m1.2.2.2.1.1.1.2.4.2.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4.2">𝑇</ci><apply id="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4.3"><minus id="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.1.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.1"></minus><ci id="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.2.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.2">𝑛</ci><cn type="integer" id="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.3.cmml" xref="A1.Ex1.m1.2.2.2.1.1.1.2.4.3.3">1</cn></apply></apply></apply></apply></apply></apply></apply><apply id="A1.Ex1.m1.2.2g.cmml" xref="A1.Ex1.m1.2.2"><lt id="A1.Ex1.m1.2.2.9.cmml" xref="A1.Ex1.m1.2.2.9"></lt><share href="#A1.Ex1.m1.2.2.2.cmml" id="A1.Ex1.m1.2.2h.cmml" xref="A1.Ex1.m1.2.2"></share><apply id="A1.Ex1.m1.2.2.10.cmml" xref="A1.Ex1.m1.2.2.10"><plus id="A1.Ex1.m1.2.2.10.1.cmml" xref="A1.Ex1.m1.2.2.10.1"></plus><apply id="A1.Ex1.m1.2.2.10.2.cmml" xref="A1.Ex1.m1.2.2.10.2"><csymbol cd="ambiguous" id="A1.Ex1.m1.2.2.10.2.1.cmml" xref="A1.Ex1.m1.2.2.10.2">subscript</csymbol><ci id="A1.Ex1.m1.2.2.10.2.2.cmml" xref="A1.Ex1.m1.2.2.10.2.2">𝑍</ci><ci id="A1.Ex1.m1.2.2.10.2.3.cmml" xref="A1.Ex1.m1.2.2.10.2.3">𝜃</ci></apply><ci id="A1.Ex1.m1.2.2.10.3.cmml" xref="A1.Ex1.m1.2.2.10.3">italic-ϵ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex1.m1.2c">Z_{\theta}-\epsilon&lt;\mathbb{E}[T_{n}]&lt;Z_{\theta}&lt;\mathbb{E}[(2n-1)T_{n}-2(n-1)T_{n-1}]&lt;Z_{\theta}+\epsilon</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A1.SS2.3.p3.4" class="ltx_p">∎</p>
</div>
</div>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Optimization Settings</h3>

<figure id="A1.T4" class="ltx_table">
<table id="A1.T4.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T4.1.1.1" class="ltx_tr">
<th id="A1.T4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Model</th>
<th id="A1.T4.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_tt">fp16</th>
<th id="A1.T4.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_tt">batch size</th>
<td id="A1.T4.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">warmup steps</td>
<td id="A1.T4.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt">max steps</td>
<td id="A1.T4.1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt">max lr</td>
<td id="A1.T4.1.1.1.7" class="ltx_td ltx_align_center ltx_border_tt">max grad norm</td>
</tr>
<tr id="A1.T4.1.2.2" class="ltx_tr">
<th id="A1.T4.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="A1.T4.1.2.2.1.1" class="ltx_text ltx_font_smallcaps">base LM</span></th>
<th id="A1.T4.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">-</th>
<th id="A1.T4.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">32</th>
<td id="A1.T4.1.2.2.4" class="ltx_td ltx_align_center ltx_border_t">2,000</td>
<td id="A1.T4.1.2.2.5" class="ltx_td ltx_align_center ltx_border_t">180,000</td>
<td id="A1.T4.1.2.2.6" class="ltx_td ltx_align_center ltx_border_t">0.0001</td>
<td id="A1.T4.1.2.2.7" class="ltx_td ltx_align_center ltx_border_t">10</td>
</tr>
<tr id="A1.T4.1.3.3" class="ltx_tr">
<th id="A1.T4.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T4.1.3.3.1.1" class="ltx_text ltx_font_smallcaps">RALM</span></th>
<th id="A1.T4.1.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">-</th>
<th id="A1.T4.1.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">64</th>
<td id="A1.T4.1.3.3.4" class="ltx_td ltx_align_center">2,000</td>
<td id="A1.T4.1.3.3.5" class="ltx_td ltx_align_center">180,000</td>
<td id="A1.T4.1.3.3.6" class="ltx_td ltx_align_center">0.0001</td>
<td id="A1.T4.1.3.3.7" class="ltx_td ltx_align_center">10</td>
</tr>
<tr id="A1.T4.1.4.4" class="ltx_tr">
<th id="A1.T4.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T4.1.4.4.1.1" class="ltx_text ltx_font_smallcaps">BALM</span></th>
<th id="A1.T4.1.4.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">-</th>
<th id="A1.T4.1.4.4.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">32</th>
<td id="A1.T4.1.4.4.4" class="ltx_td ltx_align_center">2,000</td>
<td id="A1.T4.1.4.4.5" class="ltx_td ltx_align_center">180,000</td>
<td id="A1.T4.1.4.4.6" class="ltx_td ltx_align_center">0.0001</td>
<td id="A1.T4.1.4.4.7" class="ltx_td ltx_align_center">10</td>
</tr>
<tr id="A1.T4.1.5.5" class="ltx_tr">
<th id="A1.T4.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T4.1.5.5.1.1" class="ltx_text ltx_font_smallcaps">joint UniT</span></th>
<th id="A1.T4.1.5.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">+</th>
<th id="A1.T4.1.5.5.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">64</th>
<td id="A1.T4.1.5.5.4" class="ltx_td ltx_align_center">2,000</td>
<td id="A1.T4.1.5.5.5" class="ltx_td ltx_align_center">180,000</td>
<td id="A1.T4.1.5.5.6" class="ltx_td ltx_align_center">0.0003</td>
<td id="A1.T4.1.5.5.7" class="ltx_td ltx_align_center">10</td>
</tr>
<tr id="A1.T4.1.6.6" class="ltx_tr">
<th id="A1.T4.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T4.1.6.6.1.1" class="ltx_text ltx_font_smallcaps">joint BiT-Base</span></th>
<th id="A1.T4.1.6.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">-</th>
<th id="A1.T4.1.6.6.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">60</th>
<td id="A1.T4.1.6.6.4" class="ltx_td ltx_align_center">2,000</td>
<td id="A1.T4.1.6.6.5" class="ltx_td ltx_align_center">90,000</td>
<td id="A1.T4.1.6.6.6" class="ltx_td ltx_align_center">0.00005</td>
<td id="A1.T4.1.6.6.7" class="ltx_td ltx_align_center">0.25</td>
</tr>
<tr id="A1.T4.1.7.7" class="ltx_tr">
<th id="A1.T4.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T4.1.7.7.1.1" class="ltx_text ltx_font_smallcaps">joint BiT-Base*</span></th>
<th id="A1.T4.1.7.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">-</th>
<th id="A1.T4.1.7.7.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">60</th>
<td id="A1.T4.1.7.7.4" class="ltx_td ltx_align_center">2,000</td>
<td id="A1.T4.1.7.7.5" class="ltx_td ltx_align_center">90,000</td>
<td id="A1.T4.1.7.7.6" class="ltx_td ltx_align_center">0.00005</td>
<td id="A1.T4.1.7.7.7" class="ltx_td ltx_align_center">0.25</td>
</tr>
<tr id="A1.T4.1.8.8" class="ltx_tr">
<th id="A1.T4.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T4.1.8.8.1.1" class="ltx_text ltx_font_smallcaps">joint BiT-Large*</span></th>
<th id="A1.T4.1.8.8.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">+</th>
<th id="A1.T4.1.8.8.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">64</th>
<td id="A1.T4.1.8.8.4" class="ltx_td ltx_align_center">2,000</td>
<td id="A1.T4.1.8.8.5" class="ltx_td ltx_align_center">90,000</td>
<td id="A1.T4.1.8.8.6" class="ltx_td ltx_align_center">0.0003</td>
<td id="A1.T4.1.8.8.7" class="ltx_td ltx_align_center">10</td>
</tr>
<tr id="A1.T4.1.9.9" class="ltx_tr">
<th id="A1.T4.1.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="A1.T4.1.9.9.1.1" class="ltx_text ltx_font_smallcaps">base LM-24L</span></th>
<th id="A1.T4.1.9.9.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">-</th>
<th id="A1.T4.1.9.9.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">50</th>
<td id="A1.T4.1.9.9.4" class="ltx_td ltx_align_center ltx_border_t">2,000</td>
<td id="A1.T4.1.9.9.5" class="ltx_td ltx_align_center ltx_border_t">90,000</td>
<td id="A1.T4.1.9.9.6" class="ltx_td ltx_align_center ltx_border_t">0.0003</td>
<td id="A1.T4.1.9.9.7" class="ltx_td ltx_align_center ltx_border_t">0.25</td>
</tr>
<tr id="A1.T4.1.10.10" class="ltx_tr">
<th id="A1.T4.1.10.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T4.1.10.10.1.1" class="ltx_text ltx_font_smallcaps">RALM-24L</span></th>
<th id="A1.T4.1.10.10.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">-</th>
<th id="A1.T4.1.10.10.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">28</th>
<td id="A1.T4.1.10.10.4" class="ltx_td ltx_align_center">1,000</td>
<td id="A1.T4.1.10.10.5" class="ltx_td ltx_align_center">90,000</td>
<td id="A1.T4.1.10.10.6" class="ltx_td ltx_align_center">0.00015</td>
<td id="A1.T4.1.10.10.7" class="ltx_td ltx_align_center">0.25</td>
</tr>
<tr id="A1.T4.1.11.11" class="ltx_tr">
<th id="A1.T4.1.11.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T4.1.11.11.1.1" class="ltx_text ltx_font_smallcaps">BALM-24L</span></th>
<th id="A1.T4.1.11.11.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">-</th>
<th id="A1.T4.1.11.11.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">28</th>
<td id="A1.T4.1.11.11.4" class="ltx_td ltx_align_center">2,000</td>
<td id="A1.T4.1.11.11.5" class="ltx_td ltx_align_center">90,000</td>
<td id="A1.T4.1.11.11.6" class="ltx_td ltx_align_center">0.0003</td>
<td id="A1.T4.1.11.11.7" class="ltx_td ltx_align_center">0.25</td>
</tr>
<tr id="A1.T4.1.12.12" class="ltx_tr">
<th id="A1.T4.1.12.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A1.T4.1.12.12.1.1" class="ltx_text ltx_font_smallcaps">joint UniT</span> (LM-24L)</th>
<th id="A1.T4.1.12.12.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">+</th>
<th id="A1.T4.1.12.12.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">64</th>
<td id="A1.T4.1.12.12.4" class="ltx_td ltx_align_center">2,000</td>
<td id="A1.T4.1.12.12.5" class="ltx_td ltx_align_center">180,000</td>
<td id="A1.T4.1.12.12.6" class="ltx_td ltx_align_center">0.0003</td>
<td id="A1.T4.1.12.12.7" class="ltx_td ltx_align_center">10</td>
</tr>
<tr id="A1.T4.1.13.13" class="ltx_tr">
<th id="A1.T4.1.13.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A1.T4.1.13.13.1.1" class="ltx_text ltx_font_smallcaps">joint BiT-Base</span> (LM-24L)</th>
<th id="A1.T4.1.13.13.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">-</th>
<th id="A1.T4.1.13.13.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">60</th>
<td id="A1.T4.1.13.13.4" class="ltx_td ltx_align_center">2,000</td>
<td id="A1.T4.1.13.13.5" class="ltx_td ltx_align_center">90,000</td>
<td id="A1.T4.1.13.13.6" class="ltx_td ltx_align_center">0.00005</td>
<td id="A1.T4.1.13.13.7" class="ltx_td ltx_align_center">0.25</td>
</tr>
<tr id="A1.T4.1.14.14" class="ltx_tr">
<th id="A1.T4.1.14.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A1.T4.1.14.14.1.1" class="ltx_text ltx_font_smallcaps">joint BiT-Base*</span> (LM-24L)</th>
<th id="A1.T4.1.14.14.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">-</th>
<th id="A1.T4.1.14.14.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">60</th>
<td id="A1.T4.1.14.14.4" class="ltx_td ltx_align_center">2,000</td>
<td id="A1.T4.1.14.14.5" class="ltx_td ltx_align_center">90,000</td>
<td id="A1.T4.1.14.14.6" class="ltx_td ltx_align_center">0.00005</td>
<td id="A1.T4.1.14.14.7" class="ltx_td ltx_align_center">0.25</td>
</tr>
<tr id="A1.T4.1.15.15" class="ltx_tr">
<th id="A1.T4.1.15.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A1.T4.1.15.15.1.1" class="ltx_text ltx_font_smallcaps">joint BiT-Med</span> (LM-24L)</th>
<th id="A1.T4.1.15.15.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">-</th>
<th id="A1.T4.1.15.15.3" class="ltx_td ltx_align_center ltx_th ltx_th_row">32</th>
<td id="A1.T4.1.15.15.4" class="ltx_td ltx_align_center">2,000</td>
<td id="A1.T4.1.15.15.5" class="ltx_td ltx_align_center">90,000</td>
<td id="A1.T4.1.15.15.6" class="ltx_td ltx_align_center">0.00005</td>
<td id="A1.T4.1.15.15.7" class="ltx_td ltx_align_center">0.25</td>
</tr>
<tr id="A1.T4.1.16.16" class="ltx_tr">
<th id="A1.T4.1.16.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">
<span id="A1.T4.1.16.16.1.1" class="ltx_text ltx_font_smallcaps">joint BiT-Large*</span> (LM-24L)</th>
<th id="A1.T4.1.16.16.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb">-</th>
<th id="A1.T4.1.16.16.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb">20</th>
<td id="A1.T4.1.16.16.4" class="ltx_td ltx_align_center ltx_border_bb">2,000</td>
<td id="A1.T4.1.16.16.5" class="ltx_td ltx_align_center ltx_border_bb">90,000</td>
<td id="A1.T4.1.16.16.6" class="ltx_td ltx_align_center ltx_border_bb">0.00005</td>
<td id="A1.T4.1.16.16.7" class="ltx_td ltx_align_center ltx_border_bb">0.25</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Optimization settings. We use the same setting for CC-News and Toronto Book Corpus.</figcaption>
</figure>
<figure id="A1.F4" class="ltx_figure"><img src="/html/2004.11714/assets/images/turk_screenshot_larger.png" id="A1.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="592" height="268" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Screenshot of the human evaluation.</figcaption>
</figure>
<div id="A1.SS3.p1" class="ltx_para ltx_noindent">
<p id="A1.SS3.p1.1" class="ltx_p">The optimization settings are presented in Table <a href="#A1.T4" title="Table 4 ‣ A.3 Optimization Settings ‣ Appendix A Appendix ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.</p>
</div>
</section>
<section id="A1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.4 </span>Human Evaluation</h3>

<div id="A1.SS4.p1" class="ltx_para ltx_noindent">
<p id="A1.SS4.p1.1" class="ltx_p">A screenshot of the human evaluation experiments can be found in Fig <a href="#A1.F4" title="Figure 4 ‣ A.3 Optimization Settings ‣ Appendix A Appendix ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.
Every page asks for 4 comparisons, one of which we know what the ground truth answer is.
We subsampled 333 sentences from the test set of CC-News, and asked 3 Amazon Mechanical turkers to vote.
We consider one continuation better if it gets more votes. To check the quality of the received ratings,
we performed a qualification task beforehand, where one of the continuations is real text,
and we kept the top half performing turkers for further evaluation (corresponding to higher than 66.67% accuracy
for discriminating real from LM samples – for a total of 26 qualified turkers).
Then in the actual experiment, we use one out of every four comparisons as an attention check and drop
responses if the turker did not pass the check.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A1.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.5 </span>Qualitative analysis</h3>

<div id="A1.SS5.p1" class="ltx_para ltx_noindent">
<p id="A1.SS5.p1.1" class="ltx_p">We present generation examples when our approach <span id="A1.SS5.p1.1.1" class="ltx_text ltx_font_smallcaps">Base LM</span> outperforms baseline <span id="A1.SS5.p1.1.2" class="ltx_text ltx_font_smallcaps">BALM</span> in Table <a href="#A1.T5" title="Table 5 ‣ A.5 Qualitative analysis ‣ Appendix A Appendix ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, and when our approach underperforms in Table <a href="#A1.T6" title="Table 6 ‣ A.5 Qualitative analysis ‣ Appendix A Appendix ‣ Residual Energy-Based Models for Text Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>. Here the judgment is based on human evaluation when all three turkers unanimously voted in favor of one model over the other.</p>
</div>
<figure id="A1.T5" class="ltx_table">
<table id="A1.T5.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T5.1.1.1" class="ltx_tr">
<th id="A1.T5.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_th ltx_th_column ltx_border_tt" style="width:156.5pt;">
<span id="A1.T5.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.1.1.1.1.1" class="ltx_p">Prefix</span>
</span>
</th>
<th id="A1.T5.1.1.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_th ltx_th_column ltx_border_tt" style="width:37.0pt;">
<span id="A1.T5.1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.1.1.2.1.1" class="ltx_p">Model</span>
</span>
</th>
<th id="A1.T5.1.1.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_th ltx_th_column ltx_border_tt" style="width:167.9pt;">
<span id="A1.T5.1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.1.1.3.1.1" class="ltx_p">Generation</span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T5.1.2.1" class="ltx_tr">
<td id="A1.T5.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:156.5pt;" rowspan="5">
<span id="A1.T5.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.2.1.1.1.1" class="ltx_p">
<span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note">9</span>Excerpt from <a target="_blank" href="https://chicago.cbslocal.com/2017/11/20/chance-the-rapper-jewel-osco/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://chicago.cbslocal.com/2017/11/20/chance-the-rapper-jewel-osco/</a></span></span></span>the timing of the donation couldn’t be better, and even went on to say that no one is more relevant than Chance right now. Thank you to Doug and @jewelosco for donating $1,000,000 to The New Chance Arts &amp; Literature Fund, which will ensure more students have access to arts enrichment education! #supportcps pic.twitter.com/MXZtpscU5b — SocialWorks (@SocialWorks_Chi) November 20, 2017 “In the past, CPS hasn’t really gotten a lot of money from non-profits</span>
</span>
</td>
<td id="A1.T5.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.2.1.2.1.1" class="ltx_p"><span id="A1.T5.1.2.1.2.1.1.1" class="ltx_text ltx_font_smallcaps">Base LM</span></span>
</span>
</td>
<td id="A1.T5.1.2.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.2.1.3.1.1" class="ltx_p">. And the fact that the money is coming from the government, it makes a big difference,” he said. “We’re not the biggest donor of arts education, so to</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.3.2" class="ltx_tr">
<td id="A1.T5.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.3.2.1.1.1" class="ltx_p"><span id="A1.T5.1.3.2.1.1.1.1" class="ltx_text ltx_font_smallcaps">BALM</span></span>
</span>
</td>
<td id="A1.T5.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.3.2.2.1.1" class="ltx_p">, and it doesn’t even have the kind of resources or funding for the arts or literary programs that we need to do the things that people want to support,” said Chance.</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.4.3" class="ltx_tr">
<td id="A1.T5.1.4.3.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.4.3.1.1.1" class="ltx_p"><span id="A1.T5.1.4.3.1.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base-Worst</span></span>
</span>
</td>
<td id="A1.T5.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.4.3.2.1.1" class="ltx_p">,” said Cpl. Chris Bissonnette of the Stony Brook Police Department. “So the reason is, as I think about the impact of these grants and these grants to schools</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.5.4" class="ltx_tr">
<td id="A1.T5.1.5.4.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.5.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.5.4.1.1.1" class="ltx_p"><span id="A1.T5.1.5.4.1.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base</span></span>
</span>
</td>
<td id="A1.T5.1.5.4.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.5.4.2.1.1" class="ltx_p">to fund programming. So, I think that it’s a great time for us to do something to support that,” said Chance. “We want to make sure that our programs</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.6.5" class="ltx_tr">
<td id="A1.T5.1.6.5.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.6.5.1.1.1" class="ltx_p"><span id="A1.T5.1.6.5.1.1.1.1" class="ltx_text ltx_font_smallcaps">Gold</span></span>
</span>
</td>
<td id="A1.T5.1.6.5.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.6.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.6.5.2.1.1" class="ltx_p">and charitable arms and organizations because there’s been a past of corrupt activity in the city of Chicago,” Chance said. “And so, when it comes down to it, it</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.7.6" class="ltx_tr">
<td id="A1.T5.1.7.6.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:156.5pt;" rowspan="5">
<span id="A1.T5.1.7.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.7.6.1.1.1" class="ltx_p"><span id="footnote10" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">10</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">10</sup><span class="ltx_tag ltx_tag_note">10</span>Excerpt from <a target="_blank" href="https://chiefswire.usatoday.com/2017/12/10/halftime-analysis-for-chiefs-vs-raiders-week-14/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://chiefswire.usatoday.com/2017/12/10/halftime-analysis-for-chiefs-vs-raiders-week-14/</a></span></span></span>quarter. The penalties are still somewhat of a problem but tight ends Travis Kelce and Demetrius Harris made some impressive grown-man football plays. -It was nice to see running back Kareem Hunt get in the end zone for the first time since Week 3. He must feel good to end the drought. -Kelce was visibly frustrated on the sidelines and rightfully so. The officials seem to be leaning toward Oakland with calls today. Still, Kelce should’ve caught that easy pass that he dropped. -Quarterback Alex Smith has delivered a couple of nice deep balls to</span>
</span>
</td>
<td id="A1.T5.1.7.6.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.7.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.7.6.2.1.1" class="ltx_p"><span id="A1.T5.1.7.6.2.1.1.1" class="ltx_text ltx_font_smallcaps">Base LM</span></span>
</span>
</td>
<td id="A1.T5.1.7.6.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.7.6.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.7.6.3.1.1" class="ltx_p">get his hands on the ball this week. He threw two touchdown passes on Thursday. He should get another touchdown as the season goes on. He’s got a good chance to be one of</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.8.7" class="ltx_tr">
<td id="A1.T5.1.8.7.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.8.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.8.7.1.1.1" class="ltx_p"><span id="A1.T5.1.8.7.1.1.1.1" class="ltx_text ltx_font_smallcaps">BALM</span></span>
</span>
</td>
<td id="A1.T5.1.8.7.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.8.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.8.7.2.1.1" class="ltx_p">the Chiefs and the Chiefs defense has made them pay for it. The offense has done a nice job with the ball, and they’s been a factor. It’s nice to have</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.9.8" class="ltx_tr">
<td id="A1.T5.1.9.8.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.9.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.9.8.1.1.1" class="ltx_p"><span id="A1.T5.1.9.8.1.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base-Worst</span></span>
</span>
</td>
<td id="A1.T5.1.9.8.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.9.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.9.8.2.1.1" class="ltx_p">tight end Martavis Bryant to get the ball back, but the throw from Smith to Davis Webb is incomplete, which could lead to an incompletion. -The Chiefs are now 5-2 after winning</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.10.9" class="ltx_tr">
<td id="A1.T5.1.10.9.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.10.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.10.9.1.1.1" class="ltx_p"><span id="A1.T5.1.10.9.1.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base</span></span>
</span>
</td>
<td id="A1.T5.1.10.9.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.10.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.10.9.2.1.1" class="ltx_p">help the Chiefs win this game. His completion of a deep ball to wide receiver Sammy Watkins is a nice sign. -Jalen Ramsey should be back in the lineup today. The Jaguars are going</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.11.10" class="ltx_tr">
<td id="A1.T5.1.11.10.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.11.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.11.10.1.1.1" class="ltx_p"><span id="A1.T5.1.11.10.1.1.1.1" class="ltx_text ltx_font_smallcaps">Gold</span></span>
</span>
</td>
<td id="A1.T5.1.11.10.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.11.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.11.10.2.1.1" class="ltx_p">receivers Albert Wilson and Tyreek Hill. The footwork by Wilson on the deep ball was amazing and Hill’s ball-tracking skills were on point as usual. Good things happen when Smith trusts</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.12.11" class="ltx_tr">
<td id="A1.T5.1.12.11.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb ltx_border_t" style="width:156.5pt;" rowspan="5">
<span id="A1.T5.1.12.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.12.11.1.1.1" class="ltx_p"><span id="footnote11" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">11</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">11</sup><span class="ltx_tag ltx_tag_note">11</span>Excerpt from <a target="_blank" href="https://www.glamour.com/story/kourtney-kardashian-intense-detox" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.glamour.com/story/kourtney-kardashian-intense-detox</a></span></span></span>has an amazing ability to detoxify itself namely by the liver, kidneys, and skin,” she says. Detox programs and diets do sometimes make people feel better, but that happens more often if you’re eating and/or drinking things that make feel like crap (something it doesn’t sound like Kardashian was doing in the first place). And if, somehow, you discover that you have elevated levels of mercury or lead in your body like Kardashian said she had, it’s important to figure out why, Dr. Wider says. “Finding the source is extremely important for any heavy metal, so you</span>
</span>
</td>
<td id="A1.T5.1.12.11.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.12.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.12.11.2.1.1" class="ltx_p"><span id="A1.T5.1.12.11.2.1.1.1" class="ltx_text ltx_font_smallcaps">Base LM</span></span>
</span>
</td>
<td id="A1.T5.1.12.11.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.12.11.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.12.11.3.1.1" class="ltx_p">can’t say ’no or I’ve never seen something of that value, so I’ll try and find it again and see what happens.” So don’t be afraid to find a way to detox or</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.13.12" class="ltx_tr">
<td id="A1.T5.1.13.12.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.13.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.13.12.1.1.1" class="ltx_p"><span id="A1.T5.1.13.12.1.1.1.1" class="ltx_text ltx_font_smallcaps">BALM</span></span>
</span>
</td>
<td id="A1.T5.1.13.12.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.13.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.13.12.2.1.1" class="ltx_p">want to get the full picture and not just have to look at the whole picture,” she says. “It can be a great idea to talk to your doctor to determine what’s causing your symptoms.”</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.14.13" class="ltx_tr">
<td id="A1.T5.1.14.13.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.14.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.14.13.1.1.1" class="ltx_p"><span id="A1.T5.1.14.13.1.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base-Worst</span></span>
</span>
</td>
<td id="A1.T5.1.14.13.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.14.13.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.14.13.2.1.1" class="ltx_p">can get rid of toxins that are found in foods like whole wheat bread, nuts like walnuts, walnuts and walnuts,” she says. ”If a source of the metals has elevated levels of</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.15.14" class="ltx_tr">
<td id="A1.T5.1.15.14.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.15.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.15.14.1.1.1" class="ltx_p"><span id="A1.T5.1.15.14.1.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base</span></span>
</span>
</td>
<td id="A1.T5.1.15.14.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.15.14.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.15.14.2.1.1" class="ltx_p">can’t say, ’Oh my God, it’s a lead, but I’m fine.’ But it’s definitely a problem in our bodies,” she says. “And if it’s a problem in</span>
</span>
</td>
</tr>
<tr id="A1.T5.1.16.15" class="ltx_tr">
<td id="A1.T5.1.16.15.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb ltx_border_t" style="width:37.0pt;">
<span id="A1.T5.1.16.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.16.15.1.1.1" class="ltx_p"><span id="A1.T5.1.16.15.1.1.1.1" class="ltx_text ltx_font_smallcaps">Gold</span></span>
</span>
</td>
<td id="A1.T5.1.16.15.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb ltx_border_t" style="width:167.9pt;">
<span id="A1.T5.1.16.15.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T5.1.16.15.2.1.1" class="ltx_p">can lower your exposure over time,” she says. Meaning, a detox diet probably isn’t going to help you as much as, say, reducing your intake of foods that are high in mercury (like</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Example generations when <span id="A1.T5.5.1" class="ltx_text ltx_font_smallcaps">BIT-BASE</span> outperforms <span id="A1.T5.6.2" class="ltx_text ltx_font_smallcaps">BALM</span> according to human evaluation. <span id="A1.T5.7.3" class="ltx_text ltx_font_smallcaps">BiT-Base-Worst</span> shows the LM sample with the highest energy score.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A1.T6" class="ltx_table">
<table id="A1.T6.5" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T6.5.6.1" class="ltx_tr">
<th id="A1.T6.5.6.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_th ltx_th_column ltx_border_tt" style="width:156.5pt;">
<span id="A1.T6.5.6.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.6.1.1.1.1" class="ltx_p">Prefix</span>
</span>
</th>
<th id="A1.T6.5.6.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_th ltx_th_column ltx_border_tt" style="width:37.0pt;">
<span id="A1.T6.5.6.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.6.1.2.1.1" class="ltx_p">Model</span>
</span>
</th>
<th id="A1.T6.5.6.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_th ltx_th_column ltx_border_tt" style="width:170.7pt;">
<span id="A1.T6.5.6.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.6.1.3.1.1" class="ltx_p">Generation</span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T6.5.7.1" class="ltx_tr">
<td id="A1.T6.5.7.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:156.5pt;" rowspan="5">
<span id="A1.T6.5.7.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.7.1.1.1.1" class="ltx_p"><span id="footnote12" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">12</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">12</sup><span class="ltx_tag ltx_tag_note">12</span>Excerpt from <a target="_blank" href="https://www.libraryjournal.com/?detailStory=lakewood-oh-mom-sues-library-over-teens-rough-treatment" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.libraryjournal.com/?detailStory=lakewood-oh-mom-sues-library-over-teens-rough-treatment</a></span></span></span>but as the audio only came on halfway through the incident, it did not record his initial words to the girl or her response. It was also dropped in the hallway during their interaction, so it did not record any of what happened once the two left the library. Jones brought her outside to the library’s front steps, where he waited for responding officers to arrive. They did not call for an ambulance, according to the lawsuit. Rather, when the officers arrived, they told her to spit blood on the grass instead of the concrete, and to move from the steps to a bench. Some 20 minutes</span>
</span>
</td>
<td id="A1.T6.5.7.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.7.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.7.1.2.1.1" class="ltx_p"><span id="A1.T6.5.7.1.2.1.1.1" class="ltx_text ltx_font_smallcaps">Base LM</span></span>
</span>
</td>
<td id="A1.T6.5.7.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.7.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.7.1.3.1.1" class="ltx_p">later, she was dead, according to the lawsuit. “It’s not uncommon for an officer to spit blood on an unarmed, vulnerable adult. The use of spit is illegal in a public place,”</span>
</span>
</td>
</tr>
<tr id="A1.T6.5.8.2" class="ltx_tr">
<td id="A1.T6.5.8.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.8.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.8.2.1.1.1" class="ltx_p"><span id="A1.T6.5.8.2.1.1.1.1" class="ltx_text ltx_font_smallcaps">BALM</span></span>
</span>
</td>
<td id="A1.T6.5.8.2.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.8.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.8.2.2.1.1" class="ltx_p">later, Jones was arrested for disorderly conduct and released after posting bond, the suit said. She was arrested again on May 2. In the suit, Jones is seeking more than $1 million. A</span>
</span>
</td>
</tr>
<tr id="A1.T6.5.9.3" class="ltx_tr">
<td id="A1.T6.5.9.3.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.9.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.9.3.1.1.1" class="ltx_p"><span id="A1.T6.5.9.3.1.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base-Worst</span></span>
</span>
</td>
<td id="A1.T6.5.9.3.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.9.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.9.3.2.1.1" class="ltx_p">later, he walked up on the ground, where they began to yell and yell again. When Jones refused, the officers ordered him not to leave or leave any more. Jones then pulled his gun on</span>
</span>
</td>
</tr>
<tr id="A1.T6.5.10.4" class="ltx_tr">
<td id="A1.T6.5.10.4.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.10.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.10.4.1.1.1" class="ltx_p"><span id="A1.T6.5.10.4.1.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base</span></span>
</span>
</td>
<td id="A1.T6.5.10.4.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.10.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.10.4.2.1.1" class="ltx_p">after the officers’ arrival, Jones and the girl left the library and headed to the parking lot of a nearby apartment complex. “This is the type of behavior that is most likely to cause harm to</span>
</span>
</td>
</tr>
<tr id="A1.T6.5.11.5" class="ltx_tr">
<td id="A1.T6.5.11.5.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.11.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.11.5.1.1.1" class="ltx_p"><span id="A1.T6.5.11.5.1.1.1.1" class="ltx_text ltx_font_smallcaps">Gold</span></span>
</span>
</td>
<td id="A1.T6.5.11.5.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.11.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.11.5.2.1.1" class="ltx_p">later, J.G’s mother arrived and took her daughter to Cleveland Clinic in Lakewood, where she was treated for a dislocated jaw, a head injury, and an injury to her knee.</span>
</span>
</td>
</tr>
<tr id="A1.T6.2.2" class="ltx_tr">
<td id="A1.T6.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:156.5pt;" rowspan="5">
<span id="A1.T6.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.1.1.1.1.1" class="ltx_p"><span id="footnote13" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">13</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">13</sup><span class="ltx_tag ltx_tag_note">13</span>Excerpt from <a target="_blank" href="https://www.sun-sentinel.com/community/delray-sun/fl-drf-village-academy-steam-0418-20180410-story.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.sun-sentinel.com/community/delray-sun/fl-drf-village-academy-steam-0418-20180410-story.html</a></span></span></span>, Bronson said. “The initiative provides a variety of supports to early childhood programs’ children, families and staff. The resources provided through this partnership increase the quality of the participating programs, which benefits the community and impacts our future in such a positive way,” Scott said. Visit PNCGrowUpGreat.com/donorschoose. <math id="A1.T6.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\backslash" display="inline"><semantics id="A1.T6.1.1.1.1.1.m1.1a"><mo id="A1.T6.1.1.1.1.1.m1.1.1" xref="A1.T6.1.1.1.1.1.m1.1.1.cmml">\</mo><annotation-xml encoding="MathML-Content" id="A1.T6.1.1.1.1.1.m1.1b"><ci id="A1.T6.1.1.1.1.1.m1.1.1.cmml" xref="A1.T6.1.1.1.1.1.m1.1.1">\</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.1.1.1.1.1.m1.1c">\backslash</annotation></semantics></math>nHere are Varsity sports headlines for April 13, 2018. Refresh to get the latest as we add news throughout the night as we collect scores: Best of the best in track and field Our Sentinel coverage area top performers lists for girls track and field</span>
</span>
</td>
<td id="A1.T6.2.2.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.2.2.3.1.1" class="ltx_p"><span id="A1.T6.2.2.3.1.1.1" class="ltx_text ltx_font_smallcaps">Base LM</span></span>
</span>
</td>
<td id="A1.T6.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.2.2.2.1.1" class="ltx_p">at the Varsity Track &amp; Field Invite.<math id="A1.T6.2.2.2.1.1.m1.1" class="ltx_Math" alttext="\backslash" display="inline"><semantics id="A1.T6.2.2.2.1.1.m1.1a"><mo id="A1.T6.2.2.2.1.1.m1.1.1" xref="A1.T6.2.2.2.1.1.m1.1.1.cmml">\</mo><annotation-xml encoding="MathML-Content" id="A1.T6.2.2.2.1.1.m1.1b"><ci id="A1.T6.2.2.2.1.1.m1.1.1.cmml" xref="A1.T6.2.2.2.1.1.m1.1.1">\</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.2.2.2.1.1.m1.1c">\backslash</annotation></semantics></math>nThe U.S. Army Corps of Engineers has approved $2 billion in construction work along the U.S.-Mexico boundary as a way to</span>
</span>
</td>
</tr>
<tr id="A1.T6.5.12.6" class="ltx_tr">
<td id="A1.T6.5.12.6.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.12.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.12.6.1.1.1" class="ltx_p"><span id="A1.T6.5.12.6.1.1.1.1" class="ltx_text ltx_font_smallcaps">BALM</span></span>
</span>
</td>
<td id="A1.T6.5.12.6.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.12.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.12.6.2.1.1" class="ltx_p">. Check back frequently for updates. The Sentinel also has a feature on the boys basketball and wrestling teams. Boys golf The Sentinel boys golf and wrestling teams have been one of those teams who has had some</span>
</span>
</td>
</tr>
<tr id="A1.T6.3.3" class="ltx_tr">
<td id="A1.T6.3.3.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.3.3.2.1.1" class="ltx_p"><span id="A1.T6.3.3.2.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base-Worst</span></span>
</span>
</td>
<td id="A1.T6.3.3.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.3.3.1.1.1" class="ltx_p">.<math id="A1.T6.3.3.1.1.1.m1.1" class="ltx_Math" alttext="\backslash" display="inline"><semantics id="A1.T6.3.3.1.1.1.m1.1a"><mo id="A1.T6.3.3.1.1.1.m1.1.1" xref="A1.T6.3.3.1.1.1.m1.1.1.cmml">\</mo><annotation-xml encoding="MathML-Content" id="A1.T6.3.3.1.1.1.m1.1b"><ci id="A1.T6.3.3.1.1.1.m1.1.1.cmml" xref="A1.T6.3.3.1.1.1.m1.1.1">\</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.3.3.1.1.1.m1.1c">\backslash</annotation></semantics></math>nLONDON, April 13 (IFR) - The following are some of the main factors expected to affect Spanish corporate bond prices on Wednesday. BAML: Spanish sovereign wealth fund PPV</span>
</span>
</td>
</tr>
<tr id="A1.T6.4.4" class="ltx_tr">
<td id="A1.T6.4.4.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.4.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.4.4.2.1.1" class="ltx_p"><span id="A1.T6.4.4.2.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base</span></span>
</span>
</td>
<td id="A1.T6.4.4.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.4.4.1.1.1" class="ltx_p">.<math id="A1.T6.4.4.1.1.1.m1.1" class="ltx_Math" alttext="\backslash" display="inline"><semantics id="A1.T6.4.4.1.1.1.m1.1a"><mo id="A1.T6.4.4.1.1.1.m1.1.1" xref="A1.T6.4.4.1.1.1.m1.1.1.cmml">\</mo><annotation-xml encoding="MathML-Content" id="A1.T6.4.4.1.1.1.m1.1b"><ci id="A1.T6.4.4.1.1.1.m1.1.1.cmml" xref="A1.T6.4.4.1.1.1.m1.1.1">\</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.4.4.1.1.1.m1.1c">\backslash</annotation></semantics></math>nA few weeks back, it’s been rumored that the HTC Desire was going to be the company’s last flagship phone, and now, a new leak has confirmed that it</span>
</span>
</td>
</tr>
<tr id="A1.T6.5.13.7" class="ltx_tr">
<td id="A1.T6.5.13.7.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.13.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.13.7.1.1.1" class="ltx_p"><span id="A1.T6.5.13.7.1.1.1.1" class="ltx_text ltx_font_smallcaps">Gold</span></span>
</span>
</td>
<td id="A1.T6.5.13.7.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.13.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.13.7.2.1.1" class="ltx_p">and boys track and field are updated going into the Saturday district meets. The season is heating up with more district and region races coming up next week. Click these links for girls top performers and boys top</span>
</span>
</td>
</tr>
<tr id="A1.T6.5.14.8" class="ltx_tr">
<td id="A1.T6.5.14.8.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb ltx_border_t" style="width:156.5pt;" rowspan="5">
<span id="A1.T6.5.14.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.14.8.1.1.1" class="ltx_p">
<span id="footnote14" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">14</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">14</sup><span class="ltx_tag ltx_tag_note">14</span>Excerpt from <a target="_blank" href="https://seekingalpha.com/article/4215142-apple-looks-to-recharge-tech-sector" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://seekingalpha.com/article/4215142-apple-looks-to-recharge-tech-sector</a></span></span></span>leaders meeting November 17-18 in Papua New Guinea as potential Xi-Trump meet dates. If all else fails, Trump and Xi are also expected to meet for a bit at the G20 meeting at the end of November. On the economic calendar next week, the update on jobs and the U.S. trade deficit are the headliners on November 2. Notable earnings reports: Akamai Technologies (NASDAQ:AKAM), Mondelez International (NASDAQ:MDLZ) and Olin Corp. (NYSE:OLN) on October 29; Under Armour (NYSE:</span>
</span>
</td>
<td id="A1.T6.5.14.8.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.14.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.14.8.2.1.1" class="ltx_p"><span id="A1.T6.5.14.8.2.1.1.1" class="ltx_text ltx_font_smallcaps">Base LM</span></span>
</span>
</td>
<td id="A1.T6.5.14.8.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.14.8.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.14.8.3.1.1" class="ltx_p">UAA), General Motors (NYSE:GM) on November 4; and Procter &amp; Gamble (NYSE:PG) for October. On the retail front, Lowe’s Companies (NYSE:L</span>
</span>
</td>
</tr>
<tr id="A1.T6.5.15.9" class="ltx_tr">
<td id="A1.T6.5.15.9.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.15.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.15.9.1.1.1" class="ltx_p"><span id="A1.T6.5.15.9.1.1.1.1" class="ltx_text ltx_font_smallcaps">BALM</span></span>
</span>
</td>
<td id="A1.T6.5.15.9.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.15.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.15.9.2.1.1" class="ltx_p">UA) on October 30; CVS Health (NASDAQ:CVS) on November 27; Intel Corporation (NASDAQ:INTC) on October 28; and Verizon Communications (NYSE:V</span>
</span>
</td>
</tr>
<tr id="A1.T6.5.16.10" class="ltx_tr">
<td id="A1.T6.5.16.10.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.16.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.16.10.1.1.1" class="ltx_p"><span id="A1.T6.5.16.10.1.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base-Worst</span></span>
</span>
</td>
<td id="A1.T6.5.16.10.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.16.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.16.10.2.1.1" class="ltx_p">UAA) and Adidas (OTCPK:ADDYYF; OTCQX:ADDYYFGF; OLYMP), on November 30; and Qualcomm Incorporated (NASDAQ:</span>
</span>
</td>
</tr>
<tr id="A1.T6.5.5" class="ltx_tr">
<td id="A1.T6.5.5.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.5.2.1.1" class="ltx_p"><span id="A1.T6.5.5.2.1.1.1" class="ltx_text ltx_font_smallcaps">BiT-Base</span></span>
</span>
</td>
<td id="A1.T6.5.5.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.5.1.1.1" class="ltx_p">UAA), Johnson Controls (NYSE:JCI) and Cisco Systems (NASDAQ:CSCO) on November 6.<math id="A1.T6.5.5.1.1.1.m1.1" class="ltx_Math" alttext="\backslash" display="inline"><semantics id="A1.T6.5.5.1.1.1.m1.1a"><mo id="A1.T6.5.5.1.1.1.m1.1.1" xref="A1.T6.5.5.1.1.1.m1.1.1.cmml">\</mo><annotation-xml encoding="MathML-Content" id="A1.T6.5.5.1.1.1.m1.1b"><ci id="A1.T6.5.5.1.1.1.m1.1.1.cmml" xref="A1.T6.5.5.1.1.1.m1.1.1">\</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.5.5.1.1.1.m1.1c">\backslash</annotation></semantics></math>nA woman who had to have her nose and mouth taped as punishment</span>
</span>
</td>
</tr>
<tr id="A1.T6.5.17.11" class="ltx_tr">
<td id="A1.T6.5.17.11.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb ltx_border_t" style="width:37.0pt;">
<span id="A1.T6.5.17.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.17.11.1.1.1" class="ltx_p"><span id="A1.T6.5.17.11.1.1.1.1" class="ltx_text ltx_font_smallcaps">Gold</span></span>
</span>
</td>
<td id="A1.T6.5.17.11.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb ltx_border_t" style="width:170.7pt;">
<span id="A1.T6.5.17.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T6.5.17.11.2.1.1" class="ltx_p">UAA), eBay (NASDAQ:EBAY), General Electric (NYSE:GE), Coca-Cola (NYSE:KO), Pfizer (NYSE:PFE) and Electronic Arts (NAS</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Example generations when <span id="A1.T6.9.1" class="ltx_text ltx_font_smallcaps">BIT-BASE</span> underperforms <span id="A1.T6.10.2" class="ltx_text ltx_font_smallcaps">BALM</span> according to human evaluation. <span id="A1.T6.11.3" class="ltx_text ltx_font_smallcaps">BiT-Base-Worst</span> shows the LM sample with the highest energy score.</figcaption>
</figure>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2004.11713" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2004.11714" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2004.11714">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2004.11714" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2004.11715" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Sat Mar  2 11:59:17 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2004.11714
Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation.
The dominant parametric approach is based on locally normalized models which predict one word at a time. While these
work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process.
In this work, we investigateun-normalizedenergy-based models (EBMs)  which operate not at the token but at the sequence
level. In order to make training tractable, we first work in the residual of a pretrained locally normalized
language model and second we train using noise contrastive estimation. Furthermore, since the EBM works at the
sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa.
Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity
compared to locally normalized baselines. Moreover, generation via importance sampling is very efficient and of higher quality
than the baseline models according to human evaluation.
abstract
Residual Energy-Based Models forText Generation
Abstract
2004.11714
1IntroductionThe dominant approach to parametric text generation is based on large neural auto-regressive
models. These models can be trained efficiently via maximum likelihood
and they can efficiently generate samples of remarkable quality.
Key to their success islocal normalization, i.e. they are defined in terms of a product of conditional
distributions, one for each token in the sequence. Such distributions are relatively cheap to compute with modern hardware given
the limited vocabulary size of common sub-word units like BPE.Unfortunately, local normalization also brings some drawbacks. First, the designer of the model needs to specify the order
in which tokens are generated. Second, at training time the model is conditioned on ground truth context while at test
time it is conditioned on its own generations, a discrepancy referred to as exposure bias.
Finally, while heuristics like beam search somewhat help rescore at the sequence level,
generation generally lacks long-range coherency because it is produced by the greedy selection of one token at a time
without lookahead.Energy-based models (EBMs) are a more general framework which
potentially address all these issues, as they do not require any local normalization. They only require the definition
of an energy function defined over the whole input sequence. Training aims at shaping the energy function such that
regions of high density of training data points have lower energy than elsewhere.
In principle, EBMs are ideal for modeling text as they can score the whole input at once,
they are not prone to label biasand they may enable generation of large chunks of text,
which should help improve coherency.However, so far EBMs had limited application in text generation, because
sampling from the model is intractable, and so is maximum likelihood training. The problem is that shaping the energy function
is accomplished by updating the model parameters such that the energy is decreased at the training data points (a.k.a. positive
examples)  and increased at other data points (a.k.a. negative examples) . In maximum likelihood training negatives are generated
from the model, but in text application we cannot use gradient-based MCMC methodsand
Gibbs samplingis
too slow to be practical. Generating negatives by local perturbations of the ground truth would be efficient but hardly useful for
generation purposes, when at test time the model needs to generate from scratch.Recently,carefully studied the problem of training a discriminator to distinguish human written
text from language model generations. They experimented with different language model and discriminator architectures,
training/test time corpora and concluded that the discriminator can generalize rather well to weaker language
models when the training/test corpora match.found that the learned discriminator is not robust to random perturbations,
and argued that the discriminator operates in the “residual” space of the language model.Concurrently,proposed a general approach to “de-bias” a generator,
by simply training a discriminator and using its output for importance sampling.In this work, we build upon these two works. First, we formalize the residual interpretation byand use a generative model of the form:(1) whereis a locally normalized language model which is fixed during training, andis the
energy function parameterized by. The resulting modelis globally normalized due to the energy term. Note that the same residual formulation was also used in.This formulation has multi-fold benefits. First, by incorporating a locally normalized language model,
we can leverage recent advancements in locally normalized language modeling. Second, the language model provides a natural proposal distribution for training, and training can be made efficient by using
the conditional noise contrastive
estimation objectiveas we shall see in §3.
Lastly, this formulation enables efficient evaluation and generation via importance sampling.In some sense, this last point is perhaps the central contribution of the paper, as it allows estimating perplexity of the residual EBM, and thus allows these EBMs to be compared in a standard way to other models. Indeed, in §4we show that our joint model decreases perplexity
on two large datasets, when compared
to various auto-regressive language model baselines. Finally, the EBM generations are significantly preferred by humans
according to our qualitative evaluation. To the best of our knowledge, this is the first time that an EBM has
demonstrated improved generation ability against very strong auto-regressive baselines, both in terms of estimated perplexity and through human evaluation.
introduction
Residual Energy-Based Models forText Generation
1Introduction
2004.11714
2Related WorkEnergy-based models have a long history in machine learning.
The key challenge of training is mining for good negatives. This can be accomplished explicitly by fantasizing inputs where
the energy should be increased or implicitly via global constraints such as sparsity.
Methods attempting at maximizing the likelihood of the data require to sample from the distribution induced by the model.
Unfortunately, gradient-based MCMC approaches like Hybrid Monte Carloand Langevyn dynamicsare not applicable when the input is discrete like in text applications. Other approaches like Gibbs samplingwere applied to binary inputs but do not scale well to large dictionaries once the energy function is a large
bidirectional transformer model like the one used in this work. Several variants of auto-encoders have also been investigated
for representing and generating text, but they have not shown significant improvements in terms of perplexity
and they have so far been applied to relatively small datasets only.Our approach appears similar to discriminative reranking approaches used in the parsing and machine translation
community. However, our approach provides a generative model, and parameters/hyper-parameters
are directly tuned to close the gap between the model distribution and the data distribution,
rather than relying on surrogate ranking losses. This approach is also related to other sequence level training
objectives, with the major difference that in those works training aims at improving the baseline model, but generation at test time is still greedy.Energy Networks have been used for sequence modeling. In particular, our residual modeling form and the training algorithm is the same as in, where they used an LSTM as the generator and a CNN-LSTM as the energy function, and showed significant gains compared to LSTM baselines in speech recognition. Our work builds on these prior works and develops new lower and upper bounds for the log-probability under the joint model, which makes it possible to show that the residual EBM approach gets better perplexity. We also develop an importance weighting sampling scheme used at generation time, which is focused on conditional generation as opposed to rescoring in speech recognition. The residual EBM formalism makes it very natural to use BERT for language modeling, and we show that empirically this type of approach can outperform modern state-of-the-art language modeling baselines, both in terms of perplexity, and through human evaluation.Generative Adversarial Networksalso relate to EBMs, except that in EBMs the generator is implicit and negatives
samples are produced by the discriminator itself. In our work, the pretrained locally normalized language model can be seen as
a fixed generator, like in.also share our same goal but their generator is not locally normalized and they
propose to improve the sampling from the generator by using the discriminator for rejection sampling. Similar to our work,propose to use the discriminator to de-bias the pretrained generator using importance sampling.
We adapt this work to the application of text generation. In particular, we adopt the conditional noise contrastive estimation
(NCE)  objectiveto our residual model energy function and then sample from the joint model using importance sampling. We want to note that the same formulation has been proposed in. Whileused
conditional NCE to predict the next word in a sequence, we apply it to produce a whole sequence at once with
the pretrained auto-regressive language model as the noise distribution.
body
Residual Energy-Based Models forText Generation
2Related Work
2004.11714
3Residual Energy-Based ModelsWe study the problem of conditional generation of discrete sequences.
Given a prefixwithwhereis the vocabulary,
we want to model the probabilities of generating a sequence of total length111We assume a fixedfor simplicity of analysis and implementation,
but our method generalizes to varying length generation with an end-of-sequence symbol..
The generative model is:(2) whereis a normalizing factor known as partition function. Computing the partition function is intractable
in our case since it involves a sum overterms which grow exponentially with the sequence length: in our
experiments the size of the vocabulary is 50,096 and the length of the generation is 40 tokens.
We callthe joint model, andthe residual energy function sinceis fixed throughout training. The goal of training is to learn the parameters
of the energy function such that the joint model distribution gets close to the data distribution.
For the sake of reducing clutter in the notation, we will drop the conditioning variables in the following discussion.3.1TrainingWhen the partition function is intractable, Maximum Likelihood Estimation (MLE)  requires samples from the model distribution,
which is usually approximated with Monte Carlo sampling or mean field inferencefor globally normalized models.
Unfortunately, both approaches are too computationally expensive for text applications when using large bidirectional transformer models.
For instance, if we were to employ Gibbs sampling exactly, we would need to perform at every position as many forward passes
as words in the dictionary to compute each conditional distribution.
On large datasets where training locally normalized models on multiple machines already takes days,
having such additional overhead means that the model would learn from much less data for the same amount of time,
and this is seldom a beneficial strategy for learning models that generalize well.
Therefore, we do not use either MCMC nor mean field methods, as the latter would introduce additional variational parameters
or an inference network which anyway yields an approximation to MLE learning.Instead, we train our residual energy function using Noise Contrastive Estimation (NCE) , and
more specifically its conditional version. NCE requires two distributions: The model distribution and a noise
distribution. In our case, the model distribution is the joint model of Eq.2,, while the noise distribution
is the pretrained language model,. NCE then trains a binary classifier on the difference of log-probability scores
of these two models. Since our joint model is the product of the energy function (whose parameters we want to learn)  with, the difference reduces to:. Therefore, under these modeling
assumptions of residual learning and noise model, the objective function becomes:(3) whereis a positive sequence taken from the human generated training set, andis a negative sequence drawn
from(for a given ground truth prefix) . In other words, training the energy function reduces to training a binary classifier to discriminate between real text and text
generated by an auto-regressive language model. The aim of training is to assign as negative energy as possible to real data,
and as positive energy as possible to machine generated data. Interestingly, the role of positive and negative samples is totally
symmetric in this loss function, §5will discuss the consequences of this.With the theoretical guarantee of NCE, we can show that the optimum of the above objective is reached at data
distribution with infinite amount of data and model with enough capacity, which is also proved in222FromAssumption 2, for conditional NCE the model needs to be flexible enough such that the self-normalizing property can be satisfied conditioned on any prefix..Theorem 1.Ifhas the same support as, then the objective function in Eq.3reaches its maximum at, if there exists such.Proof.This theorem directly follows from the proof in. Note that at optimum,is self-normalizing: instead of, we have. However, we still need to estimate the partition function throughout the rest of this paper, since we cannot guarantee that this optimum can be reached.
∎3.2EvaluationA commonly used protocol for evaluating generative sequence models, especially language models, is perplexity (PPL) , which is
equal to. PPL can be interpreted as the average number of
tokens the model is uncertain of at every time step.
Since the log-likelihood required by PPL relies on estimating the partition function, we derive two estimators for the log-partition functionbased on the work of.Theorem 2.Denoteas the empirical estimate ofwithsamples:, then,such thatwe have(4) The proof is given in AppendixA.2.We can use the above two estimators to estimate the lower and upper bounds of the partition function, but we want to emphasize that they are true only asymptotically (whenis sufficiently large) .
We also want to note that to get lower variance estimates we use leave-one-out strategy to estimate.
Seefor implementation details and methods to improve numeric stability.Similarly to locally normalized models, we can also factorize the probabilities of an entire sequence step by step, as, and evaluate the PPL for each generation step. By marginalizing over the future, we can derive the following
per step probabilities:(5) The step-wise probabilities in Eq.5are an instance of importance sampling.
The basicdistribution is adjusted by the probability assigned to tokenby the energy function (numerator
is clamped atwhile denominator sums over all the possible values of the token at position t) , with the additional
marginalization over all subsequent tokens up to the horizon. Since the summation involves exponentially many terms, unless, this is approximated by samples drawn by. Since both the numerator and the denominator take the same form as the partition function, we also use Eq.4to estimate the upper and lower bounds. E.g., the lower bound ofcan be obtained by using the lower bound of the numerator and the upper bound of the denominator.For, we can calculate the log probability by exhaustive enumeration. This gives us an idea of the true performance
of our model at the last step, and it also provides a sanity-check of the tightness of our estimators.1Input:number of samplesdrawn from, value ofin top-k//Get a set of samples fromsamplesamplesfromwith top-k sampling2calculate energiesfor each//Resample from the set of LM samples3samplewith probabilityreturnAlgorithm 1Top-k Joint Sampling3.3GenerationGenerating from the joint model is a non-trivial task. A naive way is to generate from the joint model auto-regressively, by marginalizing the future as in Eq.5, which we termTop-k auto-regressive sampling.
However, doing so is computationally expensive and impractical, and we only use this method for a qualitative analysis of the joint model in AppendixA.1.In order to generate efficiently, we use self-normalizing importance sampling.
Under the assumptions
that the model from which we wish to draw samples is the joint model, which is the product of the auto-regressive
model and the energy function, and that the proposal distribution is the auto-regressive model itself,
sampling proceeds simply by: a)  sampling from the auto-regressive language model, followed by b)  resampling according
to the energy function.
The algorithm is shown in Algorithm1, where we introduce an optional top-k constraint on the pretrained language
model to improve the quality of samples in the set333Adapting to other types of local constraints
such as nucleus samplingis straightforward..
Without the top-k constraint, as the number of samples goes to infinity, we would recover exact samples
from the joint model distribution.
body
Residual Energy-Based Models forText Generation
3Residual Energy-Based Models
2004.11714
4ExperimentsIn this section, we describe the experimental set up and the results we obtained by using the residual EBM for text generation,
both in terms of perplexity and generation quality.4.1Experimental SetupDatasetsWe consider two datasets: the Toronto Book Corpusand CC-News.
The former dataset consists of fiction books in 16 different genres, totaling about half a billion words.
The latter is a de-duplicated subset of the English portion of the CommonCrawl news dataset,
which totals around 16 Billion words. The book corpus is more challenging because the range of style and topics is
more diverse than CC-News. Also, the book corpus is 30 times smaller than CC-News and may pose generalization challenges because
of its smaller size.In all our experiments we use a prefix of size 120 tokens and we generate the following 40 tokens; with the notation
of Eq.2,and. For training the joint models, for efficiency we generated 16/128 samples per prefix for CC-News/Book Corpus offline, and sample uniformly from those samples at training time.BaselinesWe consider as base language model (Base LM)  used to generate negatives for the residual EBM, a transformer language
model with 12 layers,,,(we refer tofor notations) .
This is also our first baseline model.Thejointmodel has as many parameters as the sum of the number of parameters in the base LM
and the number of parameters in the energy network.
To make a fair comparison, we consider two additional baselines that have the same number of parameters
as our joint model.The first baseline is a Residual Auto-regressive Language Model baseline (RALM) :(6) wheretakes the form of another auto-regressive language model. The parameters ofare trained by
exact maximum likelihood training of.The second baseline is an auto-regressive language model of the same size of our joint model (sum of the base LM
and energy function parameters) , we dub this model Big
Auto-regressive Language Model (BALM) .BALMhas 12 layers,,,, and is trained by standard token level cross-entropy loss.Residual EBM ArchitectureWe consider two architectures for our residual EBM, both of them are based on
transformers. The first version uses causal self-attention
and is derived from the base LM, a unidirectional transformer (UniT) . It is of the same architecture asBase LM, except that in the final layer we project the mean-pooled hidden states to a scalar energy value. We initialize its parameters with a language model trained on the same dataset.The second version is instead bi-directional (BiT) , and the energy function is computed by projecting the mean-pooled top hidden states
down to a single scalar value. We consider three variants, aBiT-Basefollowing the architecture of RoBERTa-Base, and afollowing RoBERTa-Large, and aBiT-Medwith the same number of parameters asUniT(such thatJoint BiT-Medhas roughly the same number of parameters asBALM) 444We use models from the HuggingFace repository athttps://github.com/huggingface/transformers. We initialize the parameters with a trained BERT, and we useto mark usage of external data, otherwise it means that BERT was trained on our training set.
Notice how our model can be interpreted as a natural way to finetune large bidirectional pretrained models for the
text generation task.While we expectBiTto yield better results because it can fully leverage context also for intermediate tokens,
we also considerUniTto compare to the RALM baseline, which uses the same architecture and only differs in the way
parameters are trained and in the presence of local normalization.We train our models on 8 DGX nodes, each with 8 Nvidia V100s. To improve training speed, we use mixed precision training555https://github.com/NVIDIA/apex. We use the Adam optimizer, with cosine learning rate decay and learning rate warmup. To stabilize training we used gradient norm clipping. Detailed hyper-parameter settings can be found in AppendixA.3.For generation, we use top-k sampling withfor all human evaluations. We take 10,000 samples fromBase LMfor our joint sampling.4.2ResultsAutomatic EvaluationModel (#parameters) CC-NewsToronto Book CorpusValTestValTestbase LM(203M) 18.4117.5716.1618.29RALM(LM+203M) 17.0116.1715.7117.85BALM(408M) joint UniT(LM+203M) 16.42-16.4415.57-15.5815.12-15.1316.98-17.00joint BiT-Base(LM+125M) ----joint BiT-Base*(LM+125M) ----joint BiT-Large*(LM+355M) ----Base LM-24L(203M) RALM-24L(LM-24L+203M) BALM-24L(408M) joint UniT(LM-24L+203M) ---joint BiT-Base(LM-24L+125M) ----joint BiT-Base*(LM-24L+125M) ----joint BiT-Med(LM-24L+203M) ----joint BiT-Large*(LM-24L+355M) ----Table 1:Validation and test perplexity on CC-News and Toronto Book Corpus. * denotes models initialized with RoBERTa
trained on additional data. The joint model perplexity ranges areestimatedusing 100,000 samples, see
Eq.4. The number of parameters of each model is shown in parentheses.Figure 1:Perplexity gain ofJoint BiT-Medand(usingBase LM-24L)  at each position relative toBase LM-24Lon the test set of CC-News. At each position the lower and upper bounds (Eq.5estimated using the method in Eq.4, see §3.2for more details)  are estimated
using 20,000 samples. The shorter the horizon (moving to the right) , the tighter the estimation is but also the more
limited the gains compared to base LM as un-normalized models are most useful on longer generations.Model1 (baseline) Model2 (compared model) Ratep-valuebase LMjoint uniT52.85%0.16base LMjoint BiT-Base56.25%0.015base LMjoint BiT-Large*58.93%0.00084base LMBALM46.77%0.88BALMjoint UniT50.00%0.52BALMjoint BiT-Base57.89%0.0027BALMjoint BiT-Large*59.89%0.00020BALM-24Ljoint BiT-Med(24L) 56.23%0.015joint BiT-Large*(24L) Human55.21%0.036base LMBALM54.85%0.050Table 2:Human evaluation results on a subset of 333 sentences on the CC-News test set. The rate is computed as the percentage of sentences where the number of turkers preferring Model1 is strictly less than (denoted with)  or not greater than (denoted with)  those preferring Model2. Attention check is used to drop some votes, so there might exist ties. p-value is based on single-sided binomial test.Our main result is reported in Table1where we compare models in terms of their perplexity.
We can see that on both datasets, residual EBMs with causal attentionjoint UniToutperforms
the baselineRALMwith approximately the same number of parameters. The non-residual baselineBALMperforms similarly tojoint UniT, which might be due to the limitation thatis not trained jointly with the residual model in bothjoint UniTandRALM. However, by using our EBM approach, we can remove the causal attention mask and use bi-directional models, which achieves better performance than baselines andjoint UniT:
without external data,joint BiT-Basereaches a higher performance thanjoint UniTwith fewer parameters. By initializing from the state-of-the-art pretrained bi-directional transformers
RoBERTa-Base and RoBERTa-Large,joint BiT-Base*andJoint BiT-Large*reach even
better performance thanjoint BiT-Base.In the lower part of the table, we show that if we make the big language model baselineBALMdeeper (BALM-24L)  (24 layers instead of 12, for the same number of parameters)  we attain lower perplexity.
However, training the joint modelJoint BiT-Baseon the residual of a deeper language modelBASE LM-24Lyields even lower perplexity, despite having fewer parameters. By using the same number of parameters asBALM-24L,Joint Bit-Medfurther decreases perplexity. Finally, by initializing from RoBERTa-Large,joint BiT-Base*obtains the best results.One caveat of our evaluation protocol is that the perplexity bounds are only estimates,
which might not reflect the true value, particularly since the number of possible sequences grows exponentially
with the number of words that are generated.
We therefore break down perplexity per position in the generated sequences as in Eq.5,
and compare the estimated PPLs to the true enumerated
PPLs at the last position, as shown in Figure1. We find that at the final generation step, the estimated
bounds agree remarkably well with the exact values, proving that our method at least gets a reasonable PPL estimate at the last generation step, and thatJoint BiT-Medoutperforms baselines at the last generation step for sure.Human EvaluationBetter perplexity results do not necessarily imply better generations.
Besides, since generation from the residual EBM
requires approximations as in Algorithm1, the limited sample size might induce approximation errors
compared to truly sampling from the joint distribution. Therefore, we conducted human evaluations to compare generations
from the residual EBM model to generations from the baseline language models.For each prefix, we present one completion from each model, and ask humans to select the one that is a better continuation.
More details about human evaluation can be found in the AppendixA.4.
The preference rates reported in Table2confirm that indeed the generation quality ofJoint Bit-Baseandis better than both language model baselines.
Depending on the model variant, our joint model (with bidirectional EBM)  is preferred between 56% and almost 60% of the times;
interestingly, the preference rate does not change much as we compare against base LM as opposed to BALM.
In fact, humans do not seem to have a strong preference for BALM over base LM, despite the former scores two perplexity points lower.
Similarly,Joint Unitis not strongly preferred overBase LMdespite its lower perplexity score.
We surmise that unidirectional scoring functions and auto-regressive models exhibit generation artifacts which are easily
detected by humans, and these may overshadow the improvements brought by perplexity gains.4.3AnalysesFigure 2:Left: PPL estimation for jointBiT-Baseon CC-News validation set as we vary
the number of samples. Right: Percentage of Unique n-grams found in real data,
samples from the joint modelBiT-Baseand samples from the base language model.
The joint sampling is done with 10,000 samples.In this section, we analyze some of the results we obtained. First, we check whether we used a sufficient number of
samples in our perplexity estimates. Second, we assess whether the joint model produces fewer repetitions compared
to the base language model, and finally we check how well some statistics of the model and data distributions match.Number of samples.In Figure2, we vary the number of samples we take in order to
estimate PPL upper and lower bounds. Beyond 20,000 samples the upper estimate becomes very stable,
although we have to emphasize that these estimates might be biased even though the gap between lower and upper bound
closes as we take more samples.Repetitions.A typical artifact of auto-regressive language models is their tendency to repeat phrases.
It is then interesting to check whether the joint model is able to alleviate this artifact. Fig.2shows that indeed the joint model has a slightly higher percentage of unique n-grams compared to the baseline language model with,
although still not as high as the original human generated text.A necessary condition for the model to match the data distribution.If the joint modelmatches the data distribution, then statistics computed on a large population of samples
from the two distributions should also match. In particular, Fig.3show the density plots of log-likelihood scores
of the baseline language model (left)  and joint model (right)  when fed with their own samples versus samples from the test set.
We observe that the histogram of samples from the joint model matches the real data distribution more closely: The difference of means in theLM Basecase is 21.64 whereas the difference is 6.20 in the joint approach.Figure 3:Density plot of log-probability scores using the base
language model (left)  or the joint model (right) . The red curve corresponds to real samples, the black curve to samples fromBase LMand the green curve to samples fromBiT-Base. The joint model provides a much better fit than the base language model.
body
Residual Energy-Based Models forText Generation
4Experiments
2004.11714
5LimitationsIn the previous sections we highlighted the strengths of residual EBMs, namely their simplicity, efficiency both at training
and test time, and their improved perplexity scores against strong auto-regressive language model baselines. In this section, we comment on their limitations to caution the reader about when these methods are more likely to succeed
and to inform other researchers about what future avenues of research may naturally derive from this work.In order to make training efficient and side step costly negative mining using the energy function itself, the current approach
uses negatives generated from a pretrained auto-regressive language model. Therefore, our model works as long as the base
language model from which we draw samples is strong enough, and as long as the ground truth and other plausible sequences arereachableby the baseline language model.If the base language model has poor quality, then generation from our
joint model is going to be poor as well, as the joint model merely resamples generations from the original language model.
Moreover, training is going to be trivial if the base language model is poor, because the residual energy function
merely needs to detect trivial generation artifacts from the base language model. In fact, observe that the role of
positive and negative samples is symmetric in the loss of Eq.3. This means that the energy function can choose
to minimize the loss by either modeling the true data or the negative samples; since the latter have much simpler structure,
it is going to model the negative samples. Therefore, importance sampling amounts to mostly down-weighing the worst samples
from the base language model.
The consequence of this is that search with a poor base language model is going to be catastrophically inefficient, as
we would need to sample an impractically large number of negatives in order to find samples that are reasonably close
to the true data manifold.To summarize, this work makes a rather strong implicit assumption on the quality of the base language model, and it
is expected to work well only when this is rather strong. In our application, this assumption is met quite well in practice
as large auto-regressive language models trained on large datasets have improved significantly in recent years.
In general however, residual learning always carries liability to its
base model.
body
Residual Energy-Based Models forText Generation
5Limitations
2004.11714
6Conclusions and Future workWe investigated an EBM trained on the residual of a pretrained autoregressive language model.
The resulting joint model scores sequences holistically, thanks to the energy function. Training is very efficient
and consists of a binary classification task between positives from the training set and pregenerated negatives from
the fixed language model. Generation is also very efficient as it amounts to resampling from the large set of negatives
produced by the base language model. Our estimates show that the resulting model has lower perplexity than the base
language model.
Finally, this approach may be interpreted
as a natural way to finetune a large bidirectional transformer like BERT for text generation applications.In the future, we plan to investigate other ways to generate negatives that may strike a better trade-off
between the amount of compute each negative requires and their closeness to the joint model distribution.
It would also be interesting to explore other loss functions and the generation of longer pieces of text by using this
model auto-regressively at the chunk level, as opposed to the token level.
conclusion
Residual Energy-Based Models forText Generation
6Conclusions and Future work