<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2105.11921] Focus Attention: Promoting Faithfulness and Diversity in Summarization</title><meta property="og:description" content="Professional summaries are written with document-level information, such as the theme of the document, in mind. This is in contrast with most seq2seq decoders which simultaneously learn to focus on salient content, whi…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Focus Attention: Promoting Faithfulness and Diversity in Summarization">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Focus Attention: Promoting Faithfulness and Diversity in Summarization">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2105.11921">

<!--Generated on Thu Mar  7 00:22:55 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Focus Attention: Promoting Faithfulness and Diversity in Summarization</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
Rahul Aralikatte 
<br class="ltx_break">University of Copenhagen 
<br class="ltx_break"><span id="id3.2.id1" class="ltx_text ltx_font_typewriter">rahul@di.ku.dk</span> &amp;Shashi Narayan 
<br class="ltx_break">Google Research 
<br class="ltx_break"><span id="id4.3.id2" class="ltx_text ltx_font_typewriter">shashinarayan@google.com</span> <span id="id5.4.id3" class="ltx_ERROR undefined">\AND</span>Joshua Maynez 
<br class="ltx_break">Google Research 
<br class="ltx_break"><span id="id6.5.id4" class="ltx_text ltx_font_typewriter">joshuahm@google.com</span> &amp;Sascha Rothe 
<br class="ltx_break">Google Research 
<br class="ltx_break"><span id="id7.6.id5" class="ltx_text ltx_font_typewriter">rothe@google.com</span> &amp;Ryan McDonald<sup id="id8.7.id6" class="ltx_sup"><span id="id8.7.id6.1" class="ltx_text ltx_font_italic">∗</span></sup> 
<br class="ltx_break">ASAPP 
<br class="ltx_break"><span id="id9.8.id7" class="ltx_text ltx_font_typewriter">ryanmcd@asapp.com</span>
</span><span class="ltx_author_notes"> Work done when authors were interning/working at Google.</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id2.1" class="ltx_p">Professional summaries are written with document-level information, such as the theme of the document, in mind. This is in contrast with most seq2seq decoders which simultaneously learn to focus on salient content, while deciding what to generate, at each decoding step. With the motivation to narrow this gap, we introduce Focus Attention Mechanism, a simple yet effective method to encourage decoders to proactively generate tokens that are similar or topical to the input document. Further, we propose a Focus Sampling method to enable generation of diverse summaries, an area currently understudied in summarization. When evaluated on the BBC extreme summarization task, two state-of-the-art models augmented with Focus Attention generate summaries that are closer to the target and more faithful to their input documents,
outperforming their vanilla counterparts on <span id="id2.1.1" class="ltx_text ltx_font_smallcaps">rouge</span> and multiple faithfulness measures.
We also empirically demonstrate that Focus Sampling is more effective in generating diverse and faithful summaries than top-<math id="id2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="id2.1.m1.1a"><mi id="id2.1.m1.1.1" xref="id2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="id2.1.m1.1b"><ci id="id2.1.m1.1.1.cmml" xref="id2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="id2.1.m1.1c">k</annotation></semantics></math> or nucleus sampling-based decoding methods.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Document summarization — producing the shorter version of a document while preserving salient information <cite class="ltx_cite ltx_citemacro_cite">Mani (<a href="#bib.bib50" title="" class="ltx_ref">2001</a>); Nenkova and McKeown (<a href="#bib.bib56" title="" class="ltx_ref">2011</a>)</cite> — is challenging even for humans. Today, systems can generate summaries with a high level of fluency and coherence. This is due to recent advances such as sequence-to-sequence architectures (seq2seq) with attention and copy mechanism <cite class="ltx_cite ltx_citemacro_cite">Hochreiter and Schmidhuber (<a href="#bib.bib33" title="" class="ltx_ref">1997</a>); Bahdanau et al. (<a href="#bib.bib3" title="" class="ltx_ref">2015</a>); Gu et al. (<a href="#bib.bib30" title="" class="ltx_ref">2016</a>)</cite>, fully attention-based Transformer architectures <cite class="ltx_cite ltx_citemacro_cite">Vaswani et al. (<a href="#bib.bib71" title="" class="ltx_ref">2017</a>)</cite>, and large pretrained language models
<cite class="ltx_cite ltx_citemacro_cite">Devlin et al. (<a href="#bib.bib13" title="" class="ltx_ref">2019</a>); Radford et al. (<a href="#bib.bib59" title="" class="ltx_ref">2018</a>); Yang et al. (<a href="#bib.bib84" title="" class="ltx_ref">2019</a>); Liu et al. (<a href="#bib.bib47" title="" class="ltx_ref">2019</a>); Dong et al. (<a href="#bib.bib17" title="" class="ltx_ref">2019a</a>); Song et al. (<a href="#bib.bib68" title="" class="ltx_ref">2019</a>); Lewis et al. (<a href="#bib.bib42" title="" class="ltx_ref">2019</a>); Rothe et al. (<a href="#bib.bib63" title="" class="ltx_ref">2020</a>); Raffel et al. (<a href="#bib.bib60" title="" class="ltx_ref">2019</a>); Zhang et al. (<a href="#bib.bib85" title="" class="ltx_ref">2019</a>)</cite>.</p>
</div>
<figure id="S1.F1" class="ltx_figure">
<table id="S1.F1.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S1.F1.1.2.1" class="ltx_tr">
<td id="S1.F1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt" style="padding-left:2.8pt;padding-right:2.8pt;" rowspan="3">
<span id="S1.F1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.2.1.1.1.1" class="ltx_p" style="width:2.8pt;"><span id="S1.F1.1.2.1.1.1.1.1" class="ltx_text">A</span></span>
</span>
</td>
<td id="S1.F1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.2.1.2.1.1" class="ltx_p" style="width:199.2pt;"><span id="S1.F1.1.2.1.2.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Gold</span>: Australia has expelled an Israeli diplomat saying Israel was behind the forging of Australian passports linked to the murder of a Hamas operative in Dubai.</span>
</span>
</td>
</tr>
<tr id="S1.F1.1.3.2" class="ltx_tr">
<td id="S1.F1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.3.2.1.1.1" class="ltx_p" style="width:199.2pt;"><span id="S1.F1.1.3.2.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Pegasus</span>: Australia has expelled an Israeli diplomat after concluding that forged Australian passports used in the killing of a Hamas militant in Dubai were issued by Israel.</span>
</span>
</td>
</tr>
<tr id="S1.F1.1.4.3" class="ltx_tr">
<td id="S1.F1.1.4.3.1" class="ltx_td ltx_align_justify ltx_align_top" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.4.3.1.1.1" class="ltx_p" style="width:199.2pt;"><span id="S1.F1.1.4.3.1.1.1.1" class="ltx_text ltx_font_bold">Our <span id="S1.F1.1.4.3.1.1.1.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> model</span>: The Australian government has expelled an Israeli diplomat over the use of forged Australian passports in the killing of a Hamas militant in Dubai.</span>
</span>
</td>
</tr>
<tr id="S1.F1.1.1" class="ltx_tr">
<td id="S1.F1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;" rowspan="6">
<span id="S1.F1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.1.2.1.1" class="ltx_p" style="width:2.8pt;"><span id="S1.F1.1.1.2.1.1.1" class="ltx_text">B</span></span>
</span>
</td>
<td id="S1.F1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.1.1.1.1" class="ltx_p" style="width:199.2pt;"><span id="S1.F1.1.1.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Pegasus<span id="S1.F1.1.1.1.1.1.1.1" class="ltx_text ltx_font_upright"> with Top-<math id="S1.F1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S1.F1.1.1.1.1.1.1.1.m1.1a"><mi id="S1.F1.1.1.1.1.1.1.1.m1.1.1" xref="S1.F1.1.1.1.1.1.1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S1.F1.1.1.1.1.1.1.1.m1.1b"><ci id="S1.F1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S1.F1.1.1.1.1.1.1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.1.1.1.1.1.1.1.m1.1c">k</annotation></semantics></math> Sampling</span></span></span>
</span>
</td>
</tr>
<tr id="S1.F1.1.5.4" class="ltx_tr">
<td id="S1.F1.1.5.4.1" class="ltx_td ltx_align_justify ltx_align_top" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.5.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.5.4.1.1.1" class="ltx_p" style="width:199.2pt;"><span id="S1.F1.1.5.4.1.1.1.1" class="ltx_text" style="color:#E24C00;">Israel has summoned the Australian ambassador to complain</span> after the Australian government said forged passports used in the killing of a Hamas operative in Dubai belonged to Netanyahu’s foreign ministry.</span>
</span>
</td>
</tr>
<tr id="S1.F1.1.6.5" class="ltx_tr">
<td id="S1.F1.1.6.5.1" class="ltx_td ltx_align_justify ltx_align_top" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.6.5.1.1.1" class="ltx_p" style="width:199.2pt;">The Australian government has ordered Israel to withdraw an officer over the use of forged Australian passports used <span id="S1.F1.1.6.5.1.1.1.1" class="ltx_text" style="color:#E24C00;">by the 2013 murder of a Lebanese opposition figure in Dubai.</span></span>
</span>
</td>
</tr>
<tr id="S1.F1.1.7.6" class="ltx_tr">
<td id="S1.F1.1.7.6.1" class="ltx_td ltx_align_justify ltx_align_top" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.7.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.7.6.1.1.1" class="ltx_p" style="width:199.2pt;"><span id="S1.F1.1.7.6.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Pegasus<span id="S1.F1.1.7.6.1.1.1.1.1" class="ltx_text ltx_font_upright"> with Nucleus Sampling</span></span></span>
</span>
</td>
</tr>
<tr id="S1.F1.1.8.7" class="ltx_tr">
<td id="S1.F1.1.8.7.1" class="ltx_td ltx_align_justify ltx_align_top" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.8.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.8.7.1.1.1" class="ltx_p" style="width:199.2pt;"><span id="S1.F1.1.8.7.1.1.1.1" class="ltx_text" style="color:#E24C00;">Israel hasracuse withdrawn an envoy after</span> the Australian government said it concluded that Israeli agents used forged passports used to kill a Dubai <span id="S1.F1.1.8.7.1.1.1.2" class="ltx_text" style="color:#E24C00;">Bendigo businessman.</span></span>
</span>
</td>
</tr>
<tr id="S1.F1.1.9.8" class="ltx_tr">
<td id="S1.F1.1.9.8.1" class="ltx_td ltx_align_justify ltx_align_top" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.9.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.9.8.1.1.1" class="ltx_p" style="width:199.2pt;">The Australian government has recalled an Israeli diplomat over accusation that fake Australian passports <span id="S1.F1.1.9.8.1.1.1.1" class="ltx_text" style="color:#E24C00;">used 436 kilometres (300 miles) from Canberra</span> in the death of a Hamas militant <span id="S1.F1.1.9.8.1.1.1.2" class="ltx_text" style="color:#E24C00;">were stolen by Israeli agents.</span></span>
</span>
</td>
</tr>
<tr id="S1.F1.1.10.9" class="ltx_tr">
<td id="S1.F1.1.10.9.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;" rowspan="7">
<span id="S1.F1.1.10.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.10.9.1.1.1" class="ltx_p" style="width:2.8pt;"><span id="S1.F1.1.10.9.1.1.1.1" class="ltx_text">C</span></span>
</span>
</td>
<td id="S1.F1.1.10.9.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.10.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.10.9.2.1.1" class="ltx_p" style="width:199.2pt;"><span id="S1.F1.1.10.9.2.1.1.1" class="ltx_text ltx_font_bold">Our <span id="S1.F1.1.10.9.2.1.1.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> model with novel Focus Sampling</span></span>
</span>
</td>
</tr>
<tr id="S1.F1.1.11.10" class="ltx_tr">
<td id="S1.F1.1.11.10.1" class="ltx_td ltx_align_justify ltx_align_top" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.11.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.11.10.1.1.1" class="ltx_p" style="width:199.2pt;">Australia has expelled an Israeli diplomatic staff after accusing the country’s security agency, the Israeli military’s intelligence agency, of being responsible for the use of Australian visas used in the killing of a Palestinian.</span>
</span>
</td>
</tr>
<tr id="S1.F1.1.12.11" class="ltx_tr">
<td id="S1.F1.1.12.11.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S1.F1.1.12.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="S1.F1.1.12.11.1.1.1" class="ltx_p" style="width:199.2pt;">The Australian government has expelled an Israeli diplomatic staff after it said the country was responsible for the use of Australian visas used in the killing of a Palestinian in the Middle East.</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Block A shows the best predictions from <span id="S1.F1.13.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> and our <span id="S1.F1.14.2" class="ltx_text ltx_font_smallcaps">PegFame</span> (<span id="S1.F1.15.3" class="ltx_text ltx_font_smallcaps">Pegasus</span> with <span id="S1.F1.16.4" class="ltx_text ltx_font_smallcaps">Fame</span>) model, along with the <span id="S1.F1.17.5" class="ltx_text ltx_font_smallcaps">Gold</span> summary for an <span id="S1.F1.18.6" class="ltx_text ltx_font_smallcaps">XSum</span> article. Block B presents diverse summaries generated from <span id="S1.F1.19.7" class="ltx_text ltx_font_smallcaps">Pegasus</span> using top-<math id="S1.F1.3.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S1.F1.3.m1.1b"><mi id="S1.F1.3.m1.1.1" xref="S1.F1.3.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S1.F1.3.m1.1c"><ci id="S1.F1.3.m1.1.1.cmml" xref="S1.F1.3.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.3.m1.1d">k</annotation></semantics></math> and nucleus sampling. Block C shows diverse summaries generated using our <span id="S1.F1.20.8" class="ltx_text ltx_font_smallcaps">PegFame</span> model with Focus sampling. The text in <span id="S1.F1.21.9" class="ltx_text" style="color:#E24C00;">orange</span> is not supported by the input article.
</figcaption>
</figure>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">However, in terms of summary quality, many challenges remain. For example, generating summaries that are faithful to the input is an unsolved problem <cite class="ltx_cite ltx_citemacro_cite">Kryscinski et al. (<a href="#bib.bib38" title="" class="ltx_ref">2020</a>); Maynez et al. (<a href="#bib.bib52" title="" class="ltx_ref">2020</a>); Gabriel et al. (<a href="#bib.bib25" title="" class="ltx_ref">2020</a>)</cite>.
Furthermore, there can be multiple equally good summaries per source document. Neural generation models fail to account for this and
tend to generate outputs with low diversity due to standard likelihood training, approximate decoding objectives, and lack of high quality multi-reference datasets <cite class="ltx_cite ltx_citemacro_cite">Fan et al. (<a href="#bib.bib22" title="" class="ltx_ref">2018</a>); Kulikov et al. (<a href="#bib.bib41" title="" class="ltx_ref">2019</a>); Freitag et al. (<a href="#bib.bib24" title="" class="ltx_ref">2020</a>); Choi et al. (<a href="#bib.bib11" title="" class="ltx_ref">2020</a>)</cite>. Not much attention has been given to generation of diverse, yet faithful summaries – two goals are often challenging to achieve simultaneously <cite class="ltx_cite ltx_citemacro_cite">Hashimoto et al. (<a href="#bib.bib31" title="" class="ltx_ref">2019</a>)</cite>; a model can produce diverse outputs through sampling <cite class="ltx_cite ltx_citemacro_cite">Fan et al. (<a href="#bib.bib22" title="" class="ltx_ref">2018</a>); Holtzman et al. (<a href="#bib.bib34" title="" class="ltx_ref">2020</a>)</cite>, but at the cost of quality.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this paper we introduce a Focus Attention MEchanism (or <span id="S1.p3.1.1" class="ltx_text ltx_font_smallcaps">Fame</span>) to transformer-based seq2seq architectures. <span id="S1.p3.1.2" class="ltx_text ltx_font_smallcaps">Fame</span> is inspired by how humans write summaries. Specifically, <span id="S1.p3.1.3" class="ltx_text ltx_font_smallcaps">Fame</span> aims to perform source-side planning to focus the summary on supported and topical content.
<span id="S1.p3.1.4" class="ltx_text ltx_font_smallcaps">Fame</span> achieves this through a novel technique which augments standard contextual representations with a dynamic source-conditioned vocabulary biasing layer.
We present the following experimental findings:</p>
</div>
<section id="S1.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S1.SS0.SSS0.Px1.1.1" class="ltx_text ltx_font_smallcaps">Fame</span> promotes summaries faithful to the source</h4>

<div id="S1.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S1.SS0.SSS0.Px1.p1.1" class="ltx_p">When evaluated on the BBC extreme summarization task (<span id="S1.SS0.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_smallcaps">XSum</span>; <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib54" title="" class="ltx_ref">Narayan et al.</a></cite>, <cite class="ltx_cite ltx_citemacro_citeyear"><a href="#bib.bib54" title="" class="ltx_ref">2018</a></cite>), experiments with two state-of-the-art summarizers – <span id="S1.SS0.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> <cite class="ltx_cite ltx_citemacro_cite">Rothe et al. (<a href="#bib.bib63" title="" class="ltx_ref">2020</a>)</cite> and <span id="S1.SS0.SSS0.Px1.p1.1.3" class="ltx_text ltx_font_smallcaps">Pegasus</span> <cite class="ltx_cite ltx_citemacro_cite">Zhang et al. (<a href="#bib.bib85" title="" class="ltx_ref">2019</a>)</cite> –
show that both models generate summaries that are more faithful to their input documents when augmented with <span id="S1.SS0.SSS0.Px1.p1.1.4" class="ltx_text ltx_font_smallcaps">Fame</span>, in comparison with their vanilla counterparts.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>In the paper we focus on assessing <span id="footnote1.1" class="ltx_text ltx_font_smallcaps">Fame</span> on <span id="footnote1.2" class="ltx_text ltx_font_smallcaps">XSum</span>. But other summarization and text editing results can be found in Appendix <a href="#A2" title="Appendix B Abstractive Summarization Results on CNN/DailyMail ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> and <a href="#A3" title="Appendix C Text Editing Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</span></span></span>
Faithfulness is measured through a variety of previously proposed metrics. In addition, we leverage the manually annotated document-summary pairs for faithfulness from <cite class="ltx_cite ltx_citemacro_citet">Maynez et al. (<a href="#bib.bib52" title="" class="ltx_ref">2020</a>)</cite> and train a scorer which serves as an efficient proxy for expensive human evaluations. We call this metric <span id="S1.SS0.SSS0.Px1.p1.1.5" class="ltx_text ltx_font_italic">BERTFaithful</span>.</p>
</div>
</section>
<section id="S1.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S1.SS0.SSS0.Px2.1.1" class="ltx_text ltx_font_smallcaps">Fame</span> enables diverse summaries</h4>

<div id="S1.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S1.SS0.SSS0.Px2.p1.1" class="ltx_p"><span id="S1.SS0.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_smallcaps">Fame</span>, by design, supports <span id="S1.SS0.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_italic">Focus Sampling</span> – a technique that is more effective in sampling topically relevant tokens to generate diverse, yet topically consistent and faithful outputs, than other sampling methods <cite class="ltx_cite ltx_citemacro_cite">Fan et al. (<a href="#bib.bib22" title="" class="ltx_ref">2018</a>); Holtzman et al. (<a href="#bib.bib34" title="" class="ltx_ref">2020</a>)</cite>. Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> illustrates how focus sampling generates better summaries than other sampling methods. We demonstrate the effectiveness of our new Focus Sampling technique using a variety of existing diversity and faithfulness measures. Empirically, we find that optimizing for high diversity often comes at the cost of faithfulness. Thus <span id="S1.SS0.SSS0.Px2.p1.1.3" class="ltx_text ltx_font_smallcaps">Fame</span> provides a mechanism for trading-off high faithfulness with better diversity in summarization.</p>
</div>
</section>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Task-Specific Architectural Priors</h4>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p1.1" class="ltx_p">Several works enhance seq2seq architectures with task-specific priors. Pointer-generator style models <cite class="ltx_cite ltx_citemacro_cite">See et al. (<a href="#bib.bib65" title="" class="ltx_ref">2017</a>); Xu et al. (<a href="#bib.bib82" title="" class="ltx_ref">2020</a>)</cite> can accurately generate mostly extractive summaries by copying words from the source text via pointing. Text editing models <cite class="ltx_cite ltx_citemacro_cite">Malmi et al. (<a href="#bib.bib49" title="" class="ltx_ref">2019</a>); Dong et al. (<a href="#bib.bib18" title="" class="ltx_ref">2019b</a>); Mallinson et al. (<a href="#bib.bib48" title="" class="ltx_ref">2020</a>)</cite> cast text generation as a sequence tagging problem with carefully selected edit operations required for the task. Others focus on improving content selection to better constrain the model to likely input phrases <cite class="ltx_cite ltx_citemacro_cite">Gehrmann et al. (<a href="#bib.bib27" title="" class="ltx_ref">2018</a>)</cite> or by improving the representation of relevant input tokens <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a href="#bib.bib88" title="" class="ltx_ref">2017</a>)</cite>. Instead of directly modeling such priors, <span id="S2.SS0.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_smallcaps">Fame</span> learns the theme of the document through dynamic vocabulary biasing. Thus, <span id="S2.SS0.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_smallcaps">Fame</span> can be seen as a generalization of Pointer-generator or text-editing models via soft vocabulary learning. In fact, our <span id="S2.SS0.SSS0.Px1.p1.1.3" class="ltx_text ltx_font_smallcaps">Fame</span> models achieve state-of-the-art on text-editing tasks (Appendix <a href="#A3" title="Appendix C Text Editing Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>).</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Topic-Aware Generation Models</h4>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px2.p1.1" class="ltx_p">The idea of capturing document-level semantic information has been widely explored in the summarization community. <cite class="ltx_cite ltx_citemacro_citet">Barzilay and Elhadad (<a href="#bib.bib4" title="" class="ltx_ref">1997</a>)</cite> use WordNet <cite class="ltx_cite ltx_citemacro_cite">Fellbaum (<a href="#bib.bib23" title="" class="ltx_ref">1998</a>)</cite> to model a text’s content relative to a topic based on lexical chains. <cite class="ltx_cite ltx_citemacro_citet">Lin and Hovy (<a href="#bib.bib45" title="" class="ltx_ref">2000</a>)</cite> propose to learn topic signatures for summarizing documents. Recently, document-level topic information has been used for improving neural language models <cite class="ltx_cite ltx_citemacro_cite">Mikolov and Zweig (<a href="#bib.bib53" title="" class="ltx_ref">2012</a>); Ghosh et al. (<a href="#bib.bib29" title="" class="ltx_ref">2016</a>); Dieng et al. (<a href="#bib.bib15" title="" class="ltx_ref">2017</a>); Karmaker Santu et al. (<a href="#bib.bib35" title="" class="ltx_ref">2019</a>)</cite>, neural response generators <cite class="ltx_cite ltx_citemacro_cite">Xing et al. (<a href="#bib.bib80" title="" class="ltx_ref">2017</a>); Dziri et al. (<a href="#bib.bib20" title="" class="ltx_ref">2019</a>)</cite>, and not surprisingly, neural summarizers <cite class="ltx_cite ltx_citemacro_cite">Narayan et al. (<a href="#bib.bib54" title="" class="ltx_ref">2018</a>); Ailem et al. (<a href="#bib.bib1" title="" class="ltx_ref">2019</a>); Wang et al. (<a href="#bib.bib75" title="" class="ltx_ref">2020c</a>)</cite>. Both, <cite class="ltx_cite ltx_citemacro_citet">Narayan et al. (<a href="#bib.bib54" title="" class="ltx_ref">2018</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Ailem et al. (<a href="#bib.bib1" title="" class="ltx_ref">2019</a>)</cite>, use a pretrained Latent Dirichlet Allocation (LDA; <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib5" title="" class="ltx_ref">Blei et al.</a></cite>, <cite class="ltx_cite ltx_citemacro_citeyear"><a href="#bib.bib5" title="" class="ltx_ref">2003</a></cite>) model, whereas, <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib75" title="" class="ltx_ref">2020c</a>)</cite> use Poisson factor analysis <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a href="#bib.bib87" title="" class="ltx_ref">2012</a>)</cite>, to synthesize topic vectors for the input. Instead, we dynamically learn a target-induced topic distribution for the input under the assumption that the human-written summary is a good proxy for the input document.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Faithful Generation Models</h4>

<div id="S2.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px3.p1.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet">Cao et al. (<a href="#bib.bib9" title="" class="ltx_ref">2017</a>)</cite> force faithful generation by conditioning on both source text and extracted fact descriptions from the source text. <cite class="ltx_cite ltx_citemacro_citet">Song et al. (<a href="#bib.bib67" title="" class="ltx_ref">2020</a>)</cite> propose to jointly generate a sentence and its syntactic dependency parse to induce grammaticality and faithfulness. <cite class="ltx_cite ltx_citemacro_citet">Tian et al. (<a href="#bib.bib70" title="" class="ltx_ref">2019</a>)</cite> learn a confidence score to ensure that the model attends to the source whenever necessary. <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib76" title="" class="ltx_ref">2020d</a>)</cite> introduce new input-output matching and embedding similarity losses to alleviate hallucination issues.
Yet, the task of generating text that is consistent with the input remains an open problem <cite class="ltx_cite ltx_citemacro_cite">Gabriel et al. (<a href="#bib.bib25" title="" class="ltx_ref">2020</a>)</cite>.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Diverse Generation Models</h4>

<div id="S2.SS0.SSS0.Px4.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px4.p1.2" class="ltx_p">There has been a surge of interest in making language models generate more diverse and human-like outputs. <cite class="ltx_cite ltx_citemacro_citet">Vijayakumar et al. (<a href="#bib.bib72" title="" class="ltx_ref">2018</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Kulikov et al. (<a href="#bib.bib41" title="" class="ltx_ref">2019</a>)</cite> diversify beam search, using a task-specific scoring function, or constrain beam hypotheses to be sufficiently different. Others avoid text degeneration by truncating the unreliable tail of the probability distribution at each decoding step, either by sampling from the top-<math id="S2.SS0.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.1.m1.1a"><mi id="S2.SS0.SSS0.Px4.p1.1.m1.1.1" xref="S2.SS0.SSS0.Px4.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.1.m1.1b"><ci id="S2.SS0.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.1.m1.1c">k</annotation></semantics></math> tokens (<span id="S2.SS0.SSS0.Px4.p1.2.1" class="ltx_text ltx_font_italic">Top-<math id="S2.SS0.SSS0.Px4.p1.2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.2.1.m1.1a"><mi id="S2.SS0.SSS0.Px4.p1.2.1.m1.1.1" xref="S2.SS0.SSS0.Px4.p1.2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.2.1.m1.1b"><ci id="S2.SS0.SSS0.Px4.p1.2.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.2.1.m1.1c">k</annotation></semantics></math> Sampling</span>; <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib22" title="" class="ltx_ref">Fan et al.</a></cite>, <cite class="ltx_cite ltx_citemacro_citeyear"><a href="#bib.bib22" title="" class="ltx_ref">2018</a></cite>) or by sampling from a dynamic nucleus of tokens with the bulk of the probability mass (<span id="S2.SS0.SSS0.Px4.p1.2.2" class="ltx_text ltx_font_italic">Nucleus Sampling</span>; <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib34" title="" class="ltx_ref">Holtzman et al.</a></cite>, <cite class="ltx_cite ltx_citemacro_citeyear"><a href="#bib.bib34" title="" class="ltx_ref">2020</a></cite>). Others modify the training objective to make the distribution sparse <cite class="ltx_cite ltx_citemacro_cite">Martins et al. (<a href="#bib.bib51" title="" class="ltx_ref">2020</a>)</cite> or assign lower probability to unlikely generations <cite class="ltx_cite ltx_citemacro_cite">Welleck et al. (<a href="#bib.bib77" title="" class="ltx_ref">2019a</a>)</cite>.</p>
</div>
<div id="S2.SS0.SSS0.Px4.p2" class="ltx_para">
<p id="S2.SS0.SSS0.Px4.p2.1" class="ltx_p">For conditional text generation, most work focuses on generating diverse questions <cite class="ltx_cite ltx_citemacro_cite">Narayan et al. (<a href="#bib.bib55" title="" class="ltx_ref">2016</a>); Dong et al. (<a href="#bib.bib16" title="" class="ltx_ref">2017</a>); Sultan et al. (<a href="#bib.bib69" title="" class="ltx_ref">2020</a>); Wang et al. (<a href="#bib.bib74" title="" class="ltx_ref">2020b</a>)</cite> or paraphrases <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a href="#bib.bib44" title="" class="ltx_ref">2016b</a>); Dai et al. (<a href="#bib.bib12" title="" class="ltx_ref">2017</a>); Xu et al. (<a href="#bib.bib81" title="" class="ltx_ref">2018</a>); Cao and Wan (<a href="#bib.bib8" title="" class="ltx_ref">2020</a>)</cite>. Following <cite class="ltx_cite ltx_citemacro_citet">Gehrmann et al. (<a href="#bib.bib27" title="" class="ltx_ref">2018</a>)</cite>, <cite class="ltx_cite ltx_citemacro_citet">Cho et al. (<a href="#bib.bib10" title="" class="ltx_ref">2019</a>)</cite> use a mixture of experts to sample different binary masks on the source sequence for diverse content selection for summarization.</p>
</div>
<div id="S2.SS0.SSS0.Px4.p3" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px4.p3.1" class="ltx_p">Our focus sampling is similar to top-<math id="S2.SS0.SSS0.Px4.p3.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS0.SSS0.Px4.p3.1.m1.1a"><mi id="S2.SS0.SSS0.Px4.p3.1.m1.1.1" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p3.1.m1.1b"><ci id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p3.1.m1.1c">k</annotation></semantics></math> and nucleus sampling methods; in that it truncates the tail of the probability distribution. However, instead of truncating it at each decoding step, it biases the decoder proactively to generate output from a set of tokens which are topically-relevant to the input.</p>
</div>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2105.11921/assets/x1.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="448" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>A Transformer-based encoder-decoder architecture with <span id="S2.F2.2.1" class="ltx_text ltx_font_smallcaps">Fame</span>. </figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Summarization with Focus Attention</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.9" class="ltx_p">Given an input document <math id="S3.p1.1.m1.1" class="ltx_Math" alttext="X_{1:n}" display="inline"><semantics id="S3.p1.1.m1.1a"><msub id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml"><mi id="S3.p1.1.m1.1.1.2" xref="S3.p1.1.m1.1.1.2.cmml">X</mi><mrow id="S3.p1.1.m1.1.1.3" xref="S3.p1.1.m1.1.1.3.cmml"><mn id="S3.p1.1.m1.1.1.3.2" xref="S3.p1.1.m1.1.1.3.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="S3.p1.1.m1.1.1.3.1" xref="S3.p1.1.m1.1.1.3.1.cmml">:</mo><mi id="S3.p1.1.m1.1.1.3.3" xref="S3.p1.1.m1.1.1.3.3.cmml">n</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><apply id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.p1.1.m1.1.1.1.cmml" xref="S3.p1.1.m1.1.1">subscript</csymbol><ci id="S3.p1.1.m1.1.1.2.cmml" xref="S3.p1.1.m1.1.1.2">𝑋</ci><apply id="S3.p1.1.m1.1.1.3.cmml" xref="S3.p1.1.m1.1.1.3"><ci id="S3.p1.1.m1.1.1.3.1.cmml" xref="S3.p1.1.m1.1.1.3.1">:</ci><cn type="integer" id="S3.p1.1.m1.1.1.3.2.cmml" xref="S3.p1.1.m1.1.1.3.2">1</cn><ci id="S3.p1.1.m1.1.1.3.3.cmml" xref="S3.p1.1.m1.1.1.3.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">X_{1:n}</annotation></semantics></math>, we aim to generate its summary <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="Y_{1:m}" display="inline"><semantics id="S3.p1.2.m2.1a"><msub id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml"><mi id="S3.p1.2.m2.1.1.2" xref="S3.p1.2.m2.1.1.2.cmml">Y</mi><mrow id="S3.p1.2.m2.1.1.3" xref="S3.p1.2.m2.1.1.3.cmml"><mn id="S3.p1.2.m2.1.1.3.2" xref="S3.p1.2.m2.1.1.3.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="S3.p1.2.m2.1.1.3.1" xref="S3.p1.2.m2.1.1.3.1.cmml">:</mo><mi id="S3.p1.2.m2.1.1.3.3" xref="S3.p1.2.m2.1.1.3.3.cmml">m</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><apply id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.cmml" xref="S3.p1.2.m2.1.1">subscript</csymbol><ci id="S3.p1.2.m2.1.1.2.cmml" xref="S3.p1.2.m2.1.1.2">𝑌</ci><apply id="S3.p1.2.m2.1.1.3.cmml" xref="S3.p1.2.m2.1.1.3"><ci id="S3.p1.2.m2.1.1.3.1.cmml" xref="S3.p1.2.m2.1.1.3.1">:</ci><cn type="integer" id="S3.p1.2.m2.1.1.3.2.cmml" xref="S3.p1.2.m2.1.1.3.2">1</cn><ci id="S3.p1.2.m2.1.1.3.3.cmml" xref="S3.p1.2.m2.1.1.3.3">𝑚</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">Y_{1:m}</annotation></semantics></math>, where <math id="S3.p1.3.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.p1.3.m3.1a"><mi id="S3.p1.3.m3.1.1" xref="S3.p1.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.p1.3.m3.1b"><ci id="S3.p1.3.m3.1.1.cmml" xref="S3.p1.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m3.1c">n</annotation></semantics></math> and <math id="S3.p1.4.m4.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S3.p1.4.m4.1a"><mi id="S3.p1.4.m4.1.1" xref="S3.p1.4.m4.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S3.p1.4.m4.1b"><ci id="S3.p1.4.m4.1.1.cmml" xref="S3.p1.4.m4.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.4.m4.1c">m</annotation></semantics></math> are input and output sequence lengths. We address this problem using seq2seq architectures with Transformer encoder and decoder, augmented with <span id="S3.p1.9.1" class="ltx_text ltx_font_smallcaps">Fame</span>, as depicted in Figure <a href="#S2.F2" title="Figure 2 ‣ Diverse Generation Models ‣ 2 Related Work ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
<span id="S3.p1.9.2" class="ltx_text ltx_font_smallcaps">Fame</span> learns a distribution <math id="S3.p1.5.m5.1" class="ltx_Math" alttext="{\bm{t}}_{x_{i}}" display="inline"><semantics id="S3.p1.5.m5.1a"><msub id="S3.p1.5.m5.1.1" xref="S3.p1.5.m5.1.1.cmml"><mi id="S3.p1.5.m5.1.1.2" xref="S3.p1.5.m5.1.1.2.cmml">𝒕</mi><msub id="S3.p1.5.m5.1.1.3" xref="S3.p1.5.m5.1.1.3.cmml"><mi id="S3.p1.5.m5.1.1.3.2" xref="S3.p1.5.m5.1.1.3.2.cmml">x</mi><mi id="S3.p1.5.m5.1.1.3.3" xref="S3.p1.5.m5.1.1.3.3.cmml">i</mi></msub></msub><annotation-xml encoding="MathML-Content" id="S3.p1.5.m5.1b"><apply id="S3.p1.5.m5.1.1.cmml" xref="S3.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S3.p1.5.m5.1.1.1.cmml" xref="S3.p1.5.m5.1.1">subscript</csymbol><ci id="S3.p1.5.m5.1.1.2.cmml" xref="S3.p1.5.m5.1.1.2">𝒕</ci><apply id="S3.p1.5.m5.1.1.3.cmml" xref="S3.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="S3.p1.5.m5.1.1.3.1.cmml" xref="S3.p1.5.m5.1.1.3">subscript</csymbol><ci id="S3.p1.5.m5.1.1.3.2.cmml" xref="S3.p1.5.m5.1.1.3.2">𝑥</ci><ci id="S3.p1.5.m5.1.1.3.3.cmml" xref="S3.p1.5.m5.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.5.m5.1c">{\bm{t}}_{x_{i}}</annotation></semantics></math> for each input token <math id="S3.p1.6.m6.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.p1.6.m6.1a"><msub id="S3.p1.6.m6.1.1" xref="S3.p1.6.m6.1.1.cmml"><mi id="S3.p1.6.m6.1.1.2" xref="S3.p1.6.m6.1.1.2.cmml">x</mi><mi id="S3.p1.6.m6.1.1.3" xref="S3.p1.6.m6.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p1.6.m6.1b"><apply id="S3.p1.6.m6.1.1.cmml" xref="S3.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S3.p1.6.m6.1.1.1.cmml" xref="S3.p1.6.m6.1.1">subscript</csymbol><ci id="S3.p1.6.m6.1.1.2.cmml" xref="S3.p1.6.m6.1.1.2">𝑥</ci><ci id="S3.p1.6.m6.1.1.3.cmml" xref="S3.p1.6.m6.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.6.m6.1c">x_{i}</annotation></semantics></math> over the vocabulary, measuring similarity of <math id="S3.p1.7.m7.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.p1.7.m7.1a"><msub id="S3.p1.7.m7.1.1" xref="S3.p1.7.m7.1.1.cmml"><mi id="S3.p1.7.m7.1.1.2" xref="S3.p1.7.m7.1.1.2.cmml">x</mi><mi id="S3.p1.7.m7.1.1.3" xref="S3.p1.7.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p1.7.m7.1b"><apply id="S3.p1.7.m7.1.1.cmml" xref="S3.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S3.p1.7.m7.1.1.1.cmml" xref="S3.p1.7.m7.1.1">subscript</csymbol><ci id="S3.p1.7.m7.1.1.2.cmml" xref="S3.p1.7.m7.1.1.2">𝑥</ci><ci id="S3.p1.7.m7.1.1.3.cmml" xref="S3.p1.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.7.m7.1c">x_{i}</annotation></semantics></math> (in context) to the tokens in the vocabulary. The vocabulary distributions, <math id="S3.p1.8.m8.1" class="ltx_Math" alttext="{\bm{t}}_{x_{i}}" display="inline"><semantics id="S3.p1.8.m8.1a"><msub id="S3.p1.8.m8.1.1" xref="S3.p1.8.m8.1.1.cmml"><mi id="S3.p1.8.m8.1.1.2" xref="S3.p1.8.m8.1.1.2.cmml">𝒕</mi><msub id="S3.p1.8.m8.1.1.3" xref="S3.p1.8.m8.1.1.3.cmml"><mi id="S3.p1.8.m8.1.1.3.2" xref="S3.p1.8.m8.1.1.3.2.cmml">x</mi><mi id="S3.p1.8.m8.1.1.3.3" xref="S3.p1.8.m8.1.1.3.3.cmml">i</mi></msub></msub><annotation-xml encoding="MathML-Content" id="S3.p1.8.m8.1b"><apply id="S3.p1.8.m8.1.1.cmml" xref="S3.p1.8.m8.1.1"><csymbol cd="ambiguous" id="S3.p1.8.m8.1.1.1.cmml" xref="S3.p1.8.m8.1.1">subscript</csymbol><ci id="S3.p1.8.m8.1.1.2.cmml" xref="S3.p1.8.m8.1.1.2">𝒕</ci><apply id="S3.p1.8.m8.1.1.3.cmml" xref="S3.p1.8.m8.1.1.3"><csymbol cd="ambiguous" id="S3.p1.8.m8.1.1.3.1.cmml" xref="S3.p1.8.m8.1.1.3">subscript</csymbol><ci id="S3.p1.8.m8.1.1.3.2.cmml" xref="S3.p1.8.m8.1.1.3.2">𝑥</ci><ci id="S3.p1.8.m8.1.1.3.3.cmml" xref="S3.p1.8.m8.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.8.m8.1c">{\bm{t}}_{x_{i}}</annotation></semantics></math>, for all <math id="S3.p1.9.m9.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.p1.9.m9.1a"><msub id="S3.p1.9.m9.1.1" xref="S3.p1.9.m9.1.1.cmml"><mi id="S3.p1.9.m9.1.1.2" xref="S3.p1.9.m9.1.1.2.cmml">x</mi><mi id="S3.p1.9.m9.1.1.3" xref="S3.p1.9.m9.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p1.9.m9.1b"><apply id="S3.p1.9.m9.1.1.cmml" xref="S3.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S3.p1.9.m9.1.1.1.cmml" xref="S3.p1.9.m9.1.1">subscript</csymbol><ci id="S3.p1.9.m9.1.1.2.cmml" xref="S3.p1.9.m9.1.1.2">𝑥</ci><ci id="S3.p1.9.m9.1.1.3.cmml" xref="S3.p1.9.m9.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.9.m9.1c">x_{i}</annotation></semantics></math> are combined to form a dynamic vocabulary bias that is added to the decoder logits. This mechanism enhances the conditioning on the input source and encourages the decoder to generate tokens that are topically similar to the input.</p>
</div>
<section id="S3.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Transformer-based seq2seq Model</h4>

<div id="S3.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px1.p1.12" class="ltx_p">The encoder uses BERT Transformer layers with multi-headed self-attention to encode <math id="S3.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="X" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.1.m1.1a"><mi id="S3.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.cmml">X</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.1.m1.1b"><ci id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1">𝑋</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.1.m1.1c">X</annotation></semantics></math> to a vector sequence <math id="S3.SS0.SSS0.Px1.p1.2.m2.3" class="ltx_Math" alttext="{\bm{X}}={\bm{x}}_{1},\ldots,{\bm{x}}_{n}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.2.m2.3a"><mrow id="S3.SS0.SSS0.Px1.p1.2.m2.3.3" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.4" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.4.cmml">𝑿</mi><mo id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.3" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.3.cmml">=</mo><mrow id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.3.cmml"><msub id="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.2" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.2.cmml">𝒙</mi><mn id="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.3" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">…</mi><mo id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.4" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.3.cmml">,</mo><msub id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2.2" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2.2.cmml">𝒙</mi><mi id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2.3" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.cmml">n</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.2.m2.3b"><apply id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3"><eq id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.3"></eq><ci id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.4.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.4">𝑿</ci><list id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2"><apply id="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.2">𝒙</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.3">1</cn></apply><ci id="S3.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.1.1">…</ci><apply id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2.2">𝒙</ci><ci id="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.3.3.2.2.2.3">𝑛</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.2.m2.3c">{\bm{X}}={\bm{x}}_{1},\ldots,{\bm{x}}_{n}</annotation></semantics></math>, with <math id="S3.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="{\bm{x}}_{i}\in\mathbb{R}^{h}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.3.m3.1a"><mrow id="S3.SS0.SSS0.Px1.p1.3.m3.1.1" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.cmml"><msub id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2.2" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.cmml">𝒙</mi><mi id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2.3" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.1" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.1.cmml">∈</mo><msup id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3.2.cmml">ℝ</mi><mi id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3.3" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3.3.cmml">h</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.3.m3.1b"><apply id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1"><in id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.1"></in><apply id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2.2">𝒙</ci><ci id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.2.3">𝑖</ci></apply><apply id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3.2">ℝ</ci><ci id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.3.m3.1c">{\bm{x}}_{i}\in\mathbb{R}^{h}</annotation></semantics></math>, where <math id="S3.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.4.m4.1a"><mi id="S3.SS0.SSS0.Px1.p1.4.m4.1.1" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.4.m4.1b"><ci id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.4.m4.1c">h</annotation></semantics></math> is the size of hidden representation. The decoder uses an identical architecture, except that
at decoding step <math id="S3.SS0.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.5.m5.1a"><mi id="S3.SS0.SSS0.Px1.p1.5.m5.1.1" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.5.m5.1b"><ci id="S3.SS0.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.5.m5.1c">t</annotation></semantics></math>, layer <math id="S3.SS0.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.6.m6.1a"><mi id="S3.SS0.SSS0.Px1.p1.6.m6.1.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.6.m6.1b"><ci id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.6.m6.1c">l</annotation></semantics></math> adds a conditional representation <math id="S3.SS0.SSS0.Px1.p1.7.m7.1" class="ltx_Math" alttext="{\bm{y}}^{l}_{t}\in\mathbb{R}^{h}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.7.m7.1a"><mrow id="S3.SS0.SSS0.Px1.p1.7.m7.1.1" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.cmml"><msubsup id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.2.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.2.2.cmml">𝒚</mi><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.3.cmml">t</mi><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.2.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.2.3.cmml">l</mi></msubsup><mo id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.cmml">∈</mo><msup id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.2.cmml">ℝ</mi><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.3.cmml">h</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.7.m7.1b"><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1"><in id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1"></in><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2">subscript</csymbol><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.2.2">𝒚</ci><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.2.3">𝑙</ci></apply><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.3">𝑡</ci></apply><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.2">ℝ</ci><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.7.m7.1c">{\bm{y}}^{l}_{t}\in\mathbb{R}^{h}</annotation></semantics></math> for the token <math id="S3.SS0.SSS0.Px1.p1.8.m8.1" class="ltx_Math" alttext="y_{t}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.8.m8.1a"><msub id="S3.SS0.SSS0.Px1.p1.8.m8.1.1" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.2" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.2.cmml">y</mi><mi id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.3" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.8.m8.1b"><apply id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.2">𝑦</ci><ci id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.8.m8.1c">y_{t}</annotation></semantics></math> by attending to the output representation <math id="S3.SS0.SSS0.Px1.p1.9.m9.3" class="ltx_Math" alttext="{\bm{Y}}^{l-1}_{1:t-1}={\bm{y}}^{l-1}_{1},\ldots,{\bm{y}}^{l-1}_{t-1}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.9.m9.3a"><mrow id="S3.SS0.SSS0.Px1.p1.9.m9.3.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.cmml"><msubsup id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.cmml"><mi id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.2.cmml">𝒀</mi><mrow id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.cmml"><mn id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.1" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.1.cmml">:</mo><mrow id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.2.cmml">t</mi><mo id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.1" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.1.cmml">−</mo><mn id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.3.cmml">1</mn></mrow></mrow><mrow id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.2.cmml">l</mi><mo id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.1" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.1.cmml">−</mo><mn id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.3.cmml">1</mn></mrow></msubsup><mo id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.3.cmml">=</mo><mrow id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.3.cmml"><msubsup id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.2.cmml">𝒚</mi><mn id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.3.cmml">1</mn><mrow id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.2.cmml">l</mi><mo id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.1" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.1.cmml">−</mo><mn id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.3.cmml">1</mn></mrow></msubsup><mo id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.SS0.SSS0.Px1.p1.9.m9.1.1" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1.cmml">…</mi><mo id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.4" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.3.cmml">,</mo><msubsup id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.2.cmml">𝒚</mi><mrow id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.2.cmml">t</mi><mo id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.1" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.1.cmml">−</mo><mn id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.3.cmml">1</mn></mrow><mrow id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.2.cmml">l</mi><mo id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.1" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.1.cmml">−</mo><mn id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.3.cmml">1</mn></mrow></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.9.m9.3b"><apply id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3"><eq id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.3"></eq><apply id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4">subscript</csymbol><apply id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.2">𝒀</ci><apply id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3"><minus id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.1"></minus><ci id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.2">𝑙</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.2.3.3">1</cn></apply></apply><apply id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3"><ci id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.1">:</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.2">1</cn><apply id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3"><minus id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.1"></minus><ci id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.2">𝑡</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.4.3.3.3">1</cn></apply></apply></apply><list id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2"><apply id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1">subscript</csymbol><apply id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.2">𝒚</ci><apply id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3"><minus id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.1"></minus><ci id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.2">𝑙</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.2.3.3">1</cn></apply></apply><cn type="integer" id="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.2.2.1.1.1.3">1</cn></apply><ci id="S3.SS0.SSS0.Px1.p1.9.m9.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1">…</ci><apply id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2">subscript</csymbol><apply id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.2">𝒚</ci><apply id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3"><minus id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.1"></minus><ci id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.2">𝑙</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.2.3.3">1</cn></apply></apply><apply id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3"><minus id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.1"></minus><ci id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.2">𝑡</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.3.3.2.2.2.3.3">1</cn></apply></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.9.m9.3c">{\bm{Y}}^{l-1}_{1:t-1}={\bm{y}}^{l-1}_{1},\ldots,{\bm{y}}^{l-1}_{t-1}</annotation></semantics></math> generated so far through self-attention and by attending to the input contextual representation <math id="S3.SS0.SSS0.Px1.p1.10.m10.1" class="ltx_Math" alttext="{\bm{X}}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.10.m10.1a"><mi id="S3.SS0.SSS0.Px1.p1.10.m10.1.1" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.cmml">𝑿</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.10.m10.1b"><ci id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1">𝑿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.10.m10.1c">{\bm{X}}</annotation></semantics></math> through encoder-decoder attention. The probability of predicting the next token <math id="S3.SS0.SSS0.Px1.p1.11.m11.1" class="ltx_Math" alttext="y_{t}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.11.m11.1a"><msub id="S3.SS0.SSS0.Px1.p1.11.m11.1.1" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.2" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.2.cmml">y</mi><mi id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.3" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.11.m11.1b"><apply id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.2">𝑦</ci><ci id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.11.m11.1c">y_{t}</annotation></semantics></math> from a vocabulary <math id="S3.SS0.SSS0.Px1.p1.12.m12.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.12.m12.1a"><mi id="S3.SS0.SSS0.Px1.p1.12.m12.1.1" xref="S3.SS0.SSS0.Px1.p1.12.m12.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.12.m12.1b"><ci id="S3.SS0.SSS0.Px1.p1.12.m12.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.12.m12.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.12.m12.1c">V</annotation></semantics></math> is:</p>
<table id="S3.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E1.m1.3" class="ltx_Math" alttext="p(y_{t}|Y_{1:t-1},X;\theta)=\mathrm{softmax}({\bm{E}}{\bm{y}}^{L}_{t})," display="block"><semantics id="S3.E1.m1.3a"><mrow id="S3.E1.m1.3.3.1" xref="S3.E1.m1.3.3.1.1.cmml"><mrow id="S3.E1.m1.3.3.1.1" xref="S3.E1.m1.3.3.1.1.cmml"><mrow id="S3.E1.m1.3.3.1.1.1" xref="S3.E1.m1.3.3.1.1.1.cmml"><mi id="S3.E1.m1.3.3.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.3.3.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.2.cmml">​</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E1.m1.3.3.1.1.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.cmml"><msub id="S3.E1.m1.3.3.1.1.1.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.1.3.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.1.3.2" xref="S3.E1.m1.3.3.1.1.1.1.1.1.3.2.cmml">y</mi><mi id="S3.E1.m1.3.3.1.1.1.1.1.1.3.3" xref="S3.E1.m1.3.3.1.1.1.1.1.1.3.3.cmml">t</mi></msub><mo fence="false" id="S3.E1.m1.3.3.1.1.1.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.1.2.cmml">|</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.2.cmml"><msub id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.2.cmml">Y</mi><mrow id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.cmml"><mn id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.2" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.1.cmml">:</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.2" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.2.cmml">t</mi><mo id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.1.cmml">−</mo><mn id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.3" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.3.cmml">1</mn></mrow></mrow></msub><mo id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.2.cmml">,</mo><mi id="S3.E1.m1.1.1" xref="S3.E1.m1.1.1.cmml">X</mi><mo id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.2.cmml">;</mo><mi id="S3.E1.m1.2.2" xref="S3.E1.m1.2.2.cmml">θ</mi></mrow></mrow><mo stretchy="false" id="S3.E1.m1.3.3.1.1.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E1.m1.3.3.1.1.3" xref="S3.E1.m1.3.3.1.1.3.cmml">=</mo><mrow id="S3.E1.m1.3.3.1.1.2" xref="S3.E1.m1.3.3.1.1.2.cmml"><mi id="S3.E1.m1.3.3.1.1.2.3" xref="S3.E1.m1.3.3.1.1.2.3.cmml">softmax</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.3.3.1.1.2.2" xref="S3.E1.m1.3.3.1.1.2.2.cmml">​</mo><mrow id="S3.E1.m1.3.3.1.1.2.1.1" xref="S3.E1.m1.3.3.1.1.2.1.1.1.cmml"><mo stretchy="false" id="S3.E1.m1.3.3.1.1.2.1.1.2" xref="S3.E1.m1.3.3.1.1.2.1.1.1.cmml">(</mo><mrow id="S3.E1.m1.3.3.1.1.2.1.1.1" xref="S3.E1.m1.3.3.1.1.2.1.1.1.cmml"><mi id="S3.E1.m1.3.3.1.1.2.1.1.1.2" xref="S3.E1.m1.3.3.1.1.2.1.1.1.2.cmml">𝑬</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.3.3.1.1.2.1.1.1.1" xref="S3.E1.m1.3.3.1.1.2.1.1.1.1.cmml">​</mo><msubsup id="S3.E1.m1.3.3.1.1.2.1.1.1.3" xref="S3.E1.m1.3.3.1.1.2.1.1.1.3.cmml"><mi id="S3.E1.m1.3.3.1.1.2.1.1.1.3.2.2" xref="S3.E1.m1.3.3.1.1.2.1.1.1.3.2.2.cmml">𝒚</mi><mi id="S3.E1.m1.3.3.1.1.2.1.1.1.3.3" xref="S3.E1.m1.3.3.1.1.2.1.1.1.3.3.cmml">t</mi><mi id="S3.E1.m1.3.3.1.1.2.1.1.1.3.2.3" xref="S3.E1.m1.3.3.1.1.2.1.1.1.3.2.3.cmml">L</mi></msubsup></mrow><mo stretchy="false" id="S3.E1.m1.3.3.1.1.2.1.1.3" xref="S3.E1.m1.3.3.1.1.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E1.m1.3.3.1.2" xref="S3.E1.m1.3.3.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E1.m1.3b"><apply id="S3.E1.m1.3.3.1.1.cmml" xref="S3.E1.m1.3.3.1"><eq id="S3.E1.m1.3.3.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.3"></eq><apply id="S3.E1.m1.3.3.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1"><times id="S3.E1.m1.3.3.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.2"></times><ci id="S3.E1.m1.3.3.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.1.3">𝑝</ci><apply id="S3.E1.m1.3.3.1.1.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1"><csymbol cd="latexml" id="S3.E1.m1.3.3.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.2">conditional</csymbol><apply id="S3.E1.m1.3.3.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.1.1.1.1.3.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E1.m1.3.3.1.1.1.1.1.1.3.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.3.2">𝑦</ci><ci id="S3.E1.m1.3.3.1.1.1.1.1.1.3.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.3.3">𝑡</ci></apply><list id="S3.E1.m1.3.3.1.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1"><apply id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.2">𝑌</ci><apply id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3"><ci id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.1">:</ci><cn type="integer" id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.2">1</cn><apply id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3"><minus id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.1"></minus><ci id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.2">𝑡</ci><cn type="integer" id="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.1.1.3.3.3">1</cn></apply></apply></apply><ci id="S3.E1.m1.1.1.cmml" xref="S3.E1.m1.1.1">𝑋</ci><ci id="S3.E1.m1.2.2.cmml" xref="S3.E1.m1.2.2">𝜃</ci></list></apply></apply><apply id="S3.E1.m1.3.3.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.2"><times id="S3.E1.m1.3.3.1.1.2.2.cmml" xref="S3.E1.m1.3.3.1.1.2.2"></times><ci id="S3.E1.m1.3.3.1.1.2.3.cmml" xref="S3.E1.m1.3.3.1.1.2.3">softmax</ci><apply id="S3.E1.m1.3.3.1.1.2.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.2.1.1"><times id="S3.E1.m1.3.3.1.1.2.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.2.1.1.1.1"></times><ci id="S3.E1.m1.3.3.1.1.2.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.2.1.1.1.2">𝑬</ci><apply id="S3.E1.m1.3.3.1.1.2.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.2.1.1.1.3.1.cmml" xref="S3.E1.m1.3.3.1.1.2.1.1.1.3">subscript</csymbol><apply id="S3.E1.m1.3.3.1.1.2.1.1.1.3.2.cmml" xref="S3.E1.m1.3.3.1.1.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.2.1.1.1.3.2.1.cmml" xref="S3.E1.m1.3.3.1.1.2.1.1.1.3">superscript</csymbol><ci id="S3.E1.m1.3.3.1.1.2.1.1.1.3.2.2.cmml" xref="S3.E1.m1.3.3.1.1.2.1.1.1.3.2.2">𝒚</ci><ci id="S3.E1.m1.3.3.1.1.2.1.1.1.3.2.3.cmml" xref="S3.E1.m1.3.3.1.1.2.1.1.1.3.2.3">𝐿</ci></apply><ci id="S3.E1.m1.3.3.1.1.2.1.1.1.3.3.cmml" xref="S3.E1.m1.3.3.1.1.2.1.1.1.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1.m1.3c">p(y_{t}|Y_{1:t-1},X;\theta)=\mathrm{softmax}({\bm{E}}{\bm{y}}^{L}_{t}),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S3.SS0.SSS0.Px1.p1.16" class="ltx_p">where, <math id="S3.SS0.SSS0.Px1.p1.13.m1.1" class="ltx_Math" alttext="{\bm{y}}^{L}_{t}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.13.m1.1a"><msubsup id="S3.SS0.SSS0.Px1.p1.13.m1.1.1" xref="S3.SS0.SSS0.Px1.p1.13.m1.1.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.13.m1.1.1.2.2" xref="S3.SS0.SSS0.Px1.p1.13.m1.1.1.2.2.cmml">𝒚</mi><mi id="S3.SS0.SSS0.Px1.p1.13.m1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.13.m1.1.1.3.cmml">t</mi><mi id="S3.SS0.SSS0.Px1.p1.13.m1.1.1.2.3" xref="S3.SS0.SSS0.Px1.p1.13.m1.1.1.2.3.cmml">L</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.13.m1.1b"><apply id="S3.SS0.SSS0.Px1.p1.13.m1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.13.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.13.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.13.m1.1.1">subscript</csymbol><apply id="S3.SS0.SSS0.Px1.p1.13.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.13.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.13.m1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.13.m1.1.1">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.13.m1.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.13.m1.1.1.2.2">𝒚</ci><ci id="S3.SS0.SSS0.Px1.p1.13.m1.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.13.m1.1.1.2.3">𝐿</ci></apply><ci id="S3.SS0.SSS0.Px1.p1.13.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.13.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.13.m1.1c">{\bm{y}}^{L}_{t}</annotation></semantics></math> is the representation from the final decoder layer <math id="S3.SS0.SSS0.Px1.p1.14.m2.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.14.m2.1a"><mi id="S3.SS0.SSS0.Px1.p1.14.m2.1.1" xref="S3.SS0.SSS0.Px1.p1.14.m2.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.14.m2.1b"><ci id="S3.SS0.SSS0.Px1.p1.14.m2.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.14.m2.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.14.m2.1c">L</annotation></semantics></math>, <math id="S3.SS0.SSS0.Px1.p1.15.m3.1" class="ltx_Math" alttext="{\bm{E}}\in\mathbb{R}^{|V|\times h}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.15.m3.1a"><mrow id="S3.SS0.SSS0.Px1.p1.15.m3.1.2" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.15.m3.1.2.2" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.2.2.cmml">𝑬</mi><mo id="S3.SS0.SSS0.Px1.p1.15.m3.1.2.1" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.2.1.cmml">∈</mo><msup id="S3.SS0.SSS0.Px1.p1.15.m3.1.2.3" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.2.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.15.m3.1.2.3.2" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.2.3.2.cmml">ℝ</mi><mrow id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.cmml"><mrow id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.3.1.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.3.2.1" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.3.1.1.cmml">|</mo><mi id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.1.cmml">V</mi><mo rspace="0.055em" stretchy="false" id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.3.2.2" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.3.1.1.cmml">|</mo></mrow><mo rspace="0.222em" id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.2" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.2.cmml">×</mo><mi id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.4" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.4.cmml">h</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.15.m3.1b"><apply id="S3.SS0.SSS0.Px1.p1.15.m3.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.2"><in id="S3.SS0.SSS0.Px1.p1.15.m3.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.2.1"></in><ci id="S3.SS0.SSS0.Px1.p1.15.m3.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.2.2">𝑬</ci><apply id="S3.SS0.SSS0.Px1.p1.15.m3.1.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.2.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.15.m3.1.2.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.2.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.15.m3.1.2.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.2.3.2">ℝ</ci><apply id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1"><times id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.2"></times><apply id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.3.2"><abs id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.3.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.3.2.1"></abs><ci id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.1">𝑉</ci></apply><ci id="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.4.cmml" xref="S3.SS0.SSS0.Px1.p1.15.m3.1.1.1.4">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.15.m3.1c">{\bm{E}}\in\mathbb{R}^{|V|\times h}</annotation></semantics></math> the embedding matrix and <math id="S3.SS0.SSS0.Px1.p1.16.m4.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.16.m4.1a"><mi id="S3.SS0.SSS0.Px1.p1.16.m4.1.1" xref="S3.SS0.SSS0.Px1.p1.16.m4.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.16.m4.1b"><ci id="S3.SS0.SSS0.Px1.p1.16.m4.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.16.m4.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.16.m4.1c">\theta</annotation></semantics></math> the model parameters. Parameters are trained by minimizing cross-entropy at each decoding step:</p>
<table id="S3.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E2.m1.4" class="ltx_Math" alttext="L_{\mathrm{MLE}}(\theta)=-\frac{1}{m}\sum^{m}_{i=1}\log p(\hat{y}_{t}|\hat{Y}_{1:t-1},X;\theta)," display="block"><semantics id="S3.E2.m1.4a"><mrow id="S3.E2.m1.4.4.1" xref="S3.E2.m1.4.4.1.1.cmml"><mrow id="S3.E2.m1.4.4.1.1" xref="S3.E2.m1.4.4.1.1.cmml"><mrow id="S3.E2.m1.4.4.1.1.3" xref="S3.E2.m1.4.4.1.1.3.cmml"><msub id="S3.E2.m1.4.4.1.1.3.2" xref="S3.E2.m1.4.4.1.1.3.2.cmml"><mi id="S3.E2.m1.4.4.1.1.3.2.2" xref="S3.E2.m1.4.4.1.1.3.2.2.cmml">L</mi><mi id="S3.E2.m1.4.4.1.1.3.2.3" xref="S3.E2.m1.4.4.1.1.3.2.3.cmml">MLE</mi></msub><mo lspace="0em" rspace="0em" id="S3.E2.m1.4.4.1.1.3.1" xref="S3.E2.m1.4.4.1.1.3.1.cmml">​</mo><mrow id="S3.E2.m1.4.4.1.1.3.3.2" xref="S3.E2.m1.4.4.1.1.3.cmml"><mo stretchy="false" id="S3.E2.m1.4.4.1.1.3.3.2.1" xref="S3.E2.m1.4.4.1.1.3.cmml">(</mo><mi id="S3.E2.m1.1.1" xref="S3.E2.m1.1.1.cmml">θ</mi><mo stretchy="false" id="S3.E2.m1.4.4.1.1.3.3.2.2" xref="S3.E2.m1.4.4.1.1.3.cmml">)</mo></mrow></mrow><mo id="S3.E2.m1.4.4.1.1.2" xref="S3.E2.m1.4.4.1.1.2.cmml">=</mo><mrow id="S3.E2.m1.4.4.1.1.1" xref="S3.E2.m1.4.4.1.1.1.cmml"><mo id="S3.E2.m1.4.4.1.1.1a" xref="S3.E2.m1.4.4.1.1.1.cmml">−</mo><mrow id="S3.E2.m1.4.4.1.1.1.1" xref="S3.E2.m1.4.4.1.1.1.1.cmml"><mfrac id="S3.E2.m1.4.4.1.1.1.1.3" xref="S3.E2.m1.4.4.1.1.1.1.3.cmml"><mn id="S3.E2.m1.4.4.1.1.1.1.3.2" xref="S3.E2.m1.4.4.1.1.1.1.3.2.cmml">1</mn><mi id="S3.E2.m1.4.4.1.1.1.1.3.3" xref="S3.E2.m1.4.4.1.1.1.1.3.3.cmml">m</mi></mfrac><mo lspace="0em" rspace="0em" id="S3.E2.m1.4.4.1.1.1.1.2" xref="S3.E2.m1.4.4.1.1.1.1.2.cmml">​</mo><mrow id="S3.E2.m1.4.4.1.1.1.1.1" xref="S3.E2.m1.4.4.1.1.1.1.1.cmml"><munderover id="S3.E2.m1.4.4.1.1.1.1.1.2" xref="S3.E2.m1.4.4.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="S3.E2.m1.4.4.1.1.1.1.1.2.2.2" xref="S3.E2.m1.4.4.1.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="S3.E2.m1.4.4.1.1.1.1.1.2.3" xref="S3.E2.m1.4.4.1.1.1.1.1.2.3.cmml"><mi id="S3.E2.m1.4.4.1.1.1.1.1.2.3.2" xref="S3.E2.m1.4.4.1.1.1.1.1.2.3.2.cmml">i</mi><mo id="S3.E2.m1.4.4.1.1.1.1.1.2.3.1" xref="S3.E2.m1.4.4.1.1.1.1.1.2.3.1.cmml">=</mo><mn id="S3.E2.m1.4.4.1.1.1.1.1.2.3.3" xref="S3.E2.m1.4.4.1.1.1.1.1.2.3.3.cmml">1</mn></mrow><mi id="S3.E2.m1.4.4.1.1.1.1.1.2.2.3" xref="S3.E2.m1.4.4.1.1.1.1.1.2.2.3.cmml">m</mi></munderover><mrow id="S3.E2.m1.4.4.1.1.1.1.1.1" xref="S3.E2.m1.4.4.1.1.1.1.1.1.cmml"><mrow id="S3.E2.m1.4.4.1.1.1.1.1.1.3" xref="S3.E2.m1.4.4.1.1.1.1.1.1.3.cmml"><mi id="S3.E2.m1.4.4.1.1.1.1.1.1.3.1" xref="S3.E2.m1.4.4.1.1.1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S3.E2.m1.4.4.1.1.1.1.1.1.3a" xref="S3.E2.m1.4.4.1.1.1.1.1.1.3.cmml">⁡</mo><mi id="S3.E2.m1.4.4.1.1.1.1.1.1.3.2" xref="S3.E2.m1.4.4.1.1.1.1.1.1.3.2.cmml">p</mi></mrow><mo lspace="0em" rspace="0em" id="S3.E2.m1.4.4.1.1.1.1.1.1.2" xref="S3.E2.m1.4.4.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.2" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.cmml"><msub id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.cmml"><mi id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.2" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.2.cmml">y</mi><mo id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.1" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.1.cmml">^</mo></mover><mi id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.3" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.3.cmml">t</mi></msub><mo fence="false" id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.2" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.2.cmml">|</mo><mrow id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.2.cmml"><msub id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mover accent="true" id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">Y</mi><mo id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">^</mo></mover><mrow id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mn id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.1" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">:</mo><mrow id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml">t</mi><mo id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1.cmml">−</mo><mn id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3.cmml">1</mn></mrow></mrow></msub><mo id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.2.cmml">,</mo><mi id="S3.E2.m1.2.2" xref="S3.E2.m1.2.2.cmml">X</mi><mo id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.2.cmml">;</mo><mi id="S3.E2.m1.3.3" xref="S3.E2.m1.3.3.cmml">θ</mi></mrow></mrow><mo stretchy="false" id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.3" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow></mrow><mo id="S3.E2.m1.4.4.1.2" xref="S3.E2.m1.4.4.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m1.4b"><apply id="S3.E2.m1.4.4.1.1.cmml" xref="S3.E2.m1.4.4.1"><eq id="S3.E2.m1.4.4.1.1.2.cmml" xref="S3.E2.m1.4.4.1.1.2"></eq><apply id="S3.E2.m1.4.4.1.1.3.cmml" xref="S3.E2.m1.4.4.1.1.3"><times id="S3.E2.m1.4.4.1.1.3.1.cmml" xref="S3.E2.m1.4.4.1.1.3.1"></times><apply id="S3.E2.m1.4.4.1.1.3.2.cmml" xref="S3.E2.m1.4.4.1.1.3.2"><csymbol cd="ambiguous" id="S3.E2.m1.4.4.1.1.3.2.1.cmml" xref="S3.E2.m1.4.4.1.1.3.2">subscript</csymbol><ci id="S3.E2.m1.4.4.1.1.3.2.2.cmml" xref="S3.E2.m1.4.4.1.1.3.2.2">𝐿</ci><ci id="S3.E2.m1.4.4.1.1.3.2.3.cmml" xref="S3.E2.m1.4.4.1.1.3.2.3">MLE</ci></apply><ci id="S3.E2.m1.1.1.cmml" xref="S3.E2.m1.1.1">𝜃</ci></apply><apply id="S3.E2.m1.4.4.1.1.1.cmml" xref="S3.E2.m1.4.4.1.1.1"><minus id="S3.E2.m1.4.4.1.1.1.2.cmml" xref="S3.E2.m1.4.4.1.1.1"></minus><apply id="S3.E2.m1.4.4.1.1.1.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1"><times id="S3.E2.m1.4.4.1.1.1.1.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.2"></times><apply id="S3.E2.m1.4.4.1.1.1.1.3.cmml" xref="S3.E2.m1.4.4.1.1.1.1.3"><divide id="S3.E2.m1.4.4.1.1.1.1.3.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.3"></divide><cn type="integer" id="S3.E2.m1.4.4.1.1.1.1.3.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.3.2">1</cn><ci id="S3.E2.m1.4.4.1.1.1.1.3.3.cmml" xref="S3.E2.m1.4.4.1.1.1.1.3.3">𝑚</ci></apply><apply id="S3.E2.m1.4.4.1.1.1.1.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1"><apply id="S3.E2.m1.4.4.1.1.1.1.1.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E2.m1.4.4.1.1.1.1.1.2.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.2">subscript</csymbol><apply id="S3.E2.m1.4.4.1.1.1.1.1.2.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E2.m1.4.4.1.1.1.1.1.2.2.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.2">superscript</csymbol><sum id="S3.E2.m1.4.4.1.1.1.1.1.2.2.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.2.2.2"></sum><ci id="S3.E2.m1.4.4.1.1.1.1.1.2.2.3.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.2.2.3">𝑚</ci></apply><apply id="S3.E2.m1.4.4.1.1.1.1.1.2.3.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.2.3"><eq id="S3.E2.m1.4.4.1.1.1.1.1.2.3.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.2.3.1"></eq><ci id="S3.E2.m1.4.4.1.1.1.1.1.2.3.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.2.3.2">𝑖</ci><cn type="integer" id="S3.E2.m1.4.4.1.1.1.1.1.2.3.3.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.2.3.3">1</cn></apply></apply><apply id="S3.E2.m1.4.4.1.1.1.1.1.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1"><times id="S3.E2.m1.4.4.1.1.1.1.1.1.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.2"></times><apply id="S3.E2.m1.4.4.1.1.1.1.1.1.3.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.3"><log id="S3.E2.m1.4.4.1.1.1.1.1.1.3.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.3.1"></log><ci id="S3.E2.m1.4.4.1.1.1.1.1.1.3.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.3.2">𝑝</ci></apply><apply id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.2">conditional</csymbol><apply id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2"><ci id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.1">^</ci><ci id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.2">𝑦</ci></apply><ci id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.3">𝑡</ci></apply><list id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1"><apply id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><apply id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2"><ci id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2.1">^</ci><ci id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.2.2">𝑌</ci></apply><apply id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3"><ci id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.1">:</ci><cn type="integer" id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.2">1</cn><apply id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3"><minus id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1"></minus><ci id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2">𝑡</ci><cn type="integer" id="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3">1</cn></apply></apply></apply><ci id="S3.E2.m1.2.2.cmml" xref="S3.E2.m1.2.2">𝑋</ci><ci id="S3.E2.m1.3.3.cmml" xref="S3.E2.m1.3.3">𝜃</ci></list></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.4c">L_{\mathrm{MLE}}(\theta)=-\frac{1}{m}\sum^{m}_{i=1}\log p(\hat{y}_{t}|\hat{Y}_{1:t-1},X;\theta),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S3.SS0.SSS0.Px1.p1.17" class="ltx_p">where, <math id="S3.SS0.SSS0.Px1.p1.17.m1.1" class="ltx_Math" alttext="\hat{Y}_{1:m}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.17.m1.1a"><msub id="S3.SS0.SSS0.Px1.p1.17.m1.1.1" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.cmml"><mover accent="true" id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2.2" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2.2.cmml">Y</mi><mo id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2.1" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2.1.cmml">^</mo></mover><mrow id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.cmml"><mn id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.1" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.1.cmml">:</mo><mi id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.3" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.3.cmml">m</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.17.m1.1b"><apply id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1">subscript</csymbol><apply id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2"><ci id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2.1">^</ci><ci id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.2.2">𝑌</ci></apply><apply id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3"><ci id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.1">:</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.2">1</cn><ci id="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.17.m1.1.1.3.3">𝑚</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.17.m1.1c">\hat{Y}_{1:m}</annotation></semantics></math> is the human-written summary.</p>
</div>
</section>
<section id="S3.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Focus Attention MEchansim (<span id="S3.SS0.SSS0.Px2.1.1" class="ltx_text ltx_font_smallcaps">Fame</span>)</h4>

<div id="S3.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px2.p1.2" class="ltx_p">It is challenging for a decoder to obtain all relevant information from the conditional representation <math id="S3.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="{\bm{y}}^{L}_{t}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.1.m1.1a"><msubsup id="S3.SS0.SSS0.Px2.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.2.2" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1.2.2.cmml">𝒚</mi><mi id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.3" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1.3.cmml">t</mi><mi id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.2.3" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1.2.3.cmml">L</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.1.m1.1b"><apply id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1">subscript</csymbol><apply id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1.2.2">𝒚</ci><ci id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1.2.3">𝐿</ci></apply><ci id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.1.m1.1c">{\bm{y}}^{L}_{t}</annotation></semantics></math> to learn the vocabulary output logits such that predictions <math id="S3.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="y_{t}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.2.m2.1a"><msub id="S3.SS0.SSS0.Px2.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p1.2.m2.1.1.2" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1.2.cmml">y</mi><mi id="S3.SS0.SSS0.Px2.p1.2.m2.1.1.3" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.2.m2.1b"><apply id="S3.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1.2">𝑦</ci><ci id="S3.SS0.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.2.m2.1c">y_{t}</annotation></semantics></math> are consistent with the input.
Other modeling factors, specifically the decoder language model, can overwhelm model predictions.
<span id="S3.SS0.SSS0.Px2.p1.2.1" class="ltx_text ltx_font_smallcaps">Fame</span> (Figure <a href="#S2.F2" title="Figure 2 ‣ Diverse Generation Models ‣ 2 Related Work ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>) addresses this by introducing a short-circuit from the source to the vocabulary output logits via a source-conditioned bias on vocabulary items.</p>
</div>
<div id="S3.SS0.SSS0.Px2.p2" class="ltx_para">
<p id="S3.SS0.SSS0.Px2.p2.12" class="ltx_p">We take the encoder representation <math id="S3.SS0.SSS0.Px2.p2.1.m1.3" class="ltx_Math" alttext="{\bm{X}}={\bm{x}}_{1},\ldots,{\bm{x}}_{n}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.1.m1.3a"><mrow id="S3.SS0.SSS0.Px2.p2.1.m1.3.3" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.cmml"><mi id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.4" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.4.cmml">𝑿</mi><mo id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.3" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.3.cmml">=</mo><mrow id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.3.cmml"><msub id="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1" xref="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1.2" xref="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1.2.cmml">𝒙</mi><mn id="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1.3" xref="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.3" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.SS0.SSS0.Px2.p2.1.m1.1.1" xref="S3.SS0.SSS0.Px2.p2.1.m1.1.1.cmml">…</mi><mo id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.4" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.3.cmml">,</mo><msub id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2.cmml"><mi id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2.2" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2.2.cmml">𝒙</mi><mi id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2.3" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2.3.cmml">n</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.1.m1.3b"><apply id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3"><eq id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.3"></eq><ci id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.4.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.4">𝑿</ci><list id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2"><apply id="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1.2">𝒙</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.2.2.1.1.1.3">1</cn></apply><ci id="S3.SS0.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.1.1">…</ci><apply id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2.2">𝒙</ci><ci id="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.1.m1.3.3.2.2.2.3">𝑛</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.1.m1.3c">{\bm{X}}={\bm{x}}_{1},\ldots,{\bm{x}}_{n}</annotation></semantics></math> and learn a <em id="S3.SS0.SSS0.Px2.p2.12.1" class="ltx_emph ltx_font_italic">Token-level Vocabulary Distribution</em> <math id="S3.SS0.SSS0.Px2.p2.2.m2.2" class="ltx_Math" alttext="{\bm{t}}_{x_{i}}=\mathrm{gelu}({\bm{x}}_{i}{\bm{W}}_{1}){\bm{W}}_{2}{\bm{E}}\in\mathbb{R}^{|V|}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.2.m2.2a"><mrow id="S3.SS0.SSS0.Px2.p2.2.m2.2.2" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.cmml"><msub id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.cmml"><mi id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.2" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.2.cmml">𝒕</mi><msub id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3.cmml"><mi id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3.2" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3.2.cmml">x</mi><mi id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3.3" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3.3.cmml">i</mi></msub></msub><mo id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.4" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.4.cmml">=</mo><mrow id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.cmml"><mi id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.3" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.3.cmml">gelu</mi><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.2" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.2.cmml">​</mo><mrow id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.2" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.cmml">(</mo><mrow id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.cmml"><msub id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2.cmml"><mi id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2.2" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2.2.cmml">𝒙</mi><mi id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2.3" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.1" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.1.cmml">​</mo><msub id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3.2" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3.2.cmml">𝑾</mi><mn id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3.3" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3.3.cmml">1</mn></msub></mrow><mo stretchy="false" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.3" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.2a" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.2.cmml">​</mo><msub id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4.cmml"><mi id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4.2" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4.2.cmml">𝑾</mi><mn id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4.3" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.2b" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.2.cmml">​</mo><mi id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.5" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.5.cmml">𝑬</mi></mrow><mo id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.5" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.5.cmml">∈</mo><msup id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.6" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.6.cmml"><mi id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.6.2" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.6.2.cmml">ℝ</mi><mrow id="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.3" xref="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.3.1" xref="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.2.1.cmml">|</mo><mi id="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.1" xref="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.1.cmml">V</mi><mo stretchy="false" id="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.3.2" xref="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.2.1.cmml">|</mo></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.2.m2.2b"><apply id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2"><and id="S3.SS0.SSS0.Px2.p2.2.m2.2.2a.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2"></and><apply id="S3.SS0.SSS0.Px2.p2.2.m2.2.2b.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2"><eq id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.4.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.4"></eq><apply id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.2">𝒕</ci><apply id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3.2">𝑥</ci><ci id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.3.3.3">𝑖</ci></apply></apply><apply id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1"><times id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.2"></times><ci id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.3.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.3">gelu</ci><apply id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1"><times id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.1"></times><apply id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2.2">𝒙</ci><ci id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.2.3">𝑖</ci></apply><apply id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3.2">𝑾</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.1.1.1.3.3">1</cn></apply></apply><apply id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4.1.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4.2.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4.2">𝑾</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4.3.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.4.3">2</cn></apply><ci id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.5.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.5">𝑬</ci></apply></apply><apply id="S3.SS0.SSS0.Px2.p2.2.m2.2.2c.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2"><in id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.5.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.5"></in><share href="#S3.SS0.SSS0.Px2.p2.2.m2.2.2.1.cmml" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2d.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2"></share><apply id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.6.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.6"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.6.1.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.6">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.2.m2.2.2.6.2.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.2.2.6.2">ℝ</ci><apply id="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.3"><abs id="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.3.1"></abs><ci id="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.2.m2.1.1.1.1">𝑉</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.2.m2.2c">{\bm{t}}_{x_{i}}=\mathrm{gelu}({\bm{x}}_{i}{\bm{W}}_{1}){\bm{W}}_{2}{\bm{E}}\in\mathbb{R}^{|V|}</annotation></semantics></math>, for each token <math id="S3.SS0.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.3.m3.1a"><msub id="S3.SS0.SSS0.Px2.p2.3.m3.1.1" xref="S3.SS0.SSS0.Px2.p2.3.m3.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p2.3.m3.1.1.2" xref="S3.SS0.SSS0.Px2.p2.3.m3.1.1.2.cmml">x</mi><mi id="S3.SS0.SSS0.Px2.p2.3.m3.1.1.3" xref="S3.SS0.SSS0.Px2.p2.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.3.m3.1b"><apply id="S3.SS0.SSS0.Px2.p2.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.3.m3.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.3.m3.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.3.m3.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.3.m3.1.1.2">𝑥</ci><ci id="S3.SS0.SSS0.Px2.p2.3.m3.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p2.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.3.m3.1c">x_{i}</annotation></semantics></math> in the input sequence <math id="S3.SS0.SSS0.Px2.p2.4.m4.1" class="ltx_Math" alttext="{\bm{X}}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.4.m4.1a"><mi id="S3.SS0.SSS0.Px2.p2.4.m4.1.1" xref="S3.SS0.SSS0.Px2.p2.4.m4.1.1.cmml">𝑿</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.4.m4.1b"><ci id="S3.SS0.SSS0.Px2.p2.4.m4.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.4.m4.1.1">𝑿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.4.m4.1c">{\bm{X}}</annotation></semantics></math>. <math id="S3.SS0.SSS0.Px2.p2.5.m5.1" class="ltx_Math" alttext="{\bm{t}}_{x_{i}}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.5.m5.1a"><msub id="S3.SS0.SSS0.Px2.p2.5.m5.1.1" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p2.5.m5.1.1.2" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1.2.cmml">𝒕</mi><msub id="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3.2" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3.2.cmml">x</mi><mi id="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3.3" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3.3.cmml">i</mi></msub></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.5.m5.1b"><apply id="S3.SS0.SSS0.Px2.p2.5.m5.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.5.m5.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.5.m5.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1.2">𝒕</ci><apply id="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3.2">𝑥</ci><ci id="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.5.m5.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.5.m5.1c">{\bm{t}}_{x_{i}}</annotation></semantics></math> measures the contextual similarity of the input token <math id="S3.SS0.SSS0.Px2.p2.6.m6.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.6.m6.1a"><msub id="S3.SS0.SSS0.Px2.p2.6.m6.1.1" xref="S3.SS0.SSS0.Px2.p2.6.m6.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p2.6.m6.1.1.2" xref="S3.SS0.SSS0.Px2.p2.6.m6.1.1.2.cmml">x</mi><mi id="S3.SS0.SSS0.Px2.p2.6.m6.1.1.3" xref="S3.SS0.SSS0.Px2.p2.6.m6.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.6.m6.1b"><apply id="S3.SS0.SSS0.Px2.p2.6.m6.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.6.m6.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.6.m6.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.6.m6.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.6.m6.1.1.2">𝑥</ci><ci id="S3.SS0.SSS0.Px2.p2.6.m6.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p2.6.m6.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.6.m6.1c">x_{i}</annotation></semantics></math> to the tokens in the vocabulary; <math id="S3.SS0.SSS0.Px2.p2.7.m7.1" class="ltx_Math" alttext="{\bm{W}}_{1}\in\mathbb{R}^{h\times h^{\prime}}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.7.m7.1a"><mrow id="S3.SS0.SSS0.Px2.p2.7.m7.1.1" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.cmml"><msub id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2.cmml"><mi id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2.2" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2.2.cmml">𝑾</mi><mn id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2.3" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2.3.cmml">1</mn></msub><mo id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.1" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.1.cmml">∈</mo><msup id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.2" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.cmml"><mi id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.2" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.2.cmml">h</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.1" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.1.cmml">×</mo><msup id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3.cmml"><mi id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3.2" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3.2.cmml">h</mi><mo id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3.3" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3.3.cmml">′</mo></msup></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.7.m7.1b"><apply id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1"><in id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.1"></in><apply id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2.2">𝑾</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.2.3">1</cn></apply><apply id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.2">ℝ</ci><apply id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3"><times id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.1"></times><ci id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.2">ℎ</ci><apply id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3.2">ℎ</ci><ci id="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.7.m7.1.1.3.3.3.3">′</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.7.m7.1c">{\bm{W}}_{1}\in\mathbb{R}^{h\times h^{\prime}}</annotation></semantics></math> and <math id="S3.SS0.SSS0.Px2.p2.8.m8.1" class="ltx_Math" alttext="{\bm{W}}_{2}\in\mathbb{R}^{h^{\prime}\times h}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.8.m8.1a"><mrow id="S3.SS0.SSS0.Px2.p2.8.m8.1.1" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.cmml"><msub id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2.cmml"><mi id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2.2" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2.2.cmml">𝑾</mi><mn id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2.3" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2.3.cmml">2</mn></msub><mo id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.1" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.1.cmml">∈</mo><msup id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.2" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.cmml"><msup id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2.cmml"><mi id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2.2" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2.2.cmml">h</mi><mo id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2.3" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2.3.cmml">′</mo></msup><mo lspace="0.222em" rspace="0.222em" id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.1" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.1.cmml">×</mo><mi id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.3" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.3.cmml">h</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.8.m8.1b"><apply id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1"><in id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.1"></in><apply id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2.2">𝑾</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.2.3">2</cn></apply><apply id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.2">ℝ</ci><apply id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3"><times id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.1"></times><apply id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2.2">ℎ</ci><ci id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.2.3">′</ci></apply><ci id="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.8.m8.1.1.3.3.3">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.8.m8.1c">{\bm{W}}_{2}\in\mathbb{R}^{h^{\prime}\times h}</annotation></semantics></math> are parameters of newly introduced dense layers, <math id="S3.SS0.SSS0.Px2.p2.9.m9.1" class="ltx_Math" alttext="h^{\prime}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.9.m9.1a"><msup id="S3.SS0.SSS0.Px2.p2.9.m9.1.1" xref="S3.SS0.SSS0.Px2.p2.9.m9.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p2.9.m9.1.1.2" xref="S3.SS0.SSS0.Px2.p2.9.m9.1.1.2.cmml">h</mi><mo id="S3.SS0.SSS0.Px2.p2.9.m9.1.1.3" xref="S3.SS0.SSS0.Px2.p2.9.m9.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.9.m9.1b"><apply id="S3.SS0.SSS0.Px2.p2.9.m9.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.9.m9.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.9.m9.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.9.m9.1.1">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.9.m9.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.9.m9.1.1.2">ℎ</ci><ci id="S3.SS0.SSS0.Px2.p2.9.m9.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p2.9.m9.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.9.m9.1c">h^{\prime}</annotation></semantics></math> is the intermediate filter size. We define a <em id="S3.SS0.SSS0.Px2.p2.12.2" class="ltx_emph ltx_font_italic">Source-conditioned Vocabulary Distribution</em> as <math id="S3.SS0.SSS0.Px2.p2.10.m10.1" class="ltx_Math" alttext="{\bm{t}}_{X}=1/n\sum^{n}_{i=1}{\bm{t}}_{x_{i}}\in\mathbb{R}^{|V|}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.10.m10.1a"><mrow id="S3.SS0.SSS0.Px2.p2.10.m10.1.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.cmml"><msub id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2.cmml"><mi id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2.2.cmml">𝒕</mi><mi id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2.3" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2.3.cmml">X</mi></msub><mo id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.3" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.3.cmml">=</mo><mrow id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.cmml"><mrow id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.cmml"><mn id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.2.cmml">1</mn><mo id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.1" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.1.cmml">/</mo><mi id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.3" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.3.cmml">n</mi></mrow><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.1" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.1.cmml">​</mo><mrow id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.cmml"><msubsup id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.cmml"><mo id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.2.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.2.2.cmml">∑</mo><mrow id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.cmml"><mi id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.2.cmml">i</mi><mo id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.1" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.1.cmml">=</mo><mn id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.3" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.3.cmml">1</mn></mrow><mi id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.2.3" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.2.3.cmml">n</mi></msubsup><msub id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.cmml"><mi id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.2.cmml">𝒕</mi><msub id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3.cmml"><mi id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3.2.cmml">x</mi><mi id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3.3" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3.3.cmml">i</mi></msub></msub></mrow></mrow><mo id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.5" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.5.cmml">∈</mo><msup id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.6" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.6.cmml"><mi id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.6.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.6.2.cmml">ℝ</mi><mrow id="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.3" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.3.1" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.2.1.cmml">|</mo><mi id="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.1" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.1.cmml">V</mi><mo stretchy="false" id="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.3.2" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.2.1.cmml">|</mo></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.10.m10.1b"><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2"><and id="S3.SS0.SSS0.Px2.p2.10.m10.1.2a.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2"></and><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2b.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2"><eq id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.3"></eq><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2.2">𝒕</ci><ci id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.2.3">𝑋</ci></apply><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4"><times id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.1"></times><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2"><divide id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.1"></divide><cn type="integer" id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.2">1</cn><ci id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.2.3">𝑛</ci></apply><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3"><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1">subscript</csymbol><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1">superscript</csymbol><sum id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.2.2"></sum><ci id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.2.3">𝑛</ci></apply><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3"><eq id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.1"></eq><ci id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.2">𝑖</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.1.3.3">1</cn></apply></apply><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.2">𝒕</ci><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3.2">𝑥</ci><ci id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3.3.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.3.2.3.3">𝑖</ci></apply></apply></apply></apply></apply><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2c.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2"><in id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.5.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.5"></in><share href="#S3.SS0.SSS0.Px2.p2.10.m10.1.2.4.cmml" id="S3.SS0.SSS0.Px2.p2.10.m10.1.2d.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2"></share><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.6.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.6"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.6.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.6">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p2.10.m10.1.2.6.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.2.6.2">ℝ</ci><apply id="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.3"><abs id="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.3.1"></abs><ci id="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.10.m10.1.1.1.1">𝑉</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.10.m10.1c">{\bm{t}}_{X}=1/n\sum^{n}_{i=1}{\bm{t}}_{x_{i}}\in\mathbb{R}^{|V|}</annotation></semantics></math> as an average of token-level vocabulary distributions for tokens present in the input sequence <math id="S3.SS0.SSS0.Px2.p2.11.m11.1" class="ltx_Math" alttext="X" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.11.m11.1a"><mi id="S3.SS0.SSS0.Px2.p2.11.m11.1.1" xref="S3.SS0.SSS0.Px2.p2.11.m11.1.1.cmml">X</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.11.m11.1b"><ci id="S3.SS0.SSS0.Px2.p2.11.m11.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.11.m11.1.1">𝑋</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.11.m11.1c">X</annotation></semantics></math>, capturing the similarity of <math id="S3.SS0.SSS0.Px2.p2.12.m12.1" class="ltx_Math" alttext="X" display="inline"><semantics id="S3.SS0.SSS0.Px2.p2.12.m12.1a"><mi id="S3.SS0.SSS0.Px2.p2.12.m12.1.1" xref="S3.SS0.SSS0.Px2.p2.12.m12.1.1.cmml">X</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p2.12.m12.1b"><ci id="S3.SS0.SSS0.Px2.p2.12.m12.1.1.cmml" xref="S3.SS0.SSS0.Px2.p2.12.m12.1.1">𝑋</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p2.12.m12.1c">X</annotation></semantics></math> to the tokens in the vocabulary.</p>
</div>
<div id="S3.SS0.SSS0.Px2.p3" class="ltx_para">
<p id="S3.SS0.SSS0.Px2.p3.8" class="ltx_p">Let <math id="S3.SS0.SSS0.Px2.p3.1.m1.1" class="ltx_Math" alttext="{\bm{a}}^{L}_{t}\in\mathbb{R}^{n}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p3.1.m1.1a"><mrow id="S3.SS0.SSS0.Px2.p3.1.m1.1.1" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.cmml"><msubsup id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.cmml"><mi id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.2.2" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.2.2.cmml">𝒂</mi><mi id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.3" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.3.cmml">t</mi><mi id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.2.3" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.2.3.cmml">L</mi></msubsup><mo id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.1" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.1.cmml">∈</mo><msup id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3.2" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3.2.cmml">ℝ</mi><mi id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3.3" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3.3.cmml">n</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p3.1.m1.1b"><apply id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1"><in id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.1"></in><apply id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2">subscript</csymbol><apply id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.2.1.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.2.2.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.2.2">𝒂</ci><ci id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.2.3.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.2.3">𝐿</ci></apply><ci id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.2.3">𝑡</ci></apply><apply id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3.2">ℝ</ci><ci id="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px2.p3.1.m1.1.1.3.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p3.1.m1.1c">{\bm{a}}^{L}_{t}\in\mathbb{R}^{n}</annotation></semantics></math> be the encoder-decoder attention distribution over the source tokens for the output token <math id="S3.SS0.SSS0.Px2.p3.2.m2.1" class="ltx_Math" alttext="y_{t}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p3.2.m2.1a"><msub id="S3.SS0.SSS0.Px2.p3.2.m2.1.1" xref="S3.SS0.SSS0.Px2.p3.2.m2.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p3.2.m2.1.1.2" xref="S3.SS0.SSS0.Px2.p3.2.m2.1.1.2.cmml">y</mi><mi id="S3.SS0.SSS0.Px2.p3.2.m2.1.1.3" xref="S3.SS0.SSS0.Px2.p3.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p3.2.m2.1b"><apply id="S3.SS0.SSS0.Px2.p3.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.2.m2.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p3.2.m2.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p3.2.m2.1.1.2">𝑦</ci><ci id="S3.SS0.SSS0.Px2.p3.2.m2.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p3.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p3.2.m2.1c">y_{t}</annotation></semantics></math> and the final decoder layer <math id="S3.SS0.SSS0.Px2.p3.3.m3.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S3.SS0.SSS0.Px2.p3.3.m3.1a"><mi id="S3.SS0.SSS0.Px2.p3.3.m3.1.1" xref="S3.SS0.SSS0.Px2.p3.3.m3.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p3.3.m3.1b"><ci id="S3.SS0.SSS0.Px2.p3.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.3.m3.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p3.3.m3.1c">L</annotation></semantics></math>. We use <math id="S3.SS0.SSS0.Px2.p3.4.m4.1" class="ltx_Math" alttext="{\bm{a}}^{L}_{t}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p3.4.m4.1a"><msubsup id="S3.SS0.SSS0.Px2.p3.4.m4.1.1" xref="S3.SS0.SSS0.Px2.p3.4.m4.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p3.4.m4.1.1.2.2" xref="S3.SS0.SSS0.Px2.p3.4.m4.1.1.2.2.cmml">𝒂</mi><mi id="S3.SS0.SSS0.Px2.p3.4.m4.1.1.3" xref="S3.SS0.SSS0.Px2.p3.4.m4.1.1.3.cmml">t</mi><mi id="S3.SS0.SSS0.Px2.p3.4.m4.1.1.2.3" xref="S3.SS0.SSS0.Px2.p3.4.m4.1.1.2.3.cmml">L</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p3.4.m4.1b"><apply id="S3.SS0.SSS0.Px2.p3.4.m4.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.4.m4.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.4.m4.1.1">subscript</csymbol><apply id="S3.SS0.SSS0.Px2.p3.4.m4.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.4.m4.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p3.4.m4.1.1">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p3.4.m4.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px2.p3.4.m4.1.1.2.2">𝒂</ci><ci id="S3.SS0.SSS0.Px2.p3.4.m4.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px2.p3.4.m4.1.1.2.3">𝐿</ci></apply><ci id="S3.SS0.SSS0.Px2.p3.4.m4.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p3.4.m4.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p3.4.m4.1c">{\bm{a}}^{L}_{t}</annotation></semantics></math> to produce a weighted sum of the token-level vocabulary distributions to compute a dynamic vocabulary bias, or <em id="S3.SS0.SSS0.Px2.p3.8.1" class="ltx_emph ltx_font_italic">Focus Bias</em> <math id="S3.SS0.SSS0.Px2.p3.5.m5.3" class="ltx_Math" alttext="{\bm{f}}_{t}=\sum^{n}_{i=1}{\bm{a}}^{L}_{t,i}{\bm{t}}_{x_{i}}\in\mathbb{R}^{|V|}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p3.5.m5.3a"><mrow id="S3.SS0.SSS0.Px2.p3.5.m5.3.4" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.cmml"><msub id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2.cmml"><mi id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2.2.cmml">𝒇</mi><mi id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2.3" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2.3.cmml">t</mi></msub><mo rspace="0.111em" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.3" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.3.cmml">=</mo><mrow id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.cmml"><msubsup id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.cmml"><mo id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.2.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.2.2.cmml">∑</mo><mrow id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.cmml"><mi id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.2.cmml">i</mi><mo id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.1" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.1.cmml">=</mo><mn id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.3" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.3.cmml">1</mn></mrow><mi id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.2.3" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.2.3.cmml">n</mi></msubsup><mrow id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.cmml"><msubsup id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.cmml"><mi id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.2.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.2.2.cmml">𝒂</mi><mrow id="S3.SS0.SSS0.Px2.p3.5.m5.2.2.2.4" xref="S3.SS0.SSS0.Px2.p3.5.m5.2.2.2.3.cmml"><mi id="S3.SS0.SSS0.Px2.p3.5.m5.1.1.1.1" xref="S3.SS0.SSS0.Px2.p3.5.m5.1.1.1.1.cmml">t</mi><mo id="S3.SS0.SSS0.Px2.p3.5.m5.2.2.2.4.1" xref="S3.SS0.SSS0.Px2.p3.5.m5.2.2.2.3.cmml">,</mo><mi id="S3.SS0.SSS0.Px2.p3.5.m5.2.2.2.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.2.2.2.2.cmml">i</mi></mrow><mi id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.2.3" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.2.3.cmml">L</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.1" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.1.cmml">​</mo><msub id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.cmml"><mi id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.2.cmml">𝒕</mi><msub id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3.cmml"><mi id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3.2.cmml">x</mi><mi id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3.3" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3.3.cmml">i</mi></msub></msub></mrow></mrow><mo id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.5" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.5.cmml">∈</mo><msup id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.6" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.6.cmml"><mi id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.6.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.6.2.cmml">ℝ</mi><mrow id="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.3" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.3.1" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.2.1.cmml">|</mo><mi id="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.1" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.1.cmml">V</mi><mo stretchy="false" id="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.3.2" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.2.1.cmml">|</mo></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p3.5.m5.3b"><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4"><and id="S3.SS0.SSS0.Px2.p3.5.m5.3.4a.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4"></and><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4b.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4"><eq id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.3.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.3"></eq><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2.2">𝒇</ci><ci id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2.3.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.2.3">𝑡</ci></apply><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4"><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1">subscript</csymbol><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1">superscript</csymbol><sum id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.2.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.2.2"></sum><ci id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.2.3.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.2.3">𝑛</ci></apply><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3"><eq id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.1"></eq><ci id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.2">𝑖</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.3.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.1.3.3">1</cn></apply></apply><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2"><times id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.1"></times><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2">subscript</csymbol><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.2.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.2.2">𝒂</ci><ci id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.2.2.3">𝐿</ci></apply><list id="S3.SS0.SSS0.Px2.p3.5.m5.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.2.2.2.4"><ci id="S3.SS0.SSS0.Px2.p3.5.m5.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.1.1.1.1">𝑡</ci><ci id="S3.SS0.SSS0.Px2.p3.5.m5.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.2.2.2.2">𝑖</ci></list></apply><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.2">𝒕</ci><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3.2">𝑥</ci><ci id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3.3.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.2.3.3.3">𝑖</ci></apply></apply></apply></apply></apply><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4c.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4"><in id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.5.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.5"></in><share href="#S3.SS0.SSS0.Px2.p3.5.m5.3.4.4.cmml" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4d.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4"></share><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.6.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.6"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.6.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.6">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p3.5.m5.3.4.6.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.4.6.2">ℝ</ci><apply id="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.2.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.3"><abs id="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.3.1"></abs><ci id="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.5.m5.3.3.1.1">𝑉</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p3.5.m5.3c">{\bm{f}}_{t}=\sum^{n}_{i=1}{\bm{a}}^{L}_{t,i}{\bm{t}}_{x_{i}}\in\mathbb{R}^{|V|}</annotation></semantics></math> at decoding step <math id="S3.SS0.SSS0.Px2.p3.6.m6.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS0.SSS0.Px2.p3.6.m6.1a"><mi id="S3.SS0.SSS0.Px2.p3.6.m6.1.1" xref="S3.SS0.SSS0.Px2.p3.6.m6.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p3.6.m6.1b"><ci id="S3.SS0.SSS0.Px2.p3.6.m6.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.6.m6.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p3.6.m6.1c">t</annotation></semantics></math>. We modify the probability of predicting the next token <math id="S3.SS0.SSS0.Px2.p3.7.m7.1" class="ltx_Math" alttext="y_{t}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p3.7.m7.1a"><msub id="S3.SS0.SSS0.Px2.p3.7.m7.1.1" xref="S3.SS0.SSS0.Px2.p3.7.m7.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p3.7.m7.1.1.2" xref="S3.SS0.SSS0.Px2.p3.7.m7.1.1.2.cmml">y</mi><mi id="S3.SS0.SSS0.Px2.p3.7.m7.1.1.3" xref="S3.SS0.SSS0.Px2.p3.7.m7.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p3.7.m7.1b"><apply id="S3.SS0.SSS0.Px2.p3.7.m7.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p3.7.m7.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.7.m7.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p3.7.m7.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p3.7.m7.1.1.2">𝑦</ci><ci id="S3.SS0.SSS0.Px2.p3.7.m7.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p3.7.m7.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p3.7.m7.1c">y_{t}</annotation></semantics></math> from a vocabulary <math id="S3.SS0.SSS0.Px2.p3.8.m8.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S3.SS0.SSS0.Px2.p3.8.m8.1a"><mi id="S3.SS0.SSS0.Px2.p3.8.m8.1.1" xref="S3.SS0.SSS0.Px2.p3.8.m8.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p3.8.m8.1b"><ci id="S3.SS0.SSS0.Px2.p3.8.m8.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.8.m8.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p3.8.m8.1c">V</annotation></semantics></math> as:</p>
<table id="S3.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E3.m1.4" class="ltx_Math" alttext="p(y_{t}|Y_{1:t-1},X;\theta)=\mathrm{softmax}({\bm{y}}^{L}_{t}{\bm{E}}+{\bm{f}}_{t})" display="block"><semantics id="S3.E3.m1.4a"><mrow id="S3.E3.m1.4.4" xref="S3.E3.m1.4.4.cmml"><mrow id="S3.E3.m1.3.3.1" xref="S3.E3.m1.3.3.1.cmml"><mi id="S3.E3.m1.3.3.1.3" xref="S3.E3.m1.3.3.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.3.3.1.2" xref="S3.E3.m1.3.3.1.2.cmml">​</mo><mrow id="S3.E3.m1.3.3.1.1.1" xref="S3.E3.m1.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.3.3.1.1.1.2" xref="S3.E3.m1.3.3.1.1.1.1.cmml">(</mo><mrow id="S3.E3.m1.3.3.1.1.1.1" xref="S3.E3.m1.3.3.1.1.1.1.cmml"><msub id="S3.E3.m1.3.3.1.1.1.1.3" xref="S3.E3.m1.3.3.1.1.1.1.3.cmml"><mi id="S3.E3.m1.3.3.1.1.1.1.3.2" xref="S3.E3.m1.3.3.1.1.1.1.3.2.cmml">y</mi><mi id="S3.E3.m1.3.3.1.1.1.1.3.3" xref="S3.E3.m1.3.3.1.1.1.1.3.3.cmml">t</mi></msub><mo fence="false" id="S3.E3.m1.3.3.1.1.1.1.2" xref="S3.E3.m1.3.3.1.1.1.1.2.cmml">|</mo><mrow id="S3.E3.m1.3.3.1.1.1.1.1.1" xref="S3.E3.m1.3.3.1.1.1.1.1.2.cmml"><msub id="S3.E3.m1.3.3.1.1.1.1.1.1.1" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.3.3.1.1.1.1.1.1.1.2" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.2.cmml">Y</mi><mrow id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.cmml"><mn id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.2" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.1" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.1.cmml">:</mo><mrow id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.2" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.2.cmml">t</mi><mo id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.1" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.1.cmml">−</mo><mn id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.3" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.3.cmml">1</mn></mrow></mrow></msub><mo id="S3.E3.m1.3.3.1.1.1.1.1.1.2" xref="S3.E3.m1.3.3.1.1.1.1.1.2.cmml">,</mo><mi id="S3.E3.m1.1.1" xref="S3.E3.m1.1.1.cmml">X</mi><mo id="S3.E3.m1.3.3.1.1.1.1.1.1.3" xref="S3.E3.m1.3.3.1.1.1.1.1.2.cmml">;</mo><mi id="S3.E3.m1.2.2" xref="S3.E3.m1.2.2.cmml">θ</mi></mrow></mrow><mo stretchy="false" id="S3.E3.m1.3.3.1.1.1.3" xref="S3.E3.m1.3.3.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E3.m1.4.4.3" xref="S3.E3.m1.4.4.3.cmml">=</mo><mrow id="S3.E3.m1.4.4.2" xref="S3.E3.m1.4.4.2.cmml"><mi id="S3.E3.m1.4.4.2.3" xref="S3.E3.m1.4.4.2.3.cmml">softmax</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.4.4.2.2" xref="S3.E3.m1.4.4.2.2.cmml">​</mo><mrow id="S3.E3.m1.4.4.2.1.1" xref="S3.E3.m1.4.4.2.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.4.4.2.1.1.2" xref="S3.E3.m1.4.4.2.1.1.1.cmml">(</mo><mrow id="S3.E3.m1.4.4.2.1.1.1" xref="S3.E3.m1.4.4.2.1.1.1.cmml"><mrow id="S3.E3.m1.4.4.2.1.1.1.2" xref="S3.E3.m1.4.4.2.1.1.1.2.cmml"><msubsup id="S3.E3.m1.4.4.2.1.1.1.2.2" xref="S3.E3.m1.4.4.2.1.1.1.2.2.cmml"><mi id="S3.E3.m1.4.4.2.1.1.1.2.2.2.2" xref="S3.E3.m1.4.4.2.1.1.1.2.2.2.2.cmml">𝒚</mi><mi id="S3.E3.m1.4.4.2.1.1.1.2.2.3" xref="S3.E3.m1.4.4.2.1.1.1.2.2.3.cmml">t</mi><mi id="S3.E3.m1.4.4.2.1.1.1.2.2.2.3" xref="S3.E3.m1.4.4.2.1.1.1.2.2.2.3.cmml">L</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.E3.m1.4.4.2.1.1.1.2.1" xref="S3.E3.m1.4.4.2.1.1.1.2.1.cmml">​</mo><mi id="S3.E3.m1.4.4.2.1.1.1.2.3" xref="S3.E3.m1.4.4.2.1.1.1.2.3.cmml">𝑬</mi></mrow><mo id="S3.E3.m1.4.4.2.1.1.1.1" xref="S3.E3.m1.4.4.2.1.1.1.1.cmml">+</mo><msub id="S3.E3.m1.4.4.2.1.1.1.3" xref="S3.E3.m1.4.4.2.1.1.1.3.cmml"><mi id="S3.E3.m1.4.4.2.1.1.1.3.2" xref="S3.E3.m1.4.4.2.1.1.1.3.2.cmml">𝒇</mi><mi id="S3.E3.m1.4.4.2.1.1.1.3.3" xref="S3.E3.m1.4.4.2.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S3.E3.m1.4.4.2.1.1.3" xref="S3.E3.m1.4.4.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.4b"><apply id="S3.E3.m1.4.4.cmml" xref="S3.E3.m1.4.4"><eq id="S3.E3.m1.4.4.3.cmml" xref="S3.E3.m1.4.4.3"></eq><apply id="S3.E3.m1.3.3.1.cmml" xref="S3.E3.m1.3.3.1"><times id="S3.E3.m1.3.3.1.2.cmml" xref="S3.E3.m1.3.3.1.2"></times><ci id="S3.E3.m1.3.3.1.3.cmml" xref="S3.E3.m1.3.3.1.3">𝑝</ci><apply id="S3.E3.m1.3.3.1.1.1.1.cmml" xref="S3.E3.m1.3.3.1.1.1"><csymbol cd="latexml" id="S3.E3.m1.3.3.1.1.1.1.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.2">conditional</csymbol><apply id="S3.E3.m1.3.3.1.1.1.1.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.3.3.1.1.1.1.3.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.3.3.1.1.1.1.3.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.3.2">𝑦</ci><ci id="S3.E3.m1.3.3.1.1.1.1.3.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.3.3">𝑡</ci></apply><list id="S3.E3.m1.3.3.1.1.1.1.1.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1"><apply id="S3.E3.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E3.m1.3.3.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.2">𝑌</ci><apply id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3"><ci id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.1">:</ci><cn type="integer" id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.2">1</cn><apply id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3"><minus id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.1"></minus><ci id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.2">𝑡</ci><cn type="integer" id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.3.3.3">1</cn></apply></apply></apply><ci id="S3.E3.m1.1.1.cmml" xref="S3.E3.m1.1.1">𝑋</ci><ci id="S3.E3.m1.2.2.cmml" xref="S3.E3.m1.2.2">𝜃</ci></list></apply></apply><apply id="S3.E3.m1.4.4.2.cmml" xref="S3.E3.m1.4.4.2"><times id="S3.E3.m1.4.4.2.2.cmml" xref="S3.E3.m1.4.4.2.2"></times><ci id="S3.E3.m1.4.4.2.3.cmml" xref="S3.E3.m1.4.4.2.3">softmax</ci><apply id="S3.E3.m1.4.4.2.1.1.1.cmml" xref="S3.E3.m1.4.4.2.1.1"><plus id="S3.E3.m1.4.4.2.1.1.1.1.cmml" xref="S3.E3.m1.4.4.2.1.1.1.1"></plus><apply id="S3.E3.m1.4.4.2.1.1.1.2.cmml" xref="S3.E3.m1.4.4.2.1.1.1.2"><times id="S3.E3.m1.4.4.2.1.1.1.2.1.cmml" xref="S3.E3.m1.4.4.2.1.1.1.2.1"></times><apply id="S3.E3.m1.4.4.2.1.1.1.2.2.cmml" xref="S3.E3.m1.4.4.2.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.E3.m1.4.4.2.1.1.1.2.2.1.cmml" xref="S3.E3.m1.4.4.2.1.1.1.2.2">subscript</csymbol><apply id="S3.E3.m1.4.4.2.1.1.1.2.2.2.cmml" xref="S3.E3.m1.4.4.2.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.E3.m1.4.4.2.1.1.1.2.2.2.1.cmml" xref="S3.E3.m1.4.4.2.1.1.1.2.2">superscript</csymbol><ci id="S3.E3.m1.4.4.2.1.1.1.2.2.2.2.cmml" xref="S3.E3.m1.4.4.2.1.1.1.2.2.2.2">𝒚</ci><ci id="S3.E3.m1.4.4.2.1.1.1.2.2.2.3.cmml" xref="S3.E3.m1.4.4.2.1.1.1.2.2.2.3">𝐿</ci></apply><ci id="S3.E3.m1.4.4.2.1.1.1.2.2.3.cmml" xref="S3.E3.m1.4.4.2.1.1.1.2.2.3">𝑡</ci></apply><ci id="S3.E3.m1.4.4.2.1.1.1.2.3.cmml" xref="S3.E3.m1.4.4.2.1.1.1.2.3">𝑬</ci></apply><apply id="S3.E3.m1.4.4.2.1.1.1.3.cmml" xref="S3.E3.m1.4.4.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.4.4.2.1.1.1.3.1.cmml" xref="S3.E3.m1.4.4.2.1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.4.4.2.1.1.1.3.2.cmml" xref="S3.E3.m1.4.4.2.1.1.1.3.2">𝒇</ci><ci id="S3.E3.m1.4.4.2.1.1.1.3.3.cmml" xref="S3.E3.m1.4.4.2.1.1.1.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.4c">p(y_{t}|Y_{1:t-1},X;\theta)=\mathrm{softmax}({\bm{y}}^{L}_{t}{\bm{E}}+{\bm{f}}_{t})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.SS0.SSS0.Px2.p3.9" class="ltx_p">We call this <em id="S3.SS0.SSS0.Px2.p3.9.1" class="ltx_emph ltx_font_italic">Focused Probability Distribution</em>, and it modifies the output logits dynamically to put more focus on those tokens in the vocabulary which are similar to the attended tokens in <math id="S3.SS0.SSS0.Px2.p3.9.m1.1" class="ltx_Math" alttext="X" display="inline"><semantics id="S3.SS0.SSS0.Px2.p3.9.m1.1a"><mi id="S3.SS0.SSS0.Px2.p3.9.m1.1.1" xref="S3.SS0.SSS0.Px2.p3.9.m1.1.1.cmml">X</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p3.9.m1.1b"><ci id="S3.SS0.SSS0.Px2.p3.9.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p3.9.m1.1.1">𝑋</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p3.9.m1.1c">X</annotation></semantics></math>. The focus bias introduces a human-inspired control to the model where we do not generate the output in a fully abstractive manner (as in Eq. equation <a href="#S3.E1" title="In Transformer-based seq2seq Model ‣ 3 Summarization with Focus Attention ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>), but we proactively generate output tokens that are similar to the input tokens (as in Eq. equation <a href="#S3.E3" title="In Focus Attention MEchansim (Fame) ‣ 3 Summarization with Focus Attention ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>).</p>
</div>
</section>
<section id="S3.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Summary-induced Topic Focused Distribution</h4>

<div id="S3.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px3.p1.3" class="ltx_p">We aim to guide our focus bias <math id="S3.SS0.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="{\bm{f}}_{t}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.1.m1.1a"><msub id="S3.SS0.SSS0.Px3.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.cmml"><mi id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.2" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.2.cmml">𝒇</mi><mi id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.3" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.1.m1.1b"><apply id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.2">𝒇</ci><ci id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.1.m1.1c">{\bm{f}}_{t}</annotation></semantics></math> to be a better representative of the topical content relevant for the task. We achieve this by using the human-written summary <math id="S3.SS0.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="\hat{Y}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.2.m2.1a"><mover accent="true" id="S3.SS0.SSS0.Px3.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.cmml"><mi id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.2" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.2.cmml">Y</mi><mo id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.1" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.2.m2.1b"><apply id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1"><ci id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.1">^</ci><ci id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.2">𝑌</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.2.m2.1c">\hat{Y}</annotation></semantics></math> as a proxy for the topical content of the input and impose the following prior on the source-conditioned vocabulary distribution <math id="S3.SS0.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="{\bm{t}}_{X}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.3.m3.1a"><msub id="S3.SS0.SSS0.Px3.p1.3.m3.1.1" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.cmml"><mi id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.2" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.2.cmml">𝒕</mi><mi id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.3" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.3.cmml">X</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.3.m3.1b"><apply id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.2">𝒕</ci><ci id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.3.cmml" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.3.m3.1c">{\bm{t}}_{X}</annotation></semantics></math>:</p>
<table id="A5.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E4.m1.1" class="ltx_Math" alttext="\displaystyle L_{\mathrm{Topic}}(\theta)=-" display="inline"><semantics id="S3.E4.m1.1a"><mrow id="S3.E4.m1.1.2" xref="S3.E4.m1.1.2.cmml"><mrow id="S3.E4.m1.1.2.2" xref="S3.E4.m1.1.2.2.cmml"><msub id="S3.E4.m1.1.2.2.2" xref="S3.E4.m1.1.2.2.2.cmml"><mi id="S3.E4.m1.1.2.2.2.2" xref="S3.E4.m1.1.2.2.2.2.cmml">L</mi><mi id="S3.E4.m1.1.2.2.2.3" xref="S3.E4.m1.1.2.2.2.3.cmml">Topic</mi></msub><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.2.2.1" xref="S3.E4.m1.1.2.2.1.cmml">​</mo><mrow id="S3.E4.m1.1.2.2.3.2" xref="S3.E4.m1.1.2.2.cmml"><mo stretchy="false" id="S3.E4.m1.1.2.2.3.2.1" xref="S3.E4.m1.1.2.2.cmml">(</mo><mi id="S3.E4.m1.1.1" xref="S3.E4.m1.1.1.cmml">θ</mi><mo stretchy="false" id="S3.E4.m1.1.2.2.3.2.2" xref="S3.E4.m1.1.2.2.cmml">)</mo></mrow></mrow><mo rspace="0em" id="S3.E4.m1.1.2.1" xref="S3.E4.m1.1.2.1.cmml">=</mo><mo lspace="0em" id="S3.E4.m1.1.2.3" xref="S3.E4.m1.1.2.3.cmml">−</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.1b"><apply id="S3.E4.m1.1.2.cmml" xref="S3.E4.m1.1.2"><eq id="S3.E4.m1.1.2.1.cmml" xref="S3.E4.m1.1.2.1"></eq><apply id="S3.E4.m1.1.2.2.cmml" xref="S3.E4.m1.1.2.2"><times id="S3.E4.m1.1.2.2.1.cmml" xref="S3.E4.m1.1.2.2.1"></times><apply id="S3.E4.m1.1.2.2.2.cmml" xref="S3.E4.m1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E4.m1.1.2.2.2.1.cmml" xref="S3.E4.m1.1.2.2.2">subscript</csymbol><ci id="S3.E4.m1.1.2.2.2.2.cmml" xref="S3.E4.m1.1.2.2.2.2">𝐿</ci><ci id="S3.E4.m1.1.2.2.2.3.cmml" xref="S3.E4.m1.1.2.2.2.3">Topic</ci></apply><ci id="S3.E4.m1.1.1.cmml" xref="S3.E4.m1.1.1">𝜃</ci></apply><minus id="S3.E4.m1.1.2.3.cmml" xref="S3.E4.m1.1.2.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.1c">\displaystyle L_{\mathrm{Topic}}(\theta)=-</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E4.m2.5" class="ltx_math_unparsed" alttext="\displaystyle\frac{1}{|V|}\sum^{|V|}_{i=1}([v_{i}\in\hat{Y}]\log(\sigma({\bm{t}}_{X,i}))" display="inline"><semantics id="S3.E4.m2.5a"><mrow id="S3.E4.m2.5b"><mstyle displaystyle="true" id="S3.E4.m2.1.1"><mfrac id="S3.E4.m2.1.1a"><mn id="S3.E4.m2.1.1.3">1</mn><mrow id="S3.E4.m2.1.1.1.3"><mo stretchy="false" id="S3.E4.m2.1.1.1.3.1">|</mo><mi id="S3.E4.m2.1.1.1.1">V</mi><mo stretchy="false" id="S3.E4.m2.1.1.1.3.2">|</mo></mrow></mfrac></mstyle><mstyle displaystyle="true" id="S3.E4.m2.5.6"><munderover id="S3.E4.m2.5.6a"><mo movablelimits="false" id="S3.E4.m2.5.6.2.2">∑</mo><mrow id="S3.E4.m2.5.6.3"><mi id="S3.E4.m2.5.6.3.2">i</mi><mo id="S3.E4.m2.5.6.3.1">=</mo><mn id="S3.E4.m2.5.6.3.3">1</mn></mrow><mrow id="S3.E4.m2.2.2.1.3"><mo stretchy="false" id="S3.E4.m2.2.2.1.3.1">|</mo><mi id="S3.E4.m2.2.2.1.1">V</mi><mo stretchy="false" id="S3.E4.m2.2.2.1.3.2">|</mo></mrow></munderover></mstyle><mrow id="S3.E4.m2.5.7"><mo stretchy="false" id="S3.E4.m2.5.7.1">(</mo><mrow id="S3.E4.m2.5.7.2"><mo stretchy="false" id="S3.E4.m2.5.7.2.1">[</mo><msub id="S3.E4.m2.5.7.2.2"><mi id="S3.E4.m2.5.7.2.2.2">v</mi><mi id="S3.E4.m2.5.7.2.2.3">i</mi></msub><mo id="S3.E4.m2.5.7.2.3">∈</mo><mover accent="true" id="S3.E4.m2.5.7.2.4"><mi id="S3.E4.m2.5.7.2.4.2">Y</mi><mo id="S3.E4.m2.5.7.2.4.1">^</mo></mover><mo rspace="0.167em" stretchy="false" id="S3.E4.m2.5.7.2.5">]</mo></mrow><mi id="S3.E4.m2.5.5">log</mi><mrow id="S3.E4.m2.5.7.3"><mo stretchy="false" id="S3.E4.m2.5.7.3.1">(</mo><mi id="S3.E4.m2.5.7.3.2">σ</mi><mrow id="S3.E4.m2.5.7.3.3"><mo stretchy="false" id="S3.E4.m2.5.7.3.3.1">(</mo><msub id="S3.E4.m2.5.7.3.3.2"><mi id="S3.E4.m2.5.7.3.3.2.2">𝒕</mi><mrow id="S3.E4.m2.4.4.2.4"><mi id="S3.E4.m2.3.3.1.1">X</mi><mo id="S3.E4.m2.4.4.2.4.1">,</mo><mi id="S3.E4.m2.4.4.2.2">i</mi></mrow></msub><mo stretchy="false" id="S3.E4.m2.5.7.3.3.3">)</mo></mrow><mo stretchy="false" id="S3.E4.m2.5.7.3.4">)</mo></mrow></mrow></mrow><annotation encoding="application/x-tex" id="S3.E4.m2.5c">\displaystyle\frac{1}{|V|}\sum^{|V|}_{i=1}([v_{i}\in\hat{Y}]\log(\sigma({\bm{t}}_{X,i}))</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
<tbody id="S3.E5"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E5.m1.3" class="ltx_math_unparsed" alttext="\displaystyle+[v_{i}\notin\hat{Y}]\log(1-\sigma({\bm{t}}_{X,i})))." display="inline"><semantics id="S3.E5.m1.3a"><mrow id="S3.E5.m1.3b"><mo id="S3.E5.m1.3.4">+</mo><mrow id="S3.E5.m1.3.5"><mo stretchy="false" id="S3.E5.m1.3.5.1">[</mo><msub id="S3.E5.m1.3.5.2"><mi id="S3.E5.m1.3.5.2.2">v</mi><mi id="S3.E5.m1.3.5.2.3">i</mi></msub><mo id="S3.E5.m1.3.5.3">∉</mo><mover accent="true" id="S3.E5.m1.3.5.4"><mi id="S3.E5.m1.3.5.4.2">Y</mi><mo id="S3.E5.m1.3.5.4.1">^</mo></mover><mo rspace="0.167em" stretchy="false" id="S3.E5.m1.3.5.5">]</mo></mrow><mi id="S3.E5.m1.3.3">log</mi><mrow id="S3.E5.m1.3.6"><mo stretchy="false" id="S3.E5.m1.3.6.1">(</mo><mn id="S3.E5.m1.3.6.2">1</mn><mo id="S3.E5.m1.3.6.3">−</mo><mi id="S3.E5.m1.3.6.4">σ</mi><mrow id="S3.E5.m1.3.6.5"><mo stretchy="false" id="S3.E5.m1.3.6.5.1">(</mo><msub id="S3.E5.m1.3.6.5.2"><mi id="S3.E5.m1.3.6.5.2.2">𝒕</mi><mrow id="S3.E5.m1.2.2.2.4"><mi id="S3.E5.m1.1.1.1.1">X</mi><mo id="S3.E5.m1.2.2.2.4.1">,</mo><mi id="S3.E5.m1.2.2.2.2">i</mi></mrow></msub><mo stretchy="false" id="S3.E5.m1.3.6.5.3">)</mo></mrow><mo stretchy="false" id="S3.E5.m1.3.6.6">)</mo></mrow><mo stretchy="false" id="S3.E5.m1.3.7">)</mo><mo lspace="0em" id="S3.E5.m1.3.8">.</mo></mrow><annotation encoding="application/x-tex" id="S3.E5.m1.3c">\displaystyle+[v_{i}\notin\hat{Y}]\log(1-\sigma({\bm{t}}_{X,i}))).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="S3.SS0.SSS0.Px3.p1.7" class="ltx_p">We further refine Eq. equation <a href="#S3.E5" title="In Summary-induced Topic Focused Distribution ‣ 3 Summarization with Focus Attention ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> by replacing <math id="S3.SS0.SSS0.Px3.p1.4.m1.1" class="ltx_Math" alttext="\hat{Y}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.4.m1.1a"><mover accent="true" id="S3.SS0.SSS0.Px3.p1.4.m1.1.1" xref="S3.SS0.SSS0.Px3.p1.4.m1.1.1.cmml"><mi id="S3.SS0.SSS0.Px3.p1.4.m1.1.1.2" xref="S3.SS0.SSS0.Px3.p1.4.m1.1.1.2.cmml">Y</mi><mo id="S3.SS0.SSS0.Px3.p1.4.m1.1.1.1" xref="S3.SS0.SSS0.Px3.p1.4.m1.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.4.m1.1b"><apply id="S3.SS0.SSS0.Px3.p1.4.m1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.4.m1.1.1"><ci id="S3.SS0.SSS0.Px3.p1.4.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.4.m1.1.1.1">^</ci><ci id="S3.SS0.SSS0.Px3.p1.4.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p1.4.m1.1.1.2">𝑌</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.4.m1.1c">\hat{Y}</annotation></semantics></math> with <math id="S3.SS0.SSS0.Px3.p1.5.m2.1" class="ltx_Math" alttext="\hat{Y}_{c}=\hat{Y}-F" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.5.m2.1a"><mrow id="S3.SS0.SSS0.Px3.p1.5.m2.1.1" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.cmml"><msub id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.cmml"><mover accent="true" id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2.cmml"><mi id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2.2" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2.2.cmml">Y</mi><mo id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2.1" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2.1.cmml">^</mo></mover><mi id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.3" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.3.cmml">c</mi></msub><mo id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.1" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.1.cmml">=</mo><mrow id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.cmml"><mover accent="true" id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2.cmml"><mi id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2.2" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2.2.cmml">Y</mi><mo id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2.1" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2.1.cmml">^</mo></mover><mo id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.1" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.1.cmml">−</mo><mi id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.3" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.3.cmml">F</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.5.m2.1b"><apply id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1"><eq id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.1"></eq><apply id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2">subscript</csymbol><apply id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2"><ci id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2.1.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2.1">^</ci><ci id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2.2.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.2.2">𝑌</ci></apply><ci id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.2.3">𝑐</ci></apply><apply id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3"><minus id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.1"></minus><apply id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2"><ci id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2.1.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2.1">^</ci><ci id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2.2.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.2.2">𝑌</ci></apply><ci id="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m2.1.1.3.3">𝐹</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.5.m2.1c">\hat{Y}_{c}=\hat{Y}-F</annotation></semantics></math>, where <math id="S3.SS0.SSS0.Px3.p1.6.m3.1" class="ltx_Math" alttext="F" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.6.m3.1a"><mi id="S3.SS0.SSS0.Px3.p1.6.m3.1.1" xref="S3.SS0.SSS0.Px3.p1.6.m3.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.6.m3.1b"><ci id="S3.SS0.SSS0.Px3.p1.6.m3.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.6.m3.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.6.m3.1c">F</annotation></semantics></math> is a set of <math id="S3.SS0.SSS0.Px3.p1.7.m4.1" class="ltx_Math" alttext="|F|" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.7.m4.1a"><mrow id="S3.SS0.SSS0.Px3.p1.7.m4.1.2.2" xref="S3.SS0.SSS0.Px3.p1.7.m4.1.2.1.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px3.p1.7.m4.1.2.2.1" xref="S3.SS0.SSS0.Px3.p1.7.m4.1.2.1.1.cmml">|</mo><mi id="S3.SS0.SSS0.Px3.p1.7.m4.1.1" xref="S3.SS0.SSS0.Px3.p1.7.m4.1.1.cmml">F</mi><mo stretchy="false" id="S3.SS0.SSS0.Px3.p1.7.m4.1.2.2.2" xref="S3.SS0.SSS0.Px3.p1.7.m4.1.2.1.1.cmml">|</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.7.m4.1b"><apply id="S3.SS0.SSS0.Px3.p1.7.m4.1.2.1.cmml" xref="S3.SS0.SSS0.Px3.p1.7.m4.1.2.2"><abs id="S3.SS0.SSS0.Px3.p1.7.m4.1.2.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.7.m4.1.2.2.1"></abs><ci id="S3.SS0.SSS0.Px3.p1.7.m4.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.7.m4.1.1">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.7.m4.1c">|F|</annotation></semantics></math> most frequent tokens in the vocabulary,<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>which are usually articles or other function words.</span></span></span> to improve focus on
content words. Our final loss function is then</p>
<table id="S3.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E6.m1.1" class="ltx_Math" alttext="L=\lambda L_{\mathrm{MLE}}+(1-\lambda)L_{\mathrm{Topic}}," display="block"><semantics id="S3.E6.m1.1a"><mrow id="S3.E6.m1.1.1.1" xref="S3.E6.m1.1.1.1.1.cmml"><mrow id="S3.E6.m1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.cmml"><mi id="S3.E6.m1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.3.cmml">L</mi><mo id="S3.E6.m1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.2.cmml">=</mo><mrow id="S3.E6.m1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.cmml"><mrow id="S3.E6.m1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.3.cmml"><mi id="S3.E6.m1.1.1.1.1.1.3.2" xref="S3.E6.m1.1.1.1.1.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.3.1" xref="S3.E6.m1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S3.E6.m1.1.1.1.1.1.3.3" xref="S3.E6.m1.1.1.1.1.1.3.3.cmml"><mi id="S3.E6.m1.1.1.1.1.1.3.3.2" xref="S3.E6.m1.1.1.1.1.1.3.3.2.cmml">L</mi><mi id="S3.E6.m1.1.1.1.1.1.3.3.3" xref="S3.E6.m1.1.1.1.1.1.3.3.3.cmml">MLE</mi></msub></mrow><mo id="S3.E6.m1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.2.cmml">+</mo><mrow id="S3.E6.m1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.cmml"><mrow id="S3.E6.m1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.1.1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E6.m1.1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml"><mn id="S3.E6.m1.1.1.1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.cmml">1</mn><mo id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mi id="S3.E6.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.3.cmml">λ</mi></mrow><mo stretchy="false" id="S3.E6.m1.1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.2.cmml">​</mo><msub id="S3.E6.m1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.3.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.3.2" xref="S3.E6.m1.1.1.1.1.1.1.3.2.cmml">L</mi><mi id="S3.E6.m1.1.1.1.1.1.1.3.3" xref="S3.E6.m1.1.1.1.1.1.1.3.3.cmml">Topic</mi></msub></mrow></mrow></mrow><mo id="S3.E6.m1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E6.m1.1b"><apply id="S3.E6.m1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1"><eq id="S3.E6.m1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.2"></eq><ci id="S3.E6.m1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.3">𝐿</ci><apply id="S3.E6.m1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1"><plus id="S3.E6.m1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.2"></plus><apply id="S3.E6.m1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.3"><times id="S3.E6.m1.1.1.1.1.1.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.3.1"></times><ci id="S3.E6.m1.1.1.1.1.1.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.3.2">𝜆</ci><apply id="S3.E6.m1.1.1.1.1.1.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.3.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.3.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.3.3.2">𝐿</ci><ci id="S3.E6.m1.1.1.1.1.1.3.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.3.3.3">MLE</ci></apply></apply><apply id="S3.E6.m1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1"><times id="S3.E6.m1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2"></times><apply id="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1"><minus id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1"></minus><cn type="integer" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.2">1</cn><ci id="S3.E6.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.3">𝜆</ci></apply><apply id="S3.E6.m1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.2">𝐿</ci><ci id="S3.E6.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.3">Topic</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6.m1.1c">L=\lambda L_{\mathrm{MLE}}+(1-\lambda)L_{\mathrm{Topic}},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="S3.SS0.SSS0.Px3.p1.8" class="ltx_p">where, <math id="S3.SS0.SSS0.Px3.p1.8.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.8.m1.1a"><mi id="S3.SS0.SSS0.Px3.p1.8.m1.1.1" xref="S3.SS0.SSS0.Px3.p1.8.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.8.m1.1b"><ci id="S3.SS0.SSS0.Px3.p1.8.m1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.8.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.8.m1.1c">\lambda</annotation></semantics></math> is an hyper parameter.<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span><math id="footnote3.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="footnote3.m1.1b"><mi id="footnote3.m1.1.1" xref="footnote3.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="footnote3.m1.1c"><ci id="footnote3.m1.1.1.cmml" xref="footnote3.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote3.m1.1d">\lambda</annotation></semantics></math> is set to 0.5 for all experiments.</span></span></span></p>
</div>
<div id="S3.SS0.SSS0.Px3.p2" class="ltx_para">
<p id="S3.SS0.SSS0.Px3.p2.4" class="ltx_p">By enforcing <math id="S3.SS0.SSS0.Px3.p2.1.m1.1" class="ltx_Math" alttext="{\bm{t}}_{X}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p2.1.m1.1a"><msub id="S3.SS0.SSS0.Px3.p2.1.m1.1.1" xref="S3.SS0.SSS0.Px3.p2.1.m1.1.1.cmml"><mi id="S3.SS0.SSS0.Px3.p2.1.m1.1.1.2" xref="S3.SS0.SSS0.Px3.p2.1.m1.1.1.2.cmml">𝒕</mi><mi id="S3.SS0.SSS0.Px3.p2.1.m1.1.1.3" xref="S3.SS0.SSS0.Px3.p2.1.m1.1.1.3.cmml">X</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p2.1.m1.1b"><apply id="S3.SS0.SSS0.Px3.p2.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px3.p2.1.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p2.1.m1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px3.p2.1.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p2.1.m1.1.1.2">𝒕</ci><ci id="S3.SS0.SSS0.Px3.p2.1.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px3.p2.1.m1.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p2.1.m1.1c">{\bm{t}}_{X}</annotation></semantics></math> to be a topic distribution for the input <math id="S3.SS0.SSS0.Px3.p2.2.m2.1" class="ltx_Math" alttext="X" display="inline"><semantics id="S3.SS0.SSS0.Px3.p2.2.m2.1a"><mi id="S3.SS0.SSS0.Px3.p2.2.m2.1.1" xref="S3.SS0.SSS0.Px3.p2.2.m2.1.1.cmml">X</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p2.2.m2.1b"><ci id="S3.SS0.SSS0.Px3.p2.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px3.p2.2.m2.1.1">𝑋</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p2.2.m2.1c">X</annotation></semantics></math>, we encourage the focus bias <math id="S3.SS0.SSS0.Px3.p2.3.m3.1" class="ltx_Math" alttext="{\bm{f}}_{t}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p2.3.m3.1a"><msub id="S3.SS0.SSS0.Px3.p2.3.m3.1.1" xref="S3.SS0.SSS0.Px3.p2.3.m3.1.1.cmml"><mi id="S3.SS0.SSS0.Px3.p2.3.m3.1.1.2" xref="S3.SS0.SSS0.Px3.p2.3.m3.1.1.2.cmml">𝒇</mi><mi id="S3.SS0.SSS0.Px3.p2.3.m3.1.1.3" xref="S3.SS0.SSS0.Px3.p2.3.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p2.3.m3.1b"><apply id="S3.SS0.SSS0.Px3.p2.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px3.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px3.p2.3.m3.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p2.3.m3.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px3.p2.3.m3.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p2.3.m3.1.1.2">𝒇</ci><ci id="S3.SS0.SSS0.Px3.p2.3.m3.1.1.3.cmml" xref="S3.SS0.SSS0.Px3.p2.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p2.3.m3.1c">{\bm{f}}_{t}</annotation></semantics></math> to promote topically relevant tokens, and subsequently generate topically consistent outputs. Importantly, our focus bias with target-induced topic distribution is task-agnostic and less vulnerable to reference divergence issues <cite class="ltx_cite ltx_citemacro_cite">Dhingra et al. (<a href="#bib.bib14" title="" class="ltx_ref">2019</a>); Maynez et al. (<a href="#bib.bib52" title="" class="ltx_ref">2020</a>)</cite>, and can learn any property embodied in the target relevant for the task. For example, depending on the task, <math id="S3.SS0.SSS0.Px3.p2.4.m4.1" class="ltx_Math" alttext="{\bm{f}}_{t}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p2.4.m4.1a"><msub id="S3.SS0.SSS0.Px3.p2.4.m4.1.1" xref="S3.SS0.SSS0.Px3.p2.4.m4.1.1.cmml"><mi id="S3.SS0.SSS0.Px3.p2.4.m4.1.1.2" xref="S3.SS0.SSS0.Px3.p2.4.m4.1.1.2.cmml">𝒇</mi><mi id="S3.SS0.SSS0.Px3.p2.4.m4.1.1.3" xref="S3.SS0.SSS0.Px3.p2.4.m4.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p2.4.m4.1b"><apply id="S3.SS0.SSS0.Px3.p2.4.m4.1.1.cmml" xref="S3.SS0.SSS0.Px3.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px3.p2.4.m4.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p2.4.m4.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px3.p2.4.m4.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p2.4.m4.1.1.2">𝒇</ci><ci id="S3.SS0.SSS0.Px3.p2.4.m4.1.1.3.cmml" xref="S3.SS0.SSS0.Px3.p2.4.m4.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p2.4.m4.1c">{\bm{f}}_{t}</annotation></semantics></math> can learn to favour input tokens (e.g., for mostly extractive summaries) or new tokens (e.g., for mostly abstractive summaries). This is in sharp contrast to models that introduce task-specific priors, e.g., the pointer-generator network <cite class="ltx_cite ltx_citemacro_cite">See et al. (<a href="#bib.bib65" title="" class="ltx_ref">2017</a>)</cite> that can copy words from the source text, but does not do well on extreme summarization which is highly abstractive in nature <cite class="ltx_cite ltx_citemacro_cite">Narayan et al. (<a href="#bib.bib54" title="" class="ltx_ref">2018</a>)</cite>.</p>
</div>
</section>
<section id="S3.SS0.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Focus Sampling: Promoting Diversity in Faithful Generation</h4>

<div id="S3.SS0.SSS0.Px4.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px4.p1.4" class="ltx_p">We introduce <em id="S3.SS0.SSS0.Px4.p1.4.1" class="ltx_emph ltx_font_italic">Focus Sampling</em> with <span id="S3.SS0.SSS0.Px4.p1.4.2" class="ltx_text ltx_font_smallcaps">Fame</span> to construct a subset <math id="S3.SS0.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="V_{k}\subseteq V" display="inline"><semantics id="S3.SS0.SSS0.Px4.p1.1.m1.1a"><mrow id="S3.SS0.SSS0.Px4.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.cmml"><msub id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2.cmml"><mi id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2.2" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2.2.cmml">V</mi><mi id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2.3" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2.3.cmml">k</mi></msub><mo id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.1" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.1.cmml">⊆</mo><mi id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.3" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.3.cmml">V</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p1.1.m1.1b"><apply id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1"><subset id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.1"></subset><apply id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2.2">𝑉</ci><ci id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.2.3">𝑘</ci></apply><ci id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.3">𝑉</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p1.1.m1.1c">V_{k}\subseteq V</annotation></semantics></math> by
sampling <math id="S3.SS0.SSS0.Px4.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS0.SSS0.Px4.p1.2.m2.1a"><mi id="S3.SS0.SSS0.Px4.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px4.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p1.2.m2.1b"><ci id="S3.SS0.SSS0.Px4.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p1.2.m2.1c">k</annotation></semantics></math> tokens from the topic distribution <math id="S3.SS0.SSS0.Px4.p1.3.m3.1" class="ltx_Math" alttext="{\bm{t}}_{X}" display="inline"><semantics id="S3.SS0.SSS0.Px4.p1.3.m3.1a"><msub id="S3.SS0.SSS0.Px4.p1.3.m3.1.1" xref="S3.SS0.SSS0.Px4.p1.3.m3.1.1.cmml"><mi id="S3.SS0.SSS0.Px4.p1.3.m3.1.1.2" xref="S3.SS0.SSS0.Px4.p1.3.m3.1.1.2.cmml">𝒕</mi><mi id="S3.SS0.SSS0.Px4.p1.3.m3.1.1.3" xref="S3.SS0.SSS0.Px4.p1.3.m3.1.1.3.cmml">X</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p1.3.m3.1b"><apply id="S3.SS0.SSS0.Px4.p1.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px4.p1.3.m3.1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px4.p1.3.m3.1.1.2.cmml" xref="S3.SS0.SSS0.Px4.p1.3.m3.1.1.2">𝒕</ci><ci id="S3.SS0.SSS0.Px4.p1.3.m3.1.1.3.cmml" xref="S3.SS0.SSS0.Px4.p1.3.m3.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p1.3.m3.1c">{\bm{t}}_{X}</annotation></semantics></math> (<math id="S3.SS0.SSS0.Px4.p1.4.m4.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S3.SS0.SSS0.Px4.p1.4.m4.2a"><msub id="S3.SS0.SSS0.Px4.p1.4.m4.2.3" xref="S3.SS0.SSS0.Px4.p1.4.m4.2.3.cmml"><mi id="S3.SS0.SSS0.Px4.p1.4.m4.2.3.2" xref="S3.SS0.SSS0.Px4.p1.4.m4.2.3.2.cmml">Focus</mi><mrow id="S3.SS0.SSS0.Px4.p1.4.m4.2.2.2.4" xref="S3.SS0.SSS0.Px4.p1.4.m4.2.2.2.3.cmml"><mi id="S3.SS0.SSS0.Px4.p1.4.m4.1.1.1.1" xref="S3.SS0.SSS0.Px4.p1.4.m4.1.1.1.1.cmml">sample</mi><mo id="S3.SS0.SSS0.Px4.p1.4.m4.2.2.2.4.1" xref="S3.SS0.SSS0.Px4.p1.4.m4.2.2.2.3.cmml">,</mo><mi id="S3.SS0.SSS0.Px4.p1.4.m4.2.2.2.2" xref="S3.SS0.SSS0.Px4.p1.4.m4.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p1.4.m4.2b"><apply id="S3.SS0.SSS0.Px4.p1.4.m4.2.3.cmml" xref="S3.SS0.SSS0.Px4.p1.4.m4.2.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px4.p1.4.m4.2.3.1.cmml" xref="S3.SS0.SSS0.Px4.p1.4.m4.2.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px4.p1.4.m4.2.3.2.cmml" xref="S3.SS0.SSS0.Px4.p1.4.m4.2.3.2">Focus</ci><list id="S3.SS0.SSS0.Px4.p1.4.m4.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px4.p1.4.m4.2.2.2.4"><ci id="S3.SS0.SSS0.Px4.p1.4.m4.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.4.m4.1.1.1.1">sample</ci><ci id="S3.SS0.SSS0.Px4.p1.4.m4.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px4.p1.4.m4.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p1.4.m4.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>).
Then, we modify Eq. equation <a href="#S3.E3" title="In Focus Attention MEchansim (Fame) ‣ 3 Summarization with Focus Attention ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> as</p>
<table id="A5.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E7"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E7.m1.1" class="ltx_math_unparsed" alttext="\displaystyle p(y_{t}|" display="inline"><semantics id="S3.E7.m1.1a"><mrow id="S3.E7.m1.1b"><mi id="S3.E7.m1.1.1">p</mi><mrow id="S3.E7.m1.1.2"><mo stretchy="false" id="S3.E7.m1.1.2.1">(</mo><msub id="S3.E7.m1.1.2.2"><mi id="S3.E7.m1.1.2.2.2">y</mi><mi id="S3.E7.m1.1.2.2.3">t</mi></msub><mo fence="false" stretchy="false" id="S3.E7.m1.1.2.3">|</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.E7.m1.1c">\displaystyle p(y_{t}|</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E7.m2.2" class="ltx_math_unparsed" alttext="\displaystyle Y_{1:t-1},X;\theta)=" display="inline"><semantics id="S3.E7.m2.2a"><mrow id="S3.E7.m2.2b"><msub id="S3.E7.m2.2.3"><mi id="S3.E7.m2.2.3.2">Y</mi><mrow id="S3.E7.m2.2.3.3"><mn id="S3.E7.m2.2.3.3.2">1</mn><mo lspace="0.278em" rspace="0.278em" id="S3.E7.m2.2.3.3.1">:</mo><mrow id="S3.E7.m2.2.3.3.3"><mi id="S3.E7.m2.2.3.3.3.2">t</mi><mo id="S3.E7.m2.2.3.3.3.1">−</mo><mn id="S3.E7.m2.2.3.3.3.3">1</mn></mrow></mrow></msub><mo id="S3.E7.m2.2.4">,</mo><mi id="S3.E7.m2.1.1">X</mi><mo id="S3.E7.m2.2.5">;</mo><mi id="S3.E7.m2.2.2">θ</mi><mo stretchy="false" id="S3.E7.m2.2.6">)</mo><mo id="S3.E7.m2.2.7">=</mo></mrow><annotation encoding="application/x-tex" id="S3.E7.m2.2c">\displaystyle Y_{1:t-1},X;\theta)=</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
<tbody id="S3.E8"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E8.m1.4" class="ltx_Math" alttext="\displaystyle\begin{cases}\mathrm{softmax}({\bm{y}}^{L}_{t}{\bm{E}}+f_{t})_{i}&amp;\text{if }v_{i}\in V_{k}\cup F\\
0,&amp;\text{otherwise.}\end{cases}" display="inline"><semantics id="S3.E8.m1.4a"><mrow id="S3.E8.m1.4.4a" xref="S3.E8.m1.4.5.1.cmml"><mo id="S3.E8.m1.4.4a.5" xref="S3.E8.m1.4.5.1.1.cmml">{</mo><mtable columnspacing="5pt" rowspacing="0pt" id="S3.E8.m1.4.4.4a" xref="S3.E8.m1.4.5.1.cmml"><mtr id="S3.E8.m1.4.4.4aa" xref="S3.E8.m1.4.5.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="S3.E8.m1.4.4.4ab" xref="S3.E8.m1.4.5.1.cmml"><mrow id="S3.E8.m1.1.1.1.1.1.1" xref="S3.E8.m1.1.1.1.1.1.1.cmml"><mi id="S3.E8.m1.1.1.1.1.1.1.3" xref="S3.E8.m1.1.1.1.1.1.1.3.cmml">softmax</mi><mo lspace="0em" rspace="0em" id="S3.E8.m1.1.1.1.1.1.1.2" xref="S3.E8.m1.1.1.1.1.1.1.2.cmml">​</mo><msub id="S3.E8.m1.1.1.1.1.1.1.1" xref="S3.E8.m1.1.1.1.1.1.1.1.cmml"><mrow id="S3.E8.m1.1.1.1.1.1.1.1.1.1" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E8.m1.1.1.1.1.1.1.1.1.1.2" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.cmml"><msubsup id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.cmml"><mi id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.2" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml">𝒚</mi><mi id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.3" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">t</mi><mi id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.3" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.3.cmml">L</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">𝑬</mi></mrow><mo id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">+</mo><msub id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">f</mi><mi id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3.3" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S3.E8.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="S3.E8.m1.1.1.1.1.1.1.1.3" xref="S3.E8.m1.1.1.1.1.1.1.1.3.cmml">i</mi></msub></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S3.E8.m1.4.4.4ac" xref="S3.E8.m1.4.5.1.cmml"><mrow id="S3.E8.m1.2.2.2.2.2.1" xref="S3.E8.m1.2.2.2.2.2.1.cmml"><mrow id="S3.E8.m1.2.2.2.2.2.1.2" xref="S3.E8.m1.2.2.2.2.2.1.2.cmml"><mtext id="S3.E8.m1.2.2.2.2.2.1.2.2" xref="S3.E8.m1.2.2.2.2.2.1.2.2a.cmml">if </mtext><mo lspace="0em" rspace="0em" id="S3.E8.m1.2.2.2.2.2.1.2.1" xref="S3.E8.m1.2.2.2.2.2.1.2.1.cmml">​</mo><msub id="S3.E8.m1.2.2.2.2.2.1.2.3" xref="S3.E8.m1.2.2.2.2.2.1.2.3.cmml"><mi id="S3.E8.m1.2.2.2.2.2.1.2.3.2" xref="S3.E8.m1.2.2.2.2.2.1.2.3.2.cmml">v</mi><mi id="S3.E8.m1.2.2.2.2.2.1.2.3.3" xref="S3.E8.m1.2.2.2.2.2.1.2.3.3.cmml">i</mi></msub></mrow><mo id="S3.E8.m1.2.2.2.2.2.1.1" xref="S3.E8.m1.2.2.2.2.2.1.1.cmml">∈</mo><mrow id="S3.E8.m1.2.2.2.2.2.1.3" xref="S3.E8.m1.2.2.2.2.2.1.3.cmml"><msub id="S3.E8.m1.2.2.2.2.2.1.3.2" xref="S3.E8.m1.2.2.2.2.2.1.3.2.cmml"><mi id="S3.E8.m1.2.2.2.2.2.1.3.2.2" xref="S3.E8.m1.2.2.2.2.2.1.3.2.2.cmml">V</mi><mi id="S3.E8.m1.2.2.2.2.2.1.3.2.3" xref="S3.E8.m1.2.2.2.2.2.1.3.2.3.cmml">k</mi></msub><mo id="S3.E8.m1.2.2.2.2.2.1.3.1" xref="S3.E8.m1.2.2.2.2.2.1.3.1.cmml">∪</mo><mi id="S3.E8.m1.2.2.2.2.2.1.3.3" xref="S3.E8.m1.2.2.2.2.2.1.3.3.cmml">F</mi></mrow></mrow></mtd></mtr><mtr id="S3.E8.m1.4.4.4ad" xref="S3.E8.m1.4.5.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="S3.E8.m1.4.4.4ae" xref="S3.E8.m1.4.5.1.cmml"><mrow id="S3.E8.m1.3.3.3.3.1.1.3" xref="S3.E8.m1.4.5.1.cmml"><mn id="S3.E8.m1.3.3.3.3.1.1.1" xref="S3.E8.m1.3.3.3.3.1.1.1.cmml">0</mn><mo id="S3.E8.m1.3.3.3.3.1.1.3.1" xref="S3.E8.m1.4.5.1.cmml">,</mo></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S3.E8.m1.4.4.4af" xref="S3.E8.m1.4.5.1.cmml"><mtext id="S3.E8.m1.4.4.4.4.2.1" xref="S3.E8.m1.4.4.4.4.2.1a.cmml">otherwise.</mtext></mtd></mtr></mtable></mrow><annotation-xml encoding="MathML-Content" id="S3.E8.m1.4b"><apply id="S3.E8.m1.4.5.1.cmml" xref="S3.E8.m1.4.4a"><csymbol cd="latexml" id="S3.E8.m1.4.5.1.1.cmml" xref="S3.E8.m1.4.4a.5">cases</csymbol><apply id="S3.E8.m1.1.1.1.1.1.1.cmml" xref="S3.E8.m1.1.1.1.1.1.1"><times id="S3.E8.m1.1.1.1.1.1.1.2.cmml" xref="S3.E8.m1.1.1.1.1.1.1.2"></times><ci id="S3.E8.m1.1.1.1.1.1.1.3.cmml" xref="S3.E8.m1.1.1.1.1.1.1.3">softmax</ci><apply id="S3.E8.m1.1.1.1.1.1.1.1.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E8.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1">subscript</csymbol><apply id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1"><plus id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.1"></plus><apply id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2"><times id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1"></times><apply id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2">subscript</csymbol><apply id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.1.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2">superscript</csymbol><ci id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.2">𝒚</ci><ci id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.3.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.3">𝐿</ci></apply><ci id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.2.3">𝑡</ci></apply><ci id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.2.3">𝑬</ci></apply><apply id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3.2">𝑓</ci><ci id="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply><ci id="S3.E8.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E8.m1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply><apply id="S3.E8.m1.2.2.2.2.2.1.cmml" xref="S3.E8.m1.2.2.2.2.2.1"><in id="S3.E8.m1.2.2.2.2.2.1.1.cmml" xref="S3.E8.m1.2.2.2.2.2.1.1"></in><apply id="S3.E8.m1.2.2.2.2.2.1.2.cmml" xref="S3.E8.m1.2.2.2.2.2.1.2"><times id="S3.E8.m1.2.2.2.2.2.1.2.1.cmml" xref="S3.E8.m1.2.2.2.2.2.1.2.1"></times><ci id="S3.E8.m1.2.2.2.2.2.1.2.2a.cmml" xref="S3.E8.m1.2.2.2.2.2.1.2.2"><mtext id="S3.E8.m1.2.2.2.2.2.1.2.2.cmml" xref="S3.E8.m1.2.2.2.2.2.1.2.2">if </mtext></ci><apply id="S3.E8.m1.2.2.2.2.2.1.2.3.cmml" xref="S3.E8.m1.2.2.2.2.2.1.2.3"><csymbol cd="ambiguous" id="S3.E8.m1.2.2.2.2.2.1.2.3.1.cmml" xref="S3.E8.m1.2.2.2.2.2.1.2.3">subscript</csymbol><ci id="S3.E8.m1.2.2.2.2.2.1.2.3.2.cmml" xref="S3.E8.m1.2.2.2.2.2.1.2.3.2">𝑣</ci><ci id="S3.E8.m1.2.2.2.2.2.1.2.3.3.cmml" xref="S3.E8.m1.2.2.2.2.2.1.2.3.3">𝑖</ci></apply></apply><apply id="S3.E8.m1.2.2.2.2.2.1.3.cmml" xref="S3.E8.m1.2.2.2.2.2.1.3"><union id="S3.E8.m1.2.2.2.2.2.1.3.1.cmml" xref="S3.E8.m1.2.2.2.2.2.1.3.1"></union><apply id="S3.E8.m1.2.2.2.2.2.1.3.2.cmml" xref="S3.E8.m1.2.2.2.2.2.1.3.2"><csymbol cd="ambiguous" id="S3.E8.m1.2.2.2.2.2.1.3.2.1.cmml" xref="S3.E8.m1.2.2.2.2.2.1.3.2">subscript</csymbol><ci id="S3.E8.m1.2.2.2.2.2.1.3.2.2.cmml" xref="S3.E8.m1.2.2.2.2.2.1.3.2.2">𝑉</ci><ci id="S3.E8.m1.2.2.2.2.2.1.3.2.3.cmml" xref="S3.E8.m1.2.2.2.2.2.1.3.2.3">𝑘</ci></apply><ci id="S3.E8.m1.2.2.2.2.2.1.3.3.cmml" xref="S3.E8.m1.2.2.2.2.2.1.3.3">𝐹</ci></apply></apply><cn type="integer" id="S3.E8.m1.3.3.3.3.1.1.1.cmml" xref="S3.E8.m1.3.3.3.3.1.1.1">0</cn><ci id="S3.E8.m1.4.4.4.4.2.1a.cmml" xref="S3.E8.m1.4.4.4.4.2.1"><mtext id="S3.E8.m1.4.4.4.4.2.1.cmml" xref="S3.E8.m1.4.4.4.4.2.1">otherwise.</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E8.m1.4c">\displaystyle\begin{cases}\mathrm{softmax}({\bm{y}}^{L}_{t}{\bm{E}}+f_{t})_{i}&amp;\text{if }v_{i}\in V_{k}\cup F\\
0,&amp;\text{otherwise.}\end{cases}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<p id="S3.SS0.SSS0.Px4.p1.7" class="ltx_p">For document summarization, the subset <math id="S3.SS0.SSS0.Px4.p1.5.m1.1" class="ltx_Math" alttext="V_{k}" display="inline"><semantics id="S3.SS0.SSS0.Px4.p1.5.m1.1a"><msub id="S3.SS0.SSS0.Px4.p1.5.m1.1.1" xref="S3.SS0.SSS0.Px4.p1.5.m1.1.1.cmml"><mi id="S3.SS0.SSS0.Px4.p1.5.m1.1.1.2" xref="S3.SS0.SSS0.Px4.p1.5.m1.1.1.2.cmml">V</mi><mi id="S3.SS0.SSS0.Px4.p1.5.m1.1.1.3" xref="S3.SS0.SSS0.Px4.p1.5.m1.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p1.5.m1.1b"><apply id="S3.SS0.SSS0.Px4.p1.5.m1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.5.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px4.p1.5.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.5.m1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px4.p1.5.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px4.p1.5.m1.1.1.2">𝑉</ci><ci id="S3.SS0.SSS0.Px4.p1.5.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px4.p1.5.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p1.5.m1.1c">V_{k}</annotation></semantics></math> will capture topically salient tokens necessary to generate a summary; <math id="S3.SS0.SSS0.Px4.p1.6.m2.1" class="ltx_Math" alttext="F" display="inline"><semantics id="S3.SS0.SSS0.Px4.p1.6.m2.1a"><mi id="S3.SS0.SSS0.Px4.p1.6.m2.1.1" xref="S3.SS0.SSS0.Px4.p1.6.m2.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p1.6.m2.1b"><ci id="S3.SS0.SSS0.Px4.p1.6.m2.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.6.m2.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p1.6.m2.1c">F</annotation></semantics></math> is always added to <math id="S3.SS0.SSS0.Px4.p1.7.m3.1" class="ltx_Math" alttext="V_{k}" display="inline"><semantics id="S3.SS0.SSS0.Px4.p1.7.m3.1a"><msub id="S3.SS0.SSS0.Px4.p1.7.m3.1.1" xref="S3.SS0.SSS0.Px4.p1.7.m3.1.1.cmml"><mi id="S3.SS0.SSS0.Px4.p1.7.m3.1.1.2" xref="S3.SS0.SSS0.Px4.p1.7.m3.1.1.2.cmml">V</mi><mi id="S3.SS0.SSS0.Px4.p1.7.m3.1.1.3" xref="S3.SS0.SSS0.Px4.p1.7.m3.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p1.7.m3.1b"><apply id="S3.SS0.SSS0.Px4.p1.7.m3.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.7.m3.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px4.p1.7.m3.1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.7.m3.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px4.p1.7.m3.1.1.2.cmml" xref="S3.SS0.SSS0.Px4.p1.7.m3.1.1.2">𝑉</ci><ci id="S3.SS0.SSS0.Px4.p1.7.m3.1.1.3.cmml" xref="S3.SS0.SSS0.Px4.p1.7.m3.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p1.7.m3.1c">V_{k}</annotation></semantics></math> to ensure that the model has access to function words. By tuning the parameters of sampling, we can enforce the model to control the faithfulness or diversity of the outputs.</p>
</div>
<div id="S3.SS0.SSS0.Px4.p2" class="ltx_para">
<p id="S3.SS0.SSS0.Px4.p2.9" class="ltx_p">Focus sampling has similarities to top-<math id="S3.SS0.SSS0.Px4.p2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS0.SSS0.Px4.p2.1.m1.1a"><mi id="S3.SS0.SSS0.Px4.p2.1.m1.1.1" xref="S3.SS0.SSS0.Px4.p2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p2.1.m1.1b"><ci id="S3.SS0.SSS0.Px4.p2.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p2.1.m1.1c">k</annotation></semantics></math> (<math id="S3.SS0.SSS0.Px4.p2.2.m2.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S3.SS0.SSS0.Px4.p2.2.m2.2a"><msub id="S3.SS0.SSS0.Px4.p2.2.m2.2.3" xref="S3.SS0.SSS0.Px4.p2.2.m2.2.3.cmml"><mi id="S3.SS0.SSS0.Px4.p2.2.m2.2.3.2" xref="S3.SS0.SSS0.Px4.p2.2.m2.2.3.2.cmml">Div</mi><mrow id="S3.SS0.SSS0.Px4.p2.2.m2.2.2.2.4" xref="S3.SS0.SSS0.Px4.p2.2.m2.2.2.2.3.cmml"><mi id="S3.SS0.SSS0.Px4.p2.2.m2.1.1.1.1" xref="S3.SS0.SSS0.Px4.p2.2.m2.1.1.1.1.cmml">top</mi><mo id="S3.SS0.SSS0.Px4.p2.2.m2.2.2.2.4.1" xref="S3.SS0.SSS0.Px4.p2.2.m2.2.2.2.3.cmml">,</mo><mi id="S3.SS0.SSS0.Px4.p2.2.m2.2.2.2.2" xref="S3.SS0.SSS0.Px4.p2.2.m2.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p2.2.m2.2b"><apply id="S3.SS0.SSS0.Px4.p2.2.m2.2.3.cmml" xref="S3.SS0.SSS0.Px4.p2.2.m2.2.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px4.p2.2.m2.2.3.1.cmml" xref="S3.SS0.SSS0.Px4.p2.2.m2.2.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px4.p2.2.m2.2.3.2.cmml" xref="S3.SS0.SSS0.Px4.p2.2.m2.2.3.2">Div</ci><list id="S3.SS0.SSS0.Px4.p2.2.m2.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px4.p2.2.m2.2.2.2.4"><ci id="S3.SS0.SSS0.Px4.p2.2.m2.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p2.2.m2.1.1.1.1">top</ci><ci id="S3.SS0.SSS0.Px4.p2.2.m2.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px4.p2.2.m2.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p2.2.m2.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>; <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib22" title="" class="ltx_ref">Fan et al.</a></cite>, <cite class="ltx_cite ltx_citemacro_citeyear"><a href="#bib.bib22" title="" class="ltx_ref">2018</a></cite>) and nucleus sampling (<math id="S3.SS0.SSS0.Px4.p2.3.m3.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S3.SS0.SSS0.Px4.p2.3.m3.1a"><msub id="S3.SS0.SSS0.Px4.p2.3.m3.1.1" xref="S3.SS0.SSS0.Px4.p2.3.m3.1.1.cmml"><mi id="S3.SS0.SSS0.Px4.p2.3.m3.1.1.2" xref="S3.SS0.SSS0.Px4.p2.3.m3.1.1.2.cmml">Div</mi><mi id="S3.SS0.SSS0.Px4.p2.3.m3.1.1.3" xref="S3.SS0.SSS0.Px4.p2.3.m3.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p2.3.m3.1b"><apply id="S3.SS0.SSS0.Px4.p2.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px4.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px4.p2.3.m3.1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p2.3.m3.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px4.p2.3.m3.1.1.2.cmml" xref="S3.SS0.SSS0.Px4.p2.3.m3.1.1.2">Div</ci><ci id="S3.SS0.SSS0.Px4.p2.3.m3.1.1.3.cmml" xref="S3.SS0.SSS0.Px4.p2.3.m3.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p2.3.m3.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>; <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib34" title="" class="ltx_ref">Holtzman et al.</a></cite>, <cite class="ltx_cite ltx_citemacro_citeyear"><a href="#bib.bib34" title="" class="ltx_ref">2020</a></cite>); in that they all aim to promote diversity. At each decoding step, the top-<math id="S3.SS0.SSS0.Px4.p2.4.m4.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS0.SSS0.Px4.p2.4.m4.1a"><mi id="S3.SS0.SSS0.Px4.p2.4.m4.1.1" xref="S3.SS0.SSS0.Px4.p2.4.m4.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p2.4.m4.1b"><ci id="S3.SS0.SSS0.Px4.p2.4.m4.1.1.cmml" xref="S3.SS0.SSS0.Px4.p2.4.m4.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p2.4.m4.1c">k</annotation></semantics></math> sampling diversifies the generation process by sampling a token from the top <math id="S3.SS0.SSS0.Px4.p2.5.m5.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS0.SSS0.Px4.p2.5.m5.1a"><mi id="S3.SS0.SSS0.Px4.p2.5.m5.1.1" xref="S3.SS0.SSS0.Px4.p2.5.m5.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p2.5.m5.1b"><ci id="S3.SS0.SSS0.Px4.p2.5.m5.1.1.cmml" xref="S3.SS0.SSS0.Px4.p2.5.m5.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p2.5.m5.1c">k</annotation></semantics></math> tokens in the final output distribution. Similarly, nucleus sampling samples from a dynamic nucleus of tokens containing the vast majority (with a cumulative probability <math id="S3.SS0.SSS0.Px4.p2.6.m6.1" class="ltx_Math" alttext="p" display="inline"><semantics id="S3.SS0.SSS0.Px4.p2.6.m6.1a"><mi id="S3.SS0.SSS0.Px4.p2.6.m6.1.1" xref="S3.SS0.SSS0.Px4.p2.6.m6.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p2.6.m6.1b"><ci id="S3.SS0.SSS0.Px4.p2.6.m6.1.1.cmml" xref="S3.SS0.SSS0.Px4.p2.6.m6.1.1">𝑝</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p2.6.m6.1c">p</annotation></semantics></math>) of the probability distribution. Both top-<math id="S3.SS0.SSS0.Px4.p2.7.m7.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS0.SSS0.Px4.p2.7.m7.1a"><mi id="S3.SS0.SSS0.Px4.p2.7.m7.1.1" xref="S3.SS0.SSS0.Px4.p2.7.m7.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p2.7.m7.1b"><ci id="S3.SS0.SSS0.Px4.p2.7.m7.1.1.cmml" xref="S3.SS0.SSS0.Px4.p2.7.m7.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p2.7.m7.1c">k</annotation></semantics></math> and nucleus sampling shorten the tail of the output distribution at each decoding step, whereas focus sampling constrains the decoder to use a fixed and topically relevant vocabulary <math id="S3.SS0.SSS0.Px4.p2.8.m8.1" class="ltx_Math" alttext="V_{k}" display="inline"><semantics id="S3.SS0.SSS0.Px4.p2.8.m8.1a"><msub id="S3.SS0.SSS0.Px4.p2.8.m8.1.1" xref="S3.SS0.SSS0.Px4.p2.8.m8.1.1.cmml"><mi id="S3.SS0.SSS0.Px4.p2.8.m8.1.1.2" xref="S3.SS0.SSS0.Px4.p2.8.m8.1.1.2.cmml">V</mi><mi id="S3.SS0.SSS0.Px4.p2.8.m8.1.1.3" xref="S3.SS0.SSS0.Px4.p2.8.m8.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p2.8.m8.1b"><apply id="S3.SS0.SSS0.Px4.p2.8.m8.1.1.cmml" xref="S3.SS0.SSS0.Px4.p2.8.m8.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px4.p2.8.m8.1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p2.8.m8.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px4.p2.8.m8.1.1.2.cmml" xref="S3.SS0.SSS0.Px4.p2.8.m8.1.1.2">𝑉</ci><ci id="S3.SS0.SSS0.Px4.p2.8.m8.1.1.3.cmml" xref="S3.SS0.SSS0.Px4.p2.8.m8.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p2.8.m8.1c">V_{k}</annotation></semantics></math>. Unlike the other two techniques, <math id="S3.SS0.SSS0.Px4.p2.9.m9.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S3.SS0.SSS0.Px4.p2.9.m9.2a"><msub id="S3.SS0.SSS0.Px4.p2.9.m9.2.3" xref="S3.SS0.SSS0.Px4.p2.9.m9.2.3.cmml"><mi id="S3.SS0.SSS0.Px4.p2.9.m9.2.3.2" xref="S3.SS0.SSS0.Px4.p2.9.m9.2.3.2.cmml">Focus</mi><mrow id="S3.SS0.SSS0.Px4.p2.9.m9.2.2.2.4" xref="S3.SS0.SSS0.Px4.p2.9.m9.2.2.2.3.cmml"><mi id="S3.SS0.SSS0.Px4.p2.9.m9.1.1.1.1" xref="S3.SS0.SSS0.Px4.p2.9.m9.1.1.1.1.cmml">sample</mi><mo id="S3.SS0.SSS0.Px4.p2.9.m9.2.2.2.4.1" xref="S3.SS0.SSS0.Px4.p2.9.m9.2.2.2.3.cmml">,</mo><mi id="S3.SS0.SSS0.Px4.p2.9.m9.2.2.2.2" xref="S3.SS0.SSS0.Px4.p2.9.m9.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p2.9.m9.2b"><apply id="S3.SS0.SSS0.Px4.p2.9.m9.2.3.cmml" xref="S3.SS0.SSS0.Px4.p2.9.m9.2.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px4.p2.9.m9.2.3.1.cmml" xref="S3.SS0.SSS0.Px4.p2.9.m9.2.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px4.p2.9.m9.2.3.2.cmml" xref="S3.SS0.SSS0.Px4.p2.9.m9.2.3.2">Focus</ci><list id="S3.SS0.SSS0.Px4.p2.9.m9.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px4.p2.9.m9.2.2.2.4"><ci id="S3.SS0.SSS0.Px4.p2.9.m9.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p2.9.m9.1.1.1.1">sample</ci><ci id="S3.SS0.SSS0.Px4.p2.9.m9.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px4.p2.9.m9.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p2.9.m9.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math> can also benefit from standard beam search decoding, leading to superior generation that is not only diverse, but also consistent with the input document.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experimental Setup</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">In this section we present our experimental setup to assess the ability of our <span id="S4.p1.1.1" class="ltx_text ltx_font_smallcaps">Fame</span> models to generate faithful summaries and to demonstrate that focus sampling is more effective in generating diverse and faithful summaries than other sampling-based decoding methods.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Extreme Summarization</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">We evaluate <span id="S4.SS1.p1.1.1" class="ltx_text ltx_font_smallcaps">Fame</span> models on extreme document summarization (<span id="S4.SS1.p1.1.2" class="ltx_text ltx_font_smallcaps">XSum</span>; <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib54" title="" class="ltx_ref">Narayan et al.</a></cite>, <cite class="ltx_cite ltx_citemacro_citeyear"><a href="#bib.bib54" title="" class="ltx_ref">2018</a></cite>). The <span id="S4.SS1.p1.1.3" class="ltx_text ltx_font_smallcaps">XSum</span> summaries, are extreme in that the documents are summarized into single-sentence summaries. These summaries demonstrate a high level of abstractiveness, and generating them automatically requires document-level inference, abstraction, and paraphrasing. Due to their extreme nature, <span id="S4.SS1.p1.1.4" class="ltx_text ltx_font_smallcaps">XSum</span> summaries are ideal to evaluate <span id="S4.SS1.p1.1.5" class="ltx_text ltx_font_smallcaps">Fame</span> models’ ability to capture the theme of the document.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>We further experiment with long-form story highlight generation (<span id="footnote4.1" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span>; <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib32" title="" class="ltx_ref">Hermann et al.</a></cite>, <cite class="ltx_cite ltx_citemacro_citeyear"><a href="#bib.bib32" title="" class="ltx_ref">2015</a></cite>) and two text editing tasks: Sentence Fusion <cite class="ltx_cite ltx_citemacro_cite">Geva et al. (<a href="#bib.bib28" title="" class="ltx_ref">2019</a>)</cite> and Sentence Splitting <cite class="ltx_cite ltx_citemacro_cite">Botha et al. (<a href="#bib.bib7" title="" class="ltx_ref">2018</a>)</cite>. Their results can be found in Appendix <a href="#A2" title="Appendix B Abstractive Summarization Results on CNN/DailyMail ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> and <a href="#A3" title="Appendix C Text Editing Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>. Our <span id="footnote4.2" class="ltx_text ltx_font_smallcaps">Fame</span> models achieve SOTA on both text-editing tasks.</span></span></span> We use on the original cased version consisting of
204,045/11,332/11,334 training/validation/test document-summary pairs. During training, the input documents are truncated to 512 tokens. The length of the summaries
are limited to 64.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Pretrained Models with <span id="S4.SS2.1.1" class="ltx_text ltx_font_smallcaps">Fame</span>
</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">We introduce <span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_smallcaps">Fame</span> to two popular seq2seq architectures: RoBERTa initialized seq2seq (<span id="S4.SS2.p1.1.2" class="ltx_text ltx_font_smallcaps">RobertaS2S</span>, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib63" title="" class="ltx_ref">Rothe et al.</a></cite>, <cite class="ltx_cite ltx_citemacro_citeyear"><a href="#bib.bib63" title="" class="ltx_ref">2020</a></cite>) and <span id="S4.SS2.p1.1.3" class="ltx_text ltx_font_smallcaps">Pegasus</span> <cite class="ltx_cite ltx_citemacro_cite">Zhang et al. (<a href="#bib.bib85" title="" class="ltx_ref">2019</a>)</cite>. We refer <span id="S4.SS2.p1.1.4" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> models with <span id="S4.SS2.p1.1.5" class="ltx_text ltx_font_smallcaps">Fame</span> as <span id="S4.SS2.p1.1.6" class="ltx_text ltx_font_smallcaps">RobFame</span> and <span id="S4.SS2.p1.1.7" class="ltx_text ltx_font_smallcaps">Pegasus</span> with <span id="S4.SS2.p1.1.8" class="ltx_text ltx_font_smallcaps">Fame</span> with <span id="S4.SS2.p1.1.9" class="ltx_text ltx_font_smallcaps">PegFame</span>.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.1" class="ltx_p">We experiment with <span id="S4.SS2.p2.1.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span>-Large with shared encoder and decoder; it has 24 layers, a hidden size of 1024, filter size of 4096, 16 attention heads, and a vocabulary with 50K sentence pieces <cite class="ltx_cite ltx_citemacro_cite">Kudo and Richardson (<a href="#bib.bib40" title="" class="ltx_ref">2018</a>)</cite>. <span id="S4.SS2.p2.1.2" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> has around 455M parameters and <span id="S4.SS2.p2.1.3" class="ltx_text ltx_font_smallcaps">RobFame</span> has an additional 8M parameters.</p>
</div>
<div id="S4.SS2.p3" class="ltx_para">
<p id="S4.SS2.p3.1" class="ltx_p">The best-performing <span id="S4.SS2.p3.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> model from <cite class="ltx_cite ltx_citemacro_citet">Zhang et al. (<a href="#bib.bib85" title="" class="ltx_ref">2019</a>)</cite> is not directly comparable with <span id="S4.SS2.p3.1.2" class="ltx_text ltx_font_smallcaps">RobertaS2S</span>. It does not share the encoder and decoder, it only has 16 layers, a hidden size of 1024, filter size of 4096, 16 attention heads, with a total of 568M parameters, and it also uses a much larger vocabulary with 91K sentence pieces. Hence, we trained our own <span id="S4.SS2.p3.1.3" class="ltx_text ltx_font_smallcaps">Pegasus</span> model. We use the same architecture as <span id="S4.SS2.p3.1.4" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> and pretrain it on a mixture of C4 <cite class="ltx_cite ltx_citemacro_cite">Raffel et al. (<a href="#bib.bib60" title="" class="ltx_ref">2019</a>)</cite> and HugeNews <cite class="ltx_cite ltx_citemacro_cite">Zhang et al. (<a href="#bib.bib85" title="" class="ltx_ref">2019</a>)</cite> datasets with the original objective of generating salient GAP-sentences.</p>
</div>
<div id="S4.SS2.p4" class="ltx_para">
<p id="S4.SS2.p4.1" class="ltx_p">Our experiments focus on this newly trained <span id="S4.SS2.p4.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> model which has same number of parameters and vocabulary as <span id="S4.SS2.p4.1.2" class="ltx_text ltx_font_smallcaps">RobertaS2S</span>. But in contrast to <span id="S4.SS2.p4.1.3" class="ltx_text ltx_font_smallcaps">RobertaS2S</span>, the encoder-decoder attention in <span id="S4.SS2.p4.1.4" class="ltx_text ltx_font_smallcaps">Pegasus</span> is pretrained. This allows us to analyse how focus attention affects pretrained (<span id="S4.SS2.p4.1.5" class="ltx_text ltx_font_smallcaps">Pegasus</span>) vs randomly-initialized (<span id="S4.SS2.p4.1.6" class="ltx_text ltx_font_smallcaps">RobertaS2S</span>) encoder-decoder attentions.<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>See Appendix <a href="#A1" title="Appendix A Implementation and Reproducibility Details ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a> for implementation details and hyperparameter settings.</span></span></span></p>
</div>
<figure id="S4.T1" class="ltx_table">
<table id="S4.T1.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T1.1.2.1" class="ltx_tr">
<th id="S4.T1.1.2.1.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_tt" rowspan="3"><span id="S4.T1.1.2.1.1.1" class="ltx_text">Models</span></th>
<td id="S4.T1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" colspan="3" rowspan="2"><span id="S4.T1.1.2.1.2.1" class="ltx_text">Lexical Overlap (w/ ref)</span></td>
<td id="S4.T1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" rowspan="2"><span id="S4.T1.1.2.1.3.1" class="ltx_text">Sem. Sim.</span></td>
<td id="S4.T1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" colspan="4">Faithfulness</td>
<td id="S4.T1.1.2.1.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">others</td>
</tr>
<tr id="S4.T1.1.1" class="ltx_tr">
<td id="S4.T1.1.1.2" class="ltx_td ltx_align_center" rowspan="2"><span id="S4.T1.1.1.2.1" class="ltx_text">ent.</span></td>
<td id="S4.T1.1.1.3" class="ltx_td ltx_align_center" rowspan="2"><span id="S4.T1.1.1.3.1" class="ltx_text">Feqa</span></td>
<td id="S4.T1.1.1.4" class="ltx_td ltx_align_center ltx_border_r" colspan="2">BERTFaithful</td>
<td id="S4.T1.1.1.5" class="ltx_td ltx_align_center" rowspan="2"><span id="S4.T1.1.1.5.1" class="ltx_text">Len.</span></td>
<td id="S4.T1.1.1.1" class="ltx_td ltx_align_center" rowspan="2"><span id="S4.T1.1.1.1.1" class="ltx_text">Rep.(<math id="S4.T1.1.1.1.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S4.T1.1.1.1.1.m1.1a"><mo stretchy="false" id="S4.T1.1.1.1.1.m1.1.1" xref="S4.T1.1.1.1.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T1.1.1.1.1.m1.1b"><ci id="S4.T1.1.1.1.1.m1.1.1.cmml" xref="S4.T1.1.1.1.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.1.1.1.1.m1.1c">\downarrow</annotation></semantics></math>)</span></td>
<td id="S4.T1.1.1.6" class="ltx_td ltx_align_center">R1(P%)</td>
</tr>
<tr id="S4.T1.1.3.2" class="ltx_tr">
<td id="S4.T1.1.3.2.1" class="ltx_td ltx_align_center">R1</td>
<td id="S4.T1.1.3.2.2" class="ltx_td ltx_align_center">R2</td>
<td id="S4.T1.1.3.2.3" class="ltx_td ltx_align_center ltx_border_r">RL</td>
<td id="S4.T1.1.3.2.4" class="ltx_td ltx_align_center ltx_border_r">BERTSc.</td>
<td id="S4.T1.1.3.2.5" class="ltx_td ltx_align_center">%</td>
<td id="S4.T1.1.3.2.6" class="ltx_td ltx_align_center ltx_border_r">conf.</td>
<td id="S4.T1.1.3.2.7" class="ltx_td ltx_align_center">With doc.</td>
</tr>
<tr id="S4.T1.1.4.3" class="ltx_tr">
<th id="S4.T1.1.4.3.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t"><span id="S4.T1.1.4.3.1.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span></th>
<td id="S4.T1.1.4.3.2" class="ltx_td ltx_align_center ltx_border_t">41.45</td>
<td id="S4.T1.1.4.3.3" class="ltx_td ltx_align_center ltx_border_t">18.79</td>
<td id="S4.T1.1.4.3.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">33.90</td>
<td id="S4.T1.1.4.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">80.6</td>
<td id="S4.T1.1.4.3.6" class="ltx_td ltx_align_center ltx_border_t">39.1</td>
<td id="S4.T1.1.4.3.7" class="ltx_td ltx_align_center ltx_border_t">19.8</td>
<td id="S4.T1.1.4.3.8" class="ltx_td ltx_align_center ltx_border_t">21.5</td>
<td id="S4.T1.1.4.3.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.216</td>
<td id="S4.T1.1.4.3.10" class="ltx_td ltx_align_center ltx_border_t">21.2</td>
<td id="S4.T1.1.4.3.11" class="ltx_td ltx_align_center ltx_border_t">24.2</td>
<td id="S4.T1.1.4.3.12" class="ltx_td ltx_align_center ltx_border_t">71.1</td>
</tr>
<tr id="S4.T1.1.5.4" class="ltx_tr">
<th id="S4.T1.1.5.4.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r"><span id="S4.T1.1.5.4.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span></th>
<td id="S4.T1.1.5.4.2" class="ltx_td ltx_align_center">42.15</td>
<td id="S4.T1.1.5.4.3" class="ltx_td ltx_align_center">19.68</td>
<td id="S4.T1.1.5.4.4" class="ltx_td ltx_align_center ltx_border_r">34.81</td>
<td id="S4.T1.1.5.4.5" class="ltx_td ltx_align_center ltx_border_r">80.8</td>
<td id="S4.T1.1.5.4.6" class="ltx_td ltx_align_center">41.3</td>
<td id="S4.T1.1.5.4.7" class="ltx_td ltx_align_center">21.2</td>
<td id="S4.T1.1.5.4.8" class="ltx_td ltx_align_center">22.7</td>
<td id="S4.T1.1.5.4.9" class="ltx_td ltx_align_center ltx_border_r">0.226</td>
<td id="S4.T1.1.5.4.10" class="ltx_td ltx_align_center">20.8</td>
<td id="S4.T1.1.5.4.11" class="ltx_td ltx_align_center">20.7</td>
<td id="S4.T1.1.5.4.12" class="ltx_td ltx_align_center">72.5</td>
</tr>
<tr id="S4.T1.1.6.5" class="ltx_tr">
<th id="S4.T1.1.6.5.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t"><span id="S4.T1.1.6.5.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span></th>
<td id="S4.T1.1.6.5.2" class="ltx_td ltx_align_center ltx_border_t">44.85</td>
<td id="S4.T1.1.6.5.3" class="ltx_td ltx_align_center ltx_border_t">22.26</td>
<td id="S4.T1.1.6.5.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">37.03</td>
<td id="S4.T1.1.6.5.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">81.7</td>
<td id="S4.T1.1.6.5.6" class="ltx_td ltx_align_center ltx_border_t">43.6</td>
<td id="S4.T1.1.6.5.7" class="ltx_td ltx_align_center ltx_border_t">24.5</td>
<td id="S4.T1.1.6.5.8" class="ltx_td ltx_align_center ltx_border_t">27.0</td>
<td id="S4.T1.1.6.5.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.263</td>
<td id="S4.T1.1.6.5.10" class="ltx_td ltx_align_center ltx_border_t">21.1</td>
<td id="S4.T1.1.6.5.11" class="ltx_td ltx_align_center ltx_border_t">6.0</td>
<td id="S4.T1.1.6.5.12" class="ltx_td ltx_align_center ltx_border_t">73.8</td>
</tr>
<tr id="S4.T1.1.7.6" class="ltx_tr">
<th id="S4.T1.1.7.6.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_bb ltx_border_r"><span id="S4.T1.1.7.6.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span></th>
<td id="S4.T1.1.7.6.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.1.7.6.2.1" class="ltx_text ltx_font_bold">45.31</span></td>
<td id="S4.T1.1.7.6.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.1.7.6.3.1" class="ltx_text ltx_font_bold">22.75</span></td>
<td id="S4.T1.1.7.6.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S4.T1.1.7.6.4.1" class="ltx_text ltx_font_bold">37.46</span></td>
<td id="S4.T1.1.7.6.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S4.T1.1.7.6.5.1" class="ltx_text ltx_font_bold">81.9</span></td>
<td id="S4.T1.1.7.6.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.1.7.6.6.1" class="ltx_text ltx_font_bold">44.8</span></td>
<td id="S4.T1.1.7.6.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.1.7.6.7.1" class="ltx_text ltx_font_bold">24.8</span></td>
<td id="S4.T1.1.7.6.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.1.7.6.8.1" class="ltx_text ltx_font_bold">27.3</span></td>
<td id="S4.T1.1.7.6.9" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S4.T1.1.7.6.9.1" class="ltx_text ltx_font_bold">0.269</span></td>
<td id="S4.T1.1.7.6.10" class="ltx_td ltx_align_center ltx_border_bb">20.8</td>
<td id="S4.T1.1.7.6.11" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.1.7.6.11.1" class="ltx_text ltx_font_bold">5.3</span></td>
<td id="S4.T1.1.7.6.12" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.1.7.6.12.1" class="ltx_text ltx_font_bold">74.3</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Abstractive Summarization results on <span id="S4.T1.5.1" class="ltx_text ltx_font_smallcaps">XSum</span> test set comparing <span id="S4.T1.6.2" class="ltx_text ltx_font_smallcaps">Fame</span> models with their baselines. For all our models, we use standard beam decoding with a beam size of 4 to generate the single best summary for a document. Focus sampling is not used here. See Section <a href="#S4.SS3" title="4.3 Evaluation Metrics ‣ 4 Experimental Setup ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a> for details on the evaluation metrics reported. Best number for each metric is <span id="S4.T1.7.3" class="ltx_text ltx_font_bold">boldfaced</span>.</figcaption>
</figure>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Evaluation Metrics</h3>

<section id="S4.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Lexical Overlap</h4>

<div id="S4.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px1.p1.1" class="ltx_p">We report
<span id="S4.SS3.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_smallcaps">rouge</span> F1 scores <cite class="ltx_cite ltx_citemacro_cite">Lin and Hovy (<a href="#bib.bib46" title="" class="ltx_ref">2003</a>)</cite> against reference summaries; in particular, we report on <span id="S4.SS3.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_smallcaps">rouge-1</span> and <span id="S4.SS3.SSS0.Px1.p1.1.3" class="ltx_text ltx_font_smallcaps">rouge-2</span> for informativeness and <span id="S4.SS3.SSS0.Px1.p1.1.4" class="ltx_text ltx_font_smallcaps">rouge-l</span> for fluency.<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span>We lowercased candidate and reference summaries and used <span id="footnote6.1" class="ltx_text ltx_font_typewriter">pyrouge</span> with parameters “-a -c 95 -m -n 4 -w 1.2.”</span></span></span></p>
</div>
</section>
<section id="S4.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Semantic Similarity</h4>

<div id="S4.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px2.p1.1" class="ltx_p">We report <span id="S4.SS3.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_italic">BERTScore</span> <cite class="ltx_cite ltx_citemacro_cite">Zhang et al. (<a href="#bib.bib86" title="" class="ltx_ref">2020</a>)</cite> which computes the contextual similarity between a candidate and its reference summary.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Faithfulness</h4>

<div id="S4.SS3.SSS0.Px3.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px3.p1.1" class="ltx_p"><span id="S4.SS3.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">rouge</span> and BERTScore do not correlate well with faithfulness of the generated summaries <cite class="ltx_cite ltx_citemacro_cite">Maynez et al. (<a href="#bib.bib52" title="" class="ltx_ref">2020</a>)</cite>. Human evaluation is traditionally considered as the gold standard for measuring faithfulness. But recent research has shown that even human evaluation has shortcomings <cite class="ltx_cite ltx_citemacro_cite">Schoch et al. (<a href="#bib.bib64" title="" class="ltx_ref">2020</a>)</cite>. Moreover, it is prohibitively expensive. This has led to the proposal of meta-evaluation metrics for various generation tasks <cite class="ltx_cite ltx_citemacro_cite">Durmus et al. (<a href="#bib.bib19" title="" class="ltx_ref">2020</a>); Kryściński et al. (<a href="#bib.bib39" title="" class="ltx_ref">2019</a>); Sellam et al. (<a href="#bib.bib66" title="" class="ltx_ref">2020</a>); Rei et al. (<a href="#bib.bib62" title="" class="ltx_ref">2020</a>)</cite>.</p>
</div>
<div id="S4.SS3.SSS0.Px3.p2" class="ltx_para">
<p id="S4.SS3.SSS0.Px3.p2.1" class="ltx_p">We evaluate <span id="S4.SS3.SSS0.Px3.p2.1.1" class="ltx_text ltx_font_smallcaps">Fame</span> models on semantic inference metrics such as textual entailment <cite class="ltx_cite ltx_citemacro_cite">Pasunuru and Bansal (<a href="#bib.bib57" title="" class="ltx_ref">2018</a>); Welleck et al. (<a href="#bib.bib78" title="" class="ltx_ref">2019b</a>); Falke et al. (<a href="#bib.bib21" title="" class="ltx_ref">2019</a>); Kryscinski et al. (<a href="#bib.bib37" title="" class="ltx_ref">2019</a>)</cite> and question answering <cite class="ltx_cite ltx_citemacro_cite">Arumae and Liu (<a href="#bib.bib2" title="" class="ltx_ref">2019</a>); Wang et al. (<a href="#bib.bib73" title="" class="ltx_ref">2020a</a>)</cite>. In particular, we report the probability of a summary entailing (<span id="S4.SS3.SSS0.Px3.p2.1.2" class="ltx_text ltx_font_italic">ent.</span>) its input document <cite class="ltx_cite ltx_citemacro_cite">Maynez et al. (<a href="#bib.bib52" title="" class="ltx_ref">2020</a>)</cite> and QA-based <span id="S4.SS3.SSS0.Px3.p2.1.3" class="ltx_text ltx_font_italic">Feqa</span> scores <cite class="ltx_cite ltx_citemacro_cite">Durmus et al. (<a href="#bib.bib19" title="" class="ltx_ref">2020</a>)</cite>. For ent. scores, we train an entailment classifier by fine-tuning a BERT-Large pretrained model <cite class="ltx_cite ltx_citemacro_cite">Devlin et al. (<a href="#bib.bib13" title="" class="ltx_ref">2019</a>)</cite> on the Multi-NLI dataset <cite class="ltx_cite ltx_citemacro_cite">Williams et al. (<a href="#bib.bib79" title="" class="ltx_ref">2018</a>)</cite>. For Feqa, we use a fine-tuned BART <cite class="ltx_cite ltx_citemacro_cite">Lewis et al. (<a href="#bib.bib42" title="" class="ltx_ref">2019</a>)</cite> language model for question generation to generate questions from the summaries, and a BERT-base model fine-tuned on SQuAD <cite class="ltx_cite ltx_citemacro_cite">Rajpurkar et al. (<a href="#bib.bib61" title="" class="ltx_ref">2018</a>)</cite> to answer the generated questions with input document as context.<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span>We used the Feqa code available here: <a target="_blank" href="https://github.com/esdurmus/feqa/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/esdurmus/feqa/</a>.</span></span></span></p>
</div>
<div id="S4.SS3.SSS0.Px3.p3" class="ltx_para">
<p id="S4.SS3.SSS0.Px3.p3.3" class="ltx_p">In addition to <span id="S4.SS3.SSS0.Px3.p3.3.1" class="ltx_text ltx_font_italic">ent.</span> and <span id="S4.SS3.SSS0.Px3.p3.3.2" class="ltx_text ltx_font_italic">Feqa</span>, we train a scorer leveraging manually annotated document-summary pairs for faithfulness, as a surrogate for human evaluation and call this metric <span id="S4.SS3.SSS0.Px3.p3.3.3" class="ltx_text ltx_font_italic">BERTFaithful</span>.<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span>A very similar scorer was used in the GEM benchmark <cite class="ltx_cite ltx_citemacro_cite">Gehrmann et al. (<a href="#bib.bib26" title="" class="ltx_ref">2021</a>)</cite> to identify and extract the subset with faithful reference summaries from the XSum dataset <cite class="ltx_cite ltx_citemacro_cite">Narayan et al. (<a href="#bib.bib54" title="" class="ltx_ref">2018</a>)</cite>.</span></span></span>
In particular, we finetune a BERT-Base classifier on 500 manually annotated document and gold summary pairs for the XSum dataset from <cite class="ltx_cite ltx_citemacro_citet">Maynez et al. (<a href="#bib.bib52" title="" class="ltx_ref">2020</a>)</cite> to predict whether a summary is faithful to the input document or not.<span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note">9</span>Out of 500, 90% of the document-summary pairs were used for training and the rest 50 document-summary pairs were used for validation. We used the validation set to estimate Spearman’s correlation coefficients of different metrics with the human assessment for faithfulness. We found that both entailment scores (<span id="footnote9.1" class="ltx_text ltx_font_italic">ent.</span>) and <span id="footnote9.2" class="ltx_text ltx_font_italic">BERTFaithful</span> are moderately correlated with faithfulness with correlation coefficients of 0.4387 and 0.3889, respectively. As such, we believe that BERTFaithful works as an efficient proxy for expensive human evaluation for faithfulness for XSum summaries. More work is needed to understand if BERTFaithful generalizes to other datasets.</span></span></span> We report the percentage of summaries that were faithful (<math id="S4.SS3.SSS0.Px3.p3.1.m1.2" class="ltx_Math" alttext="\frac{1}{N}\sum_{i}\mathbbm{1}[p_{i}(\mbox{faithful})&gt;0.5]" display="inline"><semantics id="S4.SS3.SSS0.Px3.p3.1.m1.2a"><mrow id="S4.SS3.SSS0.Px3.p3.1.m1.2.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.cmml"><mfrac id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3.cmml"><mn id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3.2.cmml">1</mn><mi id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3.3" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3.3.cmml">N</mi></mfrac><mo lspace="0em" rspace="0em" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.2.cmml">​</mo><mrow id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.cmml"><msub id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2.cmml"><mo id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2.2.cmml">∑</mo><mi id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2.3" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2.3.cmml">i</mi></msub><mrow id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.cmml"><mn id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.3" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.3.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.2.cmml">​</mo><mrow id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.2.cmml"><mo stretchy="false" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.2.1.cmml">[</mo><mrow id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.cmml"><mrow id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.cmml"><msub id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2.cmml"><mi id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2.2.cmml">p</mi><mi id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2.3" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.1" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.1.cmml">​</mo><mrow id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.3.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.1.1a.cmml"><mo stretchy="false" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.3.2.1" xref="S4.SS3.SSS0.Px3.p3.1.m1.1.1a.cmml">(</mo><mtext id="S4.SS3.SSS0.Px3.p3.1.m1.1.1" xref="S4.SS3.SSS0.Px3.p3.1.m1.1.1.cmml">faithful</mtext><mo stretchy="false" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.3.2.2" xref="S4.SS3.SSS0.Px3.p3.1.m1.1.1a.cmml">)</mo></mrow></mrow><mo id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.1" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.1.cmml">&gt;</mo><mn id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.3" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.3.cmml">0.5</mn></mrow><mo stretchy="false" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.3" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px3.p3.1.m1.2b"><apply id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2"><times id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.2.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.2"></times><apply id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3"><divide id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3.1.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3"></divide><cn type="integer" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3.2.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3.2">1</cn><ci id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3.3.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.3.3">𝑁</ci></apply><apply id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1"><apply id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2"><csymbol cd="ambiguous" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2.1.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2">subscript</csymbol><sum id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2.2.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2.2"></sum><ci id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2.3.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.2.3">𝑖</ci></apply><apply id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1"><times id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.2.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.2"></times><cn type="integer" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.3.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.3">1</cn><apply id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.2.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1"><csymbol cd="latexml" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.2.1.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.2">delimited-[]</csymbol><apply id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1"><gt id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.1.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.1"></gt><apply id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2"><times id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.1.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.1"></times><apply id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2.1.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2">subscript</csymbol><ci id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2.2.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2.2">𝑝</ci><ci id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2.3.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.2.3">𝑖</ci></apply><ci id="S4.SS3.SSS0.Px3.p3.1.m1.1.1a.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.2.3.2"><mtext id="S4.SS3.SSS0.Px3.p3.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.1.1">faithful</mtext></ci></apply><cn type="float" id="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.3.cmml" xref="S4.SS3.SSS0.Px3.p3.1.m1.2.2.1.1.1.1.1.3">0.5</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px3.p3.1.m1.2c">\frac{1}{N}\sum_{i}\mathbbm{1}[p_{i}(\mbox{faithful})&gt;0.5]</annotation></semantics></math>) and the model’s confidence to generate faithful summaries (<math id="S4.SS3.SSS0.Px3.p3.2.m2.1" class="ltx_Math" alttext="\frac{1}{N}\sum_{i}p_{i}(\mbox{faithful})" display="inline"><semantics id="S4.SS3.SSS0.Px3.p3.2.m2.1a"><mrow id="S4.SS3.SSS0.Px3.p3.2.m2.1.2" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.cmml"><mfrac id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2.cmml"><mn id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2.2" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2.2.cmml">1</mn><mi id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2.3" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2.3.cmml">N</mi></mfrac><mo lspace="0em" rspace="0em" id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.1" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.1.cmml">​</mo><mrow id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.cmml"><msub id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1.cmml"><mo id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1.2" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1.2.cmml">∑</mo><mi id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1.3" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1.3.cmml">i</mi></msub><mrow id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.cmml"><msub id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2.cmml"><mi id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2.2" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2.2.cmml">p</mi><mi id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2.3" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.1" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.1.cmml">​</mo><mrow id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.3.2" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.1a.cmml"><mo stretchy="false" id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.3.2.1" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.1a.cmml">(</mo><mtext id="S4.SS3.SSS0.Px3.p3.2.m2.1.1" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.1.cmml">faithful</mtext><mo stretchy="false" id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.3.2.2" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.1a.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px3.p3.2.m2.1b"><apply id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2"><times id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.1.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.1"></times><apply id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2"><divide id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2.1.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2"></divide><cn type="integer" id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2.2.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2.2">1</cn><ci id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2.3.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.2.3">𝑁</ci></apply><apply id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3"><apply id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1"><csymbol cd="ambiguous" id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1.1.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1">subscript</csymbol><sum id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1.2.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1.2"></sum><ci id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1.3.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.1.3">𝑖</ci></apply><apply id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2"><times id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.1.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.1"></times><apply id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2"><csymbol cd="ambiguous" id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2.1.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2">subscript</csymbol><ci id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2.2.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2.2">𝑝</ci><ci id="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2.3.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.2.3">𝑖</ci></apply><ci id="S4.SS3.SSS0.Px3.p3.2.m2.1.1a.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.2.3.2.3.2"><mtext id="S4.SS3.SSS0.Px3.p3.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px3.p3.2.m2.1.1">faithful</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px3.p3.2.m2.1c">\frac{1}{N}\sum_{i}p_{i}(\mbox{faithful})</annotation></semantics></math>); <math id="S4.SS3.SSS0.Px3.p3.3.m3.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S4.SS3.SSS0.Px3.p3.3.m3.1a"><mi id="S4.SS3.SSS0.Px3.p3.3.m3.1.1" xref="S4.SS3.SSS0.Px3.p3.3.m3.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px3.p3.3.m3.1b"><ci id="S4.SS3.SSS0.Px3.p3.3.m3.1.1.cmml" xref="S4.SS3.SSS0.Px3.p3.3.m3.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px3.p3.3.m3.1c">N</annotation></semantics></math> is the total number of examples in the test set.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Diversity</h4>

<div id="S4.SS3.SSS0.Px4.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px4.p1.4" class="ltx_p">We report the number of times (out of <math id="S4.SS3.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.SS3.SSS0.Px4.p1.1.m1.1a"><mi id="S4.SS3.SSS0.Px4.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px4.p1.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px4.p1.1.m1.1b"><ci id="S4.SS3.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px4.p1.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px4.p1.1.m1.1c">n</annotation></semantics></math>), a model is able to generate a completely new summary (<span id="S4.SS3.SSS0.Px4.p1.4.1" class="ltx_text ltx_font_italic">Unique</span>), and <span id="S4.SS3.SSS0.Px4.p1.4.2" class="ltx_text ltx_font_italic">Distinct-N</span> <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a href="#bib.bib43" title="" class="ltx_ref">2016a</a>)</cite>, measuring the lexical diversity in the generated summaries. Distinct-N is estimated as the number of distinct <math id="S4.SS3.SSS0.Px4.p1.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.SS3.SSS0.Px4.p1.2.m2.1a"><mi id="S4.SS3.SSS0.Px4.p1.2.m2.1.1" xref="S4.SS3.SSS0.Px4.p1.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px4.p1.2.m2.1b"><ci id="S4.SS3.SSS0.Px4.p1.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px4.p1.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px4.p1.2.m2.1c">n</annotation></semantics></math>-grams of order <math id="S4.SS3.SSS0.Px4.p1.3.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.SS3.SSS0.Px4.p1.3.m3.1a"><mi id="S4.SS3.SSS0.Px4.p1.3.m3.1.1" xref="S4.SS3.SSS0.Px4.p1.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px4.p1.3.m3.1b"><ci id="S4.SS3.SSS0.Px4.p1.3.m3.1.1.cmml" xref="S4.SS3.SSS0.Px4.p1.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px4.p1.3.m3.1c">n</annotation></semantics></math> divided by the total number of <math id="S4.SS3.SSS0.Px4.p1.4.m4.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.SS3.SSS0.Px4.p1.4.m4.1a"><mi id="S4.SS3.SSS0.Px4.p1.4.m4.1.1" xref="S4.SS3.SSS0.Px4.p1.4.m4.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px4.p1.4.m4.1b"><ci id="S4.SS3.SSS0.Px4.p1.4.m4.1.1.cmml" xref="S4.SS3.SSS0.Px4.p1.4.m4.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px4.p1.4.m4.1c">n</annotation></semantics></math>-grams of the same order, in all generated summaries.</p>
</div>
<div id="S4.SS3.SSS0.Px4.p2" class="ltx_para">
<p id="S4.SS3.SSS0.Px4.p2.1" class="ltx_p">Finally, we also report the average length of summaries (<span id="S4.SS3.SSS0.Px4.p2.1.1" class="ltx_text ltx_font_italic">Len.</span>), repetition errors (<span id="S4.SS3.SSS0.Px4.p2.1.2" class="ltx_text ltx_font_italic">Rep.</span>, estimated as the percentage of summaries with at least one repetition of rare or content words), and <span id="S4.SS3.SSS0.Px4.p2.1.3" class="ltx_text ltx_font_smallcaps">rouge-1</span> precision against the input document (<span id="S4.SS3.SSS0.Px4.p2.1.4" class="ltx_text ltx_font_italic">R1, P%</span>), to better understand their quality.</p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Results</h2>

<figure id="S5.T2" class="ltx_table">
<table id="S5.T2.18" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S5.T2.18.19.1" class="ltx_tr">
<td id="S5.T2.18.19.1.1" class="ltx_td ltx_align_right ltx_border_r ltx_border_tt" rowspan="2"><span id="S5.T2.18.19.1.1.1" class="ltx_text">Metrics</span></td>
<td id="S5.T2.18.19.1.2" class="ltx_td ltx_align_center ltx_border_tt" rowspan="2"><span id="S5.T2.18.19.1.2.1" class="ltx_text">Unique</span></td>
<td id="S5.T2.18.19.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" colspan="3">Dist.-N</td>
<td id="S5.T2.18.19.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" colspan="3"><span id="S5.T2.18.19.1.4.1" class="ltx_text ltx_font_smallcaps">rouge</span></td>
<td id="S5.T2.18.19.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" rowspan="2"><span id="S5.T2.18.19.1.5.1" class="ltx_text">ent.</span></td>
<td id="S5.T2.18.19.1.6" class="ltx_td ltx_align_center ltx_border_tt" rowspan="2"><span id="S5.T2.18.19.1.6.1" class="ltx_text">BERTSc.</span></td>
</tr>
<tr id="S5.T2.18.20.2" class="ltx_tr">
<td id="S5.T2.18.20.2.1" class="ltx_td ltx_align_center">1</td>
<td id="S5.T2.18.20.2.2" class="ltx_td ltx_align_center">2</td>
<td id="S5.T2.18.20.2.3" class="ltx_td ltx_align_center ltx_border_r">3</td>
<td id="S5.T2.18.20.2.4" class="ltx_td ltx_align_center">R1</td>
<td id="S5.T2.18.20.2.5" class="ltx_td ltx_align_center">R2</td>
<td id="S5.T2.18.20.2.6" class="ltx_td ltx_align_center ltx_border_r">RL</td>
</tr>
<tr id="S5.T2.1.1" class="ltx_tr">
<td id="S5.T2.1.1.1" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">
<span id="S5.T2.1.1.1.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> (<math id="S5.T2.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.T2.1.1.1.m1.2a"><msub id="S5.T2.1.1.1.m1.2.3" xref="S5.T2.1.1.1.m1.2.3.cmml"><mi id="S5.T2.1.1.1.m1.2.3.2" xref="S5.T2.1.1.1.m1.2.3.2.cmml">Div</mi><mrow id="S5.T2.1.1.1.m1.2.2.2.4" xref="S5.T2.1.1.1.m1.2.2.2.3.cmml"><mi id="S5.T2.1.1.1.m1.1.1.1.1" xref="S5.T2.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="S5.T2.1.1.1.m1.2.2.2.4.1" xref="S5.T2.1.1.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.T2.1.1.1.m1.2.2.2.2" xref="S5.T2.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.1.1.1.m1.2b"><apply id="S5.T2.1.1.1.m1.2.3.cmml" xref="S5.T2.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="S5.T2.1.1.1.m1.2.3.1.cmml" xref="S5.T2.1.1.1.m1.2.3">subscript</csymbol><ci id="S5.T2.1.1.1.m1.2.3.2.cmml" xref="S5.T2.1.1.1.m1.2.3.2">Div</ci><list id="S5.T2.1.1.1.m1.2.2.2.3.cmml" xref="S5.T2.1.1.1.m1.2.2.2.4"><ci id="S5.T2.1.1.1.m1.1.1.1.1.cmml" xref="S5.T2.1.1.1.m1.1.1.1.1">top</ci><ci id="S5.T2.1.1.1.m1.2.2.2.2.cmml" xref="S5.T2.1.1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.1.1.1.m1.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</td>
<td id="S5.T2.1.1.2" class="ltx_td ltx_align_center ltx_border_t">9.98</td>
<td id="S5.T2.1.1.3" class="ltx_td ltx_align_center ltx_border_t">2.5</td>
<td id="S5.T2.1.1.4" class="ltx_td ltx_align_center ltx_border_t">25.0</td>
<td id="S5.T2.1.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">57.7</td>
<td id="S5.T2.1.1.6" class="ltx_td ltx_align_center ltx_border_t">33.6</td>
<td id="S5.T2.1.1.7" class="ltx_td ltx_align_center ltx_border_t">12.0</td>
<td id="S5.T2.1.1.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">26.5</td>
<td id="S5.T2.1.1.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">21.8</td>
<td id="S5.T2.1.1.10" class="ltx_td ltx_align_center ltx_border_t">76.9</td>
</tr>
<tr id="S5.T2.2.2" class="ltx_tr">
<td id="S5.T2.2.2.1" class="ltx_td ltx_align_right ltx_border_r">
<span id="S5.T2.2.2.1.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> (<math id="S5.T2.2.2.1.m1.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.T2.2.2.1.m1.1a"><msub id="S5.T2.2.2.1.m1.1.1" xref="S5.T2.2.2.1.m1.1.1.cmml"><mi id="S5.T2.2.2.1.m1.1.1.2" xref="S5.T2.2.2.1.m1.1.1.2.cmml">Div</mi><mi id="S5.T2.2.2.1.m1.1.1.3" xref="S5.T2.2.2.1.m1.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.T2.2.2.1.m1.1b"><apply id="S5.T2.2.2.1.m1.1.1.cmml" xref="S5.T2.2.2.1.m1.1.1"><csymbol cd="ambiguous" id="S5.T2.2.2.1.m1.1.1.1.cmml" xref="S5.T2.2.2.1.m1.1.1">subscript</csymbol><ci id="S5.T2.2.2.1.m1.1.1.2.cmml" xref="S5.T2.2.2.1.m1.1.1.2">Div</ci><ci id="S5.T2.2.2.1.m1.1.1.3.cmml" xref="S5.T2.2.2.1.m1.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.2.2.1.m1.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</td>
<td id="S5.T2.2.2.2" class="ltx_td ltx_align_center">9.99</td>
<td id="S5.T2.2.2.3" class="ltx_td ltx_align_center"><span id="S5.T2.2.2.3.1" class="ltx_text ltx_font_bold">4.1</span></td>
<td id="S5.T2.2.2.4" class="ltx_td ltx_align_center">30.1</td>
<td id="S5.T2.2.2.5" class="ltx_td ltx_align_center ltx_border_r">62.2</td>
<td id="S5.T2.2.2.6" class="ltx_td ltx_align_center">32.4</td>
<td id="S5.T2.2.2.7" class="ltx_td ltx_align_center">11.4</td>
<td id="S5.T2.2.2.8" class="ltx_td ltx_align_center ltx_border_r">25.6</td>
<td id="S5.T2.2.2.9" class="ltx_td ltx_align_center ltx_border_r">19.7</td>
<td id="S5.T2.2.2.10" class="ltx_td ltx_align_center">75.7</td>
</tr>
<tr id="S5.T2.3.3" class="ltx_tr">
<td id="S5.T2.3.3.1" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">
<span id="S5.T2.3.3.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (<math id="S5.T2.3.3.1.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.T2.3.3.1.m1.2a"><msub id="S5.T2.3.3.1.m1.2.3" xref="S5.T2.3.3.1.m1.2.3.cmml"><mi id="S5.T2.3.3.1.m1.2.3.2" xref="S5.T2.3.3.1.m1.2.3.2.cmml">Div</mi><mrow id="S5.T2.3.3.1.m1.2.2.2.4" xref="S5.T2.3.3.1.m1.2.2.2.3.cmml"><mi id="S5.T2.3.3.1.m1.1.1.1.1" xref="S5.T2.3.3.1.m1.1.1.1.1.cmml">top</mi><mo id="S5.T2.3.3.1.m1.2.2.2.4.1" xref="S5.T2.3.3.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.T2.3.3.1.m1.2.2.2.2" xref="S5.T2.3.3.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.3.3.1.m1.2b"><apply id="S5.T2.3.3.1.m1.2.3.cmml" xref="S5.T2.3.3.1.m1.2.3"><csymbol cd="ambiguous" id="S5.T2.3.3.1.m1.2.3.1.cmml" xref="S5.T2.3.3.1.m1.2.3">subscript</csymbol><ci id="S5.T2.3.3.1.m1.2.3.2.cmml" xref="S5.T2.3.3.1.m1.2.3.2">Div</ci><list id="S5.T2.3.3.1.m1.2.2.2.3.cmml" xref="S5.T2.3.3.1.m1.2.2.2.4"><ci id="S5.T2.3.3.1.m1.1.1.1.1.cmml" xref="S5.T2.3.3.1.m1.1.1.1.1">top</ci><ci id="S5.T2.3.3.1.m1.2.2.2.2.cmml" xref="S5.T2.3.3.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.3.3.1.m1.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</td>
<td id="S5.T2.3.3.2" class="ltx_td ltx_align_center ltx_border_t">9.99</td>
<td id="S5.T2.3.3.3" class="ltx_td ltx_align_center ltx_border_t">2.3</td>
<td id="S5.T2.3.3.4" class="ltx_td ltx_align_center ltx_border_t">25.0</td>
<td id="S5.T2.3.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">58.1</td>
<td id="S5.T2.3.3.6" class="ltx_td ltx_align_center ltx_border_t">32.7</td>
<td id="S5.T2.3.3.7" class="ltx_td ltx_align_center ltx_border_t">11.3</td>
<td id="S5.T2.3.3.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">25.7</td>
<td id="S5.T2.3.3.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">20.3</td>
<td id="S5.T2.3.3.10" class="ltx_td ltx_align_center ltx_border_t">76.6</td>
</tr>
<tr id="S5.T2.4.4" class="ltx_tr">
<td id="S5.T2.4.4.1" class="ltx_td ltx_align_right ltx_border_r">
<span id="S5.T2.4.4.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (<math id="S5.T2.4.4.1.m1.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.T2.4.4.1.m1.1a"><msub id="S5.T2.4.4.1.m1.1.1" xref="S5.T2.4.4.1.m1.1.1.cmml"><mi id="S5.T2.4.4.1.m1.1.1.2" xref="S5.T2.4.4.1.m1.1.1.2.cmml">Div</mi><mi id="S5.T2.4.4.1.m1.1.1.3" xref="S5.T2.4.4.1.m1.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.T2.4.4.1.m1.1b"><apply id="S5.T2.4.4.1.m1.1.1.cmml" xref="S5.T2.4.4.1.m1.1.1"><csymbol cd="ambiguous" id="S5.T2.4.4.1.m1.1.1.1.cmml" xref="S5.T2.4.4.1.m1.1.1">subscript</csymbol><ci id="S5.T2.4.4.1.m1.1.1.2.cmml" xref="S5.T2.4.4.1.m1.1.1.2">Div</ci><ci id="S5.T2.4.4.1.m1.1.1.3.cmml" xref="S5.T2.4.4.1.m1.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.4.4.1.m1.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</td>
<td id="S5.T2.4.4.2" class="ltx_td ltx_align_center">9.99</td>
<td id="S5.T2.4.4.3" class="ltx_td ltx_align_center"><span id="S5.T2.4.4.3.1" class="ltx_text ltx_font_bold">4.1</span></td>
<td id="S5.T2.4.4.4" class="ltx_td ltx_align_center"><span id="S5.T2.4.4.4.1" class="ltx_text ltx_font_bold">30.7</span></td>
<td id="S5.T2.4.4.5" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T2.4.4.5.1" class="ltx_text ltx_font_bold">63.2</span></td>
<td id="S5.T2.4.4.6" class="ltx_td ltx_align_center">31.3</td>
<td id="S5.T2.4.4.7" class="ltx_td ltx_align_center">10.6</td>
<td id="S5.T2.4.4.8" class="ltx_td ltx_align_center ltx_border_r">24.7</td>
<td id="S5.T2.4.4.9" class="ltx_td ltx_align_center ltx_border_r">18.0</td>
<td id="S5.T2.4.4.10" class="ltx_td ltx_align_center">75.4</td>
</tr>
<tr id="S5.T2.5.5" class="ltx_tr">
<td id="S5.T2.5.5.1" class="ltx_td ltx_align_right ltx_border_r">
<span id="S5.T2.5.5.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (<math id="S5.T2.5.5.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.T2.5.5.1.m1.2a"><msub id="S5.T2.5.5.1.m1.2.3" xref="S5.T2.5.5.1.m1.2.3.cmml"><mi id="S5.T2.5.5.1.m1.2.3.2" xref="S5.T2.5.5.1.m1.2.3.2.cmml">Focus</mi><mrow id="S5.T2.5.5.1.m1.2.2.2.4" xref="S5.T2.5.5.1.m1.2.2.2.3.cmml"><mi id="S5.T2.5.5.1.m1.1.1.1.1" xref="S5.T2.5.5.1.m1.1.1.1.1.cmml">sample</mi><mo id="S5.T2.5.5.1.m1.2.2.2.4.1" xref="S5.T2.5.5.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.T2.5.5.1.m1.2.2.2.2" xref="S5.T2.5.5.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.5.5.1.m1.2b"><apply id="S5.T2.5.5.1.m1.2.3.cmml" xref="S5.T2.5.5.1.m1.2.3"><csymbol cd="ambiguous" id="S5.T2.5.5.1.m1.2.3.1.cmml" xref="S5.T2.5.5.1.m1.2.3">subscript</csymbol><ci id="S5.T2.5.5.1.m1.2.3.2.cmml" xref="S5.T2.5.5.1.m1.2.3.2">Focus</ci><list id="S5.T2.5.5.1.m1.2.2.2.3.cmml" xref="S5.T2.5.5.1.m1.2.2.2.4"><ci id="S5.T2.5.5.1.m1.1.1.1.1.cmml" xref="S5.T2.5.5.1.m1.1.1.1.1">sample</ci><ci id="S5.T2.5.5.1.m1.2.2.2.2.cmml" xref="S5.T2.5.5.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.5.5.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>)</td>
<td id="S5.T2.5.5.2" class="ltx_td ltx_align_center">1.61</td>
<td id="S5.T2.5.5.3" class="ltx_td ltx_align_center">3.5</td>
<td id="S5.T2.5.5.4" class="ltx_td ltx_align_center">22.4</td>
<td id="S5.T2.5.5.5" class="ltx_td ltx_align_center ltx_border_r">43.9</td>
<td id="S5.T2.5.5.6" class="ltx_td ltx_align_center"><span id="S5.T2.5.5.6.1" class="ltx_text ltx_font_bold">38.0</span></td>
<td id="S5.T2.5.5.7" class="ltx_td ltx_align_center"><span id="S5.T2.5.5.7.1" class="ltx_text ltx_font_bold">15.7</span></td>
<td id="S5.T2.5.5.8" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T2.5.5.8.1" class="ltx_text ltx_font_bold">31.0</span></td>
<td id="S5.T2.5.5.9" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T2.5.5.9.1" class="ltx_text ltx_font_bold">34.3</span></td>
<td id="S5.T2.5.5.10" class="ltx_td ltx_align_center"><span id="S5.T2.5.5.10.1" class="ltx_text ltx_font_bold">78.6</span></td>
</tr>
<tr id="S5.T2.7.7" class="ltx_tr">
<td id="S5.T2.7.7.2" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">
<span id="S5.T2.7.7.2.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (<math id="S5.T2.6.6.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.T2.6.6.1.m1.2a"><msub id="S5.T2.6.6.1.m1.2.3" xref="S5.T2.6.6.1.m1.2.3.cmml"><mi id="S5.T2.6.6.1.m1.2.3.2" xref="S5.T2.6.6.1.m1.2.3.2.cmml">Focus</mi><mrow id="S5.T2.6.6.1.m1.2.2.2.4" xref="S5.T2.6.6.1.m1.2.2.2.3.cmml"><mi id="S5.T2.6.6.1.m1.1.1.1.1" xref="S5.T2.6.6.1.m1.1.1.1.1.cmml">sample</mi><mo id="S5.T2.6.6.1.m1.2.2.2.4.1" xref="S5.T2.6.6.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.T2.6.6.1.m1.2.2.2.2" xref="S5.T2.6.6.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.6.6.1.m1.2b"><apply id="S5.T2.6.6.1.m1.2.3.cmml" xref="S5.T2.6.6.1.m1.2.3"><csymbol cd="ambiguous" id="S5.T2.6.6.1.m1.2.3.1.cmml" xref="S5.T2.6.6.1.m1.2.3">subscript</csymbol><ci id="S5.T2.6.6.1.m1.2.3.2.cmml" xref="S5.T2.6.6.1.m1.2.3.2">Focus</ci><list id="S5.T2.6.6.1.m1.2.2.2.3.cmml" xref="S5.T2.6.6.1.m1.2.2.2.4"><ci id="S5.T2.6.6.1.m1.1.1.1.1.cmml" xref="S5.T2.6.6.1.m1.1.1.1.1">sample</ci><ci id="S5.T2.6.6.1.m1.2.2.2.2.cmml" xref="S5.T2.6.6.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.6.6.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, <math id="S5.T2.7.7.2.m2.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.T2.7.7.2.m2.2a"><msub id="S5.T2.7.7.2.m2.2.3" xref="S5.T2.7.7.2.m2.2.3.cmml"><mi id="S5.T2.7.7.2.m2.2.3.2" xref="S5.T2.7.7.2.m2.2.3.2.cmml">Div</mi><mrow id="S5.T2.7.7.2.m2.2.2.2.4" xref="S5.T2.7.7.2.m2.2.2.2.3.cmml"><mi id="S5.T2.7.7.2.m2.1.1.1.1" xref="S5.T2.7.7.2.m2.1.1.1.1.cmml">top</mi><mo id="S5.T2.7.7.2.m2.2.2.2.4.1" xref="S5.T2.7.7.2.m2.2.2.2.3.cmml">,</mo><mi id="S5.T2.7.7.2.m2.2.2.2.2" xref="S5.T2.7.7.2.m2.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.7.7.2.m2.2b"><apply id="S5.T2.7.7.2.m2.2.3.cmml" xref="S5.T2.7.7.2.m2.2.3"><csymbol cd="ambiguous" id="S5.T2.7.7.2.m2.2.3.1.cmml" xref="S5.T2.7.7.2.m2.2.3">subscript</csymbol><ci id="S5.T2.7.7.2.m2.2.3.2.cmml" xref="S5.T2.7.7.2.m2.2.3.2">Div</ci><list id="S5.T2.7.7.2.m2.2.2.2.3.cmml" xref="S5.T2.7.7.2.m2.2.2.2.4"><ci id="S5.T2.7.7.2.m2.1.1.1.1.cmml" xref="S5.T2.7.7.2.m2.1.1.1.1">top</ci><ci id="S5.T2.7.7.2.m2.2.2.2.2.cmml" xref="S5.T2.7.7.2.m2.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.7.7.2.m2.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</td>
<td id="S5.T2.7.7.3" class="ltx_td ltx_align_center ltx_border_t">9.99</td>
<td id="S5.T2.7.7.4" class="ltx_td ltx_align_center ltx_border_t">2.1</td>
<td id="S5.T2.7.7.5" class="ltx_td ltx_align_center ltx_border_t">20.3</td>
<td id="S5.T2.7.7.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">51.8</td>
<td id="S5.T2.7.7.7" class="ltx_td ltx_align_center ltx_border_t">31.8</td>
<td id="S5.T2.7.7.8" class="ltx_td ltx_align_center ltx_border_t">10.2</td>
<td id="S5.T2.7.7.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">24.7</td>
<td id="S5.T2.7.7.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">24.3</td>
<td id="S5.T2.7.7.11" class="ltx_td ltx_align_center ltx_border_t">75.4</td>
</tr>
<tr id="S5.T2.9.9" class="ltx_tr">
<td id="S5.T2.9.9.2" class="ltx_td ltx_align_right ltx_border_r">
<span id="S5.T2.9.9.2.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (<math id="S5.T2.8.8.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.T2.8.8.1.m1.2a"><msub id="S5.T2.8.8.1.m1.2.3" xref="S5.T2.8.8.1.m1.2.3.cmml"><mi id="S5.T2.8.8.1.m1.2.3.2" xref="S5.T2.8.8.1.m1.2.3.2.cmml">Focus</mi><mrow id="S5.T2.8.8.1.m1.2.2.2.4" xref="S5.T2.8.8.1.m1.2.2.2.3.cmml"><mi id="S5.T2.8.8.1.m1.1.1.1.1" xref="S5.T2.8.8.1.m1.1.1.1.1.cmml">sample</mi><mo id="S5.T2.8.8.1.m1.2.2.2.4.1" xref="S5.T2.8.8.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.T2.8.8.1.m1.2.2.2.2" xref="S5.T2.8.8.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.8.8.1.m1.2b"><apply id="S5.T2.8.8.1.m1.2.3.cmml" xref="S5.T2.8.8.1.m1.2.3"><csymbol cd="ambiguous" id="S5.T2.8.8.1.m1.2.3.1.cmml" xref="S5.T2.8.8.1.m1.2.3">subscript</csymbol><ci id="S5.T2.8.8.1.m1.2.3.2.cmml" xref="S5.T2.8.8.1.m1.2.3.2">Focus</ci><list id="S5.T2.8.8.1.m1.2.2.2.3.cmml" xref="S5.T2.8.8.1.m1.2.2.2.4"><ci id="S5.T2.8.8.1.m1.1.1.1.1.cmml" xref="S5.T2.8.8.1.m1.1.1.1.1">sample</ci><ci id="S5.T2.8.8.1.m1.2.2.2.2.cmml" xref="S5.T2.8.8.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.8.8.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, <math id="S5.T2.9.9.2.m2.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.T2.9.9.2.m2.1a"><msub id="S5.T2.9.9.2.m2.1.1" xref="S5.T2.9.9.2.m2.1.1.cmml"><mi id="S5.T2.9.9.2.m2.1.1.2" xref="S5.T2.9.9.2.m2.1.1.2.cmml">Div</mi><mi id="S5.T2.9.9.2.m2.1.1.3" xref="S5.T2.9.9.2.m2.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.T2.9.9.2.m2.1b"><apply id="S5.T2.9.9.2.m2.1.1.cmml" xref="S5.T2.9.9.2.m2.1.1"><csymbol cd="ambiguous" id="S5.T2.9.9.2.m2.1.1.1.cmml" xref="S5.T2.9.9.2.m2.1.1">subscript</csymbol><ci id="S5.T2.9.9.2.m2.1.1.2.cmml" xref="S5.T2.9.9.2.m2.1.1.2">Div</ci><ci id="S5.T2.9.9.2.m2.1.1.3.cmml" xref="S5.T2.9.9.2.m2.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.9.9.2.m2.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</td>
<td id="S5.T2.9.9.3" class="ltx_td ltx_align_center">9.98</td>
<td id="S5.T2.9.9.4" class="ltx_td ltx_align_center">1.9</td>
<td id="S5.T2.9.9.5" class="ltx_td ltx_align_center">18.4</td>
<td id="S5.T2.9.9.6" class="ltx_td ltx_align_center ltx_border_r">48.2</td>
<td id="S5.T2.9.9.7" class="ltx_td ltx_align_center">32.9</td>
<td id="S5.T2.9.9.8" class="ltx_td ltx_align_center">11.1</td>
<td id="S5.T2.9.9.9" class="ltx_td ltx_align_center ltx_border_r">25.8</td>
<td id="S5.T2.9.9.10" class="ltx_td ltx_align_center ltx_border_r">25.9</td>
<td id="S5.T2.9.9.11" class="ltx_td ltx_align_center">76.1</td>
</tr>
<tr id="S5.T2.10.10" class="ltx_tr">
<td id="S5.T2.10.10.1" class="ltx_td ltx_align_right ltx_border_r ltx_border_tt">
<span id="S5.T2.10.10.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> (<math id="S5.T2.10.10.1.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.T2.10.10.1.m1.2a"><msub id="S5.T2.10.10.1.m1.2.3" xref="S5.T2.10.10.1.m1.2.3.cmml"><mi id="S5.T2.10.10.1.m1.2.3.2" xref="S5.T2.10.10.1.m1.2.3.2.cmml">Div</mi><mrow id="S5.T2.10.10.1.m1.2.2.2.4" xref="S5.T2.10.10.1.m1.2.2.2.3.cmml"><mi id="S5.T2.10.10.1.m1.1.1.1.1" xref="S5.T2.10.10.1.m1.1.1.1.1.cmml">top</mi><mo id="S5.T2.10.10.1.m1.2.2.2.4.1" xref="S5.T2.10.10.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.T2.10.10.1.m1.2.2.2.2" xref="S5.T2.10.10.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.10.10.1.m1.2b"><apply id="S5.T2.10.10.1.m1.2.3.cmml" xref="S5.T2.10.10.1.m1.2.3"><csymbol cd="ambiguous" id="S5.T2.10.10.1.m1.2.3.1.cmml" xref="S5.T2.10.10.1.m1.2.3">subscript</csymbol><ci id="S5.T2.10.10.1.m1.2.3.2.cmml" xref="S5.T2.10.10.1.m1.2.3.2">Div</ci><list id="S5.T2.10.10.1.m1.2.2.2.3.cmml" xref="S5.T2.10.10.1.m1.2.2.2.4"><ci id="S5.T2.10.10.1.m1.1.1.1.1.cmml" xref="S5.T2.10.10.1.m1.1.1.1.1">top</ci><ci id="S5.T2.10.10.1.m1.2.2.2.2.cmml" xref="S5.T2.10.10.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.10.10.1.m1.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</td>
<td id="S5.T2.10.10.2" class="ltx_td ltx_align_center ltx_border_tt">9.98</td>
<td id="S5.T2.10.10.3" class="ltx_td ltx_align_center ltx_border_tt">1.9</td>
<td id="S5.T2.10.10.4" class="ltx_td ltx_align_center ltx_border_tt">23.2</td>
<td id="S5.T2.10.10.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">55.3</td>
<td id="S5.T2.10.10.6" class="ltx_td ltx_align_center ltx_border_tt">36.6</td>
<td id="S5.T2.10.10.7" class="ltx_td ltx_align_center ltx_border_tt">14.3</td>
<td id="S5.T2.10.10.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">28.8</td>
<td id="S5.T2.10.10.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">27.7</td>
<td id="S5.T2.10.10.10" class="ltx_td ltx_align_center ltx_border_tt">78.4</td>
</tr>
<tr id="S5.T2.11.11" class="ltx_tr">
<td id="S5.T2.11.11.1" class="ltx_td ltx_align_right ltx_border_r">
<span id="S5.T2.11.11.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> (<math id="S5.T2.11.11.1.m1.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.T2.11.11.1.m1.1a"><msub id="S5.T2.11.11.1.m1.1.1" xref="S5.T2.11.11.1.m1.1.1.cmml"><mi id="S5.T2.11.11.1.m1.1.1.2" xref="S5.T2.11.11.1.m1.1.1.2.cmml">Div</mi><mi id="S5.T2.11.11.1.m1.1.1.3" xref="S5.T2.11.11.1.m1.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.T2.11.11.1.m1.1b"><apply id="S5.T2.11.11.1.m1.1.1.cmml" xref="S5.T2.11.11.1.m1.1.1"><csymbol cd="ambiguous" id="S5.T2.11.11.1.m1.1.1.1.cmml" xref="S5.T2.11.11.1.m1.1.1">subscript</csymbol><ci id="S5.T2.11.11.1.m1.1.1.2.cmml" xref="S5.T2.11.11.1.m1.1.1.2">Div</ci><ci id="S5.T2.11.11.1.m1.1.1.3.cmml" xref="S5.T2.11.11.1.m1.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.11.11.1.m1.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</td>
<td id="S5.T2.11.11.2" class="ltx_td ltx_align_center">9.99</td>
<td id="S5.T2.11.11.3" class="ltx_td ltx_align_center"><span id="S5.T2.11.11.3.1" class="ltx_text ltx_font_bold">3.8</span></td>
<td id="S5.T2.11.11.4" class="ltx_td ltx_align_center"><span id="S5.T2.11.11.4.1" class="ltx_text ltx_font_bold">30.5</span></td>
<td id="S5.T2.11.11.5" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T2.11.11.5.1" class="ltx_text ltx_font_bold">63.1</span></td>
<td id="S5.T2.11.11.6" class="ltx_td ltx_align_center">34.1</td>
<td id="S5.T2.11.11.7" class="ltx_td ltx_align_center">12.8</td>
<td id="S5.T2.11.11.8" class="ltx_td ltx_align_center ltx_border_r">26.9</td>
<td id="S5.T2.11.11.9" class="ltx_td ltx_align_center ltx_border_r">22.7</td>
<td id="S5.T2.11.11.10" class="ltx_td ltx_align_center">76.5</td>
</tr>
<tr id="S5.T2.12.12" class="ltx_tr">
<td id="S5.T2.12.12.1" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">
<span id="S5.T2.12.12.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (<math id="S5.T2.12.12.1.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.T2.12.12.1.m1.2a"><msub id="S5.T2.12.12.1.m1.2.3" xref="S5.T2.12.12.1.m1.2.3.cmml"><mi id="S5.T2.12.12.1.m1.2.3.2" xref="S5.T2.12.12.1.m1.2.3.2.cmml">Div</mi><mrow id="S5.T2.12.12.1.m1.2.2.2.4" xref="S5.T2.12.12.1.m1.2.2.2.3.cmml"><mi id="S5.T2.12.12.1.m1.1.1.1.1" xref="S5.T2.12.12.1.m1.1.1.1.1.cmml">top</mi><mo id="S5.T2.12.12.1.m1.2.2.2.4.1" xref="S5.T2.12.12.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.T2.12.12.1.m1.2.2.2.2" xref="S5.T2.12.12.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.12.12.1.m1.2b"><apply id="S5.T2.12.12.1.m1.2.3.cmml" xref="S5.T2.12.12.1.m1.2.3"><csymbol cd="ambiguous" id="S5.T2.12.12.1.m1.2.3.1.cmml" xref="S5.T2.12.12.1.m1.2.3">subscript</csymbol><ci id="S5.T2.12.12.1.m1.2.3.2.cmml" xref="S5.T2.12.12.1.m1.2.3.2">Div</ci><list id="S5.T2.12.12.1.m1.2.2.2.3.cmml" xref="S5.T2.12.12.1.m1.2.2.2.4"><ci id="S5.T2.12.12.1.m1.1.1.1.1.cmml" xref="S5.T2.12.12.1.m1.1.1.1.1">top</ci><ci id="S5.T2.12.12.1.m1.2.2.2.2.cmml" xref="S5.T2.12.12.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.12.12.1.m1.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</td>
<td id="S5.T2.12.12.2" class="ltx_td ltx_align_center ltx_border_t">9.98</td>
<td id="S5.T2.12.12.3" class="ltx_td ltx_align_center ltx_border_t">1.9</td>
<td id="S5.T2.12.12.4" class="ltx_td ltx_align_center ltx_border_t">23.2</td>
<td id="S5.T2.12.12.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">55.5</td>
<td id="S5.T2.12.12.6" class="ltx_td ltx_align_center ltx_border_t">36.7</td>
<td id="S5.T2.12.12.7" class="ltx_td ltx_align_center ltx_border_t">14.5</td>
<td id="S5.T2.12.12.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">29.0</td>
<td id="S5.T2.12.12.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">28.5</td>
<td id="S5.T2.12.12.10" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T2.12.12.10.1" class="ltx_text ltx_font_bold">78.5</span></td>
</tr>
<tr id="S5.T2.13.13" class="ltx_tr">
<td id="S5.T2.13.13.1" class="ltx_td ltx_align_right ltx_border_r">
<span id="S5.T2.13.13.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (<math id="S5.T2.13.13.1.m1.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.T2.13.13.1.m1.1a"><msub id="S5.T2.13.13.1.m1.1.1" xref="S5.T2.13.13.1.m1.1.1.cmml"><mi id="S5.T2.13.13.1.m1.1.1.2" xref="S5.T2.13.13.1.m1.1.1.2.cmml">Div</mi><mi id="S5.T2.13.13.1.m1.1.1.3" xref="S5.T2.13.13.1.m1.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.T2.13.13.1.m1.1b"><apply id="S5.T2.13.13.1.m1.1.1.cmml" xref="S5.T2.13.13.1.m1.1.1"><csymbol cd="ambiguous" id="S5.T2.13.13.1.m1.1.1.1.cmml" xref="S5.T2.13.13.1.m1.1.1">subscript</csymbol><ci id="S5.T2.13.13.1.m1.1.1.2.cmml" xref="S5.T2.13.13.1.m1.1.1.2">Div</ci><ci id="S5.T2.13.13.1.m1.1.1.3.cmml" xref="S5.T2.13.13.1.m1.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.13.13.1.m1.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</td>
<td id="S5.T2.13.13.2" class="ltx_td ltx_align_center">9.99</td>
<td id="S5.T2.13.13.3" class="ltx_td ltx_align_center"><span id="S5.T2.13.13.3.1" class="ltx_text ltx_font_bold">3.8</span></td>
<td id="S5.T2.13.13.4" class="ltx_td ltx_align_center">30.4</td>
<td id="S5.T2.13.13.5" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T2.13.13.5.1" class="ltx_text ltx_font_bold">63.1</span></td>
<td id="S5.T2.13.13.6" class="ltx_td ltx_align_center">34.2</td>
<td id="S5.T2.13.13.7" class="ltx_td ltx_align_center">12.8</td>
<td id="S5.T2.13.13.8" class="ltx_td ltx_align_center ltx_border_r">27.0</td>
<td id="S5.T2.13.13.9" class="ltx_td ltx_align_center ltx_border_r">23.2</td>
<td id="S5.T2.13.13.10" class="ltx_td ltx_align_center">76.6</td>
</tr>
<tr id="S5.T2.14.14" class="ltx_tr">
<td id="S5.T2.14.14.1" class="ltx_td ltx_align_right ltx_border_r">
<span id="S5.T2.14.14.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (<math id="S5.T2.14.14.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.T2.14.14.1.m1.2a"><msub id="S5.T2.14.14.1.m1.2.3" xref="S5.T2.14.14.1.m1.2.3.cmml"><mi id="S5.T2.14.14.1.m1.2.3.2" xref="S5.T2.14.14.1.m1.2.3.2.cmml">Focus</mi><mrow id="S5.T2.14.14.1.m1.2.2.2.4" xref="S5.T2.14.14.1.m1.2.2.2.3.cmml"><mi id="S5.T2.14.14.1.m1.1.1.1.1" xref="S5.T2.14.14.1.m1.1.1.1.1.cmml">sample</mi><mo id="S5.T2.14.14.1.m1.2.2.2.4.1" xref="S5.T2.14.14.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.T2.14.14.1.m1.2.2.2.2" xref="S5.T2.14.14.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.14.14.1.m1.2b"><apply id="S5.T2.14.14.1.m1.2.3.cmml" xref="S5.T2.14.14.1.m1.2.3"><csymbol cd="ambiguous" id="S5.T2.14.14.1.m1.2.3.1.cmml" xref="S5.T2.14.14.1.m1.2.3">subscript</csymbol><ci id="S5.T2.14.14.1.m1.2.3.2.cmml" xref="S5.T2.14.14.1.m1.2.3.2">Focus</ci><list id="S5.T2.14.14.1.m1.2.2.2.3.cmml" xref="S5.T2.14.14.1.m1.2.2.2.4"><ci id="S5.T2.14.14.1.m1.1.1.1.1.cmml" xref="S5.T2.14.14.1.m1.1.1.1.1">sample</ci><ci id="S5.T2.14.14.1.m1.2.2.2.2.cmml" xref="S5.T2.14.14.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.14.14.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>)</td>
<td id="S5.T2.14.14.2" class="ltx_td ltx_align_center">2.77</td>
<td id="S5.T2.14.14.3" class="ltx_td ltx_align_center">2.4</td>
<td id="S5.T2.14.14.4" class="ltx_td ltx_align_center">16.5</td>
<td id="S5.T2.14.14.5" class="ltx_td ltx_align_center ltx_border_r">34.2</td>
<td id="S5.T2.14.14.6" class="ltx_td ltx_align_center"><span id="S5.T2.14.14.6.1" class="ltx_text ltx_font_bold">37.5</span></td>
<td id="S5.T2.14.14.7" class="ltx_td ltx_align_center"><span id="S5.T2.14.14.7.1" class="ltx_text ltx_font_bold">15.4</span></td>
<td id="S5.T2.14.14.8" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T2.14.14.8.1" class="ltx_text ltx_font_bold">30.3</span></td>
<td id="S5.T2.14.14.9" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T2.14.14.9.1" class="ltx_text ltx_font_bold">33.6</span></td>
<td id="S5.T2.14.14.10" class="ltx_td ltx_align_center">77.9</td>
</tr>
<tr id="S5.T2.16.16" class="ltx_tr">
<td id="S5.T2.16.16.2" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">
<span id="S5.T2.16.16.2.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (<math id="S5.T2.15.15.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.T2.15.15.1.m1.2a"><msub id="S5.T2.15.15.1.m1.2.3" xref="S5.T2.15.15.1.m1.2.3.cmml"><mi id="S5.T2.15.15.1.m1.2.3.2" xref="S5.T2.15.15.1.m1.2.3.2.cmml">Focus</mi><mrow id="S5.T2.15.15.1.m1.2.2.2.4" xref="S5.T2.15.15.1.m1.2.2.2.3.cmml"><mi id="S5.T2.15.15.1.m1.1.1.1.1" xref="S5.T2.15.15.1.m1.1.1.1.1.cmml">sample</mi><mo id="S5.T2.15.15.1.m1.2.2.2.4.1" xref="S5.T2.15.15.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.T2.15.15.1.m1.2.2.2.2" xref="S5.T2.15.15.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.15.15.1.m1.2b"><apply id="S5.T2.15.15.1.m1.2.3.cmml" xref="S5.T2.15.15.1.m1.2.3"><csymbol cd="ambiguous" id="S5.T2.15.15.1.m1.2.3.1.cmml" xref="S5.T2.15.15.1.m1.2.3">subscript</csymbol><ci id="S5.T2.15.15.1.m1.2.3.2.cmml" xref="S5.T2.15.15.1.m1.2.3.2">Focus</ci><list id="S5.T2.15.15.1.m1.2.2.2.3.cmml" xref="S5.T2.15.15.1.m1.2.2.2.4"><ci id="S5.T2.15.15.1.m1.1.1.1.1.cmml" xref="S5.T2.15.15.1.m1.1.1.1.1">sample</ci><ci id="S5.T2.15.15.1.m1.2.2.2.2.cmml" xref="S5.T2.15.15.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.15.15.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, <math id="S5.T2.16.16.2.m2.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.T2.16.16.2.m2.2a"><msub id="S5.T2.16.16.2.m2.2.3" xref="S5.T2.16.16.2.m2.2.3.cmml"><mi id="S5.T2.16.16.2.m2.2.3.2" xref="S5.T2.16.16.2.m2.2.3.2.cmml">Div</mi><mrow id="S5.T2.16.16.2.m2.2.2.2.4" xref="S5.T2.16.16.2.m2.2.2.2.3.cmml"><mi id="S5.T2.16.16.2.m2.1.1.1.1" xref="S5.T2.16.16.2.m2.1.1.1.1.cmml">top</mi><mo id="S5.T2.16.16.2.m2.2.2.2.4.1" xref="S5.T2.16.16.2.m2.2.2.2.3.cmml">,</mo><mi id="S5.T2.16.16.2.m2.2.2.2.2" xref="S5.T2.16.16.2.m2.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.16.16.2.m2.2b"><apply id="S5.T2.16.16.2.m2.2.3.cmml" xref="S5.T2.16.16.2.m2.2.3"><csymbol cd="ambiguous" id="S5.T2.16.16.2.m2.2.3.1.cmml" xref="S5.T2.16.16.2.m2.2.3">subscript</csymbol><ci id="S5.T2.16.16.2.m2.2.3.2.cmml" xref="S5.T2.16.16.2.m2.2.3.2">Div</ci><list id="S5.T2.16.16.2.m2.2.2.2.3.cmml" xref="S5.T2.16.16.2.m2.2.2.2.4"><ci id="S5.T2.16.16.2.m2.1.1.1.1.cmml" xref="S5.T2.16.16.2.m2.1.1.1.1">top</ci><ci id="S5.T2.16.16.2.m2.2.2.2.2.cmml" xref="S5.T2.16.16.2.m2.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.16.16.2.m2.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</td>
<td id="S5.T2.16.16.3" class="ltx_td ltx_align_center ltx_border_t">8.99</td>
<td id="S5.T2.16.16.4" class="ltx_td ltx_align_center ltx_border_t">2.8</td>
<td id="S5.T2.16.16.5" class="ltx_td ltx_align_center ltx_border_t">23.0</td>
<td id="S5.T2.16.16.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">54.7</td>
<td id="S5.T2.16.16.7" class="ltx_td ltx_align_center ltx_border_t">31.5</td>
<td id="S5.T2.16.16.8" class="ltx_td ltx_align_center ltx_border_t">10.3</td>
<td id="S5.T2.16.16.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">24.4</td>
<td id="S5.T2.16.16.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">22.8</td>
<td id="S5.T2.16.16.11" class="ltx_td ltx_align_center ltx_border_t">74.7</td>
</tr>
<tr id="S5.T2.18.18" class="ltx_tr">
<td id="S5.T2.18.18.2" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r">
<span id="S5.T2.18.18.2.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (<math id="S5.T2.17.17.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.T2.17.17.1.m1.2a"><msub id="S5.T2.17.17.1.m1.2.3" xref="S5.T2.17.17.1.m1.2.3.cmml"><mi id="S5.T2.17.17.1.m1.2.3.2" xref="S5.T2.17.17.1.m1.2.3.2.cmml">Focus</mi><mrow id="S5.T2.17.17.1.m1.2.2.2.4" xref="S5.T2.17.17.1.m1.2.2.2.3.cmml"><mi id="S5.T2.17.17.1.m1.1.1.1.1" xref="S5.T2.17.17.1.m1.1.1.1.1.cmml">sample</mi><mo id="S5.T2.17.17.1.m1.2.2.2.4.1" xref="S5.T2.17.17.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.T2.17.17.1.m1.2.2.2.2" xref="S5.T2.17.17.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.T2.17.17.1.m1.2b"><apply id="S5.T2.17.17.1.m1.2.3.cmml" xref="S5.T2.17.17.1.m1.2.3"><csymbol cd="ambiguous" id="S5.T2.17.17.1.m1.2.3.1.cmml" xref="S5.T2.17.17.1.m1.2.3">subscript</csymbol><ci id="S5.T2.17.17.1.m1.2.3.2.cmml" xref="S5.T2.17.17.1.m1.2.3.2">Focus</ci><list id="S5.T2.17.17.1.m1.2.2.2.3.cmml" xref="S5.T2.17.17.1.m1.2.2.2.4"><ci id="S5.T2.17.17.1.m1.1.1.1.1.cmml" xref="S5.T2.17.17.1.m1.1.1.1.1">sample</ci><ci id="S5.T2.17.17.1.m1.2.2.2.2.cmml" xref="S5.T2.17.17.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.17.17.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, <math id="S5.T2.18.18.2.m2.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.T2.18.18.2.m2.1a"><msub id="S5.T2.18.18.2.m2.1.1" xref="S5.T2.18.18.2.m2.1.1.cmml"><mi id="S5.T2.18.18.2.m2.1.1.2" xref="S5.T2.18.18.2.m2.1.1.2.cmml">Div</mi><mi id="S5.T2.18.18.2.m2.1.1.3" xref="S5.T2.18.18.2.m2.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.T2.18.18.2.m2.1b"><apply id="S5.T2.18.18.2.m2.1.1.cmml" xref="S5.T2.18.18.2.m2.1.1"><csymbol cd="ambiguous" id="S5.T2.18.18.2.m2.1.1.1.cmml" xref="S5.T2.18.18.2.m2.1.1">subscript</csymbol><ci id="S5.T2.18.18.2.m2.1.1.2.cmml" xref="S5.T2.18.18.2.m2.1.1.2">Div</ci><ci id="S5.T2.18.18.2.m2.1.1.3.cmml" xref="S5.T2.18.18.2.m2.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T2.18.18.2.m2.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</td>
<td id="S5.T2.18.18.3" class="ltx_td ltx_align_center ltx_border_bb">9.98</td>
<td id="S5.T2.18.18.4" class="ltx_td ltx_align_center ltx_border_bb">2.6</td>
<td id="S5.T2.18.18.5" class="ltx_td ltx_align_center ltx_border_bb">20.8</td>
<td id="S5.T2.18.18.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">50.9</td>
<td id="S5.T2.18.18.7" class="ltx_td ltx_align_center ltx_border_bb">32.5</td>
<td id="S5.T2.18.18.8" class="ltx_td ltx_align_center ltx_border_bb">11.0</td>
<td id="S5.T2.18.18.9" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">25.3</td>
<td id="S5.T2.18.18.10" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">24.8</td>
<td id="S5.T2.18.18.11" class="ltx_td ltx_align_center ltx_border_bb">75.3</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Assessment of diversity, relevance and faithfulness with focus sampling on the <span id="S5.T2.20.1" class="ltx_text ltx_font_smallcaps">XSum</span> test set.</figcaption>
</figure>
<section id="S5.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S5.SS0.SSS0.Px1.1.1" class="ltx_text ltx_font_smallcaps">Fame</span> Summaries are More Fluent, Informative and Faithful.</h4>

<div id="S5.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px1.p1.1" class="ltx_p">Table <a href="#S4.T1" title="Table 1 ‣ 4.2 Pretrained Models with Fame ‣ 4 Experimental Setup ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> presents results comparing our <span id="S5.SS0.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_smallcaps">Fame</span> models, <span id="S5.SS0.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_smallcaps">RobFame</span> and <span id="S5.SS0.SSS0.Px1.p1.1.3" class="ltx_text ltx_font_smallcaps">PegFame</span>, against their counterparts <span id="S5.SS0.SSS0.Px1.p1.1.4" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> and <span id="S5.SS0.SSS0.Px1.p1.1.5" class="ltx_text ltx_font_smallcaps">Pegasus</span>, respectively. Both <span id="S5.SS0.SSS0.Px1.p1.1.6" class="ltx_text ltx_font_smallcaps">Fame</span> models clearly outperform their vanilla counterparts in terms of generating summaries that are more fluent (see RL and Rep.), more informative (see R1, R2 and BERTSc.) and more faithful (see ent., Feqa and BERTFaithful). Among all four models, <span id="S5.SS0.SSS0.Px1.p1.1.7" class="ltx_text ltx_font_smallcaps">PegFame</span> summaries are most fluent, informative and faithful.</p>
</div>
<div id="S5.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="S5.SS0.SSS0.Px1.p2.1" class="ltx_p">We further did pairwise comparisons for all measures in Table <a href="#S4.T1" title="Table 1 ‣ 4.2 Pretrained Models with Fame ‣ 4 Experimental Setup ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and found that all differences are statistically significant except for BERTScore and faithfulness measures between <span id="S5.SS0.SSS0.Px1.p2.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> and <span id="S5.SS0.SSS0.Px1.p2.1.2" class="ltx_text ltx_font_smallcaps">PegFame</span>.<span id="footnote10" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">10</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">10</sup><span class="ltx_tag ltx_tag_note">10</span>All significance tests in this work are pairwise comparisons (one-way ANOVA with posthoc Tukey HSD tests; <math id="footnote10.m1.1" class="ltx_Math" alttext="p&lt;0.01" display="inline"><semantics id="footnote10.m1.1b"><mrow id="footnote10.m1.1.1" xref="footnote10.m1.1.1.cmml"><mi id="footnote10.m1.1.1.2" xref="footnote10.m1.1.1.2.cmml">p</mi><mo id="footnote10.m1.1.1.1" xref="footnote10.m1.1.1.1.cmml">&lt;</mo><mn id="footnote10.m1.1.1.3" xref="footnote10.m1.1.1.3.cmml">0.01</mn></mrow><annotation-xml encoding="MathML-Content" id="footnote10.m1.1c"><apply id="footnote10.m1.1.1.cmml" xref="footnote10.m1.1.1"><lt id="footnote10.m1.1.1.1.cmml" xref="footnote10.m1.1.1.1"></lt><ci id="footnote10.m1.1.1.2.cmml" xref="footnote10.m1.1.1.2">𝑝</ci><cn type="float" id="footnote10.m1.1.1.3.cmml" xref="footnote10.m1.1.1.3">0.01</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote10.m1.1d">p&lt;0.01</annotation></semantics></math>).</span></span></span> These assessments demonstrate that <span id="S5.SS0.SSS0.Px1.p2.1.3" class="ltx_text ltx_font_smallcaps">Fame</span> models aid both <span id="S5.SS0.SSS0.Px1.p2.1.4" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> and <span id="S5.SS0.SSS0.Px1.p2.1.5" class="ltx_text ltx_font_smallcaps">Pegasus</span> in generating fluent, faithful and relevant summaries, but are more effective in <span id="S5.SS0.SSS0.Px1.p2.1.6" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> than in <span id="S5.SS0.SSS0.Px1.p2.1.7" class="ltx_text ltx_font_smallcaps">Pegasus</span> for extreme summarization.</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Generating Diverse and Faithful Summaries with Focus Sampling.</h4>

<div id="S5.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px2.p1.14" class="ltx_p">Table <a href="#S5.T2" title="Table 2 ‣ 5 Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> presents results assessing focus sampling (<math id="S5.SS0.SSS0.Px2.p1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.1.m1.2a"><msub id="S5.SS0.SSS0.Px2.p1.1.m1.2.3" xref="S5.SS0.SSS0.Px2.p1.1.m1.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.1.m1.2.3.2" xref="S5.SS0.SSS0.Px2.p1.1.m1.2.3.2.cmml">Focus</mi><mrow id="S5.SS0.SSS0.Px2.p1.1.m1.2.2.2.4" xref="S5.SS0.SSS0.Px2.p1.1.m1.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.1.m1.1.1.1.1" xref="S5.SS0.SSS0.Px2.p1.1.m1.1.1.1.1.cmml">sample</mi><mo id="S5.SS0.SSS0.Px2.p1.1.m1.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p1.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p1.1.m1.2.2.2.2" xref="S5.SS0.SSS0.Px2.p1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.1.m1.2b"><apply id="S5.SS0.SSS0.Px2.p1.1.m1.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.1.m1.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p1.1.m1.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p1.1.m1.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p1.1.m1.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p1.1.m1.2.3.2">Focus</ci><list id="S5.SS0.SSS0.Px2.p1.1.m1.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.1.m1.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p1.1.m1.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.1.m1.1.1.1.1">sample</ci><ci id="S5.SS0.SSS0.Px2.p1.1.m1.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>), top-<math id="S5.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.2.m2.1a"><mi id="S5.SS0.SSS0.Px2.p1.2.m2.1.1" xref="S5.SS0.SSS0.Px2.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.2.m2.1b"><ci id="S5.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.2.m2.1c">k</annotation></semantics></math> sampling (<math id="S5.SS0.SSS0.Px2.p1.3.m3.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.3.m3.2a"><msub id="S5.SS0.SSS0.Px2.p1.3.m3.2.3" xref="S5.SS0.SSS0.Px2.p1.3.m3.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.3.m3.2.3.2" xref="S5.SS0.SSS0.Px2.p1.3.m3.2.3.2.cmml">Div</mi><mrow id="S5.SS0.SSS0.Px2.p1.3.m3.2.2.2.4" xref="S5.SS0.SSS0.Px2.p1.3.m3.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.3.m3.1.1.1.1" xref="S5.SS0.SSS0.Px2.p1.3.m3.1.1.1.1.cmml">top</mi><mo id="S5.SS0.SSS0.Px2.p1.3.m3.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p1.3.m3.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p1.3.m3.2.2.2.2" xref="S5.SS0.SSS0.Px2.p1.3.m3.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.3.m3.2b"><apply id="S5.SS0.SSS0.Px2.p1.3.m3.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.3.m3.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p1.3.m3.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p1.3.m3.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p1.3.m3.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p1.3.m3.2.3.2">Div</ci><list id="S5.SS0.SSS0.Px2.p1.3.m3.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.3.m3.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p1.3.m3.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.3.m3.1.1.1.1">top</ci><ci id="S5.SS0.SSS0.Px2.p1.3.m3.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p1.3.m3.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.3.m3.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>) and nucleus sampling (<math id="S5.SS0.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.4.m4.1a"><msub id="S5.SS0.SSS0.Px2.p1.4.m4.1.1" xref="S5.SS0.SSS0.Px2.p1.4.m4.1.1.cmml"><mi id="S5.SS0.SSS0.Px2.p1.4.m4.1.1.2" xref="S5.SS0.SSS0.Px2.p1.4.m4.1.1.2.cmml">Div</mi><mi id="S5.SS0.SSS0.Px2.p1.4.m4.1.1.3" xref="S5.SS0.SSS0.Px2.p1.4.m4.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.4.m4.1b"><apply id="S5.SS0.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.4.m4.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="S5.SS0.SSS0.Px2.p1.4.m4.1.1.2">Div</ci><ci id="S5.SS0.SSS0.Px2.p1.4.m4.1.1.3.cmml" xref="S5.SS0.SSS0.Px2.p1.4.m4.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.4.m4.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>), for their abilities to generate diverse and faithful summaries. For <math id="S5.SS0.SSS0.Px2.p1.5.m5.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.5.m5.2a"><msub id="S5.SS0.SSS0.Px2.p1.5.m5.2.3" xref="S5.SS0.SSS0.Px2.p1.5.m5.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.5.m5.2.3.2" xref="S5.SS0.SSS0.Px2.p1.5.m5.2.3.2.cmml">Focus</mi><mrow id="S5.SS0.SSS0.Px2.p1.5.m5.2.2.2.4" xref="S5.SS0.SSS0.Px2.p1.5.m5.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.5.m5.1.1.1.1" xref="S5.SS0.SSS0.Px2.p1.5.m5.1.1.1.1.cmml">sample</mi><mo id="S5.SS0.SSS0.Px2.p1.5.m5.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p1.5.m5.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p1.5.m5.2.2.2.2" xref="S5.SS0.SSS0.Px2.p1.5.m5.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.5.m5.2b"><apply id="S5.SS0.SSS0.Px2.p1.5.m5.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.5.m5.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p1.5.m5.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p1.5.m5.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p1.5.m5.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p1.5.m5.2.3.2">Focus</ci><list id="S5.SS0.SSS0.Px2.p1.5.m5.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.5.m5.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p1.5.m5.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.5.m5.1.1.1.1">sample</ci><ci id="S5.SS0.SSS0.Px2.p1.5.m5.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p1.5.m5.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.5.m5.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, we choose <math id="S5.SS0.SSS0.Px2.p1.6.m6.2" class="ltx_Math" alttext="k=10,000" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.6.m6.2a"><mrow id="S5.SS0.SSS0.Px2.p1.6.m6.2.3" xref="S5.SS0.SSS0.Px2.p1.6.m6.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.6.m6.2.3.2" xref="S5.SS0.SSS0.Px2.p1.6.m6.2.3.2.cmml">k</mi><mo id="S5.SS0.SSS0.Px2.p1.6.m6.2.3.1" xref="S5.SS0.SSS0.Px2.p1.6.m6.2.3.1.cmml">=</mo><mrow id="S5.SS0.SSS0.Px2.p1.6.m6.2.3.3.2" xref="S5.SS0.SSS0.Px2.p1.6.m6.2.3.3.1.cmml"><mn id="S5.SS0.SSS0.Px2.p1.6.m6.1.1" xref="S5.SS0.SSS0.Px2.p1.6.m6.1.1.cmml">10</mn><mo id="S5.SS0.SSS0.Px2.p1.6.m6.2.3.3.2.1" xref="S5.SS0.SSS0.Px2.p1.6.m6.2.3.3.1.cmml">,</mo><mn id="S5.SS0.SSS0.Px2.p1.6.m6.2.2" xref="S5.SS0.SSS0.Px2.p1.6.m6.2.2.cmml">000</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.6.m6.2b"><apply id="S5.SS0.SSS0.Px2.p1.6.m6.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.6.m6.2.3"><eq id="S5.SS0.SSS0.Px2.p1.6.m6.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p1.6.m6.2.3.1"></eq><ci id="S5.SS0.SSS0.Px2.p1.6.m6.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p1.6.m6.2.3.2">𝑘</ci><list id="S5.SS0.SSS0.Px2.p1.6.m6.2.3.3.1.cmml" xref="S5.SS0.SSS0.Px2.p1.6.m6.2.3.3.2"><cn type="integer" id="S5.SS0.SSS0.Px2.p1.6.m6.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.6.m6.1.1">10</cn><cn type="integer" id="S5.SS0.SSS0.Px2.p1.6.m6.2.2.cmml" xref="S5.SS0.SSS0.Px2.p1.6.m6.2.2">000</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.6.m6.2c">k=10,000</annotation></semantics></math>. We follow <cite class="ltx_cite ltx_citemacro_citet">Holtzman et al. (<a href="#bib.bib34" title="" class="ltx_ref">2020</a>)</cite> and choose <math id="S5.SS0.SSS0.Px2.p1.7.m7.1" class="ltx_Math" alttext="k=640" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.7.m7.1a"><mrow id="S5.SS0.SSS0.Px2.p1.7.m7.1.1" xref="S5.SS0.SSS0.Px2.p1.7.m7.1.1.cmml"><mi id="S5.SS0.SSS0.Px2.p1.7.m7.1.1.2" xref="S5.SS0.SSS0.Px2.p1.7.m7.1.1.2.cmml">k</mi><mo id="S5.SS0.SSS0.Px2.p1.7.m7.1.1.1" xref="S5.SS0.SSS0.Px2.p1.7.m7.1.1.1.cmml">=</mo><mn id="S5.SS0.SSS0.Px2.p1.7.m7.1.1.3" xref="S5.SS0.SSS0.Px2.p1.7.m7.1.1.3.cmml">640</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.7.m7.1b"><apply id="S5.SS0.SSS0.Px2.p1.7.m7.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.7.m7.1.1"><eq id="S5.SS0.SSS0.Px2.p1.7.m7.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.7.m7.1.1.1"></eq><ci id="S5.SS0.SSS0.Px2.p1.7.m7.1.1.2.cmml" xref="S5.SS0.SSS0.Px2.p1.7.m7.1.1.2">𝑘</ci><cn type="integer" id="S5.SS0.SSS0.Px2.p1.7.m7.1.1.3.cmml" xref="S5.SS0.SSS0.Px2.p1.7.m7.1.1.3">640</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.7.m7.1c">k=640</annotation></semantics></math>
and the nucleus probability <math id="S5.SS0.SSS0.Px2.p1.8.m8.1" class="ltx_Math" alttext="p=0.95" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.8.m8.1a"><mrow id="S5.SS0.SSS0.Px2.p1.8.m8.1.1" xref="S5.SS0.SSS0.Px2.p1.8.m8.1.1.cmml"><mi id="S5.SS0.SSS0.Px2.p1.8.m8.1.1.2" xref="S5.SS0.SSS0.Px2.p1.8.m8.1.1.2.cmml">p</mi><mo id="S5.SS0.SSS0.Px2.p1.8.m8.1.1.1" xref="S5.SS0.SSS0.Px2.p1.8.m8.1.1.1.cmml">=</mo><mn id="S5.SS0.SSS0.Px2.p1.8.m8.1.1.3" xref="S5.SS0.SSS0.Px2.p1.8.m8.1.1.3.cmml">0.95</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.8.m8.1b"><apply id="S5.SS0.SSS0.Px2.p1.8.m8.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.8.m8.1.1"><eq id="S5.SS0.SSS0.Px2.p1.8.m8.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.8.m8.1.1.1"></eq><ci id="S5.SS0.SSS0.Px2.p1.8.m8.1.1.2.cmml" xref="S5.SS0.SSS0.Px2.p1.8.m8.1.1.2">𝑝</ci><cn type="float" id="S5.SS0.SSS0.Px2.p1.8.m8.1.1.3.cmml" xref="S5.SS0.SSS0.Px2.p1.8.m8.1.1.3">0.95</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.8.m8.1c">p=0.95</annotation></semantics></math>, for <math id="S5.SS0.SSS0.Px2.p1.9.m9.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.9.m9.2a"><msub id="S5.SS0.SSS0.Px2.p1.9.m9.2.3" xref="S5.SS0.SSS0.Px2.p1.9.m9.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.9.m9.2.3.2" xref="S5.SS0.SSS0.Px2.p1.9.m9.2.3.2.cmml">Div</mi><mrow id="S5.SS0.SSS0.Px2.p1.9.m9.2.2.2.4" xref="S5.SS0.SSS0.Px2.p1.9.m9.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.9.m9.1.1.1.1" xref="S5.SS0.SSS0.Px2.p1.9.m9.1.1.1.1.cmml">top</mi><mo id="S5.SS0.SSS0.Px2.p1.9.m9.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p1.9.m9.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p1.9.m9.2.2.2.2" xref="S5.SS0.SSS0.Px2.p1.9.m9.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.9.m9.2b"><apply id="S5.SS0.SSS0.Px2.p1.9.m9.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.9.m9.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p1.9.m9.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p1.9.m9.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p1.9.m9.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p1.9.m9.2.3.2">Div</ci><list id="S5.SS0.SSS0.Px2.p1.9.m9.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.9.m9.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p1.9.m9.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.9.m9.1.1.1.1">top</ci><ci id="S5.SS0.SSS0.Px2.p1.9.m9.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p1.9.m9.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.9.m9.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math> and <math id="S5.SS0.SSS0.Px2.p1.10.m10.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.10.m10.1a"><msub id="S5.SS0.SSS0.Px2.p1.10.m10.1.1" xref="S5.SS0.SSS0.Px2.p1.10.m10.1.1.cmml"><mi id="S5.SS0.SSS0.Px2.p1.10.m10.1.1.2" xref="S5.SS0.SSS0.Px2.p1.10.m10.1.1.2.cmml">Div</mi><mi id="S5.SS0.SSS0.Px2.p1.10.m10.1.1.3" xref="S5.SS0.SSS0.Px2.p1.10.m10.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.10.m10.1b"><apply id="S5.SS0.SSS0.Px2.p1.10.m10.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.10.m10.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p1.10.m10.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.10.m10.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p1.10.m10.1.1.2.cmml" xref="S5.SS0.SSS0.Px2.p1.10.m10.1.1.2">Div</ci><ci id="S5.SS0.SSS0.Px2.p1.10.m10.1.1.3.cmml" xref="S5.SS0.SSS0.Px2.p1.10.m10.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.10.m10.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>, respectively. For <math id="S5.SS0.SSS0.Px2.p1.11.m11.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.11.m11.2a"><msub id="S5.SS0.SSS0.Px2.p1.11.m11.2.3" xref="S5.SS0.SSS0.Px2.p1.11.m11.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.11.m11.2.3.2" xref="S5.SS0.SSS0.Px2.p1.11.m11.2.3.2.cmml">Focus</mi><mrow id="S5.SS0.SSS0.Px2.p1.11.m11.2.2.2.4" xref="S5.SS0.SSS0.Px2.p1.11.m11.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.11.m11.1.1.1.1" xref="S5.SS0.SSS0.Px2.p1.11.m11.1.1.1.1.cmml">sample</mi><mo id="S5.SS0.SSS0.Px2.p1.11.m11.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p1.11.m11.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p1.11.m11.2.2.2.2" xref="S5.SS0.SSS0.Px2.p1.11.m11.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.11.m11.2b"><apply id="S5.SS0.SSS0.Px2.p1.11.m11.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.11.m11.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p1.11.m11.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p1.11.m11.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p1.11.m11.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p1.11.m11.2.3.2">Focus</ci><list id="S5.SS0.SSS0.Px2.p1.11.m11.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.11.m11.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p1.11.m11.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.11.m11.1.1.1.1">sample</ci><ci id="S5.SS0.SSS0.Px2.p1.11.m11.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p1.11.m11.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.11.m11.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, we decode with a beam size of 4. We also report <math id="S5.SS0.SSS0.Px2.p1.12.m12.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.12.m12.2a"><msub id="S5.SS0.SSS0.Px2.p1.12.m12.2.3" xref="S5.SS0.SSS0.Px2.p1.12.m12.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.12.m12.2.3.2" xref="S5.SS0.SSS0.Px2.p1.12.m12.2.3.2.cmml">Focus</mi><mrow id="S5.SS0.SSS0.Px2.p1.12.m12.2.2.2.4" xref="S5.SS0.SSS0.Px2.p1.12.m12.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.12.m12.1.1.1.1" xref="S5.SS0.SSS0.Px2.p1.12.m12.1.1.1.1.cmml">sample</mi><mo id="S5.SS0.SSS0.Px2.p1.12.m12.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p1.12.m12.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p1.12.m12.2.2.2.2" xref="S5.SS0.SSS0.Px2.p1.12.m12.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.12.m12.2b"><apply id="S5.SS0.SSS0.Px2.p1.12.m12.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.12.m12.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p1.12.m12.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p1.12.m12.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p1.12.m12.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p1.12.m12.2.3.2">Focus</ci><list id="S5.SS0.SSS0.Px2.p1.12.m12.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.12.m12.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p1.12.m12.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.12.m12.1.1.1.1">sample</ci><ci id="S5.SS0.SSS0.Px2.p1.12.m12.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p1.12.m12.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.12.m12.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math> with <math id="S5.SS0.SSS0.Px2.p1.13.m13.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.13.m13.2a"><msub id="S5.SS0.SSS0.Px2.p1.13.m13.2.3" xref="S5.SS0.SSS0.Px2.p1.13.m13.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.13.m13.2.3.2" xref="S5.SS0.SSS0.Px2.p1.13.m13.2.3.2.cmml">Div</mi><mrow id="S5.SS0.SSS0.Px2.p1.13.m13.2.2.2.4" xref="S5.SS0.SSS0.Px2.p1.13.m13.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p1.13.m13.1.1.1.1" xref="S5.SS0.SSS0.Px2.p1.13.m13.1.1.1.1.cmml">top</mi><mo id="S5.SS0.SSS0.Px2.p1.13.m13.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p1.13.m13.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p1.13.m13.2.2.2.2" xref="S5.SS0.SSS0.Px2.p1.13.m13.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.13.m13.2b"><apply id="S5.SS0.SSS0.Px2.p1.13.m13.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.13.m13.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p1.13.m13.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p1.13.m13.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p1.13.m13.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p1.13.m13.2.3.2">Div</ci><list id="S5.SS0.SSS0.Px2.p1.13.m13.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p1.13.m13.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p1.13.m13.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.13.m13.1.1.1.1">top</ci><ci id="S5.SS0.SSS0.Px2.p1.13.m13.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p1.13.m13.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.13.m13.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math> and <math id="S5.SS0.SSS0.Px2.p1.14.m14.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.14.m14.1a"><msub id="S5.SS0.SSS0.Px2.p1.14.m14.1.1" xref="S5.SS0.SSS0.Px2.p1.14.m14.1.1.cmml"><mi id="S5.SS0.SSS0.Px2.p1.14.m14.1.1.2" xref="S5.SS0.SSS0.Px2.p1.14.m14.1.1.2.cmml">Div</mi><mi id="S5.SS0.SSS0.Px2.p1.14.m14.1.1.3" xref="S5.SS0.SSS0.Px2.p1.14.m14.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.14.m14.1b"><apply id="S5.SS0.SSS0.Px2.p1.14.m14.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.14.m14.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p1.14.m14.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.14.m14.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p1.14.m14.1.1.2.cmml" xref="S5.SS0.SSS0.Px2.p1.14.m14.1.1.2">Div</ci><ci id="S5.SS0.SSS0.Px2.p1.14.m14.1.1.3.cmml" xref="S5.SS0.SSS0.Px2.p1.14.m14.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.14.m14.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math> to assess if they can benefit one-another.
In each setting we sample 10 summaries for each input document.
For all metrics, we report the average over all 10 samples.<span id="footnote11" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">11</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">11</sup><span class="ltx_tag ltx_tag_note">11</span>Feqa and BERTFaithful scores are dropped due to time constraints.</span></span></span></p>
</div>
<div id="S5.SS0.SSS0.Px2.p2" class="ltx_para">
<p id="S5.SS0.SSS0.Px2.p2.6" class="ltx_p">Both <math id="S5.SS0.SSS0.Px2.p2.1.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p2.1.m1.2a"><msub id="S5.SS0.SSS0.Px2.p2.1.m1.2.3" xref="S5.SS0.SSS0.Px2.p2.1.m1.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p2.1.m1.2.3.2" xref="S5.SS0.SSS0.Px2.p2.1.m1.2.3.2.cmml">Div</mi><mrow id="S5.SS0.SSS0.Px2.p2.1.m1.2.2.2.4" xref="S5.SS0.SSS0.Px2.p2.1.m1.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p2.1.m1.1.1.1.1" xref="S5.SS0.SSS0.Px2.p2.1.m1.1.1.1.1.cmml">top</mi><mo id="S5.SS0.SSS0.Px2.p2.1.m1.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p2.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p2.1.m1.2.2.2.2" xref="S5.SS0.SSS0.Px2.p2.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p2.1.m1.2b"><apply id="S5.SS0.SSS0.Px2.p2.1.m1.2.3.cmml" xref="S5.SS0.SSS0.Px2.p2.1.m1.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p2.1.m1.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p2.1.m1.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p2.1.m1.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p2.1.m1.2.3.2">Div</ci><list id="S5.SS0.SSS0.Px2.p2.1.m1.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p2.1.m1.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p2.1.m1.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p2.1.m1.1.1.1.1">top</ci><ci id="S5.SS0.SSS0.Px2.p2.1.m1.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p2.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p2.1.m1.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math> and <math id="S5.SS0.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p2.2.m2.1a"><msub id="S5.SS0.SSS0.Px2.p2.2.m2.1.1" xref="S5.SS0.SSS0.Px2.p2.2.m2.1.1.cmml"><mi id="S5.SS0.SSS0.Px2.p2.2.m2.1.1.2" xref="S5.SS0.SSS0.Px2.p2.2.m2.1.1.2.cmml">Div</mi><mi id="S5.SS0.SSS0.Px2.p2.2.m2.1.1.3" xref="S5.SS0.SSS0.Px2.p2.2.m2.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p2.2.m2.1b"><apply id="S5.SS0.SSS0.Px2.p2.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px2.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p2.2.m2.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p2.2.m2.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p2.2.m2.1.1.2.cmml" xref="S5.SS0.SSS0.Px2.p2.2.m2.1.1.2">Div</ci><ci id="S5.SS0.SSS0.Px2.p2.2.m2.1.1.3.cmml" xref="S5.SS0.SSS0.Px2.p2.2.m2.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p2.2.m2.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math> almost always generate a new summary. In comparison <math id="S5.SS0.SSS0.Px2.p2.3.m3.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p2.3.m3.2a"><msub id="S5.SS0.SSS0.Px2.p2.3.m3.2.3" xref="S5.SS0.SSS0.Px2.p2.3.m3.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p2.3.m3.2.3.2" xref="S5.SS0.SSS0.Px2.p2.3.m3.2.3.2.cmml">Focus</mi><mrow id="S5.SS0.SSS0.Px2.p2.3.m3.2.2.2.4" xref="S5.SS0.SSS0.Px2.p2.3.m3.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p2.3.m3.1.1.1.1" xref="S5.SS0.SSS0.Px2.p2.3.m3.1.1.1.1.cmml">sample</mi><mo id="S5.SS0.SSS0.Px2.p2.3.m3.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p2.3.m3.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p2.3.m3.2.2.2.2" xref="S5.SS0.SSS0.Px2.p2.3.m3.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p2.3.m3.2b"><apply id="S5.SS0.SSS0.Px2.p2.3.m3.2.3.cmml" xref="S5.SS0.SSS0.Px2.p2.3.m3.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p2.3.m3.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p2.3.m3.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p2.3.m3.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p2.3.m3.2.3.2">Focus</ci><list id="S5.SS0.SSS0.Px2.p2.3.m3.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p2.3.m3.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p2.3.m3.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p2.3.m3.1.1.1.1">sample</ci><ci id="S5.SS0.SSS0.Px2.p2.3.m3.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p2.3.m3.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p2.3.m3.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math> generates 1.61 and 2.77 unique summaries using <span id="S5.SS0.SSS0.Px2.p2.6.1" class="ltx_text ltx_font_smallcaps">RobFame</span> and <span id="S5.SS0.SSS0.Px2.p2.6.2" class="ltx_text ltx_font_smallcaps">PegFame</span> models, respectively. <math id="S5.SS0.SSS0.Px2.p2.4.m4.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p2.4.m4.1a"><msub id="S5.SS0.SSS0.Px2.p2.4.m4.1.1" xref="S5.SS0.SSS0.Px2.p2.4.m4.1.1.cmml"><mi id="S5.SS0.SSS0.Px2.p2.4.m4.1.1.2" xref="S5.SS0.SSS0.Px2.p2.4.m4.1.1.2.cmml">Div</mi><mi id="S5.SS0.SSS0.Px2.p2.4.m4.1.1.3" xref="S5.SS0.SSS0.Px2.p2.4.m4.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p2.4.m4.1b"><apply id="S5.SS0.SSS0.Px2.p2.4.m4.1.1.cmml" xref="S5.SS0.SSS0.Px2.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p2.4.m4.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p2.4.m4.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p2.4.m4.1.1.2.cmml" xref="S5.SS0.SSS0.Px2.p2.4.m4.1.1.2">Div</ci><ci id="S5.SS0.SSS0.Px2.p2.4.m4.1.1.3.cmml" xref="S5.SS0.SSS0.Px2.p2.4.m4.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p2.4.m4.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math> tends to generate the most distinct unigrams, bigrams, and trigrams. Interestingly, <math id="S5.SS0.SSS0.Px2.p2.5.m5.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p2.5.m5.2a"><msub id="S5.SS0.SSS0.Px2.p2.5.m5.2.3" xref="S5.SS0.SSS0.Px2.p2.5.m5.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p2.5.m5.2.3.2" xref="S5.SS0.SSS0.Px2.p2.5.m5.2.3.2.cmml">Focus</mi><mrow id="S5.SS0.SSS0.Px2.p2.5.m5.2.2.2.4" xref="S5.SS0.SSS0.Px2.p2.5.m5.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p2.5.m5.1.1.1.1" xref="S5.SS0.SSS0.Px2.p2.5.m5.1.1.1.1.cmml">sample</mi><mo id="S5.SS0.SSS0.Px2.p2.5.m5.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p2.5.m5.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p2.5.m5.2.2.2.2" xref="S5.SS0.SSS0.Px2.p2.5.m5.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p2.5.m5.2b"><apply id="S5.SS0.SSS0.Px2.p2.5.m5.2.3.cmml" xref="S5.SS0.SSS0.Px2.p2.5.m5.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p2.5.m5.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p2.5.m5.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p2.5.m5.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p2.5.m5.2.3.2">Focus</ci><list id="S5.SS0.SSS0.Px2.p2.5.m5.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p2.5.m5.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p2.5.m5.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p2.5.m5.1.1.1.1">sample</ci><ci id="S5.SS0.SSS0.Px2.p2.5.m5.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p2.5.m5.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p2.5.m5.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math> summaries have a more diverse collection of unigrams than in <math id="S5.SS0.SSS0.Px2.p2.6.m6.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p2.6.m6.2a"><msub id="S5.SS0.SSS0.Px2.p2.6.m6.2.3" xref="S5.SS0.SSS0.Px2.p2.6.m6.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p2.6.m6.2.3.2" xref="S5.SS0.SSS0.Px2.p2.6.m6.2.3.2.cmml">Div</mi><mrow id="S5.SS0.SSS0.Px2.p2.6.m6.2.2.2.4" xref="S5.SS0.SSS0.Px2.p2.6.m6.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p2.6.m6.1.1.1.1" xref="S5.SS0.SSS0.Px2.p2.6.m6.1.1.1.1.cmml">top</mi><mo id="S5.SS0.SSS0.Px2.p2.6.m6.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p2.6.m6.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p2.6.m6.2.2.2.2" xref="S5.SS0.SSS0.Px2.p2.6.m6.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p2.6.m6.2b"><apply id="S5.SS0.SSS0.Px2.p2.6.m6.2.3.cmml" xref="S5.SS0.SSS0.Px2.p2.6.m6.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p2.6.m6.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p2.6.m6.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p2.6.m6.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p2.6.m6.2.3.2">Div</ci><list id="S5.SS0.SSS0.Px2.p2.6.m6.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p2.6.m6.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p2.6.m6.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p2.6.m6.1.1.1.1">top</ci><ci id="S5.SS0.SSS0.Px2.p2.6.m6.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p2.6.m6.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p2.6.m6.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math> summaries (3.5% vs 2.3% for <span id="S5.SS0.SSS0.Px2.p2.6.3" class="ltx_text ltx_font_smallcaps">RobFame</span> and 2.4% vs 1.9% for <span id="S5.SS0.SSS0.Px2.p2.6.4" class="ltx_text ltx_font_smallcaps">PegFame</span>).</p>
</div>
<div id="S5.SS0.SSS0.Px2.p3" class="ltx_para">
<p id="S5.SS0.SSS0.Px2.p3.7" class="ltx_p">The high diversity in <math id="S5.SS0.SSS0.Px2.p3.1.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p3.1.m1.2a"><msub id="S5.SS0.SSS0.Px2.p3.1.m1.2.3" xref="S5.SS0.SSS0.Px2.p3.1.m1.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p3.1.m1.2.3.2" xref="S5.SS0.SSS0.Px2.p3.1.m1.2.3.2.cmml">Div</mi><mrow id="S5.SS0.SSS0.Px2.p3.1.m1.2.2.2.4" xref="S5.SS0.SSS0.Px2.p3.1.m1.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p3.1.m1.1.1.1.1" xref="S5.SS0.SSS0.Px2.p3.1.m1.1.1.1.1.cmml">top</mi><mo id="S5.SS0.SSS0.Px2.p3.1.m1.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p3.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p3.1.m1.2.2.2.2" xref="S5.SS0.SSS0.Px2.p3.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p3.1.m1.2b"><apply id="S5.SS0.SSS0.Px2.p3.1.m1.2.3.cmml" xref="S5.SS0.SSS0.Px2.p3.1.m1.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p3.1.m1.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p3.1.m1.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p3.1.m1.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p3.1.m1.2.3.2">Div</ci><list id="S5.SS0.SSS0.Px2.p3.1.m1.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p3.1.m1.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p3.1.m1.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p3.1.m1.1.1.1.1">top</ci><ci id="S5.SS0.SSS0.Px2.p3.1.m1.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p3.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p3.1.m1.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math> and <math id="S5.SS0.SSS0.Px2.p3.2.m2.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p3.2.m2.1a"><msub id="S5.SS0.SSS0.Px2.p3.2.m2.1.1" xref="S5.SS0.SSS0.Px2.p3.2.m2.1.1.cmml"><mi id="S5.SS0.SSS0.Px2.p3.2.m2.1.1.2" xref="S5.SS0.SSS0.Px2.p3.2.m2.1.1.2.cmml">Div</mi><mi id="S5.SS0.SSS0.Px2.p3.2.m2.1.1.3" xref="S5.SS0.SSS0.Px2.p3.2.m2.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p3.2.m2.1b"><apply id="S5.SS0.SSS0.Px2.p3.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p3.2.m2.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p3.2.m2.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p3.2.m2.1.1.2.cmml" xref="S5.SS0.SSS0.Px2.p3.2.m2.1.1.2">Div</ci><ci id="S5.SS0.SSS0.Px2.p3.2.m2.1.1.3.cmml" xref="S5.SS0.SSS0.Px2.p3.2.m2.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p3.2.m2.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math> comes at the cost of faithfulness; summaries generated with these sampling techniques have poor entailment scores. <math id="S5.SS0.SSS0.Px2.p3.3.m3.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p3.3.m3.2a"><msub id="S5.SS0.SSS0.Px2.p3.3.m3.2.3" xref="S5.SS0.SSS0.Px2.p3.3.m3.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p3.3.m3.2.3.2" xref="S5.SS0.SSS0.Px2.p3.3.m3.2.3.2.cmml">Focus</mi><mrow id="S5.SS0.SSS0.Px2.p3.3.m3.2.2.2.4" xref="S5.SS0.SSS0.Px2.p3.3.m3.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p3.3.m3.1.1.1.1" xref="S5.SS0.SSS0.Px2.p3.3.m3.1.1.1.1.cmml">sample</mi><mo id="S5.SS0.SSS0.Px2.p3.3.m3.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p3.3.m3.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p3.3.m3.2.2.2.2" xref="S5.SS0.SSS0.Px2.p3.3.m3.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p3.3.m3.2b"><apply id="S5.SS0.SSS0.Px2.p3.3.m3.2.3.cmml" xref="S5.SS0.SSS0.Px2.p3.3.m3.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p3.3.m3.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p3.3.m3.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p3.3.m3.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p3.3.m3.2.3.2">Focus</ci><list id="S5.SS0.SSS0.Px2.p3.3.m3.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p3.3.m3.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p3.3.m3.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p3.3.m3.1.1.1.1">sample</ci><ci id="S5.SS0.SSS0.Px2.p3.3.m3.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p3.3.m3.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p3.3.m3.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, on the other hand, generates summaries which entail documents the most. It also has the highest <span id="S5.SS0.SSS0.Px2.p3.7.1" class="ltx_text ltx_font_smallcaps">rouge</span> scores across the board. Some of the generated examples can be seen in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. More predictions from other models can be found in Appendix <a href="#A5" title="Appendix E Diverse Summarization with Div_{top,𝑘}, Div_nucleus and Focus_{sample,𝑘} ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>. Augmenting <math id="S5.SS0.SSS0.Px2.p3.4.m4.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p3.4.m4.2a"><msub id="S5.SS0.SSS0.Px2.p3.4.m4.2.3" xref="S5.SS0.SSS0.Px2.p3.4.m4.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p3.4.m4.2.3.2" xref="S5.SS0.SSS0.Px2.p3.4.m4.2.3.2.cmml">Div</mi><mrow id="S5.SS0.SSS0.Px2.p3.4.m4.2.2.2.4" xref="S5.SS0.SSS0.Px2.p3.4.m4.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p3.4.m4.1.1.1.1" xref="S5.SS0.SSS0.Px2.p3.4.m4.1.1.1.1.cmml">top</mi><mo id="S5.SS0.SSS0.Px2.p3.4.m4.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p3.4.m4.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p3.4.m4.2.2.2.2" xref="S5.SS0.SSS0.Px2.p3.4.m4.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p3.4.m4.2b"><apply id="S5.SS0.SSS0.Px2.p3.4.m4.2.3.cmml" xref="S5.SS0.SSS0.Px2.p3.4.m4.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p3.4.m4.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p3.4.m4.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p3.4.m4.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p3.4.m4.2.3.2">Div</ci><list id="S5.SS0.SSS0.Px2.p3.4.m4.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p3.4.m4.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p3.4.m4.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p3.4.m4.1.1.1.1">top</ci><ci id="S5.SS0.SSS0.Px2.p3.4.m4.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p3.4.m4.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p3.4.m4.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math> and <math id="S5.SS0.SSS0.Px2.p3.5.m5.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p3.5.m5.1a"><msub id="S5.SS0.SSS0.Px2.p3.5.m5.1.1" xref="S5.SS0.SSS0.Px2.p3.5.m5.1.1.cmml"><mi id="S5.SS0.SSS0.Px2.p3.5.m5.1.1.2" xref="S5.SS0.SSS0.Px2.p3.5.m5.1.1.2.cmml">Div</mi><mi id="S5.SS0.SSS0.Px2.p3.5.m5.1.1.3" xref="S5.SS0.SSS0.Px2.p3.5.m5.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p3.5.m5.1b"><apply id="S5.SS0.SSS0.Px2.p3.5.m5.1.1.cmml" xref="S5.SS0.SSS0.Px2.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p3.5.m5.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p3.5.m5.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p3.5.m5.1.1.2.cmml" xref="S5.SS0.SSS0.Px2.p3.5.m5.1.1.2">Div</ci><ci id="S5.SS0.SSS0.Px2.p3.5.m5.1.1.3.cmml" xref="S5.SS0.SSS0.Px2.p3.5.m5.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p3.5.m5.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math> with <math id="S5.SS0.SSS0.Px2.p3.6.m6.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p3.6.m6.2a"><msub id="S5.SS0.SSS0.Px2.p3.6.m6.2.3" xref="S5.SS0.SSS0.Px2.p3.6.m6.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p3.6.m6.2.3.2" xref="S5.SS0.SSS0.Px2.p3.6.m6.2.3.2.cmml">Focus</mi><mrow id="S5.SS0.SSS0.Px2.p3.6.m6.2.2.2.4" xref="S5.SS0.SSS0.Px2.p3.6.m6.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p3.6.m6.1.1.1.1" xref="S5.SS0.SSS0.Px2.p3.6.m6.1.1.1.1.cmml">sample</mi><mo id="S5.SS0.SSS0.Px2.p3.6.m6.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p3.6.m6.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p3.6.m6.2.2.2.2" xref="S5.SS0.SSS0.Px2.p3.6.m6.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p3.6.m6.2b"><apply id="S5.SS0.SSS0.Px2.p3.6.m6.2.3.cmml" xref="S5.SS0.SSS0.Px2.p3.6.m6.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p3.6.m6.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p3.6.m6.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p3.6.m6.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p3.6.m6.2.3.2">Focus</ci><list id="S5.SS0.SSS0.Px2.p3.6.m6.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p3.6.m6.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p3.6.m6.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p3.6.m6.1.1.1.1">sample</ci><ci id="S5.SS0.SSS0.Px2.p3.6.m6.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p3.6.m6.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p3.6.m6.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math> is not desirable because, though it increases diversity in terms of uniqueness and Distinct-<math id="S5.SS0.SSS0.Px2.p3.7.m7.1" class="ltx_Math" alttext="3" display="inline"><semantics id="S5.SS0.SSS0.Px2.p3.7.m7.1a"><mn id="S5.SS0.SSS0.Px2.p3.7.m7.1.1" xref="S5.SS0.SSS0.Px2.p3.7.m7.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p3.7.m7.1b"><cn type="integer" id="S5.SS0.SSS0.Px2.p3.7.m7.1.1.cmml" xref="S5.SS0.SSS0.Px2.p3.7.m7.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p3.7.m7.1c">3</annotation></semantics></math> scores, faithfulness suffers again.</p>
</div>
<div id="S5.SS0.SSS0.Px2.p4" class="ltx_para">
<p id="S5.SS0.SSS0.Px2.p4.4" class="ltx_p">Comparing results in Table <a href="#S5.T2" title="Table 2 ‣ 5 Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> to the results in Table <a href="#S4.T1" title="Table 1 ‣ 4.2 Pretrained Models with Fame ‣ 4 Experimental Setup ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, it is clear that diversity comes at the cost of quality (e.g., RL/ent. scores for <span id="S5.SS0.SSS0.Px2.p4.4.1" class="ltx_text ltx_font_smallcaps">RobFame</span> and <span id="S5.SS0.SSS0.Px2.p4.4.2" class="ltx_text ltx_font_smallcaps">RobFame</span>-<math id="S5.SS0.SSS0.Px2.p4.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p4.1.m1.2a"><msub id="S5.SS0.SSS0.Px2.p4.1.m1.2.3" xref="S5.SS0.SSS0.Px2.p4.1.m1.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p4.1.m1.2.3.2" xref="S5.SS0.SSS0.Px2.p4.1.m1.2.3.2.cmml">Focus</mi><mrow id="S5.SS0.SSS0.Px2.p4.1.m1.2.2.2.4" xref="S5.SS0.SSS0.Px2.p4.1.m1.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p4.1.m1.1.1.1.1" xref="S5.SS0.SSS0.Px2.p4.1.m1.1.1.1.1.cmml">sample</mi><mo id="S5.SS0.SSS0.Px2.p4.1.m1.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p4.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p4.1.m1.2.2.2.2" xref="S5.SS0.SSS0.Px2.p4.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p4.1.m1.2b"><apply id="S5.SS0.SSS0.Px2.p4.1.m1.2.3.cmml" xref="S5.SS0.SSS0.Px2.p4.1.m1.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p4.1.m1.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p4.1.m1.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p4.1.m1.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p4.1.m1.2.3.2">Focus</ci><list id="S5.SS0.SSS0.Px2.p4.1.m1.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p4.1.m1.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p4.1.m1.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p4.1.m1.1.1.1.1">sample</ci><ci id="S5.SS0.SSS0.Px2.p4.1.m1.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p4.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p4.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math> are 34.81/41.3 and 31.0/34.3, respectively). However, <math id="S5.SS0.SSS0.Px2.p4.2.m2.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p4.2.m2.2a"><msub id="S5.SS0.SSS0.Px2.p4.2.m2.2.3" xref="S5.SS0.SSS0.Px2.p4.2.m2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p4.2.m2.2.3.2" xref="S5.SS0.SSS0.Px2.p4.2.m2.2.3.2.cmml">Focus</mi><mrow id="S5.SS0.SSS0.Px2.p4.2.m2.2.2.2.4" xref="S5.SS0.SSS0.Px2.p4.2.m2.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p4.2.m2.1.1.1.1" xref="S5.SS0.SSS0.Px2.p4.2.m2.1.1.1.1.cmml">sample</mi><mo id="S5.SS0.SSS0.Px2.p4.2.m2.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p4.2.m2.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p4.2.m2.2.2.2.2" xref="S5.SS0.SSS0.Px2.p4.2.m2.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p4.2.m2.2b"><apply id="S5.SS0.SSS0.Px2.p4.2.m2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p4.2.m2.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p4.2.m2.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p4.2.m2.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p4.2.m2.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p4.2.m2.2.3.2">Focus</ci><list id="S5.SS0.SSS0.Px2.p4.2.m2.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p4.2.m2.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p4.2.m2.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p4.2.m2.1.1.1.1">sample</ci><ci id="S5.SS0.SSS0.Px2.p4.2.m2.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p4.2.m2.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p4.2.m2.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math> is superior to both <math id="S5.SS0.SSS0.Px2.p4.3.m3.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p4.3.m3.2a"><msub id="S5.SS0.SSS0.Px2.p4.3.m3.2.3" xref="S5.SS0.SSS0.Px2.p4.3.m3.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p4.3.m3.2.3.2" xref="S5.SS0.SSS0.Px2.p4.3.m3.2.3.2.cmml">Div</mi><mrow id="S5.SS0.SSS0.Px2.p4.3.m3.2.2.2.4" xref="S5.SS0.SSS0.Px2.p4.3.m3.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px2.p4.3.m3.1.1.1.1" xref="S5.SS0.SSS0.Px2.p4.3.m3.1.1.1.1.cmml">top</mi><mo id="S5.SS0.SSS0.Px2.p4.3.m3.2.2.2.4.1" xref="S5.SS0.SSS0.Px2.p4.3.m3.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px2.p4.3.m3.2.2.2.2" xref="S5.SS0.SSS0.Px2.p4.3.m3.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p4.3.m3.2b"><apply id="S5.SS0.SSS0.Px2.p4.3.m3.2.3.cmml" xref="S5.SS0.SSS0.Px2.p4.3.m3.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p4.3.m3.2.3.1.cmml" xref="S5.SS0.SSS0.Px2.p4.3.m3.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p4.3.m3.2.3.2.cmml" xref="S5.SS0.SSS0.Px2.p4.3.m3.2.3.2">Div</ci><list id="S5.SS0.SSS0.Px2.p4.3.m3.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px2.p4.3.m3.2.2.2.4"><ci id="S5.SS0.SSS0.Px2.p4.3.m3.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p4.3.m3.1.1.1.1">top</ci><ci id="S5.SS0.SSS0.Px2.p4.3.m3.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px2.p4.3.m3.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p4.3.m3.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math> and <math id="S5.SS0.SSS0.Px2.p4.4.m4.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p4.4.m4.1a"><msub id="S5.SS0.SSS0.Px2.p4.4.m4.1.1" xref="S5.SS0.SSS0.Px2.p4.4.m4.1.1.cmml"><mi id="S5.SS0.SSS0.Px2.p4.4.m4.1.1.2" xref="S5.SS0.SSS0.Px2.p4.4.m4.1.1.2.cmml">Div</mi><mi id="S5.SS0.SSS0.Px2.p4.4.m4.1.1.3" xref="S5.SS0.SSS0.Px2.p4.4.m4.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p4.4.m4.1b"><apply id="S5.SS0.SSS0.Px2.p4.4.m4.1.1.cmml" xref="S5.SS0.SSS0.Px2.p4.4.m4.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p4.4.m4.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p4.4.m4.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px2.p4.4.m4.1.1.2.cmml" xref="S5.SS0.SSS0.Px2.p4.4.m4.1.1.2">Div</ci><ci id="S5.SS0.SSS0.Px2.p4.4.m4.1.1.3.cmml" xref="S5.SS0.SSS0.Px2.p4.4.m4.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p4.4.m4.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math> in generating better quality summaries.</p>
</div>
<figure id="S5.F3" class="ltx_figure"><img src="/html/2105.11921/assets/imgs/top40.png" id="S5.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="468" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Top 40 sentence pieces and their logits from topic distribution <math id="S5.F3.2.m1.1" class="ltx_Math" alttext="t_{X}" display="inline"><semantics id="S5.F3.2.m1.1b"><msub id="S5.F3.2.m1.1.1" xref="S5.F3.2.m1.1.1.cmml"><mi id="S5.F3.2.m1.1.1.2" xref="S5.F3.2.m1.1.1.2.cmml">t</mi><mi id="S5.F3.2.m1.1.1.3" xref="S5.F3.2.m1.1.1.3.cmml">X</mi></msub><annotation-xml encoding="MathML-Content" id="S5.F3.2.m1.1c"><apply id="S5.F3.2.m1.1.1.cmml" xref="S5.F3.2.m1.1.1"><csymbol cd="ambiguous" id="S5.F3.2.m1.1.1.1.cmml" xref="S5.F3.2.m1.1.1">subscript</csymbol><ci id="S5.F3.2.m1.1.1.2.cmml" xref="S5.F3.2.m1.1.1.2">𝑡</ci><ci id="S5.F3.2.m1.1.1.3.cmml" xref="S5.F3.2.m1.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.F3.2.m1.1d">t_{X}</annotation></semantics></math> in <span id="S5.F3.6.1" class="ltx_text ltx_font_smallcaps">RobFame</span> and <span id="S5.F3.7.2" class="ltx_text ltx_font_smallcaps">PegFame</span> for the <span id="S5.F3.8.3" class="ltx_text ltx_font_smallcaps">XSum</span> article discussed in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
</figcaption>
</figure>
<figure id="S5.F4" class="ltx_figure"><img src="/html/2105.11921/assets/imgs/topk_subplots.png" id="S5.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="349" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span><span id="S5.F4.9.1" class="ltx_text ltx_font_smallcaps">rouge-1</span> F1 scores of <span id="S5.F4.10.2" class="ltx_text ltx_font_smallcaps">RobFame</span> and <span id="S5.F4.11.3" class="ltx_text ltx_font_smallcaps">PegFame</span> models with different top-<math id="S5.F4.2.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.F4.2.m1.1b"><mi id="S5.F4.2.m1.1.1" xref="S5.F4.2.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.F4.2.m1.1c"><ci id="S5.F4.2.m1.1.1.cmml" xref="S5.F4.2.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F4.2.m1.1d">k</annotation></semantics></math> vocabularies (Eq. equation <a href="#S3.E8" title="In Focus Sampling: Promoting Diversity in Faithful Generation ‣ 3 Summarization with Focus Attention ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>) on the <span id="S5.F4.12.4" class="ltx_text ltx_font_smallcaps">XSum</span> test set. Similar patters are observed for <span id="S5.F4.13.5" class="ltx_text ltx_font_smallcaps">rouge-2</span> and <span id="S5.F4.14.6" class="ltx_text ltx_font_smallcaps">rouge-l</span> scores.</figcaption>
</figure>
</section>
<section id="S5.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Focus Attention and Sampling Work Differently in <span id="S5.SS0.SSS0.Px3.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> and <span id="S5.SS0.SSS0.Px3.2.2" class="ltx_text ltx_font_smallcaps">PegFame</span>.</h4>

<div id="S5.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px3.p1.1" class="ltx_p">Since both encoder-decoder and focus attention parameters of <span id="S5.SS0.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> are randomly initialized, they learn to compliment each other and learn a peaky topic distribution. On the other hand, since <span id="S5.SS0.SSS0.Px3.p1.1.2" class="ltx_text ltx_font_smallcaps">PegFame</span>’s encoder-decoder attention is pre-trained, there is a push-pull effect between it and focus attention. This results in a smoother topic distribution, as seen in Figure <a href="#S5.F3" title="Figure 3 ‣ Generating Diverse and Faithful Summaries with Focus Sampling. ‣ 5 Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.<span id="footnote12" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">12</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">12</sup><span class="ltx_tag ltx_tag_note">12</span>This difference in topic distributions is consistent across the whole test set. We compute the peakiness score of a topic distribution as the slope of the line connecting logits of the top-1st token to the top-100th token. The average peakiness scores across the <span id="footnote12.1" class="ltx_text ltx_font_smallcaps">XSum</span> testset for <span id="footnote12.2" class="ltx_text ltx_font_smallcaps">RobFame</span> and <span id="footnote12.3" class="ltx_text ltx_font_smallcaps">PegFame</span> are 1.25 (51<sup id="footnote12.4" class="ltx_sup"><span id="footnote12.4.1" class="ltx_text ltx_font_italic">∘</span></sup>) and 0.45 (24.3<sup id="footnote12.5" class="ltx_sup"><span id="footnote12.5.1" class="ltx_text ltx_font_italic">∘</span></sup>), respectively.</span></span></span></p>
</div>
<div id="S5.SS0.SSS0.Px3.p2" class="ltx_para">
<p id="S5.SS0.SSS0.Px3.p2.3" class="ltx_p">Although we see that both models’ token sets capture the target intent well, the peaky distribution of <span id="S5.SS0.SSS0.Px3.p2.3.1" class="ltx_text ltx_font_smallcaps">RobFame</span> enables more accurate predictions than that of <span id="S5.SS0.SSS0.Px3.p2.3.2" class="ltx_text ltx_font_smallcaps">PegFame</span>, in a controlled generation setting. A comparison is presented in Figure <a href="#S5.F4" title="Figure 4 ‣ Generating Diverse and Faithful Summaries with Focus Sampling. ‣ 5 Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> where we show how <span id="S5.SS0.SSS0.Px3.p2.3.3" class="ltx_text ltx_font_smallcaps">rouge-1</span> scores vary when we use only top-<math id="S5.SS0.SSS0.Px3.p2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS0.SSS0.Px3.p2.1.m1.1a"><mi id="S5.SS0.SSS0.Px3.p2.1.m1.1.1" xref="S5.SS0.SSS0.Px3.p2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px3.p2.1.m1.1b"><ci id="S5.SS0.SSS0.Px3.p2.1.m1.1.1.cmml" xref="S5.SS0.SSS0.Px3.p2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px3.p2.1.m1.1c">k</annotation></semantics></math> tokens from <math id="S5.SS0.SSS0.Px3.p2.2.m2.1" class="ltx_Math" alttext="t_{X}" display="inline"><semantics id="S5.SS0.SSS0.Px3.p2.2.m2.1a"><msub id="S5.SS0.SSS0.Px3.p2.2.m2.1.1" xref="S5.SS0.SSS0.Px3.p2.2.m2.1.1.cmml"><mi id="S5.SS0.SSS0.Px3.p2.2.m2.1.1.2" xref="S5.SS0.SSS0.Px3.p2.2.m2.1.1.2.cmml">t</mi><mi id="S5.SS0.SSS0.Px3.p2.2.m2.1.1.3" xref="S5.SS0.SSS0.Px3.p2.2.m2.1.1.3.cmml">X</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px3.p2.2.m2.1b"><apply id="S5.SS0.SSS0.Px3.p2.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px3.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px3.p2.2.m2.1.1.1.cmml" xref="S5.SS0.SSS0.Px3.p2.2.m2.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px3.p2.2.m2.1.1.2.cmml" xref="S5.SS0.SSS0.Px3.p2.2.m2.1.1.2">𝑡</ci><ci id="S5.SS0.SSS0.Px3.p2.2.m2.1.1.3.cmml" xref="S5.SS0.SSS0.Px3.p2.2.m2.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px3.p2.2.m2.1c">t_{X}</annotation></semantics></math> for generation.<span id="footnote13" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">13</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">13</sup><span class="ltx_tag ltx_tag_note">13</span>Additional results and model predictions for these experiments can be found in Appendix <a href="#A4" title="Appendix D Controlled Generation with focus attention using Top-𝑘 tokens ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a>.</span></span></span> We observe that <span id="S5.SS0.SSS0.Px3.p2.3.4" class="ltx_text ltx_font_smallcaps">RobFame</span> consistently outperforms <span id="S5.SS0.SSS0.Px3.p2.3.5" class="ltx_text ltx_font_smallcaps">PegFame</span> with the lower values of <math id="S5.SS0.SSS0.Px3.p2.3.m3.5" class="ltx_Math" alttext="k\in\{50,100,200,500,1000\}" display="inline"><semantics id="S5.SS0.SSS0.Px3.p2.3.m3.5a"><mrow id="S5.SS0.SSS0.Px3.p2.3.m3.5.6" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.cmml"><mi id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.2" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.2.cmml">k</mi><mo id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.1" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.1.cmml">∈</mo><mrow id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.2" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.1.cmml"><mo stretchy="false" id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.2.1" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.1.cmml">{</mo><mn id="S5.SS0.SSS0.Px3.p2.3.m3.1.1" xref="S5.SS0.SSS0.Px3.p2.3.m3.1.1.cmml">50</mn><mo id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.2.2" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.1.cmml">,</mo><mn id="S5.SS0.SSS0.Px3.p2.3.m3.2.2" xref="S5.SS0.SSS0.Px3.p2.3.m3.2.2.cmml">100</mn><mo id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.2.3" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.1.cmml">,</mo><mn id="S5.SS0.SSS0.Px3.p2.3.m3.3.3" xref="S5.SS0.SSS0.Px3.p2.3.m3.3.3.cmml">200</mn><mo id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.2.4" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.1.cmml">,</mo><mn id="S5.SS0.SSS0.Px3.p2.3.m3.4.4" xref="S5.SS0.SSS0.Px3.p2.3.m3.4.4.cmml">500</mn><mo id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.2.5" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.1.cmml">,</mo><mn id="S5.SS0.SSS0.Px3.p2.3.m3.5.5" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.5.cmml">1000</mn><mo stretchy="false" id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.2.6" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px3.p2.3.m3.5b"><apply id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.cmml" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6"><in id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.1.cmml" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.1"></in><ci id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.2.cmml" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.2">𝑘</ci><set id="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.1.cmml" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.6.3.2"><cn type="integer" id="S5.SS0.SSS0.Px3.p2.3.m3.1.1.cmml" xref="S5.SS0.SSS0.Px3.p2.3.m3.1.1">50</cn><cn type="integer" id="S5.SS0.SSS0.Px3.p2.3.m3.2.2.cmml" xref="S5.SS0.SSS0.Px3.p2.3.m3.2.2">100</cn><cn type="integer" id="S5.SS0.SSS0.Px3.p2.3.m3.3.3.cmml" xref="S5.SS0.SSS0.Px3.p2.3.m3.3.3">200</cn><cn type="integer" id="S5.SS0.SSS0.Px3.p2.3.m3.4.4.cmml" xref="S5.SS0.SSS0.Px3.p2.3.m3.4.4">500</cn><cn type="integer" id="S5.SS0.SSS0.Px3.p2.3.m3.5.5.cmml" xref="S5.SS0.SSS0.Px3.p2.3.m3.5.5">1000</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px3.p2.3.m3.5c">k\in\{50,100,200,500,1000\}</annotation></semantics></math>.</p>
</div>
<figure id="S5.T3" class="ltx_table">
<table id="S5.T3.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S5.T3.1.1.1" class="ltx_tr">
<th id="S5.T3.1.1.1.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_tt">Models</th>
<td id="S5.T3.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt">R1</td>
<td id="S5.T3.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt">R2</td>
<td id="S5.T3.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">RL</td>
</tr>
<tr id="S5.T3.1.2.2" class="ltx_tr">
<th id="S5.T3.1.2.2.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t">Lead</th>
<td id="S5.T3.1.2.2.2" class="ltx_td ltx_align_center ltx_border_t">16.30</td>
<td id="S5.T3.1.2.2.3" class="ltx_td ltx_align_center ltx_border_t">1.61</td>
<td id="S5.T3.1.2.2.4" class="ltx_td ltx_align_center ltx_border_t">11.95</td>
</tr>
<tr id="S5.T3.1.3.3" class="ltx_tr">
<th id="S5.T3.1.3.3.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">PtGen <cite class="ltx_cite ltx_citemacro_cite">See et al. (<a href="#bib.bib65" title="" class="ltx_ref">2017</a>)</cite>
</th>
<td id="S5.T3.1.3.3.2" class="ltx_td ltx_align_center">29.70</td>
<td id="S5.T3.1.3.3.3" class="ltx_td ltx_align_center">9.21</td>
<td id="S5.T3.1.3.3.4" class="ltx_td ltx_align_center">23.24</td>
</tr>
<tr id="S5.T3.1.4.4" class="ltx_tr">
<th id="S5.T3.1.4.4.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">ConvS2S <cite class="ltx_cite ltx_citemacro_cite">Narayan et al. (<a href="#bib.bib54" title="" class="ltx_ref">2018</a>)</cite>
</th>
<td id="S5.T3.1.4.4.2" class="ltx_td ltx_align_center">31.89</td>
<td id="S5.T3.1.4.4.3" class="ltx_td ltx_align_center">11.54</td>
<td id="S5.T3.1.4.4.4" class="ltx_td ltx_align_center">25.75</td>
</tr>
<tr id="S5.T3.1.5.5" class="ltx_tr">
<th id="S5.T3.1.5.5.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">MMN <cite class="ltx_cite ltx_citemacro_cite">Kim et al. (<a href="#bib.bib36" title="" class="ltx_ref">2019</a>)</cite>
</th>
<td id="S5.T3.1.5.5.2" class="ltx_td ltx_align_center">32.00</td>
<td id="S5.T3.1.5.5.3" class="ltx_td ltx_align_center">12.10</td>
<td id="S5.T3.1.5.5.4" class="ltx_td ltx_align_center">26.00</td>
</tr>
<tr id="S5.T3.1.6.6" class="ltx_tr">
<th id="S5.T3.1.6.6.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">MASS <cite class="ltx_cite ltx_citemacro_cite">Song et al. (<a href="#bib.bib68" title="" class="ltx_ref">2019</a>)</cite>
</th>
<td id="S5.T3.1.6.6.2" class="ltx_td ltx_align_center">39.75</td>
<td id="S5.T3.1.6.6.3" class="ltx_td ltx_align_center">17.24</td>
<td id="S5.T3.1.6.6.4" class="ltx_td ltx_align_center">31.95</td>
</tr>
<tr id="S5.T3.1.7.7" class="ltx_tr">
<th id="S5.T3.1.7.7.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">BART <cite class="ltx_cite ltx_citemacro_cite">Lewis et al. (<a href="#bib.bib42" title="" class="ltx_ref">2019</a>)</cite>
</th>
<td id="S5.T3.1.7.7.2" class="ltx_td ltx_align_center">45.14</td>
<td id="S5.T3.1.7.7.3" class="ltx_td ltx_align_center">22.27</td>
<td id="S5.T3.1.7.7.4" class="ltx_td ltx_align_center">37.25</td>
</tr>
<tr id="S5.T3.1.8.8" class="ltx_tr">
<th id="S5.T3.1.8.8.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="S5.T3.1.8.8.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> <cite class="ltx_cite ltx_citemacro_cite">Zhang et al. (<a href="#bib.bib85" title="" class="ltx_ref">2019</a>)</cite>
</th>
<td id="S5.T3.1.8.8.2" class="ltx_td ltx_align_center"><span id="S5.T3.1.8.8.2.1" class="ltx_text ltx_font_bold ltx_framed ltx_framed_underline">47.21</span></td>
<td id="S5.T3.1.8.8.3" class="ltx_td ltx_align_center"><span id="S5.T3.1.8.8.3.1" class="ltx_text ltx_font_bold ltx_framed ltx_framed_underline">24.56</span></td>
<td id="S5.T3.1.8.8.4" class="ltx_td ltx_align_center"><span id="S5.T3.1.8.8.4.1" class="ltx_text ltx_font_bold ltx_framed ltx_framed_underline">39.25</span></td>
</tr>
<tr id="S5.T3.1.9.9" class="ltx_tr">
<th id="S5.T3.1.9.9.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t">
<span id="S5.T3.1.9.9.1.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> <cite class="ltx_cite ltx_citemacro_cite">Rothe et al. (<a href="#bib.bib63" title="" class="ltx_ref">2020</a>)</cite>
</th>
<td id="S5.T3.1.9.9.2" class="ltx_td ltx_align_center ltx_border_t">41.45</td>
<td id="S5.T3.1.9.9.3" class="ltx_td ltx_align_center ltx_border_t">18.79</td>
<td id="S5.T3.1.9.9.4" class="ltx_td ltx_align_center ltx_border_t">33.90</td>
</tr>
<tr id="S5.T3.1.10.10" class="ltx_tr">
<th id="S5.T3.1.10.10.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="S5.T3.1.10.10.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (w/o Eq. equation <a href="#S3.E5" title="In Summary-induced Topic Focused Distribution ‣ 3 Summarization with Focus Attention ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>)</th>
<td id="S5.T3.1.10.10.2" class="ltx_td ltx_align_center">41.27</td>
<td id="S5.T3.1.10.10.3" class="ltx_td ltx_align_center">18.86</td>
<td id="S5.T3.1.10.10.4" class="ltx_td ltx_align_center">33.90</td>
</tr>
<tr id="S5.T3.1.11.11" class="ltx_tr">
<th id="S5.T3.1.11.11.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r"><span id="S5.T3.1.11.11.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span></th>
<td id="S5.T3.1.11.11.2" class="ltx_td ltx_align_center">42.15</td>
<td id="S5.T3.1.11.11.3" class="ltx_td ltx_align_center">19.68</td>
<td id="S5.T3.1.11.11.4" class="ltx_td ltx_align_center">34.81</td>
</tr>
<tr id="S5.T3.1.12.12" class="ltx_tr">
<th id="S5.T3.1.12.12.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r"><span id="S5.T3.1.12.12.1.1" class="ltx_text ltx_font_smallcaps">Oracle</span></th>
<td id="S5.T3.1.12.12.2" class="ltx_td ltx_align_center"><span id="S5.T3.1.12.12.2.1" class="ltx_text ltx_font_bold">72.22</span></td>
<td id="S5.T3.1.12.12.3" class="ltx_td ltx_align_center"><span id="S5.T3.1.12.12.3.1" class="ltx_text ltx_font_bold">42.22</span></td>
<td id="S5.T3.1.12.12.4" class="ltx_td ltx_align_center"><span id="S5.T3.1.12.12.4.1" class="ltx_text ltx_font_bold">53.89</span></td>
</tr>
<tr id="S5.T3.1.13.13" class="ltx_tr">
<th id="S5.T3.1.13.13.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t">
<span id="S5.T3.1.13.13.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> (ours)</th>
<td id="S5.T3.1.13.13.2" class="ltx_td ltx_align_center ltx_border_t">44.85</td>
<td id="S5.T3.1.13.13.3" class="ltx_td ltx_align_center ltx_border_t">22.26</td>
<td id="S5.T3.1.13.13.4" class="ltx_td ltx_align_center ltx_border_t">37.03</td>
</tr>
<tr id="S5.T3.1.14.14" class="ltx_tr">
<th id="S5.T3.1.14.14.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="S5.T3.1.14.14.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (w/o Eq. equation <a href="#S3.E5" title="In Summary-induced Topic Focused Distribution ‣ 3 Summarization with Focus Attention ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>)</th>
<td id="S5.T3.1.14.14.2" class="ltx_td ltx_align_center">44.54</td>
<td id="S5.T3.1.14.14.3" class="ltx_td ltx_align_center">22.00</td>
<td id="S5.T3.1.14.14.4" class="ltx_td ltx_align_center">36.83</td>
</tr>
<tr id="S5.T3.1.15.15" class="ltx_tr">
<th id="S5.T3.1.15.15.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r"><span id="S5.T3.1.15.15.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span></th>
<td id="S5.T3.1.15.15.2" class="ltx_td ltx_align_center">45.31</td>
<td id="S5.T3.1.15.15.3" class="ltx_td ltx_align_center">22.75</td>
<td id="S5.T3.1.15.15.4" class="ltx_td ltx_align_center">37.46</td>
</tr>
<tr id="S5.T3.1.16.16" class="ltx_tr">
<th id="S5.T3.1.16.16.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_bb ltx_border_r"><span id="S5.T3.1.16.16.1.1" class="ltx_text ltx_font_smallcaps">Oracle</span></th>
<td id="S5.T3.1.16.16.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T3.1.16.16.2.1" class="ltx_text ltx_font_bold">82.39</span></td>
<td id="S5.T3.1.16.16.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T3.1.16.16.3.1" class="ltx_text ltx_font_bold">60.61</span></td>
<td id="S5.T3.1.16.16.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T3.1.16.16.4.1" class="ltx_text ltx_font_bold">69.19</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Ablations and SOTA comparisons on <span id="S5.T3.6.1" class="ltx_text ltx_font_smallcaps">XSum</span> dataset. The <span id="S5.T3.7.2" class="ltx_text ltx_font_bold ltx_framed ltx_framed_underline">underlined bold</span> results are from the best performing models from literature and the <span id="S5.T3.8.3" class="ltx_text ltx_font_bold">bold</span> results are the best performing <span id="S5.T3.9.4" class="ltx_text ltx_font_smallcaps">Fame</span> models.</figcaption>
</figure>
<div id="S5.SS0.SSS0.Px3.p3" class="ltx_para">
<p id="S5.SS0.SSS0.Px3.p3.3" class="ltx_p">Further, we observe that <span id="S5.SS0.SSS0.Px3.p3.3.1" class="ltx_text ltx_font_smallcaps">RobFame</span> generates fewer unique summaries (1.61 vs 2.77) but has higher Distinct-N scores (3.5/22.4/43.9 vs 2.4/16.5/34.2) than <span id="S5.SS0.SSS0.Px3.p3.3.2" class="ltx_text ltx_font_smallcaps">PegFame</span>, with <math id="S5.SS0.SSS0.Px3.p3.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="S5.SS0.SSS0.Px3.p3.1.m1.2a"><msub id="S5.SS0.SSS0.Px3.p3.1.m1.2.3" xref="S5.SS0.SSS0.Px3.p3.1.m1.2.3.cmml"><mi id="S5.SS0.SSS0.Px3.p3.1.m1.2.3.2" xref="S5.SS0.SSS0.Px3.p3.1.m1.2.3.2.cmml">Focus</mi><mrow id="S5.SS0.SSS0.Px3.p3.1.m1.2.2.2.4" xref="S5.SS0.SSS0.Px3.p3.1.m1.2.2.2.3.cmml"><mi id="S5.SS0.SSS0.Px3.p3.1.m1.1.1.1.1" xref="S5.SS0.SSS0.Px3.p3.1.m1.1.1.1.1.cmml">sample</mi><mo id="S5.SS0.SSS0.Px3.p3.1.m1.2.2.2.4.1" xref="S5.SS0.SSS0.Px3.p3.1.m1.2.2.2.3.cmml">,</mo><mi id="S5.SS0.SSS0.Px3.p3.1.m1.2.2.2.2" xref="S5.SS0.SSS0.Px3.p3.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px3.p3.1.m1.2b"><apply id="S5.SS0.SSS0.Px3.p3.1.m1.2.3.cmml" xref="S5.SS0.SSS0.Px3.p3.1.m1.2.3"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px3.p3.1.m1.2.3.1.cmml" xref="S5.SS0.SSS0.Px3.p3.1.m1.2.3">subscript</csymbol><ci id="S5.SS0.SSS0.Px3.p3.1.m1.2.3.2.cmml" xref="S5.SS0.SSS0.Px3.p3.1.m1.2.3.2">Focus</ci><list id="S5.SS0.SSS0.Px3.p3.1.m1.2.2.2.3.cmml" xref="S5.SS0.SSS0.Px3.p3.1.m1.2.2.2.4"><ci id="S5.SS0.SSS0.Px3.p3.1.m1.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px3.p3.1.m1.1.1.1.1">sample</ci><ci id="S5.SS0.SSS0.Px3.p3.1.m1.2.2.2.2.cmml" xref="S5.SS0.SSS0.Px3.p3.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px3.p3.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math> in Table <a href="#S5.T2" title="Table 2 ‣ 5 Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. This can be again be attributed to how <span id="S5.SS0.SSS0.Px3.p3.3.3" class="ltx_text ltx_font_smallcaps">Fame</span> works differently in <span id="S5.SS0.SSS0.Px3.p3.3.4" class="ltx_text ltx_font_smallcaps">RobFame</span> and <span id="S5.SS0.SSS0.Px3.p3.3.5" class="ltx_text ltx_font_smallcaps">PegFame</span>. When <math id="S5.SS0.SSS0.Px3.p3.2.m2.1" class="ltx_Math" alttext="V_{k}" display="inline"><semantics id="S5.SS0.SSS0.Px3.p3.2.m2.1a"><msub id="S5.SS0.SSS0.Px3.p3.2.m2.1.1" xref="S5.SS0.SSS0.Px3.p3.2.m2.1.1.cmml"><mi id="S5.SS0.SSS0.Px3.p3.2.m2.1.1.2" xref="S5.SS0.SSS0.Px3.p3.2.m2.1.1.2.cmml">V</mi><mi id="S5.SS0.SSS0.Px3.p3.2.m2.1.1.3" xref="S5.SS0.SSS0.Px3.p3.2.m2.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px3.p3.2.m2.1b"><apply id="S5.SS0.SSS0.Px3.p3.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px3.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px3.p3.2.m2.1.1.1.cmml" xref="S5.SS0.SSS0.Px3.p3.2.m2.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px3.p3.2.m2.1.1.2.cmml" xref="S5.SS0.SSS0.Px3.p3.2.m2.1.1.2">𝑉</ci><ci id="S5.SS0.SSS0.Px3.p3.2.m2.1.1.3.cmml" xref="S5.SS0.SSS0.Px3.p3.2.m2.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px3.p3.2.m2.1c">V_{k}</annotation></semantics></math> is sampled from <span id="S5.SS0.SSS0.Px3.p3.3.6" class="ltx_text ltx_font_smallcaps">RobFame</span>’s peaky distribution, the beam search decoding often tends to generate similar summaries (leading to a lower Uniqueness score) as the sampled <math id="S5.SS0.SSS0.Px3.p3.3.m3.1" class="ltx_Math" alttext="V_{k}" display="inline"><semantics id="S5.SS0.SSS0.Px3.p3.3.m3.1a"><msub id="S5.SS0.SSS0.Px3.p3.3.m3.1.1" xref="S5.SS0.SSS0.Px3.p3.3.m3.1.1.cmml"><mi id="S5.SS0.SSS0.Px3.p3.3.m3.1.1.2" xref="S5.SS0.SSS0.Px3.p3.3.m3.1.1.2.cmml">V</mi><mi id="S5.SS0.SSS0.Px3.p3.3.m3.1.1.3" xref="S5.SS0.SSS0.Px3.p3.3.m3.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px3.p3.3.m3.1b"><apply id="S5.SS0.SSS0.Px3.p3.3.m3.1.1.cmml" xref="S5.SS0.SSS0.Px3.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px3.p3.3.m3.1.1.1.cmml" xref="S5.SS0.SSS0.Px3.p3.3.m3.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px3.p3.3.m3.1.1.2.cmml" xref="S5.SS0.SSS0.Px3.p3.3.m3.1.1.2">𝑉</ci><ci id="S5.SS0.SSS0.Px3.p3.3.m3.1.1.3.cmml" xref="S5.SS0.SSS0.Px3.p3.3.m3.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px3.p3.3.m3.1c">V_{k}</annotation></semantics></math>s do not diverge by much from each other. But when it does diverge, the decoder tends to generate completely new summaries (leading to higher Distinct-N scores).</p>
</div>
<div id="S5.SS0.SSS0.Px3.p4" class="ltx_para">
<p id="S5.SS0.SSS0.Px3.p4.2" class="ltx_p">Currently, we set <math id="S5.SS0.SSS0.Px3.p4.1.m1.2" class="ltx_Math" alttext="k=10,000" display="inline"><semantics id="S5.SS0.SSS0.Px3.p4.1.m1.2a"><mrow id="S5.SS0.SSS0.Px3.p4.1.m1.2.3" xref="S5.SS0.SSS0.Px3.p4.1.m1.2.3.cmml"><mi id="S5.SS0.SSS0.Px3.p4.1.m1.2.3.2" xref="S5.SS0.SSS0.Px3.p4.1.m1.2.3.2.cmml">k</mi><mo id="S5.SS0.SSS0.Px3.p4.1.m1.2.3.1" xref="S5.SS0.SSS0.Px3.p4.1.m1.2.3.1.cmml">=</mo><mrow id="S5.SS0.SSS0.Px3.p4.1.m1.2.3.3.2" xref="S5.SS0.SSS0.Px3.p4.1.m1.2.3.3.1.cmml"><mn id="S5.SS0.SSS0.Px3.p4.1.m1.1.1" xref="S5.SS0.SSS0.Px3.p4.1.m1.1.1.cmml">10</mn><mo id="S5.SS0.SSS0.Px3.p4.1.m1.2.3.3.2.1" xref="S5.SS0.SSS0.Px3.p4.1.m1.2.3.3.1.cmml">,</mo><mn id="S5.SS0.SSS0.Px3.p4.1.m1.2.2" xref="S5.SS0.SSS0.Px3.p4.1.m1.2.2.cmml">000</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px3.p4.1.m1.2b"><apply id="S5.SS0.SSS0.Px3.p4.1.m1.2.3.cmml" xref="S5.SS0.SSS0.Px3.p4.1.m1.2.3"><eq id="S5.SS0.SSS0.Px3.p4.1.m1.2.3.1.cmml" xref="S5.SS0.SSS0.Px3.p4.1.m1.2.3.1"></eq><ci id="S5.SS0.SSS0.Px3.p4.1.m1.2.3.2.cmml" xref="S5.SS0.SSS0.Px3.p4.1.m1.2.3.2">𝑘</ci><list id="S5.SS0.SSS0.Px3.p4.1.m1.2.3.3.1.cmml" xref="S5.SS0.SSS0.Px3.p4.1.m1.2.3.3.2"><cn type="integer" id="S5.SS0.SSS0.Px3.p4.1.m1.1.1.cmml" xref="S5.SS0.SSS0.Px3.p4.1.m1.1.1">10</cn><cn type="integer" id="S5.SS0.SSS0.Px3.p4.1.m1.2.2.cmml" xref="S5.SS0.SSS0.Px3.p4.1.m1.2.2">000</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px3.p4.1.m1.2c">k=10,000</annotation></semantics></math> for our focus sampling experiments following our observations in Figure <a href="#S5.F4" title="Figure 4 ‣ Generating Diverse and Faithful Summaries with Focus Sampling. ‣ 5 Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. Future work will focus on how to better leverage trade-off between diversity and faithfulness by
controlling the peakiness of the topic distribution <math id="S5.SS0.SSS0.Px3.p4.2.m2.1" class="ltx_Math" alttext="t_{X}" display="inline"><semantics id="S5.SS0.SSS0.Px3.p4.2.m2.1a"><msub id="S5.SS0.SSS0.Px3.p4.2.m2.1.1" xref="S5.SS0.SSS0.Px3.p4.2.m2.1.1.cmml"><mi id="S5.SS0.SSS0.Px3.p4.2.m2.1.1.2" xref="S5.SS0.SSS0.Px3.p4.2.m2.1.1.2.cmml">t</mi><mi id="S5.SS0.SSS0.Px3.p4.2.m2.1.1.3" xref="S5.SS0.SSS0.Px3.p4.2.m2.1.1.3.cmml">X</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px3.p4.2.m2.1b"><apply id="S5.SS0.SSS0.Px3.p4.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px3.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px3.p4.2.m2.1.1.1.cmml" xref="S5.SS0.SSS0.Px3.p4.2.m2.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px3.p4.2.m2.1.1.2.cmml" xref="S5.SS0.SSS0.Px3.p4.2.m2.1.1.2">𝑡</ci><ci id="S5.SS0.SSS0.Px3.p4.2.m2.1.1.3.cmml" xref="S5.SS0.SSS0.Px3.p4.2.m2.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px3.p4.2.m2.1c">t_{X}</annotation></semantics></math>.</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Ablations and SOTA Comparisons</h4>

<div id="S5.SS0.SSS0.Px4.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px4.p1.1" class="ltx_p">We emphasize that <span id="S5.SS0.SSS0.Px4.p1.1.1" class="ltx_text ltx_font_smallcaps">Fame</span> or focus sampling does not aim to improve on state-of-the-results in terms of <span id="S5.SS0.SSS0.Px4.p1.1.2" class="ltx_text ltx_font_smallcaps">rouge</span>, but to generate more faithful or diverse summaries while maintaining their quality. For completeness, we compare our <span id="S5.SS0.SSS0.Px4.p1.1.3" class="ltx_text ltx_font_smallcaps">RobFame</span> and <span id="S5.SS0.SSS0.Px4.p1.1.4" class="ltx_text ltx_font_smallcaps">PegFame</span> models to their ablations and other state-of-the-art models on <span id="S5.SS0.SSS0.Px4.p1.1.5" class="ltx_text ltx_font_smallcaps">XSum</span> in Table <a href="#S5.T3" title="Table 3 ‣ Focus Attention and Sampling Work Differently in RobFame and PegFame. ‣ 5 Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</p>
</div>
<div id="S5.SS0.SSS0.Px4.p2" class="ltx_para">
<p id="S5.SS0.SSS0.Px4.p2.1" class="ltx_p">We report <span id="S5.SS0.SSS0.Px4.p2.1.1" class="ltx_text ltx_font_smallcaps">rouge</span> scores for <span id="S5.SS0.SSS0.Px4.p2.1.2" class="ltx_text ltx_font_smallcaps">Fame</span> in the ideal scenario (<span id="S5.SS0.SSS0.Px4.p2.1.3" class="ltx_text ltx_font_smallcaps">Oracle</span>) where it focuses on all the correct tokens in the input, i.e., the topic distribution <math id="S5.SS0.SSS0.Px4.p2.1.m1.1" class="ltx_Math" alttext="{\bm{t}}_{X}" display="inline"><semantics id="S5.SS0.SSS0.Px4.p2.1.m1.1a"><msub id="S5.SS0.SSS0.Px4.p2.1.m1.1.1" xref="S5.SS0.SSS0.Px4.p2.1.m1.1.1.cmml"><mi id="S5.SS0.SSS0.Px4.p2.1.m1.1.1.2" xref="S5.SS0.SSS0.Px4.p2.1.m1.1.1.2.cmml">𝒕</mi><mi id="S5.SS0.SSS0.Px4.p2.1.m1.1.1.3" xref="S5.SS0.SSS0.Px4.p2.1.m1.1.1.3.cmml">X</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px4.p2.1.m1.1b"><apply id="S5.SS0.SSS0.Px4.p2.1.m1.1.1.cmml" xref="S5.SS0.SSS0.Px4.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px4.p2.1.m1.1.1.1.cmml" xref="S5.SS0.SSS0.Px4.p2.1.m1.1.1">subscript</csymbol><ci id="S5.SS0.SSS0.Px4.p2.1.m1.1.1.2.cmml" xref="S5.SS0.SSS0.Px4.p2.1.m1.1.1.2">𝒕</ci><ci id="S5.SS0.SSS0.Px4.p2.1.m1.1.1.3.cmml" xref="S5.SS0.SSS0.Px4.p2.1.m1.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px4.p2.1.m1.1c">{\bm{t}}_{X}</annotation></semantics></math> is identical to the distribution observed in the reference summary.
These models generate summaries with very high <span id="S5.SS0.SSS0.Px4.p2.1.4" class="ltx_text ltx_font_smallcaps">rouge</span> scores when the model is given the correct tokens to focus on. The gap between the <span id="S5.SS0.SSS0.Px4.p2.1.5" class="ltx_text ltx_font_smallcaps">Oracle</span> and <span id="S5.SS0.SSS0.Px4.p2.1.6" class="ltx_text ltx_font_smallcaps">Fame</span> scores suggests that there is still a lot of work to be done in this space.
Focus attention without any topical supervision (models w/o Eq. equation <a href="#S3.E5" title="In Summary-induced Topic Focused Distribution ‣ 3 Summarization with Focus Attention ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>) is not significantly better than the baselines. But <span id="S5.SS0.SSS0.Px4.p2.1.7" class="ltx_text ltx_font_smallcaps">RobFame</span> and <span id="S5.SS0.SSS0.Px4.p2.1.8" class="ltx_text ltx_font_smallcaps">PegFame</span> (trained with joint supervision in Eq. equation <a href="#S3.E6" title="In Summary-induced Topic Focused Distribution ‣ 3 Summarization with Focus Attention ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>) significantly outperform <span id="S5.SS0.SSS0.Px4.p2.1.9" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> and <span id="S5.SS0.SSS0.Px4.p2.1.10" class="ltx_text ltx_font_smallcaps">Pegasus</span>, respectively.</p>
</div>
<div id="S5.SS0.SSS0.Px4.p3" class="ltx_para">
<p id="S5.SS0.SSS0.Px4.p3.1" class="ltx_p">Our best model <span id="S5.SS0.SSS0.Px4.p3.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> performs better than PtGen <cite class="ltx_cite ltx_citemacro_cite">See et al. (<a href="#bib.bib65" title="" class="ltx_ref">2017</a>)</cite>,
ConvS2S <cite class="ltx_cite ltx_citemacro_cite">Narayan et al. (<a href="#bib.bib54" title="" class="ltx_ref">2018</a>)</cite>, MMN <cite class="ltx_cite ltx_citemacro_cite">Kim et al. (<a href="#bib.bib36" title="" class="ltx_ref">2019</a>)</cite>,
MASS <cite class="ltx_cite ltx_citemacro_cite">Song et al. (<a href="#bib.bib68" title="" class="ltx_ref">2019</a>)</cite> and BART <cite class="ltx_cite ltx_citemacro_cite">Lewis et al. (<a href="#bib.bib42" title="" class="ltx_ref">2019</a>)</cite>, but worse when the original <span id="S5.SS0.SSS0.Px4.p3.1.2" class="ltx_text ltx_font_smallcaps">Pegasus</span> <cite class="ltx_cite ltx_citemacro_cite">Zhang et al. (<a href="#bib.bib85" title="" class="ltx_ref">2019</a>)</cite>. This can be expected as the number of parameters in <span id="S5.SS0.SSS0.Px4.p3.1.3" class="ltx_text ltx_font_smallcaps">PegFame</span> is far less than that in the original <span id="S5.SS0.SSS0.Px4.p3.1.4" class="ltx_text ltx_font_smallcaps">Pegasus</span>.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">We introduced <span id="S6.p1.1.1" class="ltx_text ltx_font_smallcaps">Fame</span>, a new attention mechanism which dynamically biases the decoder to proactively generate tokens that are topically similar to the input. <span id="S6.p1.1.2" class="ltx_text ltx_font_smallcaps">Fame</span> enhances the faithfulness of existing state-of-the-art abstract summarization models while improving their overall <span id="S6.p1.1.3" class="ltx_text ltx_font_smallcaps">rouge</span> scores. Finally, our newly introduced focus sampling technique is a better alternative to top-<math id="S6.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S6.p1.1.m1.1a"><mi id="S6.p1.1.m1.1.1" xref="S6.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S6.p1.1.m1.1b"><ci id="S6.p1.1.m1.1.1.cmml" xref="S6.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.1.m1.1c">k</annotation></semantics></math> or nucleus sampling to generate diverse set of faithful summaries.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgements</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">We thank Sebastian Gehrmann, Slav Petrov, the reviewers, and the action editor for their invaluable feedback.</p>
</div>
</section>
<section id="Sx2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Ethical Considerations</h2>

<div id="Sx2.p1" class="ltx_para">
<p id="Sx2.p1.1" class="ltx_p">The nature of text generation leads to multiple ethical considerations when applied to applications. The main failure mode is that the model can learn to mimic target properties in the training data that are not desirable.</p>
</div>
<section id="Sx2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Faithfulness and Factuality</h4>

<div id="Sx2.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="Sx2.SS0.SSS0.Px1.p1.1" class="ltx_p">Since models create new text, there is the danger that they may neither be faithful to the source material nor factual. This can be exacerbated when the data itself has highly abstractive targets, which require the model to generate words not seen in the source material during training. This often leads the model to generate content inconsistent with the source material <cite class="ltx_cite ltx_citemacro_cite">Kryscinski et al. (<a href="#bib.bib38" title="" class="ltx_ref">2020</a>); Maynez et al. (<a href="#bib.bib52" title="" class="ltx_ref">2020</a>); Gabriel et al. (<a href="#bib.bib25" title="" class="ltx_ref">2020</a>)</cite>.</p>
</div>
</section>
<section id="Sx2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Trustworthy Data</h4>

<div id="Sx2.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="Sx2.SS0.SSS0.Px2.p1.1" class="ltx_p">If the data itself is not trustworthy (comes from suspect or malicious sources) the model itself will naturally become untrustworthy as it will ultimately learn the language and topics of the training data. For instance, if the training data is about Obama birther conspiracies, and the model is asked to generate information about the early life of Obama, there is a risk that such false claims will be predicted by the model.</p>
</div>
</section>
<section id="Sx2.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Bias in Data</h4>

<div id="Sx2.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="Sx2.SS0.SSS0.Px3.p1.1" class="ltx_p">Similarly, biases in the data around gender, race, etc., risk being propagated in the model predictions, which is common for most NLP tasks. This is especially true when the models are trained from non-contemporary data that do not represent current norms and practices <cite class="ltx_cite ltx_citemacro_cite">Blodgett et al. (<a href="#bib.bib6" title="" class="ltx_ref">2020</a>)</cite>.</p>
</div>
<div id="Sx2.SS0.SSS0.Px3.p2" class="ltx_para">
<p id="Sx2.SS0.SSS0.Px3.p2.1" class="ltx_p">The above considerations are non-malicious, in that the model is merely learning to behave as its underlying source material. If users of such models are not aware of these issues and do not account for them, e.g., with better data selection, evaluation, etc., then the generated text can be damaging.

<br class="ltx_break">
<br class="ltx_break">Generation models can also be misused in malicious ways. These include generating fake news, spam, and other text meant to mislead large parts of the general population.</p>
</div>
</section>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ailem et al. (2019)</span>
<span class="ltx_bibblock">
Melissa Ailem, Bowen Zhang, and Fei Sha. 2019.

</span>
<span class="ltx_bibblock">Topic augmented generator for abstractive summarization.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1908.07026.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Arumae and Liu (2019)</span>
<span class="ltx_bibblock">
Kristjan Arumae and Fei Liu. 2019.

</span>
<span class="ltx_bibblock">Guiding extractive summarization with question-answering rewards.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies</em>, pages 2566–2577, Minneapolis, Minnesota.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bahdanau et al. (2015)</span>
<span class="ltx_bibblock">
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.

</span>
<span class="ltx_bibblock">Neural machine translation by jointly learning to align and
translate.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1409.0473.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Barzilay and Elhadad (1997)</span>
<span class="ltx_bibblock">
Regina Barzilay and Michael Elhadad. 1997.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aclweb.org/anthology/W97-0703" title="" class="ltx_ref ltx_href">Using lexical
chains for text summarization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Intelligent Scalable Text Summarization</em>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Blei et al. (2003)</span>
<span class="ltx_bibblock">
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003.

</span>
<span class="ltx_bibblock">Latent dirichlet allocation.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">The Journal of Machine Learning Research</em>, 3:993–1022.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Blodgett et al. (2020)</span>
<span class="ltx_bibblock">
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.acl-main.485" title="" class="ltx_ref ltx_href">Language
(technology) is power: A critical survey of “bias” in NLP</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics</em>, pages 5454–5476, Online. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Botha et al. (2018)</span>
<span class="ltx_bibblock">
Jan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das.
2018.

</span>
<span class="ltx_bibblock">Learning to split and rephrase from Wikipedia edit history.

</span>
<span class="ltx_bibblock">In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing</em>, pages 732–737, Brussels, Belgium. Association
for Computational Linguistics.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cao and Wan (2020)</span>
<span class="ltx_bibblock">
Yue Cao and Xiaojun Wan. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aclweb.org/anthology/2020.findings-emnlp.218" title="" class="ltx_ref ltx_href">DivGAN: Towards diverse paraphrase generation via diversified generative
adversarial network</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics:
EMNLP 2020</em>, pages 2411–2421, Online. Association for Computational
Linguistics.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cao et al. (2017)</span>
<span class="ltx_bibblock">
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/1711.04434" title="" class="ltx_ref ltx_href">Faithful to the original:
Fact aware neural abstractive summarization</a>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cho et al. (2019)</span>
<span class="ltx_bibblock">
Jaemin Cho, Minjoon Seo, and Hannaneh Hajishirzi. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/D19-1308" title="" class="ltx_ref ltx_href">Mixture content
selection for diverse sequence generation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)</em>, pages 3121–3131, Hong Kong,
China. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Choi et al. (2020)</span>
<span class="ltx_bibblock">
Byung-Ju Choi, Jimin Hong, David Park, and Sang Wan Lee. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aclweb.org/anthology/2020.emnlp-main.737" title="" class="ltx_ref ltx_href">F^2-softmax: Diversifying neural text generation via frequency
factorized softmax</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</em>, pages 9167–9182, Online. Association
for Computational Linguistics.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dai et al. (2017)</span>
<span class="ltx_bibblock">
Bo Dai, Dahua Lin, Raquel Urtasun, and Sanja Fidler. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/1703.06029" title="" class="ltx_ref ltx_href">Towards diverse and natural
image descriptions via a conditional GAN</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1703.06029.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al. (2019)</span>
<span class="ltx_bibblock">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.

</span>
<span class="ltx_bibblock">BERT: Pre-training of deep bidirectional transformers for language
understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies</em>, pages 4171–4186, Minneapolis, Minnesota. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dhingra et al. (2019)</span>
<span class="ltx_bibblock">
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and
William Cohen. 2019.

</span>
<span class="ltx_bibblock">Handling divergent reference texts when evaluating table-to-text
generation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics</em>, pages 4884–4895, Florence, Italy.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dieng et al. (2017)</span>
<span class="ltx_bibblock">
Adji B. Dieng, Chong Wang, Jianfeng Gao, and John Paisley. 2017.

</span>
<span class="ltx_bibblock">TopicRNN: A recurrent neural network with long-range semantic
dependency.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 5th International Conference on Learning
Representations</em>, Toulon, France.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dong et al. (2017)</span>
<span class="ltx_bibblock">
Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/D17-1091" title="" class="ltx_ref ltx_href">Learning to paraphrase
for question answering</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2017 Conference on Empirical Methods in
Natural Language Processing</em>, pages 875–886, Copenhagen, Denmark.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dong et al. (2019a)</span>
<span class="ltx_bibblock">
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao,
Ming Zhou, and Hsiao-Wuen Hon. 2019a.

</span>
<span class="ltx_bibblock">Unified language model pre-training for natural language
understanding and generation.

</span>
<span class="ltx_bibblock">In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Buc,
E. Fox, and R. Garnett, editors, <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information
Processing Systems 32</em>, pages 13042–13054. Curran Associates, Inc.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dong et al. (2019b)</span>
<span class="ltx_bibblock">
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung.
2019b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/P19-1331" title="" class="ltx_ref ltx_href">EditNTS: An neural
programmer-interpreter model for sentence simplification through explicit
editing</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics</em>, pages 3393–3402, Florence, Italy.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Durmus et al. (2020)</span>
<span class="ltx_bibblock">
Esin Durmus, He He, and Mona Diab. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.acl-main.454" title="" class="ltx_ref ltx_href">FEQA: A
question answering evaluation framework for faithfulness assessment in
abstractive summarization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics</em>, pages 5055–5070, Online. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dziri et al. (2019)</span>
<span class="ltx_bibblock">
Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar Zaiane. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/W19-4103" title="" class="ltx_ref ltx_href">Augmenting neural
response generation with context-aware topical attention</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Proceedings of the First Workshop on NLP for Conversational
AI</em>, pages 18–31, Florence, Italy. Association for Computational
Linguistics.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Falke et al. (2019)</span>
<span class="ltx_bibblock">
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna
Gurevych. 2019.

</span>
<span class="ltx_bibblock">Ranking generated summaries by correctness: An interesting but
challenging application for natural language inference.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics</em>, pages 2214–2220, Florence, Italy.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fan et al. (2018)</span>
<span class="ltx_bibblock">
Angela Fan, Mike Lewis, and Yann Dauphin. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/P18-1082" title="" class="ltx_ref ltx_href">Hierarchical neural
story generation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)</em>, pages 889–898,
Melbourne, Australia. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fellbaum (1998)</span>
<span class="ltx_bibblock">
Christiane Fellbaum, editor. 1998.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">WordNet: an electronic lexical database</em>.

</span>
<span class="ltx_bibblock">MIT Press.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Freitag et al. (2020)</span>
<span class="ltx_bibblock">
Markus Freitag, David Grangier, and Isaac Caswell. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aclweb.org/anthology/2020.emnlp-main.5" title="" class="ltx_ref ltx_href">BLEU
might be guilty but references are not innocent</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</em>, pages 61–71, Online. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gabriel et al. (2020)</span>
<span class="ltx_bibblock">
Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao.
2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2010.12834" title="" class="ltx_ref ltx_href">Go figure! a meta evaluation
of factuality in summarization</a>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gehrmann et al. (2021)</span>
<span class="ltx_bibblock">
Sebastian Gehrmann, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka
Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu,
Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin
Durmus, Ondrej Dusek, Chris Emezue, Varun Gangal, Cristina Garbacea,
Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji,
Shailza Jolly, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela,
Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique
Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin
Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey
Osei, Ankur P. Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas
Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault
Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla
Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila
Yerukola, and Jiawei Zhou. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2102.01672" title="" class="ltx_ref ltx_href">The GEM benchmark:
Natural language generation, its evaluation and metrics</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2102.01672.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gehrmann et al. (2018)</span>
<span class="ltx_bibblock">
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018.

</span>
<span class="ltx_bibblock">Bottom-up abstractive summarization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing</em>, pages 4098–4109, Brussels, Belgium.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Geva et al. (2019)</span>
<span class="ltx_bibblock">
Mor Geva, Eric Malmi, Idan Szpektor, and Jonathan Berant. 2019.

</span>
<span class="ltx_bibblock">DiscoFuse: A large-scale dataset for discourse-based sentence
fusion.

</span>
<span class="ltx_bibblock">In <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies</em>, pages 3443–3455, Minneapolis, Minnesota. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ghosh et al. (2016)</span>
<span class="ltx_bibblock">
Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, and Larry
Heck. 2016.

</span>
<span class="ltx_bibblock">Contextual LSTM (CLSTM) models for large scale NLP tasks.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1602.06291.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gu et al. (2016)</span>
<span class="ltx_bibblock">
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/P16-1154" title="" class="ltx_ref ltx_href">Incorporating copying
mechanism in sequence-to-sequence learning</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)</em>, pages 1631–1640,
Berlin, Germany. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hashimoto et al. (2019)</span>
<span class="ltx_bibblock">
Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/N19-1169" title="" class="ltx_ref ltx_href">Unifying human and
statistical evaluation for natural language generation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)</em>, pages 1689–1701,
Minneapolis, Minnesota. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hermann et al. (2015)</span>
<span class="ltx_bibblock">
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will
Kay, Mustafa Suleyman, and Phil Blunsom. 2015.

</span>
<span class="ltx_bibblock">Teaching machines to read and comprehend.

</span>
<span class="ltx_bibblock">In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett,
editors, <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems 28</em>, pages
1693–1701. Curran Associates, Inc.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hochreiter and Schmidhuber (1997)</span>
<span class="ltx_bibblock">
Sepp Hochreiter and Jürgen Schmidhuber. 1997.

</span>
<span class="ltx_bibblock">Long short-term memory.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Neural computation</em>, 9(8):1735–1780.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Holtzman et al. (2020)</span>
<span class="ltx_bibblock">
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=rygGQyrFvH" title="" class="ltx_ref ltx_href">The curious case
of neural text degeneration</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Karmaker Santu et al. (2019)</span>
<span class="ltx_bibblock">
Shubhra Kanti Karmaker Santu, Kalyan Veeramachaneni, and Chengxiang Zhai. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/K19-1073" title="" class="ltx_ref ltx_href">TILM: Neural language
models with evolving topical influence</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 23rd Conference on Computational Natural
Language Learning (CoNLL)</em>, pages 778–788, Hong Kong, China. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kim et al. (2019)</span>
<span class="ltx_bibblock">
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019.

</span>
<span class="ltx_bibblock">Abstractive summarization of Reddit posts with multi-level memory
networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies</em>, pages 2519–2531, Minneapolis, Minnesota. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kryscinski et al. (2019)</span>
<span class="ltx_bibblock">
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.

</span>
<span class="ltx_bibblock">Evaluating the factual consistency of abstractive text summarization.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1910.12840.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kryscinski et al. (2020)</span>
<span class="ltx_bibblock">
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aclweb.org/anthology/2020.emnlp-main.750" title="" class="ltx_ref ltx_href">Evaluating the factual consistency of abstractive text summarization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</em>, pages 9332–9346, Online. Association
for Computational Linguistics.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kryściński et al. (2019)</span>
<span class="ltx_bibblock">
Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/1910.12840" title="" class="ltx_ref ltx_href">Evaluating the factual
consistency of abstractive text summarization</a>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kudo and Richardson (2018)</span>
<span class="ltx_bibblock">
Taku Kudo and John Richardson. 2018.

</span>
<span class="ltx_bibblock">SentencePiece: A simple and language independent subword
tokenizer and detokenizer for neural text processing.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations</em>, pages 66–71, Brussels,
Belgium. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kulikov et al. (2019)</span>
<span class="ltx_bibblock">
Ilia Kulikov, Alexander Miller, Kyunghyun Cho, and Jason Weston. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/W19-8609" title="" class="ltx_ref ltx_href">Importance of search
and evaluation strategies in neural dialogue modeling</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 12th International Conference on Natural
Language Generation</em>, pages 76–87, Tokyo, Japan. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lewis et al. (2019)</span>
<span class="ltx_bibblock">
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed,
Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.

</span>
<span class="ltx_bibblock">BART: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1910.13461.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2016a)</span>
<span class="ltx_bibblock">
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan.
2016a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/N16-1014" title="" class="ltx_ref ltx_href">A diversity-promoting
objective function for neural conversation models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies</em>, pages 110–119, San Diego, California. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2016b)</span>
<span class="ltx_bibblock">
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/1611.08562" title="" class="ltx_ref ltx_href">A simple, fast diverse
decoding algorithm for neural generation</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1611.08562.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin and Hovy (2000)</span>
<span class="ltx_bibblock">
Chin-Yew Lin and Eduard Hovy. 2000.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aclweb.org/anthology/C00-1072" title="" class="ltx_ref ltx_href">The automated
acquisition of topic signatures for text summarization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">COLING 2000 Volume 1: The 18th International Conference on
Computational Linguistics</em>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin and Hovy (2003)</span>
<span class="ltx_bibblock">
Chin Yew Lin and Eduard Hovy. 2003.

</span>
<span class="ltx_bibblock">Automatic evaluation of summaries using n-gram co-occurrence
statistics.

</span>
<span class="ltx_bibblock">In <em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2003 Human Language Technology Conference
of the North American Chapter of the Association for Computational
Linguistics</em>, pages 150–157.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2019)</span>
<span class="ltx_bibblock">
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.

</span>
<span class="ltx_bibblock">RoBERTa: A robustly optimized BERT pretraining approach.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1907.11692.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mallinson et al. (2020)</span>
<span class="ltx_bibblock">
Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2003.10687" title="" class="ltx_ref ltx_href">Felix: Flexible text editing
through tagging and insertion</a>.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Malmi et al. (2019)</span>
<span class="ltx_bibblock">
Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei
Severyn. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/D19-1510" title="" class="ltx_ref ltx_href">Encode, tag, realize:
High-precision text editing</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)</em>, pages 5054–5065, Hong Kong,
China. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mani (2001)</span>
<span class="ltx_bibblock">
Inderjeet Mani. 2001.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">Automatic summarization</em>, volume 3.

</span>
<span class="ltx_bibblock">John Benjamins Publishing.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Martins et al. (2020)</span>
<span class="ltx_bibblock">
Pedro Henrique Martins, Zita Marinho, and André F. T. Martins. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aclweb.org/anthology/2020.emnlp-main.348" title="" class="ltx_ref ltx_href">Sparse
text generation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</em>, pages 4252–4273, Online. Association
for Computational Linguistics.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Maynez et al. (2020)</span>
<span class="ltx_bibblock">
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.acl-main.173" title="" class="ltx_ref ltx_href">On
faithfulness and factuality in abstractive summarization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics</em>, pages 1906–1919, Online. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mikolov and Zweig (2012)</span>
<span class="ltx_bibblock">
Tomas Mikolov and Geoffrey Zweig. 2012.

</span>
<span class="ltx_bibblock">Context dependent recurrent neural network language model.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Spoken Language Technology Workshop</em>,
pages 234–239. IEEE.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Narayan et al. (2018)</span>
<span class="ltx_bibblock">
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.

</span>
<span class="ltx_bibblock">Don’t give me the details, just the summary! topic-aware
convolutional neural networks for extreme summarization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing</em>, pages 1797–1807, Brussels, Belgium.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Narayan et al. (2016)</span>
<span class="ltx_bibblock">
Shashi Narayan, Siva Reddy, and Shay B. Cohen. 2016.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/W16-6625" title="" class="ltx_ref ltx_href">Paraphrase generation
from latent-variable PCFGs for semantic parsing</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 9th International Natural Language
Generation conference</em>, pages 153–162, Edinburgh, UK. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nenkova and McKeown (2011)</span>
<span class="ltx_bibblock">
Ani Nenkova and Kathleen McKeown. 2011.

</span>
<span class="ltx_bibblock">Automatic summarization.

</span>
<span class="ltx_bibblock"><em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">Foundations and Trends in Information Retrieval</em>,
5(2–3):103–233.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pasunuru and Bansal (2018)</span>
<span class="ltx_bibblock">
Ramakanth Pasunuru and Mohit Bansal. 2018.

</span>
<span class="ltx_bibblock">Multi-reward reinforced summarization with saliency and entailment.

</span>
<span class="ltx_bibblock">In <em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies</em>, pages 646–653. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Qi et al. (2020)</span>
<span class="ltx_bibblock">
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei
Zhang, and Ming Zhou. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aclweb.org/anthology/2020.findings-emnlp.217" title="" class="ltx_ref ltx_href">ProphetNet: Predicting future n-gram for
sequence-to-SequencePre-training</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib58.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics:
EMNLP 2020</em>, pages 2401–2410, Online. Association for Computational
Linguistics.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2018)</span>
<span class="ltx_bibblock">
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.

</span>
<span class="ltx_bibblock">Improving language understanding by generative pre-training.

</span>
<span class="ltx_bibblock"><em id="bib.bib59.1.1" class="ltx_emph ltx_font_italic">Technical report, OpenAI</em>.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2019)</span>
<span class="ltx_bibblock">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text
transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib60.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1901.07291.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rajpurkar et al. (2018)</span>
<span class="ltx_bibblock">
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.

</span>
<span class="ltx_bibblock">Know what you don’t know: Unanswerable questions for SQuAD.

</span>
<span class="ltx_bibblock">In <em id="bib.bib61.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics</em>, pages 784–789, Melbourne, Australia.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rei et al. (2020)</span>
<span class="ltx_bibblock">
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.emnlp-main.213" title="" class="ltx_ref ltx_href">COMET: A
neural framework for MT evaluation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib62.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</em>, pages 2685–2702, Online. Association
for Computational Linguistics.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rothe et al. (2020)</span>
<span class="ltx_bibblock">
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020.

</span>
<span class="ltx_bibblock">Leveraging pre-trained checkpoints for sequence generation tasks.

</span>
<span class="ltx_bibblock"><em id="bib.bib63.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>,
8:264–280.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schoch et al. (2020)</span>
<span class="ltx_bibblock">
Stephanie Schoch, Diyi Yang, and Yangfeng Ji. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aclweb.org/anthology/2020.evalnlgeval-1.2" title="" class="ltx_ref ltx_href">“this is a problem, don’t you agree?” framing and bias in human
evaluation for natural language generation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib64.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 1st Workshop on Evaluating NLG
Evaluation</em>, pages 10–16, Online (Dublin, Ireland). Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">See et al. (2017)</span>
<span class="ltx_bibblock">
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.

</span>
<span class="ltx_bibblock">Get to the point: Summarization with pointer-generator networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib65.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 55th Annual Meeting of the Association
for Computational Linguistics</em>, pages 1073–1083, Vancouver, Canada.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sellam et al. (2020)</span>
<span class="ltx_bibblock">
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.acl-main.704" title="" class="ltx_ref ltx_href">BLEURT:
Learning robust metrics for text generation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib66.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics</em>, pages 7881–7892, Online. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Song et al. (2020)</span>
<span class="ltx_bibblock">
Kaiqiang Song, Logan Lebanoff, Q. Guo, Xipeng Qiu, X. Xue, Chen Li, Dong Yu,
and Fei Liu. 2020.

</span>
<span class="ltx_bibblock">Joint parsing and generation for abstractive summarization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib67.1.1" class="ltx_emph ltx_font_italic">AAAI</em>.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Song et al. (2019)</span>
<span class="ltx_bibblock">
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019.

</span>
<span class="ltx_bibblock">MASS: Masked sequence to sequence pre-training for language
generation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib68.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 36th International Conference on Machine
Learning</em>, volume 97, pages 5926–5936. PMLR.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sultan et al. (2020)</span>
<span class="ltx_bibblock">
Md Arafat Sultan, Shubham Chandel, Ramón Fernandez Astudillo, and Vittorio
Castelli. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.acl-main.500" title="" class="ltx_ref ltx_href">On the
importance of diversity in question generation for QA</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib69.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics</em>, pages 5651–5656, Online. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tian et al. (2019)</span>
<span class="ltx_bibblock">
Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P. Parikh. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/1910.08684" title="" class="ltx_ref ltx_href">Sticking to the facts:
Confident decoding for faithful data-to-text generation</a>.

</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. (2017)</span>
<span class="ltx_bibblock">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock">In <em id="bib.bib71.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems 30</em>, pages
5998–6008.

</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vijayakumar et al. (2018)</span>
<span class="ltx_bibblock">
Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun,
Stefan Lee, David J. Crandall, and Dhruv Batra. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17329" title="" class="ltx_ref ltx_href">Diverse
beam search for improved description of complex scenes</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib72.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Thirty-Second AAAI Conference on
Artificial Intelligence, (AAAI-18), the 30th innovative Applications of
Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on
Educational Advances in Artificial Intelligence (EAAI-18), New Orleans,
Louisiana, USA, February 2-7, 2018</em>, pages 7371–7379. AAAI Press.

</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2020a)</span>
<span class="ltx_bibblock">
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.acl-main.450" title="" class="ltx_ref ltx_href">Asking and
answering questions to evaluate the factual consistency of summaries</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib73.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics</em>, pages 5008–5020, Online. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2020b)</span>
<span class="ltx_bibblock">
Zhen Wang, Siwei Rao, Jie Zhang, Zhen Qin, Guangjian Tian, and Jun Wang.
2020b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aclweb.org/anthology/2020.findings-emnlp.194" title="" class="ltx_ref ltx_href">Diversify question generation with continuous content selectors and question
type modeling</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib74.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics:
EMNLP 2020</em>, pages 2134–2143, Online. Association for Computational
Linguistics.

</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2020c)</span>
<span class="ltx_bibblock">
Zhengjue Wang, Zhibin Duan, Hao Zhang, Chaojie Wang, Long Tian, Bo Chen, and
Mingyuan Zhou. 2020c.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.aclweb.org/anthology/2020.emnlp-main.35" title="" class="ltx_ref ltx_href">Friendly
topic assistant for transformer based abstractive summarization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib75.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</em>, pages 485–497, Online. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib76" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2020d)</span>
<span class="ltx_bibblock">
Zhenyi Wang, Xiaoyang Wang, Bang An, Dong Yu, and Changyou Chen.
2020d.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2005.00969" title="" class="ltx_ref ltx_href">Towards faithful neural
table-to-text generation with content-matching constraints</a>.

</span>
</li>
<li id="bib.bib77" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Welleck et al. (2019a)</span>
<span class="ltx_bibblock">
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and
Jason Weston. 2019a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/1908.04319" title="" class="ltx_ref ltx_href">Neural text generation with
unlikelihood training</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib77.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1908.04319.

</span>
</li>
<li id="bib.bib78" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Welleck et al. (2019b)</span>
<span class="ltx_bibblock">
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho.
2019b.

</span>
<span class="ltx_bibblock">Dialogue natural language inference.

</span>
<span class="ltx_bibblock">In <em id="bib.bib78.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics</em>, pages 3731–3741, Florence, Italy.

</span>
</li>
<li id="bib.bib79" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Williams et al. (2018)</span>
<span class="ltx_bibblock">
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.

</span>
<span class="ltx_bibblock">A broad-coverage challenge corpus for sentence understanding through
inference.

</span>
<span class="ltx_bibblock">In <em id="bib.bib79.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies</em>, pages 1112–1122. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib80" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xing et al. (2017)</span>
<span class="ltx_bibblock">
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma.
2017.

</span>
<span class="ltx_bibblock">Topic aware neural response generation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib80.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Thirty-First AAAI Conference on
Artificial Intelligence</em>, AAAI’17, page 3351–3357. AAAI Press.

</span>
</li>
<li id="bib.bib81" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2018)</span>
<span class="ltx_bibblock">
Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/D18-1428" title="" class="ltx_ref ltx_href">Diversity-promoting
GAN: A cross-entropy based generative adversarial network for diversified
text generation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib81.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing</em>, pages 3940–3949, Brussels, Belgium.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib82" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2020)</span>
<span class="ltx_bibblock">
Song Xu, Haoran Li, Peng Yuan, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.acl-main.125" title="" class="ltx_ref ltx_href">Self-attention
guided copy mechanism for abstractive summarization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib82.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics</em>, pages 1355–1362, Online. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib83" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2016)</span>
<span class="ltx_bibblock">
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch.
2016.

</span>
<span class="ltx_bibblock">Optimizing statistical machine translation for text simplification.

</span>
<span class="ltx_bibblock"><em id="bib.bib83.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>,
4:401–415.

</span>
</li>
<li id="bib.bib84" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2019)</span>
<span class="ltx_bibblock">
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov,
and Quoc V. Le. 2019.

</span>
<span class="ltx_bibblock">XLNet: Generalized autoregressive pretraining for language
understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib84.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1906.08237.

</span>
</li>
<li id="bib.bib85" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2019)</span>
<span class="ltx_bibblock">
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/1912.08777" title="" class="ltx_ref ltx_href">Pegasus: Pre-training with
extracted gap-sentences for abstractive summarization</a>.

</span>
</li>
<li id="bib.bib86" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2020)</span>
<span class="ltx_bibblock">
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi.
2020.

</span>
<span class="ltx_bibblock">BERTScore: Evaluating text generation with BERT.

</span>
<span class="ltx_bibblock">In <em id="bib.bib86.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 8th International Conference on Learning
Representations</em>, Virtual Conference, Formerly Addis Ababa Ethiopia.

</span>
</li>
<li id="bib.bib87" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2012)</span>
<span class="ltx_bibblock">
Mingyuan Zhou, Lauren Hannah, David Dunson, and Lawrence Carin. 2012.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://proceedings.mlr.press/v22/zhou12c.html" title="" class="ltx_ref ltx_href">Beta-negative
binomial process and poisson factor analysis</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib87.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Fifteenth International Conference on
Artificial Intelligence and Statistics</em>, volume 22 of <em id="bib.bib87.2.2" class="ltx_emph ltx_font_italic">Proceedings of
Machine Learning Research</em>, pages 1462–1471, La Palma, Canary Islands. PMLR.

</span>
</li>
<li id="bib.bib88" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2017)</span>
<span class="ltx_bibblock">
Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/P17-1101" title="" class="ltx_ref ltx_href">Selective encoding for
abstractive sentence summarization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib88.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 55th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)</em>, pages 1095–1104,
Vancouver, Canada. Association for Computational Linguistics.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Implementation and Reproducibility Details</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">Following <cite class="ltx_cite ltx_citemacro_citet">Rothe et al. (<a href="#bib.bib63" title="" class="ltx_ref">2020</a>)</cite>, the encoder and decoder of <span id="A1.p1.1.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> and <span id="A1.p1.1.2" class="ltx_text ltx_font_smallcaps">RobFame</span> models are initialized with public RoBERTa checkpoints. The encoder and decoder parameters are shared in both cases. Only the encoder-decoder attention parameters are initialized randomly. For <span id="A1.p1.1.3" class="ltx_text ltx_font_smallcaps">RobFame</span>, the focus attention parameters are also randomly initialized. We experiment with large RoBERTa checkpoints with 24 layers, a hidden size of 1024, filter size of 4096, 16 attention heads, and a vocabulary with 50K sentence pieces <cite class="ltx_cite ltx_citemacro_cite">Kudo and Richardson (<a href="#bib.bib40" title="" class="ltx_ref">2018</a>)</cite>. <span id="A1.p1.1.4" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> has around 455M parameters and <span id="A1.p1.1.5" class="ltx_text ltx_font_smallcaps">RobFame</span> has 463M parameters, with an additional 8M parameters. Our <span id="A1.p1.1.6" class="ltx_text ltx_font_smallcaps">Pegasus</span> and <span id="A1.p1.1.7" class="ltx_text ltx_font_smallcaps">PegFame</span> implementation also have the same configuration, except for the encoder-decoder attention parameters which are pretrained.</p>
</div>
<div id="A1.p2" class="ltx_para">
<p id="A1.p2.1" class="ltx_p">We used Cloud TPU v3 accelerators for training. All models are fine-tuned on the target task using Adam with a learning rate of 0.05. We use a linear learning rate warm up with 40k steps, normalized by the square root of the hidden size, and a square root decay. We do not perform any tuning on these hyperparameters. We use a global batch size of 128 document-summary pairs. We adapt to different number of training steps depending on the training data sizes. Models are trained for 400k and 200k steps for <span id="A1.p2.1.1" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span> and <span id="A1.p2.1.2" class="ltx_text ltx_font_smallcaps">XSum</span> respectively, saving check-points every 1000 steps. We choose the best model based on <span id="A1.p2.1.3" class="ltx_text ltx_font_smallcaps">rouge-l</span> performance on the respective validation set.</p>
</div>
<div id="A1.p3" class="ltx_para">
<p id="A1.p3.4" class="ltx_p">The vocabulary for functional tokens <math id="A1.p3.1.m1.1" class="ltx_Math" alttext="F" display="inline"><semantics id="A1.p3.1.m1.1a"><mi id="A1.p3.1.m1.1.1" xref="A1.p3.1.m1.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="A1.p3.1.m1.1b"><ci id="A1.p3.1.m1.1.1.cmml" xref="A1.p3.1.m1.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p3.1.m1.1c">F</annotation></semantics></math> is constructed by taking the most frequent sentence pieces in the training set. We tune <math id="A1.p3.2.m2.1" class="ltx_Math" alttext="|F|" display="inline"><semantics id="A1.p3.2.m2.1a"><mrow id="A1.p3.2.m2.1.2.2" xref="A1.p3.2.m2.1.2.1.cmml"><mo stretchy="false" id="A1.p3.2.m2.1.2.2.1" xref="A1.p3.2.m2.1.2.1.1.cmml">|</mo><mi id="A1.p3.2.m2.1.1" xref="A1.p3.2.m2.1.1.cmml">F</mi><mo stretchy="false" id="A1.p3.2.m2.1.2.2.2" xref="A1.p3.2.m2.1.2.1.1.cmml">|</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.p3.2.m2.1b"><apply id="A1.p3.2.m2.1.2.1.cmml" xref="A1.p3.2.m2.1.2.2"><abs id="A1.p3.2.m2.1.2.1.1.cmml" xref="A1.p3.2.m2.1.2.2.1"></abs><ci id="A1.p3.2.m2.1.1.cmml" xref="A1.p3.2.m2.1.1">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p3.2.m2.1c">|F|</annotation></semantics></math> using the respective validation sets; for <span id="A1.p3.4.1" class="ltx_text ltx_font_smallcaps">XSum</span>, we choose <math id="A1.p3.3.m3.1" class="ltx_Math" alttext="f=500" display="inline"><semantics id="A1.p3.3.m3.1a"><mrow id="A1.p3.3.m3.1.1" xref="A1.p3.3.m3.1.1.cmml"><mi id="A1.p3.3.m3.1.1.2" xref="A1.p3.3.m3.1.1.2.cmml">f</mi><mo id="A1.p3.3.m3.1.1.1" xref="A1.p3.3.m3.1.1.1.cmml">=</mo><mn id="A1.p3.3.m3.1.1.3" xref="A1.p3.3.m3.1.1.3.cmml">500</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p3.3.m3.1b"><apply id="A1.p3.3.m3.1.1.cmml" xref="A1.p3.3.m3.1.1"><eq id="A1.p3.3.m3.1.1.1.cmml" xref="A1.p3.3.m3.1.1.1"></eq><ci id="A1.p3.3.m3.1.1.2.cmml" xref="A1.p3.3.m3.1.1.2">𝑓</ci><cn type="integer" id="A1.p3.3.m3.1.1.3.cmml" xref="A1.p3.3.m3.1.1.3">500</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p3.3.m3.1c">f=500</annotation></semantics></math> frequent sentence pieces and for <span id="A1.p3.4.2" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span>, <math id="A1.p3.4.m4.1" class="ltx_Math" alttext="f=1000" display="inline"><semantics id="A1.p3.4.m4.1a"><mrow id="A1.p3.4.m4.1.1" xref="A1.p3.4.m4.1.1.cmml"><mi id="A1.p3.4.m4.1.1.2" xref="A1.p3.4.m4.1.1.2.cmml">f</mi><mo id="A1.p3.4.m4.1.1.1" xref="A1.p3.4.m4.1.1.1.cmml">=</mo><mn id="A1.p3.4.m4.1.1.3" xref="A1.p3.4.m4.1.1.3.cmml">1000</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p3.4.m4.1b"><apply id="A1.p3.4.m4.1.1.cmml" xref="A1.p3.4.m4.1.1"><eq id="A1.p3.4.m4.1.1.1.cmml" xref="A1.p3.4.m4.1.1.1"></eq><ci id="A1.p3.4.m4.1.1.2.cmml" xref="A1.p3.4.m4.1.1.2">𝑓</ci><cn type="integer" id="A1.p3.4.m4.1.1.3.cmml" xref="A1.p3.4.m4.1.1.3">1000</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p3.4.m4.1c">f=1000</annotation></semantics></math>. For all our experiments with the <span id="A1.p3.4.3" class="ltx_text ltx_font_smallcaps">Fame</span> models, the beam size is set to 4.</p>
</div>
<div id="A1.p4" class="ltx_para">
<p id="A1.p4.1" class="ltx_p">We use Cloud TPU v3 accelerators for computing entailment scores which takes about 20 minutes for the two datasets’ test sets. Question generation and answering for Feqa are run on a NVIDIA V100 GPU, and it takes between 8-12 hours for one setting of each test set.</p>
</div>
<figure id="A1.T4" class="ltx_table">
<table id="A1.T4.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T4.1.1.1" class="ltx_tr">
<th id="A1.T4.1.1.1.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A1.T4.1.1.1.1.1" class="ltx_text">Models</span></th>
<td id="A1.T4.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="A1.T4.1.1.1.2.1" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span></td>
</tr>
<tr id="A1.T4.1.2.2" class="ltx_tr">
<td id="A1.T4.1.2.2.1" class="ltx_td ltx_align_center">R1</td>
<td id="A1.T4.1.2.2.2" class="ltx_td ltx_align_center">R2</td>
<td id="A1.T4.1.2.2.3" class="ltx_td ltx_align_center">RL</td>
</tr>
<tr id="A1.T4.1.3.3" class="ltx_tr">
<th id="A1.T4.1.3.3.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t">Lead</th>
<td id="A1.T4.1.3.3.2" class="ltx_td ltx_align_center ltx_border_t">39.60</td>
<td id="A1.T4.1.3.3.3" class="ltx_td ltx_align_center ltx_border_t">17.70</td>
<td id="A1.T4.1.3.3.4" class="ltx_td ltx_align_center ltx_border_t">36.20</td>
</tr>
<tr id="A1.T4.1.4.4" class="ltx_tr">
<th id="A1.T4.1.4.4.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">PtGen <cite class="ltx_cite ltx_citemacro_cite">See et al. (<a href="#bib.bib65" title="" class="ltx_ref">2017</a>)</cite>
</th>
<td id="A1.T4.1.4.4.2" class="ltx_td ltx_align_center">39.53</td>
<td id="A1.T4.1.4.4.3" class="ltx_td ltx_align_center">17.28</td>
<td id="A1.T4.1.4.4.4" class="ltx_td ltx_align_center">36.38</td>
</tr>
<tr id="A1.T4.1.5.5" class="ltx_tr">
<th id="A1.T4.1.5.5.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">Bottom-Up <cite class="ltx_cite ltx_citemacro_cite">Gehrmann et al. (<a href="#bib.bib27" title="" class="ltx_ref">2018</a>)</cite>
</th>
<td id="A1.T4.1.5.5.2" class="ltx_td ltx_align_center">41.22</td>
<td id="A1.T4.1.5.5.3" class="ltx_td ltx_align_center">18.68</td>
<td id="A1.T4.1.5.5.4" class="ltx_td ltx_align_center">38.34</td>
</tr>
<tr id="A1.T4.1.6.6" class="ltx_tr">
<th id="A1.T4.1.6.6.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">SAGCopy <cite class="ltx_cite ltx_citemacro_cite">Xu et al. (<a href="#bib.bib82" title="" class="ltx_ref">2020</a>)</cite>
</th>
<td id="A1.T4.1.6.6.2" class="ltx_td ltx_align_center">42.53</td>
<td id="A1.T4.1.6.6.3" class="ltx_td ltx_align_center">19.92</td>
<td id="A1.T4.1.6.6.4" class="ltx_td ltx_align_center">39.44</td>
</tr>
<tr id="A1.T4.1.7.7" class="ltx_tr">
<th id="A1.T4.1.7.7.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">MASS <cite class="ltx_cite ltx_citemacro_cite">Song et al. (<a href="#bib.bib68" title="" class="ltx_ref">2019</a>)</cite>
</th>
<td id="A1.T4.1.7.7.2" class="ltx_td ltx_align_center">42.12</td>
<td id="A1.T4.1.7.7.3" class="ltx_td ltx_align_center">19.50</td>
<td id="A1.T4.1.7.7.4" class="ltx_td ltx_align_center">39.01</td>
</tr>
<tr id="A1.T4.1.8.8" class="ltx_tr">
<th id="A1.T4.1.8.8.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">UniLM <cite class="ltx_cite ltx_citemacro_cite">Dong et al. (<a href="#bib.bib17" title="" class="ltx_ref">2019a</a>)</cite>
</th>
<td id="A1.T4.1.8.8.2" class="ltx_td ltx_align_center">43.33</td>
<td id="A1.T4.1.8.8.3" class="ltx_td ltx_align_center">20.21</td>
<td id="A1.T4.1.8.8.4" class="ltx_td ltx_align_center">40.51</td>
</tr>
<tr id="A1.T4.1.9.9" class="ltx_tr">
<th id="A1.T4.1.9.9.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">BART <cite class="ltx_cite ltx_citemacro_cite">Lewis et al. (<a href="#bib.bib42" title="" class="ltx_ref">2019</a>)</cite>
</th>
<td id="A1.T4.1.9.9.2" class="ltx_td ltx_align_center">44.16</td>
<td id="A1.T4.1.9.9.3" class="ltx_td ltx_align_center">21.28</td>
<td id="A1.T4.1.9.9.4" class="ltx_td ltx_align_center">40.90</td>
</tr>
<tr id="A1.T4.1.10.10" class="ltx_tr">
<th id="A1.T4.1.10.10.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">T5 <cite class="ltx_cite ltx_citemacro_cite">Raffel et al. (<a href="#bib.bib60" title="" class="ltx_ref">2019</a>)</cite>
</th>
<td id="A1.T4.1.10.10.2" class="ltx_td ltx_align_center">43.52</td>
<td id="A1.T4.1.10.10.3" class="ltx_td ltx_align_center"><span id="A1.T4.1.10.10.3.1" class="ltx_text ltx_font_bold ltx_framed ltx_framed_underline">21.55</span></td>
<td id="A1.T4.1.10.10.4" class="ltx_td ltx_align_center">40.69</td>
</tr>
<tr id="A1.T4.1.11.11" class="ltx_tr">
<th id="A1.T4.1.11.11.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A1.T4.1.11.11.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> (C4, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib85" title="" class="ltx_ref">Zhang et al.</a></cite>, <cite class="ltx_cite ltx_citemacro_citeyear"><a href="#bib.bib85" title="" class="ltx_ref">2019</a></cite>)</th>
<td id="A1.T4.1.11.11.2" class="ltx_td ltx_align_center">43.90</td>
<td id="A1.T4.1.11.11.3" class="ltx_td ltx_align_center">21.20</td>
<td id="A1.T4.1.11.11.4" class="ltx_td ltx_align_center">40.76</td>
</tr>
<tr id="A1.T4.1.12.12" class="ltx_tr">
<th id="A1.T4.1.12.12.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A1.T4.1.12.12.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> (HugeNews, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib85" title="" class="ltx_ref">Zhang et al.</a></cite>, <cite class="ltx_cite ltx_citemacro_citeyear"><a href="#bib.bib85" title="" class="ltx_ref">2019</a></cite>)</th>
<td id="A1.T4.1.12.12.2" class="ltx_td ltx_align_center">44.17</td>
<td id="A1.T4.1.12.12.3" class="ltx_td ltx_align_center">21.47</td>
<td id="A1.T4.1.12.12.4" class="ltx_td ltx_align_center">41.11</td>
</tr>
<tr id="A1.T4.1.13.13" class="ltx_tr">
<th id="A1.T4.1.13.13.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">ProphetNet <cite class="ltx_cite ltx_citemacro_cite">Qi et al. (<a href="#bib.bib58" title="" class="ltx_ref">2020</a>)</cite>
</th>
<td id="A1.T4.1.13.13.2" class="ltx_td ltx_align_center"><span id="A1.T4.1.13.13.2.1" class="ltx_text ltx_font_bold ltx_framed ltx_framed_underline">44.20</span></td>
<td id="A1.T4.1.13.13.3" class="ltx_td ltx_align_center">21.17</td>
<td id="A1.T4.1.13.13.4" class="ltx_td ltx_align_center"><span id="A1.T4.1.13.13.4.1" class="ltx_text ltx_font_bold ltx_framed ltx_framed_underline">41.30</span></td>
</tr>
<tr id="A1.T4.1.14.14" class="ltx_tr">
<th id="A1.T4.1.14.14.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t">
<span id="A1.T4.1.14.14.1.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> <cite class="ltx_cite ltx_citemacro_cite">Rothe et al. (<a href="#bib.bib63" title="" class="ltx_ref">2020</a>)</cite>
</th>
<td id="A1.T4.1.14.14.2" class="ltx_td ltx_align_center ltx_border_t">39.88</td>
<td id="A1.T4.1.14.14.3" class="ltx_td ltx_align_center ltx_border_t">18.66</td>
<td id="A1.T4.1.14.14.4" class="ltx_td ltx_align_center ltx_border_t">37.22</td>
</tr>
<tr id="A1.T4.1.15.15" class="ltx_tr">
<th id="A1.T4.1.15.15.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A1.T4.1.15.15.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (ours)</th>
<td id="A1.T4.1.15.15.2" class="ltx_td ltx_align_center">40.27</td>
<td id="A1.T4.1.15.15.3" class="ltx_td ltx_align_center">18.43</td>
<td id="A1.T4.1.15.15.4" class="ltx_td ltx_align_center">37.51</td>
</tr>
<tr id="A1.T4.1.16.16" class="ltx_tr">
<th id="A1.T4.1.16.16.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A1.T4.1.16.16.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> (ours)</th>
<td id="A1.T4.1.16.16.2" class="ltx_td ltx_align_center">42.62</td>
<td id="A1.T4.1.16.16.3" class="ltx_td ltx_align_center">20.38</td>
<td id="A1.T4.1.16.16.4" class="ltx_td ltx_align_center">39.61</td>
</tr>
<tr id="A1.T4.1.17.17" class="ltx_tr">
<th id="A1.T4.1.17.17.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_bb ltx_border_r">
<span id="A1.T4.1.17.17.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (ours)</th>
<td id="A1.T4.1.17.17.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T4.1.17.17.2.1" class="ltx_text ltx_font_bold">42.95</span></td>
<td id="A1.T4.1.17.17.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T4.1.17.17.3.1" class="ltx_text ltx_font_bold">20.79</span></td>
<td id="A1.T4.1.17.17.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T4.1.17.17.4.1" class="ltx_text ltx_font_bold">39.90</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Abstractive summarization results on <span id="A1.T4.6.1" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span> datasets. The <span id="A1.T4.7.2" class="ltx_text ltx_font_bold ltx_framed ltx_framed_underline">underlined bold</span> results are from the best performing models from literature and the <span id="A1.T4.8.3" class="ltx_text ltx_font_bold">bold</span> results are the best performing <span id="A1.T4.9.4" class="ltx_text ltx_font_smallcaps">Fame</span> models.</figcaption>
</figure>
<figure id="A1.T5" class="ltx_table">
<table id="A1.T5.3" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T5.1.1" class="ltx_tr">
<th id="A1.T5.1.1.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_r ltx_border_tt" rowspan="2"><span id="A1.T5.1.1.2.1" class="ltx_text">Models</span></th>
<th id="A1.T5.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="A1.T5.1.1.3.1" class="ltx_text">Len.</span></th>
<th id="A1.T5.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Rep.</th>
<th id="A1.T5.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">R1(P%)</th>
<th id="A1.T5.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="2">doc. <math id="A1.T5.1.1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="A1.T5.1.1.1.m1.1a"><mo stretchy="false" id="A1.T5.1.1.1.m1.1.1" xref="A1.T5.1.1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="A1.T5.1.1.1.m1.1b"><ci id="A1.T5.1.1.1.m1.1.1.cmml" xref="A1.T5.1.1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.1.1.1.m1.1c">\rightarrow</annotation></semantics></math> sum.</th>
<th id="A1.T5.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="2">Feqa</th>
<th id="A1.T5.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="A1.T5.1.1.7.1" class="ltx_text">BERTSc.</span></th>
</tr>
<tr id="A1.T5.3.3" class="ltx_tr">
<th id="A1.T5.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">%</th>
<th id="A1.T5.3.3.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">With doc.</th>
<th id="A1.T5.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column">ent. (<math id="A1.T5.2.2.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A1.T5.2.2.1.m1.1a"><mo stretchy="false" id="A1.T5.2.2.1.m1.1.1" xref="A1.T5.2.2.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A1.T5.2.2.1.m1.1b"><ci id="A1.T5.2.2.1.m1.1.1.cmml" xref="A1.T5.2.2.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.2.2.1.m1.1c">\uparrow</annotation></semantics></math>)</th>
<th id="A1.T5.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">
<math id="A1.T5.3.3.2.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T5.3.3.2.m1.1a"><mo id="A1.T5.3.3.2.m1.1.1" xref="A1.T5.3.3.2.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T5.3.3.2.m1.1b"><not id="A1.T5.3.3.2.m1.1.1.cmml" xref="A1.T5.3.3.2.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.3.3.2.m1.1c">\neg</annotation></semantics></math> cont.</th>
<th id="A1.T5.3.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column">acc.</th>
<th id="A1.T5.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">avg.(#Q)</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T5.3.4.1" class="ltx_tr">
<td id="A1.T5.3.4.1.1" class="ltx_td ltx_align_right ltx_border_r ltx_border_t"><span id="A1.T5.3.4.1.1.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span></td>
<td id="A1.T5.3.4.1.2" class="ltx_td ltx_align_center ltx_border_t">52.1</td>
<td id="A1.T5.3.4.1.3" class="ltx_td ltx_align_center ltx_border_t">77.6</td>
<td id="A1.T5.3.4.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">92.7</td>
<td id="A1.T5.3.4.1.5" class="ltx_td ltx_align_center ltx_border_t">88.8</td>
<td id="A1.T5.3.4.1.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">96.4</td>
<td id="A1.T5.3.4.1.7" class="ltx_td ltx_align_center ltx_border_t">37.3</td>
<td id="A1.T5.3.4.1.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">18.1</td>
<td id="A1.T5.3.4.1.9" class="ltx_td ltx_align_center ltx_border_t">76.0</td>
</tr>
<tr id="A1.T5.3.5.2" class="ltx_tr">
<td id="A1.T5.3.5.2.1" class="ltx_td ltx_align_right ltx_border_r"><span id="A1.T5.3.5.2.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span></td>
<td id="A1.T5.3.5.2.2" class="ltx_td ltx_align_center">55.5</td>
<td id="A1.T5.3.5.2.3" class="ltx_td ltx_align_center">79.6</td>
<td id="A1.T5.3.5.2.4" class="ltx_td ltx_align_center ltx_border_r">92.5</td>
<td id="A1.T5.3.5.2.5" class="ltx_td ltx_align_center">87.3</td>
<td id="A1.T5.3.5.2.6" class="ltx_td ltx_align_center ltx_border_r">96.3</td>
<td id="A1.T5.3.5.2.7" class="ltx_td ltx_align_center">35.2</td>
<td id="A1.T5.3.5.2.8" class="ltx_td ltx_align_center ltx_border_r">19.3</td>
<td id="A1.T5.3.5.2.9" class="ltx_td ltx_align_center">76.1</td>
</tr>
<tr id="A1.T5.3.6.3" class="ltx_tr">
<td id="A1.T5.3.6.3.1" class="ltx_td ltx_align_right ltx_border_r"><span id="A1.T5.3.6.3.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span></td>
<td id="A1.T5.3.6.3.2" class="ltx_td ltx_align_center">58.1</td>
<td id="A1.T5.3.6.3.3" class="ltx_td ltx_align_center">69.4</td>
<td id="A1.T5.3.6.3.4" class="ltx_td ltx_align_center ltx_border_r">95.0</td>
<td id="A1.T5.3.6.3.5" class="ltx_td ltx_align_center">90.9</td>
<td id="A1.T5.3.6.3.6" class="ltx_td ltx_align_center ltx_border_r">97.5</td>
<td id="A1.T5.3.6.3.7" class="ltx_td ltx_align_center">40.3</td>
<td id="A1.T5.3.6.3.8" class="ltx_td ltx_align_center ltx_border_r">21.0</td>
<td id="A1.T5.3.6.3.9" class="ltx_td ltx_align_center">76.8</td>
</tr>
<tr id="A1.T5.3.7.4" class="ltx_tr">
<td id="A1.T5.3.7.4.1" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r"><span id="A1.T5.3.7.4.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span></td>
<td id="A1.T5.3.7.4.2" class="ltx_td ltx_align_center ltx_border_bb">58.5</td>
<td id="A1.T5.3.7.4.3" class="ltx_td ltx_align_center ltx_border_bb">71.0</td>
<td id="A1.T5.3.7.4.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">95.3</td>
<td id="A1.T5.3.7.4.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T5.3.7.4.5.1" class="ltx_text ltx_font_bold">91.0</span></td>
<td id="A1.T5.3.7.4.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="A1.T5.3.7.4.6.1" class="ltx_text ltx_font_bold">97.6</span></td>
<td id="A1.T5.3.7.4.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T5.3.7.4.7.1" class="ltx_text ltx_font_bold">41.1</span></td>
<td id="A1.T5.3.7.4.8" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">21.1</td>
<td id="A1.T5.3.7.4.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T5.3.7.4.9.1" class="ltx_text ltx_font_bold">76.9</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Faithfulness and qualitative assessment of summaries on <span id="A1.T5.5.1" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span> dataset.</figcaption>
</figure>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Abstractive Summarization Results on CNN/DailyMail</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.3" class="ltx_p">The <span id="A2.p1.3.1" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span> dataset <cite class="ltx_cite ltx_citemacro_cite">Hermann et al. (<a href="#bib.bib32" title="" class="ltx_ref">2015</a>)</cite> consists of 287,227/13,368/11,490 training/validation/test document-summary pairs. The <span id="A2.p1.3.2" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span> summaries are in the form of bullet-point story highlights and exhibit a high degree of extraction, requiring the models to learn to copy from the source documents. The <span id="A2.p1.3.3" class="ltx_text ltx_font_smallcaps">XSum</span> summaries, on the other hand, are extreme, in that the documents are summarized into single-sentence summaries with a high level of abstractiveness. For comparison, the <span id="A2.p1.3.4" class="ltx_text ltx_font_smallcaps">XSum</span> summaries show a much larger percentages of novel constructions than found in <span id="A2.p1.3.5" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span> summaries (<math id="A2.p1.1.m1.1" class="ltx_Math" alttext="35.8/83.5/95.5/98.5" display="inline"><semantics id="A2.p1.1.m1.1a"><mrow id="A2.p1.1.m1.1.1" xref="A2.p1.1.m1.1.1.cmml"><mn id="A2.p1.1.m1.1.1.2" xref="A2.p1.1.m1.1.1.2.cmml">35.8</mn><mo id="A2.p1.1.m1.1.1.1" xref="A2.p1.1.m1.1.1.1.cmml">/</mo><mn id="A2.p1.1.m1.1.1.3" xref="A2.p1.1.m1.1.1.3.cmml">83.5</mn><mo id="A2.p1.1.m1.1.1.1a" xref="A2.p1.1.m1.1.1.1.cmml">/</mo><mn id="A2.p1.1.m1.1.1.4" xref="A2.p1.1.m1.1.1.4.cmml">95.5</mn><mo id="A2.p1.1.m1.1.1.1b" xref="A2.p1.1.m1.1.1.1.cmml">/</mo><mn id="A2.p1.1.m1.1.1.5" xref="A2.p1.1.m1.1.1.5.cmml">98.5</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.1.m1.1b"><apply id="A2.p1.1.m1.1.1.cmml" xref="A2.p1.1.m1.1.1"><divide id="A2.p1.1.m1.1.1.1.cmml" xref="A2.p1.1.m1.1.1.1"></divide><cn type="float" id="A2.p1.1.m1.1.1.2.cmml" xref="A2.p1.1.m1.1.1.2">35.8</cn><cn type="float" id="A2.p1.1.m1.1.1.3.cmml" xref="A2.p1.1.m1.1.1.3">83.5</cn><cn type="float" id="A2.p1.1.m1.1.1.4.cmml" xref="A2.p1.1.m1.1.1.4">95.5</cn><cn type="float" id="A2.p1.1.m1.1.1.5.cmml" xref="A2.p1.1.m1.1.1.5">98.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.1.m1.1c">35.8/83.5/95.5/98.5</annotation></semantics></math> vs <math id="A2.p1.2.m2.1" class="ltx_Math" alttext="16.8/54.3/72.4/80.4" display="inline"><semantics id="A2.p1.2.m2.1a"><mrow id="A2.p1.2.m2.1.1" xref="A2.p1.2.m2.1.1.cmml"><mn id="A2.p1.2.m2.1.1.2" xref="A2.p1.2.m2.1.1.2.cmml">16.8</mn><mo id="A2.p1.2.m2.1.1.1" xref="A2.p1.2.m2.1.1.1.cmml">/</mo><mn id="A2.p1.2.m2.1.1.3" xref="A2.p1.2.m2.1.1.3.cmml">54.3</mn><mo id="A2.p1.2.m2.1.1.1a" xref="A2.p1.2.m2.1.1.1.cmml">/</mo><mn id="A2.p1.2.m2.1.1.4" xref="A2.p1.2.m2.1.1.4.cmml">72.4</mn><mo id="A2.p1.2.m2.1.1.1b" xref="A2.p1.2.m2.1.1.1.cmml">/</mo><mn id="A2.p1.2.m2.1.1.5" xref="A2.p1.2.m2.1.1.5.cmml">80.4</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.2.m2.1b"><apply id="A2.p1.2.m2.1.1.cmml" xref="A2.p1.2.m2.1.1"><divide id="A2.p1.2.m2.1.1.1.cmml" xref="A2.p1.2.m2.1.1.1"></divide><cn type="float" id="A2.p1.2.m2.1.1.2.cmml" xref="A2.p1.2.m2.1.1.2">16.8</cn><cn type="float" id="A2.p1.2.m2.1.1.3.cmml" xref="A2.p1.2.m2.1.1.3">54.3</cn><cn type="float" id="A2.p1.2.m2.1.1.4.cmml" xref="A2.p1.2.m2.1.1.4">72.4</cn><cn type="float" id="A2.p1.2.m2.1.1.5.cmml" xref="A2.p1.2.m2.1.1.5">80.4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.2.m2.1c">16.8/54.3/72.4/80.4</annotation></semantics></math> novel <math id="A2.p1.3.m3.1" class="ltx_Math" alttext="1/2/3/4" display="inline"><semantics id="A2.p1.3.m3.1a"><mrow id="A2.p1.3.m3.1.1" xref="A2.p1.3.m3.1.1.cmml"><mn id="A2.p1.3.m3.1.1.2" xref="A2.p1.3.m3.1.1.2.cmml">1</mn><mo id="A2.p1.3.m3.1.1.1" xref="A2.p1.3.m3.1.1.1.cmml">/</mo><mn id="A2.p1.3.m3.1.1.3" xref="A2.p1.3.m3.1.1.3.cmml">2</mn><mo id="A2.p1.3.m3.1.1.1a" xref="A2.p1.3.m3.1.1.1.cmml">/</mo><mn id="A2.p1.3.m3.1.1.4" xref="A2.p1.3.m3.1.1.4.cmml">3</mn><mo id="A2.p1.3.m3.1.1.1b" xref="A2.p1.3.m3.1.1.1.cmml">/</mo><mn id="A2.p1.3.m3.1.1.5" xref="A2.p1.3.m3.1.1.5.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.3.m3.1b"><apply id="A2.p1.3.m3.1.1.cmml" xref="A2.p1.3.m3.1.1"><divide id="A2.p1.3.m3.1.1.1.cmml" xref="A2.p1.3.m3.1.1.1"></divide><cn type="integer" id="A2.p1.3.m3.1.1.2.cmml" xref="A2.p1.3.m3.1.1.2">1</cn><cn type="integer" id="A2.p1.3.m3.1.1.3.cmml" xref="A2.p1.3.m3.1.1.3">2</cn><cn type="integer" id="A2.p1.3.m3.1.1.4.cmml" xref="A2.p1.3.m3.1.1.4">3</cn><cn type="integer" id="A2.p1.3.m3.1.1.5.cmml" xref="A2.p1.3.m3.1.1.5">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.3.m3.1c">1/2/3/4</annotation></semantics></math>-grams). We use the original cased version. During training, the input documents are truncated to 512 tokens and the length of the summaries are limited to 128 tokens.</p>
</div>
<div id="A2.p2" class="ltx_para">
<p id="A2.p2.1" class="ltx_p">Table <a href="#A1.T4" title="Table 4 ‣ Appendix A Implementation and Reproducibility Details ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> and <a href="#A1.T5" title="Table 5 ‣ Appendix A Implementation and Reproducibility Details ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> present complete results for <span id="A2.p2.1.1" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span> dataset. We see similar kind of improvements as observed in Table <a href="#S4.T1" title="Table 1 ‣ 4.2 Pretrained Models with Fame ‣ 4 Experimental Setup ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, except for <span id="A2.p2.1.2" class="ltx_text ltx_font_smallcaps">rouge-2</span> for <span id="A2.p2.1.3" class="ltx_text ltx_font_smallcaps">RobFame</span> which is 0.23 points worse than the <span id="A2.p2.1.4" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> baseline.
Our best model <span id="A2.p2.1.5" class="ltx_text ltx_font_smallcaps">PegFame</span> performs better than both copy mechanism models: LSTM-based PtGen <cite class="ltx_cite ltx_citemacro_cite">See et al. (<a href="#bib.bib65" title="" class="ltx_ref">2017</a>)</cite> and Transformer-based SAGCopy <cite class="ltx_cite ltx_citemacro_cite">Xu et al. (<a href="#bib.bib82" title="" class="ltx_ref">2020</a>)</cite>. <span id="A2.p2.1.6" class="ltx_text ltx_font_smallcaps">PegFame</span> performs worse when compared with T5 <cite class="ltx_cite ltx_citemacro_cite">Raffel et al. (<a href="#bib.bib60" title="" class="ltx_ref">2019</a>)</cite>, the original <span id="A2.p2.1.7" class="ltx_text ltx_font_smallcaps">Pegasus</span> <cite class="ltx_cite ltx_citemacro_cite">Zhang et al. (<a href="#bib.bib85" title="" class="ltx_ref">2019</a>)</cite> and ProphetNet <cite class="ltx_cite ltx_citemacro_cite">Qi et al. (<a href="#bib.bib58" title="" class="ltx_ref">2020</a>)</cite>. This can be expected as the number of parameters in <span id="A2.p2.1.8" class="ltx_text ltx_font_smallcaps">PegFame</span> is almost half of T5 or ProphetNet, and is 100M less than that in the original <span id="A2.p2.1.9" class="ltx_text ltx_font_smallcaps">Pegasus</span>.</p>
</div>
<div id="A2.p3" class="ltx_para">
<p id="A2.p3.1" class="ltx_p"><span id="A2.p3.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> performs worse than <span id="A2.p3.1.2" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> on both ent. and Feqa measures for <span id="A2.p3.1.3" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span>, similar to <span id="A2.p3.1.4" class="ltx_text ltx_font_smallcaps">rouge-2</span> in Table <a href="#A1.T4" title="Table 4 ‣ Appendix A Implementation and Reproducibility Details ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. We hypothesize that this is due to the extractive nature of the <span id="A2.p3.1.5" class="ltx_text ltx_font_smallcaps">Cnn/Dm</span> dataset and the fact that it is not able to copy tokens from the input to the necessary extent as the encoder-decoder attention is not pre-trained. Moreover, Feqa scores for <span id="A2.p3.1.6" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> and <span id="A2.p3.1.7" class="ltx_text ltx_font_smallcaps">RobFame</span> may not be fully comparable due to variation in their summary lengths and the number of Feqa questions generated; the <span id="A2.p3.1.8" class="ltx_text ltx_font_smallcaps">RobFame</span> summaries, on average, are 3 words longer and generate 1.2 more questions than that of <span id="A2.p3.1.9" class="ltx_text ltx_font_smallcaps">RobertaS2S</span>.
Nevertheless, we don’t see this kind of drop in <math id="A2.p3.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A2.p3.1.m1.1a"><mo id="A2.p3.1.m1.1.1" xref="A2.p3.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A2.p3.1.m1.1b"><not id="A2.p3.1.m1.1.1.cmml" xref="A2.p3.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A2.p3.1.m1.1c">\neg</annotation></semantics></math>cont. scores (i.e., summary not contradicting, either entailed by or neutral to the document) and BERTScores.</p>
</div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Text Editing Results</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">We also train the <span id="A3.p1.1.1" class="ltx_text ltx_font_smallcaps">Fame</span> models on two text editing tasks: (i) for sentence fusion – the problem of combining multiple sentences into a single coherent sentence – we used the “balanced Wikipedia” portion of the DiscoFuse dataset <cite class="ltx_cite ltx_citemacro_citep">(Geva et al., <a href="#bib.bib28" title="" class="ltx_ref">2019</a>)</cite>, and (ii) for split-and-rephrase – the reverse task of sentence fusion – we used the WikiSplit dataset <cite class="ltx_cite ltx_citemacro_citep">(Botha et al., <a href="#bib.bib7" title="" class="ltx_ref">2018</a>)</cite>, which consists of 1M examples of sentence splits extracted from the Wikipedia edit history. As the name suggests, both text editing tasks require a low degree of abstraction.</p>
</div>
<div id="A3.p2" class="ltx_para">
<p id="A3.p2.1" class="ltx_p">For both the tasks, we train the models for 300k steps with a global batch size of 256. The input and output are padded to a length of 128, which covers 100% of the training, evaluation and test data. The vocabulary for functional tokens <math id="A3.p2.1.m1.1" class="ltx_Math" alttext="F" display="inline"><semantics id="A3.p2.1.m1.1a"><mi id="A3.p2.1.m1.1.1" xref="A3.p2.1.m1.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="A3.p2.1.m1.1b"><ci id="A3.p2.1.m1.1.1.cmml" xref="A3.p2.1.m1.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.p2.1.m1.1c">F</annotation></semantics></math> is constructed by taking the top 100 and 500 sentence pieces for DiscoFuse and WikiSplit respectively.</p>
</div>
<figure id="A3.T6" class="ltx_table">
<table id="A3.T6.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A3.T6.1.1.1" class="ltx_tr">
<th id="A3.T6.1.1.1.1" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt">DiscoFuse</th>
<th id="A3.T6.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Exact</th>
<th id="A3.T6.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">SARI</th>
<th id="A3.T6.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">BLEU</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A3.T6.1.2.1" class="ltx_tr">
<th id="A3.T6.1.2.1.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_cite">Geva et al. (<a href="#bib.bib28" title="" class="ltx_ref">2019</a>)</cite></th>
<td id="A3.T6.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">51.1</td>
<td id="A3.T6.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">84.5</td>
<td id="A3.T6.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">–</td>
</tr>
<tr id="A3.T6.1.3.2" class="ltx_tr">
<th id="A3.T6.1.3.2.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">LaserTagger <cite class="ltx_cite ltx_citemacro_cite">Malmi et al. (<a href="#bib.bib49" title="" class="ltx_ref">2019</a>)</cite>
</th>
<td id="A3.T6.1.3.2.2" class="ltx_td ltx_align_center">53.8</td>
<td id="A3.T6.1.3.2.3" class="ltx_td ltx_align_center">85.5</td>
<td id="A3.T6.1.3.2.4" class="ltx_td ltx_align_center">–</td>
</tr>
<tr id="A3.T6.1.4.3" class="ltx_tr">
<th id="A3.T6.1.4.3.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">Felix <cite class="ltx_cite ltx_citemacro_cite">Mallinson et al. (<a href="#bib.bib48" title="" class="ltx_ref">2020</a>)</cite>
</th>
<td id="A3.T6.1.4.3.2" class="ltx_td ltx_align_center">61.3</td>
<td id="A3.T6.1.4.3.3" class="ltx_td ltx_align_center">88.8</td>
<td id="A3.T6.1.4.3.4" class="ltx_td ltx_align_center">–</td>
</tr>
<tr id="A3.T6.1.5.4" class="ltx_tr">
<th id="A3.T6.1.5.4.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A3.T6.1.5.4.1.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> <cite class="ltx_cite ltx_citemacro_cite">Rothe et al. (<a href="#bib.bib63" title="" class="ltx_ref">2020</a>)</cite>
</th>
<td id="A3.T6.1.5.4.2" class="ltx_td ltx_align_center">66.6</td>
<td id="A3.T6.1.5.4.3" class="ltx_td ltx_align_center">90.3</td>
<td id="A3.T6.1.5.4.4" class="ltx_td ltx_align_center">–</td>
</tr>
<tr id="A3.T6.1.6.5" class="ltx_tr">
<th id="A3.T6.1.6.5.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A3.T6.1.6.5.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> (ours)</th>
<td id="A3.T6.1.6.5.2" class="ltx_td ltx_align_center"><span id="A3.T6.1.6.5.2.1" class="ltx_text ltx_framed ltx_framed_underline">67.4</span></td>
<td id="A3.T6.1.6.5.3" class="ltx_td ltx_align_center"><span id="A3.T6.1.6.5.3.1" class="ltx_text ltx_framed ltx_framed_underline">90.5</span></td>
<td id="A3.T6.1.6.5.4" class="ltx_td ltx_align_center">95.8</td>
</tr>
<tr id="A3.T6.1.7.6" class="ltx_tr">
<th id="A3.T6.1.7.6.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A3.T6.1.7.6.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (ours)</th>
<td id="A3.T6.1.7.6.2" class="ltx_td ltx_align_center"><span id="A3.T6.1.7.6.2.1" class="ltx_text ltx_font_bold">67.8</span></td>
<td id="A3.T6.1.7.6.3" class="ltx_td ltx_align_center"><span id="A3.T6.1.7.6.3.1" class="ltx_text ltx_font_bold">90.7</span></td>
<td id="A3.T6.1.7.6.4" class="ltx_td ltx_align_center"><span id="A3.T6.1.7.6.4.1" class="ltx_text ltx_font_bold">95.9</span></td>
</tr>
<tr id="A3.T6.1.8.7" class="ltx_tr">
<th id="A3.T6.1.8.7.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t">WikiSplit</th>
<td id="A3.T6.1.8.7.2" class="ltx_td ltx_align_center ltx_border_t">Exact</td>
<td id="A3.T6.1.8.7.3" class="ltx_td ltx_align_center ltx_border_t">SARI</td>
<td id="A3.T6.1.8.7.4" class="ltx_td ltx_align_center ltx_border_t">BLEU</td>
</tr>
<tr id="A3.T6.1.9.8" class="ltx_tr">
<th id="A3.T6.1.9.8.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_cite">Botha et al. (<a href="#bib.bib7" title="" class="ltx_ref">2018</a>)</cite></th>
<td id="A3.T6.1.9.8.2" class="ltx_td ltx_align_center ltx_border_t">14.3</td>
<td id="A3.T6.1.9.8.3" class="ltx_td ltx_align_center ltx_border_t">61.5</td>
<td id="A3.T6.1.9.8.4" class="ltx_td ltx_align_center ltx_border_t">76.4</td>
</tr>
<tr id="A3.T6.1.10.9" class="ltx_tr">
<th id="A3.T6.1.10.9.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">LaseTagger <cite class="ltx_cite ltx_citemacro_cite">Malmi et al. (<a href="#bib.bib49" title="" class="ltx_ref">2019</a>)</cite>
</th>
<td id="A3.T6.1.10.9.2" class="ltx_td ltx_align_center">15.2</td>
<td id="A3.T6.1.10.9.3" class="ltx_td ltx_align_center">61.7</td>
<td id="A3.T6.1.10.9.4" class="ltx_td ltx_align_center">76.3</td>
</tr>
<tr id="A3.T6.1.11.10" class="ltx_tr">
<th id="A3.T6.1.11.10.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A3.T6.1.11.10.1.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> <cite class="ltx_cite ltx_citemacro_cite">Rothe et al. (<a href="#bib.bib63" title="" class="ltx_ref">2020</a>)</cite>
</th>
<td id="A3.T6.1.11.10.2" class="ltx_td ltx_align_center">16.4</td>
<td id="A3.T6.1.11.10.3" class="ltx_td ltx_align_center">63.8</td>
<td id="A3.T6.1.11.10.4" class="ltx_td ltx_align_center"><span id="A3.T6.1.11.10.4.1" class="ltx_text ltx_font_bold">77.4</span></td>
</tr>
<tr id="A3.T6.1.12.11" class="ltx_tr">
<th id="A3.T6.1.12.11.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A3.T6.1.12.11.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> (ours)</th>
<td id="A3.T6.1.12.11.2" class="ltx_td ltx_align_center"><span id="A3.T6.1.12.11.2.1" class="ltx_text ltx_framed ltx_framed_underline">16.6</span></td>
<td id="A3.T6.1.12.11.3" class="ltx_td ltx_align_center"><span id="A3.T6.1.12.11.3.1" class="ltx_text ltx_font_bold">64.1</span></td>
<td id="A3.T6.1.12.11.4" class="ltx_td ltx_align_center"><span id="A3.T6.1.12.11.4.1" class="ltx_text ltx_font_bold">77.4</span></td>
</tr>
<tr id="A3.T6.1.13.12" class="ltx_tr">
<th id="A3.T6.1.13.12.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_bb ltx_border_r">
<span id="A3.T6.1.13.12.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (ours)</th>
<td id="A3.T6.1.13.12.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A3.T6.1.13.12.2.1" class="ltx_text ltx_font_bold">16.8</span></td>
<td id="A3.T6.1.13.12.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A3.T6.1.13.12.3.1" class="ltx_text ltx_font_bold">64.1</span></td>
<td id="A3.T6.1.13.12.4" class="ltx_td ltx_align_center ltx_border_bb">77.3</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Text editing results on Discofuse and WikiSplit. The <span id="A3.T6.4.1" class="ltx_text ltx_framed ltx_framed_underline">underlined</span> scores beat the current state-of-the-art and the <span id="A3.T6.5.2" class="ltx_text ltx_font_bold">bold</span> scores are the new state-of-the-art.</figcaption>
</figure>
<div id="A3.p3" class="ltx_para">
<p id="A3.p3.1" class="ltx_p">We report corpus-level BLEU<span id="footnote14" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">14</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">14</sup><span class="ltx_tag ltx_tag_note">14</span>We use NLTK v3.2.2 with case sensitive scoring to estimate BLEU scores.</span></span></span>, the exact match accuracy, and SARI scores <cite class="ltx_cite ltx_citemacro_citep">(Xu et al., <a href="#bib.bib83" title="" class="ltx_ref">2016</a>)</cite><span id="footnote15" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">15</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">15</sup><span class="ltx_tag ltx_tag_note">15</span>SARI
is a lexical similarity metric which compares the model’s output to multiple references and the input in order to assess the model’s ability to add, delete, and keep an <math id="footnote15.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="footnote15.m1.1b"><mi id="footnote15.m1.1.1" xref="footnote15.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="footnote15.m1.1c"><ci id="footnote15.m1.1.1.cmml" xref="footnote15.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote15.m1.1d">n</annotation></semantics></math>-gram. It’s implementation is available at: <a target="_blank" href="https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/sari_hook.py" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/sari_hook.py</a>.</span></span></span>. The results can be seen in Table <a href="#A3.T6" title="Table 6 ‣ Appendix C Text Editing Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>. The vanilla <span id="A3.p3.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> model already beats the current state-of-the-art on both DiscoFuse and WikiSplit. The <span id="A3.p3.1.2" class="ltx_text ltx_font_smallcaps">PegFame</span> model performs better, albeit by a small margin, on all metrics on DiscoFuse. On WikiSplit, it has a higher exact match accuracy while maintaining the SARI score and performs 0.1 BLEU worse than <span id="A3.p3.1.3" class="ltx_text ltx_font_smallcaps">Pegasus</span>.</p>
</div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Controlled Generation with focus attention using Top-<math id="A4.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A4.1.m1.1b"><mi id="A4.1.m1.1.1" xref="A4.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A4.1.m1.1c"><ci id="A4.1.m1.1.1.cmml" xref="A4.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.1.m1.1d">k</annotation></semantics></math> tokens</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.6" class="ltx_p">Table <a href="#A5.T7" title="Table 7 ‣ Appendix E Diverse Summarization with Div_{top,𝑘}, Div_nucleus and Focus_{sample,𝑘} ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> presents results from our controlled summary generation experiments with top-<math id="A4.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A4.p1.1.m1.1a"><mi id="A4.p1.1.m1.1.1" xref="A4.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A4.p1.1.m1.1b"><ci id="A4.p1.1.m1.1.1.cmml" xref="A4.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.1.m1.1c">k</annotation></semantics></math> tokens from <math id="A4.p1.2.m2.1" class="ltx_Math" alttext="t_{X}" display="inline"><semantics id="A4.p1.2.m2.1a"><msub id="A4.p1.2.m2.1.1" xref="A4.p1.2.m2.1.1.cmml"><mi id="A4.p1.2.m2.1.1.2" xref="A4.p1.2.m2.1.1.2.cmml">t</mi><mi id="A4.p1.2.m2.1.1.3" xref="A4.p1.2.m2.1.1.3.cmml">X</mi></msub><annotation-xml encoding="MathML-Content" id="A4.p1.2.m2.1b"><apply id="A4.p1.2.m2.1.1.cmml" xref="A4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A4.p1.2.m2.1.1.1.cmml" xref="A4.p1.2.m2.1.1">subscript</csymbol><ci id="A4.p1.2.m2.1.1.2.cmml" xref="A4.p1.2.m2.1.1.2">𝑡</ci><ci id="A4.p1.2.m2.1.1.3.cmml" xref="A4.p1.2.m2.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.2.m2.1c">t_{X}</annotation></semantics></math> using focus attention (<math id="A4.p1.3.m3.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k}" display="inline"><semantics id="A4.p1.3.m3.2a"><msub id="A4.p1.3.m3.2.3" xref="A4.p1.3.m3.2.3.cmml"><mi id="A4.p1.3.m3.2.3.2" xref="A4.p1.3.m3.2.3.2.cmml">Focus</mi><mrow id="A4.p1.3.m3.2.2.2.4" xref="A4.p1.3.m3.2.2.2.3.cmml"><mi id="A4.p1.3.m3.1.1.1.1" xref="A4.p1.3.m3.1.1.1.1.cmml">top</mi><mo id="A4.p1.3.m3.2.2.2.4.1" xref="A4.p1.3.m3.2.2.2.3.cmml">,</mo><mi id="A4.p1.3.m3.2.2.2.2" xref="A4.p1.3.m3.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A4.p1.3.m3.2b"><apply id="A4.p1.3.m3.2.3.cmml" xref="A4.p1.3.m3.2.3"><csymbol cd="ambiguous" id="A4.p1.3.m3.2.3.1.cmml" xref="A4.p1.3.m3.2.3">subscript</csymbol><ci id="A4.p1.3.m3.2.3.2.cmml" xref="A4.p1.3.m3.2.3.2">Focus</ci><list id="A4.p1.3.m3.2.2.2.3.cmml" xref="A4.p1.3.m3.2.2.2.4"><ci id="A4.p1.3.m3.1.1.1.1.cmml" xref="A4.p1.3.m3.1.1.1.1">top</ci><ci id="A4.p1.3.m3.2.2.2.2.cmml" xref="A4.p1.3.m3.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.3.m3.2c">\mathrm{Focus}_{\mathrm{top},k}</annotation></semantics></math>) on the <span id="A4.p1.6.1" class="ltx_text ltx_font_smallcaps">XSum</span> test set. In Figures <a href="#S5.F3" title="Figure 3 ‣ Generating Diverse and Faithful Summaries with Focus Sampling. ‣ 5 Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> and <a href="#S5.F4" title="Figure 4 ‣ Generating Diverse and Faithful Summaries with Focus Sampling. ‣ 5 Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we describe how <span id="A4.p1.6.2" class="ltx_text ltx_font_smallcaps">RobFame</span> consistently outperforms <span id="A4.p1.6.3" class="ltx_text ltx_font_smallcaps">PegFame</span> at lower values of <math id="A4.p1.4.m4.5" class="ltx_Math" alttext="k\in\{50,100,200,500,1000\}" display="inline"><semantics id="A4.p1.4.m4.5a"><mrow id="A4.p1.4.m4.5.6" xref="A4.p1.4.m4.5.6.cmml"><mi id="A4.p1.4.m4.5.6.2" xref="A4.p1.4.m4.5.6.2.cmml">k</mi><mo id="A4.p1.4.m4.5.6.1" xref="A4.p1.4.m4.5.6.1.cmml">∈</mo><mrow id="A4.p1.4.m4.5.6.3.2" xref="A4.p1.4.m4.5.6.3.1.cmml"><mo stretchy="false" id="A4.p1.4.m4.5.6.3.2.1" xref="A4.p1.4.m4.5.6.3.1.cmml">{</mo><mn id="A4.p1.4.m4.1.1" xref="A4.p1.4.m4.1.1.cmml">50</mn><mo id="A4.p1.4.m4.5.6.3.2.2" xref="A4.p1.4.m4.5.6.3.1.cmml">,</mo><mn id="A4.p1.4.m4.2.2" xref="A4.p1.4.m4.2.2.cmml">100</mn><mo id="A4.p1.4.m4.5.6.3.2.3" xref="A4.p1.4.m4.5.6.3.1.cmml">,</mo><mn id="A4.p1.4.m4.3.3" xref="A4.p1.4.m4.3.3.cmml">200</mn><mo id="A4.p1.4.m4.5.6.3.2.4" xref="A4.p1.4.m4.5.6.3.1.cmml">,</mo><mn id="A4.p1.4.m4.4.4" xref="A4.p1.4.m4.4.4.cmml">500</mn><mo id="A4.p1.4.m4.5.6.3.2.5" xref="A4.p1.4.m4.5.6.3.1.cmml">,</mo><mn id="A4.p1.4.m4.5.5" xref="A4.p1.4.m4.5.5.cmml">1000</mn><mo stretchy="false" id="A4.p1.4.m4.5.6.3.2.6" xref="A4.p1.4.m4.5.6.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p1.4.m4.5b"><apply id="A4.p1.4.m4.5.6.cmml" xref="A4.p1.4.m4.5.6"><in id="A4.p1.4.m4.5.6.1.cmml" xref="A4.p1.4.m4.5.6.1"></in><ci id="A4.p1.4.m4.5.6.2.cmml" xref="A4.p1.4.m4.5.6.2">𝑘</ci><set id="A4.p1.4.m4.5.6.3.1.cmml" xref="A4.p1.4.m4.5.6.3.2"><cn type="integer" id="A4.p1.4.m4.1.1.cmml" xref="A4.p1.4.m4.1.1">50</cn><cn type="integer" id="A4.p1.4.m4.2.2.cmml" xref="A4.p1.4.m4.2.2">100</cn><cn type="integer" id="A4.p1.4.m4.3.3.cmml" xref="A4.p1.4.m4.3.3">200</cn><cn type="integer" id="A4.p1.4.m4.4.4.cmml" xref="A4.p1.4.m4.4.4">500</cn><cn type="integer" id="A4.p1.4.m4.5.5.cmml" xref="A4.p1.4.m4.5.5">1000</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.4.m4.5c">k\in\{50,100,200,500,1000\}</annotation></semantics></math> due to their peaky and smooth <math id="A4.p1.5.m5.1" class="ltx_Math" alttext="t_{X}" display="inline"><semantics id="A4.p1.5.m5.1a"><msub id="A4.p1.5.m5.1.1" xref="A4.p1.5.m5.1.1.cmml"><mi id="A4.p1.5.m5.1.1.2" xref="A4.p1.5.m5.1.1.2.cmml">t</mi><mi id="A4.p1.5.m5.1.1.3" xref="A4.p1.5.m5.1.1.3.cmml">X</mi></msub><annotation-xml encoding="MathML-Content" id="A4.p1.5.m5.1b"><apply id="A4.p1.5.m5.1.1.cmml" xref="A4.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A4.p1.5.m5.1.1.1.cmml" xref="A4.p1.5.m5.1.1">subscript</csymbol><ci id="A4.p1.5.m5.1.1.2.cmml" xref="A4.p1.5.m5.1.1.2">𝑡</ci><ci id="A4.p1.5.m5.1.1.3.cmml" xref="A4.p1.5.m5.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.5.m5.1c">t_{X}</annotation></semantics></math>, respectively. While Figure <a href="#S5.F4" title="Figure 4 ‣ Generating Diverse and Faithful Summaries with Focus Sampling. ‣ 5 Results ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> only plots <span id="A4.p1.6.4" class="ltx_text ltx_font_smallcaps">rouge</span>-1 F1 scores, Table <a href="#A5.T7" title="Table 7 ‣ Appendix E Diverse Summarization with Div_{top,𝑘}, Div_nucleus and Focus_{sample,𝑘} ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> additionally reports <span id="A4.p1.6.5" class="ltx_text ltx_font_smallcaps">rouge</span>-2, <span id="A4.p1.6.6" class="ltx_text ltx_font_smallcaps">rouge</span>-L, entailment, Feqa, and BERTScores. Figure <a href="#A5.F6" title="Figure 6 ‣ Appendix E Diverse Summarization with Div_{top,𝑘}, Div_nucleus and Focus_{sample,𝑘} ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> presents predictions from models using <math id="A4.p1.6.m6.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k}" display="inline"><semantics id="A4.p1.6.m6.2a"><msub id="A4.p1.6.m6.2.3" xref="A4.p1.6.m6.2.3.cmml"><mi id="A4.p1.6.m6.2.3.2" xref="A4.p1.6.m6.2.3.2.cmml">Focus</mi><mrow id="A4.p1.6.m6.2.2.2.4" xref="A4.p1.6.m6.2.2.2.3.cmml"><mi id="A4.p1.6.m6.1.1.1.1" xref="A4.p1.6.m6.1.1.1.1.cmml">top</mi><mo id="A4.p1.6.m6.2.2.2.4.1" xref="A4.p1.6.m6.2.2.2.3.cmml">,</mo><mi id="A4.p1.6.m6.2.2.2.2" xref="A4.p1.6.m6.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A4.p1.6.m6.2b"><apply id="A4.p1.6.m6.2.3.cmml" xref="A4.p1.6.m6.2.3"><csymbol cd="ambiguous" id="A4.p1.6.m6.2.3.1.cmml" xref="A4.p1.6.m6.2.3">subscript</csymbol><ci id="A4.p1.6.m6.2.3.2.cmml" xref="A4.p1.6.m6.2.3.2">Focus</ci><list id="A4.p1.6.m6.2.2.2.3.cmml" xref="A4.p1.6.m6.2.2.2.4"><ci id="A4.p1.6.m6.1.1.1.1.cmml" xref="A4.p1.6.m6.1.1.1.1">top</ci><ci id="A4.p1.6.m6.2.2.2.2.cmml" xref="A4.p1.6.m6.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.6.m6.2c">\mathrm{Focus}_{\mathrm{top},k}</annotation></semantics></math> for the article presented in Figures <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and <a href="#A5.F5" title="Figure 5 ‣ Appendix E Diverse Summarization with Div_{top,𝑘}, Div_nucleus and Focus_{sample,𝑘} ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Diverse Summarization with <math id="A5.1.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="A5.1.m1.2b"><msub id="A5.1.m1.2.3" xref="A5.1.m1.2.3.cmml"><mi id="A5.1.m1.2.3.2" xref="A5.1.m1.2.3.2.cmml">Div</mi><mrow id="A5.1.m1.2.2.2.4" xref="A5.1.m1.2.2.2.3.cmml"><mi id="A5.1.m1.1.1.1.1" xref="A5.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.1.m1.2.2.2.4.1" xref="A5.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.1.m1.2.2.2.2" xref="A5.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.1.m1.2c"><apply id="A5.1.m1.2.3.cmml" xref="A5.1.m1.2.3"><csymbol cd="ambiguous" id="A5.1.m1.2.3.1.cmml" xref="A5.1.m1.2.3">subscript</csymbol><ci id="A5.1.m1.2.3.2.cmml" xref="A5.1.m1.2.3.2">Div</ci><list id="A5.1.m1.2.2.2.3.cmml" xref="A5.1.m1.2.2.2.4"><ci id="A5.1.m1.1.1.1.1.cmml" xref="A5.1.m1.1.1.1.1">top</ci><ci id="A5.1.m1.2.2.2.2.cmml" xref="A5.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.1.m1.2d">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>, <math id="A5.2.m2.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="A5.2.m2.1b"><msub id="A5.2.m2.1.1" xref="A5.2.m2.1.1.cmml"><mi id="A5.2.m2.1.1.2" xref="A5.2.m2.1.1.2.cmml">Div</mi><mi id="A5.2.m2.1.1.3" xref="A5.2.m2.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="A5.2.m2.1c"><apply id="A5.2.m2.1.1.cmml" xref="A5.2.m2.1.1"><csymbol cd="ambiguous" id="A5.2.m2.1.1.1.cmml" xref="A5.2.m2.1.1">subscript</csymbol><ci id="A5.2.m2.1.1.2.cmml" xref="A5.2.m2.1.1.2">Div</ci><ci id="A5.2.m2.1.1.3.cmml" xref="A5.2.m2.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.2.m2.1d">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math> and <math id="A5.3.m3.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="A5.3.m3.2b"><msub id="A5.3.m3.2.3" xref="A5.3.m3.2.3.cmml"><mi id="A5.3.m3.2.3.2" xref="A5.3.m3.2.3.2.cmml">Focus</mi><mrow id="A5.3.m3.2.2.2.4" xref="A5.3.m3.2.2.2.3.cmml"><mi id="A5.3.m3.1.1.1.1" xref="A5.3.m3.1.1.1.1.cmml">sample</mi><mo id="A5.3.m3.2.2.2.4.1" xref="A5.3.m3.2.2.2.3.cmml">,</mo><mi id="A5.3.m3.2.2.2.2" xref="A5.3.m3.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.3.m3.2c"><apply id="A5.3.m3.2.3.cmml" xref="A5.3.m3.2.3"><csymbol cd="ambiguous" id="A5.3.m3.2.3.1.cmml" xref="A5.3.m3.2.3">subscript</csymbol><ci id="A5.3.m3.2.3.2.cmml" xref="A5.3.m3.2.3.2">Focus</ci><list id="A5.3.m3.2.2.2.3.cmml" xref="A5.3.m3.2.2.2.4"><ci id="A5.3.m3.1.1.1.1.cmml" xref="A5.3.m3.1.1.1.1">sample</ci><ci id="A5.3.m3.2.2.2.2.cmml" xref="A5.3.m3.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.3.m3.2d">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>
</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.3" class="ltx_p">Figures <a href="#A5.F7" title="Figure 7 ‣ Appendix E Diverse Summarization with Div_{top,𝑘}, Div_nucleus and Focus_{sample,𝑘} ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, <a href="#A5.F8" title="Figure 8 ‣ Appendix E Diverse Summarization with Div_{top,𝑘}, Div_nucleus and Focus_{sample,𝑘} ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>, <a href="#A5.F9" title="Figure 9 ‣ Appendix E Diverse Summarization with Div_{top,𝑘}, Div_nucleus and Focus_{sample,𝑘} ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>, <a href="#A5.F10" title="Figure 10 ‣ Appendix E Diverse Summarization with Div_{top,𝑘}, Div_nucleus and Focus_{sample,𝑘} ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> and <a href="#A5.F11" title="Figure 11 ‣ Appendix E Diverse Summarization with Div_{top,𝑘}, Div_nucleus and Focus_{sample,𝑘} ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> show the diverse summaries generated using <math id="A5.p1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="A5.p1.1.m1.2a"><msub id="A5.p1.1.m1.2.3" xref="A5.p1.1.m1.2.3.cmml"><mi id="A5.p1.1.m1.2.3.2" xref="A5.p1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.p1.1.m1.2.2.2.4" xref="A5.p1.1.m1.2.2.2.3.cmml"><mi id="A5.p1.1.m1.1.1.1.1" xref="A5.p1.1.m1.1.1.1.1.cmml">sample</mi><mo id="A5.p1.1.m1.2.2.2.4.1" xref="A5.p1.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.p1.1.m1.2.2.2.2" xref="A5.p1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.p1.1.m1.2b"><apply id="A5.p1.1.m1.2.3.cmml" xref="A5.p1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.p1.1.m1.2.3.1.cmml" xref="A5.p1.1.m1.2.3">subscript</csymbol><ci id="A5.p1.1.m1.2.3.2.cmml" xref="A5.p1.1.m1.2.3.2">Focus</ci><list id="A5.p1.1.m1.2.2.2.3.cmml" xref="A5.p1.1.m1.2.2.2.4"><ci id="A5.p1.1.m1.1.1.1.1.cmml" xref="A5.p1.1.m1.1.1.1.1">sample</ci><ci id="A5.p1.1.m1.2.2.2.2.cmml" xref="A5.p1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, <math id="A5.p1.2.m2.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="A5.p1.2.m2.2a"><msub id="A5.p1.2.m2.2.3" xref="A5.p1.2.m2.2.3.cmml"><mi id="A5.p1.2.m2.2.3.2" xref="A5.p1.2.m2.2.3.2.cmml">Div</mi><mrow id="A5.p1.2.m2.2.2.2.4" xref="A5.p1.2.m2.2.2.2.3.cmml"><mi id="A5.p1.2.m2.1.1.1.1" xref="A5.p1.2.m2.1.1.1.1.cmml">top</mi><mo id="A5.p1.2.m2.2.2.2.4.1" xref="A5.p1.2.m2.2.2.2.3.cmml">,</mo><mi id="A5.p1.2.m2.2.2.2.2" xref="A5.p1.2.m2.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.p1.2.m2.2b"><apply id="A5.p1.2.m2.2.3.cmml" xref="A5.p1.2.m2.2.3"><csymbol cd="ambiguous" id="A5.p1.2.m2.2.3.1.cmml" xref="A5.p1.2.m2.2.3">subscript</csymbol><ci id="A5.p1.2.m2.2.3.2.cmml" xref="A5.p1.2.m2.2.3.2">Div</ci><list id="A5.p1.2.m2.2.2.2.3.cmml" xref="A5.p1.2.m2.2.2.2.4"><ci id="A5.p1.2.m2.1.1.1.1.cmml" xref="A5.p1.2.m2.1.1.1.1">top</ci><ci id="A5.p1.2.m2.2.2.2.2.cmml" xref="A5.p1.2.m2.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.2.m2.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math> and <math id="A5.p1.3.m3.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="A5.p1.3.m3.1a"><msub id="A5.p1.3.m3.1.1" xref="A5.p1.3.m3.1.1.cmml"><mi id="A5.p1.3.m3.1.1.2" xref="A5.p1.3.m3.1.1.2.cmml">Div</mi><mi id="A5.p1.3.m3.1.1.3" xref="A5.p1.3.m3.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="A5.p1.3.m3.1b"><apply id="A5.p1.3.m3.1.1.cmml" xref="A5.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A5.p1.3.m3.1.1.1.cmml" xref="A5.p1.3.m3.1.1">subscript</csymbol><ci id="A5.p1.3.m3.1.1.2.cmml" xref="A5.p1.3.m3.1.1.2">Div</ci><ci id="A5.p1.3.m3.1.1.3.cmml" xref="A5.p1.3.m3.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.3.m3.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math> sampling methods for the article shown in Figure <a href="#A5.F5" title="Figure 5 ‣ Appendix E Diverse Summarization with Div_{top,𝑘}, Div_nucleus and Focus_{sample,𝑘} ‣ Focus Attention: Promoting Faithfulness and Diversity in Summarization" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<figure id="A5.T7" class="ltx_table">
<table id="A5.T7.12" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A5.T7.12.13.1" class="ltx_tr">
<th id="A5.T7.12.13.1.1" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A5.T7.12.13.1.1.1" class="ltx_text">Metrics</span></th>
<th id="A5.T7.12.13.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="3"><span id="A5.T7.12.13.1.2.1" class="ltx_text ltx_font_smallcaps">rouge</span></th>
<th id="A5.T7.12.13.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="A5.T7.12.13.1.3.1" class="ltx_text">ent.</span></th>
<th id="A5.T7.12.13.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="A5.T7.12.13.1.4.1" class="ltx_text">Feqa</span></th>
<th id="A5.T7.12.13.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="A5.T7.12.13.1.5.1" class="ltx_text">BERTScore</span></th>
</tr>
<tr id="A5.T7.12.14.2" class="ltx_tr">
<th id="A5.T7.12.14.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column">R1</th>
<th id="A5.T7.12.14.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">R2</th>
<th id="A5.T7.12.14.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">RL</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A5.T7.12.15.1" class="ltx_tr">
<th id="A5.T7.12.15.1.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t"><span id="A5.T7.12.15.1.1.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span></th>
<td id="A5.T7.12.15.1.2" class="ltx_td ltx_align_center ltx_border_t">41.45</td>
<td id="A5.T7.12.15.1.3" class="ltx_td ltx_align_center ltx_border_t">18.79</td>
<td id="A5.T7.12.15.1.4" class="ltx_td ltx_align_center ltx_border_t">33.90</td>
<td id="A5.T7.12.15.1.5" class="ltx_td ltx_align_center ltx_border_t">39.1</td>
<td id="A5.T7.12.15.1.6" class="ltx_td ltx_align_center ltx_border_t">19.8</td>
<td id="A5.T7.12.15.1.7" class="ltx_td ltx_align_center ltx_border_t">80.6</td>
</tr>
<tr id="A5.T7.12.16.2" class="ltx_tr">
<th id="A5.T7.12.16.2.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r"><span id="A5.T7.12.16.2.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span></th>
<td id="A5.T7.12.16.2.2" class="ltx_td ltx_align_center"><span id="A5.T7.12.16.2.2.1" class="ltx_text ltx_font_bold">42.15</span></td>
<td id="A5.T7.12.16.2.3" class="ltx_td ltx_align_center"><span id="A5.T7.12.16.2.3.1" class="ltx_text ltx_font_bold">19.68</span></td>
<td id="A5.T7.12.16.2.4" class="ltx_td ltx_align_center"><span id="A5.T7.12.16.2.4.1" class="ltx_text ltx_font_bold">34.81</span></td>
<td id="A5.T7.12.16.2.5" class="ltx_td ltx_align_center"><span id="A5.T7.12.16.2.5.1" class="ltx_text ltx_font_bold">41.3</span></td>
<td id="A5.T7.12.16.2.6" class="ltx_td ltx_align_center"><span id="A5.T7.12.16.2.6.1" class="ltx_text ltx_font_bold">21.2</span></td>
<td id="A5.T7.12.16.2.7" class="ltx_td ltx_align_center"><span id="A5.T7.12.16.2.7.1" class="ltx_text ltx_font_bold">80.8</span></td>
</tr>
<tr id="A5.T7.1.1" class="ltx_tr">
<th id="A5.T7.1.1.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A5.T7.1.1.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (<math id="A5.T7.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=50}" display="inline"><semantics id="A5.T7.1.1.1.m1.2a"><msub id="A5.T7.1.1.1.m1.2.3" xref="A5.T7.1.1.1.m1.2.3.cmml"><mi id="A5.T7.1.1.1.m1.2.3.2" xref="A5.T7.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.1.1.1.m1.2.2.2" xref="A5.T7.1.1.1.m1.2.2.2.cmml"><mrow id="A5.T7.1.1.1.m1.2.2.2.4.2" xref="A5.T7.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.1.1.1.m1.1.1.1.1" xref="A5.T7.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.1.1.1.m1.2.2.2.4.2.1" xref="A5.T7.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.1.1.1.m1.2.2.2.2" xref="A5.T7.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.1.1.1.m1.2.2.2.3" xref="A5.T7.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.1.1.1.m1.2.2.2.5" xref="A5.T7.1.1.1.m1.2.2.2.5.cmml">50</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.1.1.1.m1.2b"><apply id="A5.T7.1.1.1.m1.2.3.cmml" xref="A5.T7.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.1.1.1.m1.2.3.1.cmml" xref="A5.T7.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.T7.1.1.1.m1.2.3.2.cmml" xref="A5.T7.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.T7.1.1.1.m1.2.2.2.cmml" xref="A5.T7.1.1.1.m1.2.2.2"><eq id="A5.T7.1.1.1.m1.2.2.2.3.cmml" xref="A5.T7.1.1.1.m1.2.2.2.3"></eq><list id="A5.T7.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.T7.1.1.1.m1.2.2.2.4.2"><ci id="A5.T7.1.1.1.m1.1.1.1.1.cmml" xref="A5.T7.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.T7.1.1.1.m1.2.2.2.2.cmml" xref="A5.T7.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.1.1.1.m1.2.2.2.5.cmml" xref="A5.T7.1.1.1.m1.2.2.2.5">50</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=50}</annotation></semantics></math>)</th>
<td id="A5.T7.1.1.2" class="ltx_td ltx_align_center">30.90</td>
<td id="A5.T7.1.1.3" class="ltx_td ltx_align_center">10.60</td>
<td id="A5.T7.1.1.4" class="ltx_td ltx_align_center">24.85</td>
<td id="A5.T7.1.1.5" class="ltx_td ltx_align_center">27.1</td>
<td id="A5.T7.1.1.6" class="ltx_td ltx_align_center">10.6</td>
<td id="A5.T7.1.1.7" class="ltx_td ltx_align_center">74.2</td>
</tr>
<tr id="A5.T7.2.2" class="ltx_tr">
<th id="A5.T7.2.2.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A5.T7.2.2.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (<math id="A5.T7.2.2.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=100}" display="inline"><semantics id="A5.T7.2.2.1.m1.2a"><msub id="A5.T7.2.2.1.m1.2.3" xref="A5.T7.2.2.1.m1.2.3.cmml"><mi id="A5.T7.2.2.1.m1.2.3.2" xref="A5.T7.2.2.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.2.2.1.m1.2.2.2" xref="A5.T7.2.2.1.m1.2.2.2.cmml"><mrow id="A5.T7.2.2.1.m1.2.2.2.4.2" xref="A5.T7.2.2.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.2.2.1.m1.1.1.1.1" xref="A5.T7.2.2.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.2.2.1.m1.2.2.2.4.2.1" xref="A5.T7.2.2.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.2.2.1.m1.2.2.2.2" xref="A5.T7.2.2.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.2.2.1.m1.2.2.2.3" xref="A5.T7.2.2.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.2.2.1.m1.2.2.2.5" xref="A5.T7.2.2.1.m1.2.2.2.5.cmml">100</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.2.2.1.m1.2b"><apply id="A5.T7.2.2.1.m1.2.3.cmml" xref="A5.T7.2.2.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.2.2.1.m1.2.3.1.cmml" xref="A5.T7.2.2.1.m1.2.3">subscript</csymbol><ci id="A5.T7.2.2.1.m1.2.3.2.cmml" xref="A5.T7.2.2.1.m1.2.3.2">Focus</ci><apply id="A5.T7.2.2.1.m1.2.2.2.cmml" xref="A5.T7.2.2.1.m1.2.2.2"><eq id="A5.T7.2.2.1.m1.2.2.2.3.cmml" xref="A5.T7.2.2.1.m1.2.2.2.3"></eq><list id="A5.T7.2.2.1.m1.2.2.2.4.1.cmml" xref="A5.T7.2.2.1.m1.2.2.2.4.2"><ci id="A5.T7.2.2.1.m1.1.1.1.1.cmml" xref="A5.T7.2.2.1.m1.1.1.1.1">top</ci><ci id="A5.T7.2.2.1.m1.2.2.2.2.cmml" xref="A5.T7.2.2.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.2.2.1.m1.2.2.2.5.cmml" xref="A5.T7.2.2.1.m1.2.2.2.5">100</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.2.2.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=100}</annotation></semantics></math>)</th>
<td id="A5.T7.2.2.2" class="ltx_td ltx_align_center">33.62</td>
<td id="A5.T7.2.2.3" class="ltx_td ltx_align_center">12.39</td>
<td id="A5.T7.2.2.4" class="ltx_td ltx_align_center">27.14</td>
<td id="A5.T7.2.2.5" class="ltx_td ltx_align_center">30.3</td>
<td id="A5.T7.2.2.6" class="ltx_td ltx_align_center">12.4</td>
<td id="A5.T7.2.2.7" class="ltx_td ltx_align_center">74.2</td>
</tr>
<tr id="A5.T7.3.3" class="ltx_tr">
<th id="A5.T7.3.3.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A5.T7.3.3.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (<math id="A5.T7.3.3.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=200}" display="inline"><semantics id="A5.T7.3.3.1.m1.2a"><msub id="A5.T7.3.3.1.m1.2.3" xref="A5.T7.3.3.1.m1.2.3.cmml"><mi id="A5.T7.3.3.1.m1.2.3.2" xref="A5.T7.3.3.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.3.3.1.m1.2.2.2" xref="A5.T7.3.3.1.m1.2.2.2.cmml"><mrow id="A5.T7.3.3.1.m1.2.2.2.4.2" xref="A5.T7.3.3.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.3.3.1.m1.1.1.1.1" xref="A5.T7.3.3.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.3.3.1.m1.2.2.2.4.2.1" xref="A5.T7.3.3.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.3.3.1.m1.2.2.2.2" xref="A5.T7.3.3.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.3.3.1.m1.2.2.2.3" xref="A5.T7.3.3.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.3.3.1.m1.2.2.2.5" xref="A5.T7.3.3.1.m1.2.2.2.5.cmml">200</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.3.3.1.m1.2b"><apply id="A5.T7.3.3.1.m1.2.3.cmml" xref="A5.T7.3.3.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.3.3.1.m1.2.3.1.cmml" xref="A5.T7.3.3.1.m1.2.3">subscript</csymbol><ci id="A5.T7.3.3.1.m1.2.3.2.cmml" xref="A5.T7.3.3.1.m1.2.3.2">Focus</ci><apply id="A5.T7.3.3.1.m1.2.2.2.cmml" xref="A5.T7.3.3.1.m1.2.2.2"><eq id="A5.T7.3.3.1.m1.2.2.2.3.cmml" xref="A5.T7.3.3.1.m1.2.2.2.3"></eq><list id="A5.T7.3.3.1.m1.2.2.2.4.1.cmml" xref="A5.T7.3.3.1.m1.2.2.2.4.2"><ci id="A5.T7.3.3.1.m1.1.1.1.1.cmml" xref="A5.T7.3.3.1.m1.1.1.1.1">top</ci><ci id="A5.T7.3.3.1.m1.2.2.2.2.cmml" xref="A5.T7.3.3.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.3.3.1.m1.2.2.2.5.cmml" xref="A5.T7.3.3.1.m1.2.2.2.5">200</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.3.3.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=200}</annotation></semantics></math>)</th>
<td id="A5.T7.3.3.2" class="ltx_td ltx_align_center">35.99</td>
<td id="A5.T7.3.3.3" class="ltx_td ltx_align_center">14.12</td>
<td id="A5.T7.3.3.4" class="ltx_td ltx_align_center">29.23</td>
<td id="A5.T7.3.3.5" class="ltx_td ltx_align_center">32.4</td>
<td id="A5.T7.3.3.6" class="ltx_td ltx_align_center">13.9</td>
<td id="A5.T7.3.3.7" class="ltx_td ltx_align_center">77.3</td>
</tr>
<tr id="A5.T7.4.4" class="ltx_tr">
<th id="A5.T7.4.4.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A5.T7.4.4.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (<math id="A5.T7.4.4.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=500}" display="inline"><semantics id="A5.T7.4.4.1.m1.2a"><msub id="A5.T7.4.4.1.m1.2.3" xref="A5.T7.4.4.1.m1.2.3.cmml"><mi id="A5.T7.4.4.1.m1.2.3.2" xref="A5.T7.4.4.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.4.4.1.m1.2.2.2" xref="A5.T7.4.4.1.m1.2.2.2.cmml"><mrow id="A5.T7.4.4.1.m1.2.2.2.4.2" xref="A5.T7.4.4.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.4.4.1.m1.1.1.1.1" xref="A5.T7.4.4.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.4.4.1.m1.2.2.2.4.2.1" xref="A5.T7.4.4.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.4.4.1.m1.2.2.2.2" xref="A5.T7.4.4.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.4.4.1.m1.2.2.2.3" xref="A5.T7.4.4.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.4.4.1.m1.2.2.2.5" xref="A5.T7.4.4.1.m1.2.2.2.5.cmml">500</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.4.4.1.m1.2b"><apply id="A5.T7.4.4.1.m1.2.3.cmml" xref="A5.T7.4.4.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.4.4.1.m1.2.3.1.cmml" xref="A5.T7.4.4.1.m1.2.3">subscript</csymbol><ci id="A5.T7.4.4.1.m1.2.3.2.cmml" xref="A5.T7.4.4.1.m1.2.3.2">Focus</ci><apply id="A5.T7.4.4.1.m1.2.2.2.cmml" xref="A5.T7.4.4.1.m1.2.2.2"><eq id="A5.T7.4.4.1.m1.2.2.2.3.cmml" xref="A5.T7.4.4.1.m1.2.2.2.3"></eq><list id="A5.T7.4.4.1.m1.2.2.2.4.1.cmml" xref="A5.T7.4.4.1.m1.2.2.2.4.2"><ci id="A5.T7.4.4.1.m1.1.1.1.1.cmml" xref="A5.T7.4.4.1.m1.1.1.1.1">top</ci><ci id="A5.T7.4.4.1.m1.2.2.2.2.cmml" xref="A5.T7.4.4.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.4.4.1.m1.2.2.2.5.cmml" xref="A5.T7.4.4.1.m1.2.2.2.5">500</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.4.4.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=500}</annotation></semantics></math>)</th>
<td id="A5.T7.4.4.2" class="ltx_td ltx_align_center">38.29</td>
<td id="A5.T7.4.4.3" class="ltx_td ltx_align_center">16.04</td>
<td id="A5.T7.4.4.4" class="ltx_td ltx_align_center">31.30</td>
<td id="A5.T7.4.4.5" class="ltx_td ltx_align_center">35.8</td>
<td id="A5.T7.4.4.6" class="ltx_td ltx_align_center">15.9</td>
<td id="A5.T7.4.4.7" class="ltx_td ltx_align_center">78.6</td>
</tr>
<tr id="A5.T7.5.5" class="ltx_tr">
<th id="A5.T7.5.5.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A5.T7.5.5.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (<math id="A5.T7.5.5.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=1000}" display="inline"><semantics id="A5.T7.5.5.1.m1.2a"><msub id="A5.T7.5.5.1.m1.2.3" xref="A5.T7.5.5.1.m1.2.3.cmml"><mi id="A5.T7.5.5.1.m1.2.3.2" xref="A5.T7.5.5.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.5.5.1.m1.2.2.2" xref="A5.T7.5.5.1.m1.2.2.2.cmml"><mrow id="A5.T7.5.5.1.m1.2.2.2.4.2" xref="A5.T7.5.5.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.5.5.1.m1.1.1.1.1" xref="A5.T7.5.5.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.5.5.1.m1.2.2.2.4.2.1" xref="A5.T7.5.5.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.5.5.1.m1.2.2.2.2" xref="A5.T7.5.5.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.5.5.1.m1.2.2.2.3" xref="A5.T7.5.5.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.5.5.1.m1.2.2.2.5" xref="A5.T7.5.5.1.m1.2.2.2.5.cmml">1000</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.5.5.1.m1.2b"><apply id="A5.T7.5.5.1.m1.2.3.cmml" xref="A5.T7.5.5.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.5.5.1.m1.2.3.1.cmml" xref="A5.T7.5.5.1.m1.2.3">subscript</csymbol><ci id="A5.T7.5.5.1.m1.2.3.2.cmml" xref="A5.T7.5.5.1.m1.2.3.2">Focus</ci><apply id="A5.T7.5.5.1.m1.2.2.2.cmml" xref="A5.T7.5.5.1.m1.2.2.2"><eq id="A5.T7.5.5.1.m1.2.2.2.3.cmml" xref="A5.T7.5.5.1.m1.2.2.2.3"></eq><list id="A5.T7.5.5.1.m1.2.2.2.4.1.cmml" xref="A5.T7.5.5.1.m1.2.2.2.4.2"><ci id="A5.T7.5.5.1.m1.1.1.1.1.cmml" xref="A5.T7.5.5.1.m1.1.1.1.1">top</ci><ci id="A5.T7.5.5.1.m1.2.2.2.2.cmml" xref="A5.T7.5.5.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.5.5.1.m1.2.2.2.5.cmml" xref="A5.T7.5.5.1.m1.2.2.2.5">1000</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.5.5.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=1000}</annotation></semantics></math>)</th>
<td id="A5.T7.5.5.2" class="ltx_td ltx_align_center">39.58</td>
<td id="A5.T7.5.5.3" class="ltx_td ltx_align_center">17.18</td>
<td id="A5.T7.5.5.4" class="ltx_td ltx_align_center">32.49</td>
<td id="A5.T7.5.5.5" class="ltx_td ltx_align_center">37.3</td>
<td id="A5.T7.5.5.6" class="ltx_td ltx_align_center">17.3</td>
<td id="A5.T7.5.5.7" class="ltx_td ltx_align_center">79.3</td>
</tr>
<tr id="A5.T7.6.6" class="ltx_tr">
<th id="A5.T7.6.6.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A5.T7.6.6.1.1" class="ltx_text ltx_font_smallcaps">RobFame</span> (<math id="A5.T7.6.6.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=10000}" display="inline"><semantics id="A5.T7.6.6.1.m1.2a"><msub id="A5.T7.6.6.1.m1.2.3" xref="A5.T7.6.6.1.m1.2.3.cmml"><mi id="A5.T7.6.6.1.m1.2.3.2" xref="A5.T7.6.6.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.6.6.1.m1.2.2.2" xref="A5.T7.6.6.1.m1.2.2.2.cmml"><mrow id="A5.T7.6.6.1.m1.2.2.2.4.2" xref="A5.T7.6.6.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.6.6.1.m1.1.1.1.1" xref="A5.T7.6.6.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.6.6.1.m1.2.2.2.4.2.1" xref="A5.T7.6.6.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.6.6.1.m1.2.2.2.2" xref="A5.T7.6.6.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.6.6.1.m1.2.2.2.3" xref="A5.T7.6.6.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.6.6.1.m1.2.2.2.5" xref="A5.T7.6.6.1.m1.2.2.2.5.cmml">10000</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.6.6.1.m1.2b"><apply id="A5.T7.6.6.1.m1.2.3.cmml" xref="A5.T7.6.6.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.6.6.1.m1.2.3.1.cmml" xref="A5.T7.6.6.1.m1.2.3">subscript</csymbol><ci id="A5.T7.6.6.1.m1.2.3.2.cmml" xref="A5.T7.6.6.1.m1.2.3.2">Focus</ci><apply id="A5.T7.6.6.1.m1.2.2.2.cmml" xref="A5.T7.6.6.1.m1.2.2.2"><eq id="A5.T7.6.6.1.m1.2.2.2.3.cmml" xref="A5.T7.6.6.1.m1.2.2.2.3"></eq><list id="A5.T7.6.6.1.m1.2.2.2.4.1.cmml" xref="A5.T7.6.6.1.m1.2.2.2.4.2"><ci id="A5.T7.6.6.1.m1.1.1.1.1.cmml" xref="A5.T7.6.6.1.m1.1.1.1.1">top</ci><ci id="A5.T7.6.6.1.m1.2.2.2.2.cmml" xref="A5.T7.6.6.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.6.6.1.m1.2.2.2.5.cmml" xref="A5.T7.6.6.1.m1.2.2.2.5">10000</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.6.6.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=10000}</annotation></semantics></math>)</th>
<td id="A5.T7.6.6.2" class="ltx_td ltx_align_center">41.58</td>
<td id="A5.T7.6.6.3" class="ltx_td ltx_align_center">19.13</td>
<td id="A5.T7.6.6.4" class="ltx_td ltx_align_center">34.30</td>
<td id="A5.T7.6.6.5" class="ltx_td ltx_align_center">40.7</td>
<td id="A5.T7.6.6.6" class="ltx_td ltx_align_center">20.2</td>
<td id="A5.T7.6.6.7" class="ltx_td ltx_align_center">80.5</td>
</tr>
<tr id="A5.T7.12.17.3" class="ltx_tr">
<th id="A5.T7.12.17.3.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r ltx_border_t"><span id="A5.T7.12.17.3.1.1" class="ltx_text ltx_font_smallcaps">Pegasus</span></th>
<td id="A5.T7.12.17.3.2" class="ltx_td ltx_align_center ltx_border_t">44.85</td>
<td id="A5.T7.12.17.3.3" class="ltx_td ltx_align_center ltx_border_t">22.26</td>
<td id="A5.T7.12.17.3.4" class="ltx_td ltx_align_center ltx_border_t">37.03</td>
<td id="A5.T7.12.17.3.5" class="ltx_td ltx_align_center ltx_border_t">43.6</td>
<td id="A5.T7.12.17.3.6" class="ltx_td ltx_align_center ltx_border_t">24.5</td>
<td id="A5.T7.12.17.3.7" class="ltx_td ltx_align_center ltx_border_t">81.7</td>
</tr>
<tr id="A5.T7.12.18.4" class="ltx_tr">
<th id="A5.T7.12.18.4.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r"><span id="A5.T7.12.18.4.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span></th>
<td id="A5.T7.12.18.4.2" class="ltx_td ltx_align_center"><span id="A5.T7.12.18.4.2.1" class="ltx_text ltx_font_bold">45.31</span></td>
<td id="A5.T7.12.18.4.3" class="ltx_td ltx_align_center"><span id="A5.T7.12.18.4.3.1" class="ltx_text ltx_font_bold">22.75</span></td>
<td id="A5.T7.12.18.4.4" class="ltx_td ltx_align_center"><span id="A5.T7.12.18.4.4.1" class="ltx_text ltx_font_bold">37.46</span></td>
<td id="A5.T7.12.18.4.5" class="ltx_td ltx_align_center"><span id="A5.T7.12.18.4.5.1" class="ltx_text ltx_font_bold">44.8</span></td>
<td id="A5.T7.12.18.4.6" class="ltx_td ltx_align_center"><span id="A5.T7.12.18.4.6.1" class="ltx_text ltx_font_bold">24.8</span></td>
<td id="A5.T7.12.18.4.7" class="ltx_td ltx_align_center"><span id="A5.T7.12.18.4.7.1" class="ltx_text ltx_font_bold">81.9</span></td>
</tr>
<tr id="A5.T7.7.7" class="ltx_tr">
<th id="A5.T7.7.7.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A5.T7.7.7.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (<math id="A5.T7.7.7.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=50}" display="inline"><semantics id="A5.T7.7.7.1.m1.2a"><msub id="A5.T7.7.7.1.m1.2.3" xref="A5.T7.7.7.1.m1.2.3.cmml"><mi id="A5.T7.7.7.1.m1.2.3.2" xref="A5.T7.7.7.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.7.7.1.m1.2.2.2" xref="A5.T7.7.7.1.m1.2.2.2.cmml"><mrow id="A5.T7.7.7.1.m1.2.2.2.4.2" xref="A5.T7.7.7.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.7.7.1.m1.1.1.1.1" xref="A5.T7.7.7.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.7.7.1.m1.2.2.2.4.2.1" xref="A5.T7.7.7.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.7.7.1.m1.2.2.2.2" xref="A5.T7.7.7.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.7.7.1.m1.2.2.2.3" xref="A5.T7.7.7.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.7.7.1.m1.2.2.2.5" xref="A5.T7.7.7.1.m1.2.2.2.5.cmml">50</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.7.7.1.m1.2b"><apply id="A5.T7.7.7.1.m1.2.3.cmml" xref="A5.T7.7.7.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.7.7.1.m1.2.3.1.cmml" xref="A5.T7.7.7.1.m1.2.3">subscript</csymbol><ci id="A5.T7.7.7.1.m1.2.3.2.cmml" xref="A5.T7.7.7.1.m1.2.3.2">Focus</ci><apply id="A5.T7.7.7.1.m1.2.2.2.cmml" xref="A5.T7.7.7.1.m1.2.2.2"><eq id="A5.T7.7.7.1.m1.2.2.2.3.cmml" xref="A5.T7.7.7.1.m1.2.2.2.3"></eq><list id="A5.T7.7.7.1.m1.2.2.2.4.1.cmml" xref="A5.T7.7.7.1.m1.2.2.2.4.2"><ci id="A5.T7.7.7.1.m1.1.1.1.1.cmml" xref="A5.T7.7.7.1.m1.1.1.1.1">top</ci><ci id="A5.T7.7.7.1.m1.2.2.2.2.cmml" xref="A5.T7.7.7.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.7.7.1.m1.2.2.2.5.cmml" xref="A5.T7.7.7.1.m1.2.2.2.5">50</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.7.7.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=50}</annotation></semantics></math>)</th>
<td id="A5.T7.7.7.2" class="ltx_td ltx_align_center">24.30</td>
<td id="A5.T7.7.7.3" class="ltx_td ltx_align_center">7.52</td>
<td id="A5.T7.7.7.4" class="ltx_td ltx_align_center">19.32</td>
<td id="A5.T7.7.7.5" class="ltx_td ltx_align_center">20.8</td>
<td id="A5.T7.7.7.6" class="ltx_td ltx_align_center">8.0</td>
<td id="A5.T7.7.7.7" class="ltx_td ltx_align_center">68.8</td>
</tr>
<tr id="A5.T7.8.8" class="ltx_tr">
<th id="A5.T7.8.8.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A5.T7.8.8.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (<math id="A5.T7.8.8.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=100}" display="inline"><semantics id="A5.T7.8.8.1.m1.2a"><msub id="A5.T7.8.8.1.m1.2.3" xref="A5.T7.8.8.1.m1.2.3.cmml"><mi id="A5.T7.8.8.1.m1.2.3.2" xref="A5.T7.8.8.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.8.8.1.m1.2.2.2" xref="A5.T7.8.8.1.m1.2.2.2.cmml"><mrow id="A5.T7.8.8.1.m1.2.2.2.4.2" xref="A5.T7.8.8.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.8.8.1.m1.1.1.1.1" xref="A5.T7.8.8.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.8.8.1.m1.2.2.2.4.2.1" xref="A5.T7.8.8.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.8.8.1.m1.2.2.2.2" xref="A5.T7.8.8.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.8.8.1.m1.2.2.2.3" xref="A5.T7.8.8.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.8.8.1.m1.2.2.2.5" xref="A5.T7.8.8.1.m1.2.2.2.5.cmml">100</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.8.8.1.m1.2b"><apply id="A5.T7.8.8.1.m1.2.3.cmml" xref="A5.T7.8.8.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.8.8.1.m1.2.3.1.cmml" xref="A5.T7.8.8.1.m1.2.3">subscript</csymbol><ci id="A5.T7.8.8.1.m1.2.3.2.cmml" xref="A5.T7.8.8.1.m1.2.3.2">Focus</ci><apply id="A5.T7.8.8.1.m1.2.2.2.cmml" xref="A5.T7.8.8.1.m1.2.2.2"><eq id="A5.T7.8.8.1.m1.2.2.2.3.cmml" xref="A5.T7.8.8.1.m1.2.2.2.3"></eq><list id="A5.T7.8.8.1.m1.2.2.2.4.1.cmml" xref="A5.T7.8.8.1.m1.2.2.2.4.2"><ci id="A5.T7.8.8.1.m1.1.1.1.1.cmml" xref="A5.T7.8.8.1.m1.1.1.1.1">top</ci><ci id="A5.T7.8.8.1.m1.2.2.2.2.cmml" xref="A5.T7.8.8.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.8.8.1.m1.2.2.2.5.cmml" xref="A5.T7.8.8.1.m1.2.2.2.5">100</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.8.8.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=100}</annotation></semantics></math>)</th>
<td id="A5.T7.8.8.2" class="ltx_td ltx_align_center">27.77</td>
<td id="A5.T7.8.8.3" class="ltx_td ltx_align_center">9.26</td>
<td id="A5.T7.8.8.4" class="ltx_td ltx_align_center">22.09</td>
<td id="A5.T7.8.8.5" class="ltx_td ltx_align_center">24.1</td>
<td id="A5.T7.8.8.6" class="ltx_td ltx_align_center">9.3</td>
<td id="A5.T7.8.8.7" class="ltx_td ltx_align_center">71.3</td>
</tr>
<tr id="A5.T7.9.9" class="ltx_tr">
<th id="A5.T7.9.9.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A5.T7.9.9.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (<math id="A5.T7.9.9.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=200}" display="inline"><semantics id="A5.T7.9.9.1.m1.2a"><msub id="A5.T7.9.9.1.m1.2.3" xref="A5.T7.9.9.1.m1.2.3.cmml"><mi id="A5.T7.9.9.1.m1.2.3.2" xref="A5.T7.9.9.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.9.9.1.m1.2.2.2" xref="A5.T7.9.9.1.m1.2.2.2.cmml"><mrow id="A5.T7.9.9.1.m1.2.2.2.4.2" xref="A5.T7.9.9.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.9.9.1.m1.1.1.1.1" xref="A5.T7.9.9.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.9.9.1.m1.2.2.2.4.2.1" xref="A5.T7.9.9.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.9.9.1.m1.2.2.2.2" xref="A5.T7.9.9.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.9.9.1.m1.2.2.2.3" xref="A5.T7.9.9.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.9.9.1.m1.2.2.2.5" xref="A5.T7.9.9.1.m1.2.2.2.5.cmml">200</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.9.9.1.m1.2b"><apply id="A5.T7.9.9.1.m1.2.3.cmml" xref="A5.T7.9.9.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.9.9.1.m1.2.3.1.cmml" xref="A5.T7.9.9.1.m1.2.3">subscript</csymbol><ci id="A5.T7.9.9.1.m1.2.3.2.cmml" xref="A5.T7.9.9.1.m1.2.3.2">Focus</ci><apply id="A5.T7.9.9.1.m1.2.2.2.cmml" xref="A5.T7.9.9.1.m1.2.2.2"><eq id="A5.T7.9.9.1.m1.2.2.2.3.cmml" xref="A5.T7.9.9.1.m1.2.2.2.3"></eq><list id="A5.T7.9.9.1.m1.2.2.2.4.1.cmml" xref="A5.T7.9.9.1.m1.2.2.2.4.2"><ci id="A5.T7.9.9.1.m1.1.1.1.1.cmml" xref="A5.T7.9.9.1.m1.1.1.1.1">top</ci><ci id="A5.T7.9.9.1.m1.2.2.2.2.cmml" xref="A5.T7.9.9.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.9.9.1.m1.2.2.2.5.cmml" xref="A5.T7.9.9.1.m1.2.2.2.5">200</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.9.9.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=200}</annotation></semantics></math>)</th>
<td id="A5.T7.9.9.2" class="ltx_td ltx_align_center">31.05</td>
<td id="A5.T7.9.9.3" class="ltx_td ltx_align_center">11.14</td>
<td id="A5.T7.9.9.4" class="ltx_td ltx_align_center">24.82</td>
<td id="A5.T7.9.9.5" class="ltx_td ltx_align_center">27.0</td>
<td id="A5.T7.9.9.6" class="ltx_td ltx_align_center">10.8</td>
<td id="A5.T7.9.9.7" class="ltx_td ltx_align_center">73.6</td>
</tr>
<tr id="A5.T7.10.10" class="ltx_tr">
<th id="A5.T7.10.10.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A5.T7.10.10.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (<math id="A5.T7.10.10.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=500}" display="inline"><semantics id="A5.T7.10.10.1.m1.2a"><msub id="A5.T7.10.10.1.m1.2.3" xref="A5.T7.10.10.1.m1.2.3.cmml"><mi id="A5.T7.10.10.1.m1.2.3.2" xref="A5.T7.10.10.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.10.10.1.m1.2.2.2" xref="A5.T7.10.10.1.m1.2.2.2.cmml"><mrow id="A5.T7.10.10.1.m1.2.2.2.4.2" xref="A5.T7.10.10.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.10.10.1.m1.1.1.1.1" xref="A5.T7.10.10.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.10.10.1.m1.2.2.2.4.2.1" xref="A5.T7.10.10.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.10.10.1.m1.2.2.2.2" xref="A5.T7.10.10.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.10.10.1.m1.2.2.2.3" xref="A5.T7.10.10.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.10.10.1.m1.2.2.2.5" xref="A5.T7.10.10.1.m1.2.2.2.5.cmml">500</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.10.10.1.m1.2b"><apply id="A5.T7.10.10.1.m1.2.3.cmml" xref="A5.T7.10.10.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.10.10.1.m1.2.3.1.cmml" xref="A5.T7.10.10.1.m1.2.3">subscript</csymbol><ci id="A5.T7.10.10.1.m1.2.3.2.cmml" xref="A5.T7.10.10.1.m1.2.3.2">Focus</ci><apply id="A5.T7.10.10.1.m1.2.2.2.cmml" xref="A5.T7.10.10.1.m1.2.2.2"><eq id="A5.T7.10.10.1.m1.2.2.2.3.cmml" xref="A5.T7.10.10.1.m1.2.2.2.3"></eq><list id="A5.T7.10.10.1.m1.2.2.2.4.1.cmml" xref="A5.T7.10.10.1.m1.2.2.2.4.2"><ci id="A5.T7.10.10.1.m1.1.1.1.1.cmml" xref="A5.T7.10.10.1.m1.1.1.1.1">top</ci><ci id="A5.T7.10.10.1.m1.2.2.2.2.cmml" xref="A5.T7.10.10.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.10.10.1.m1.2.2.2.5.cmml" xref="A5.T7.10.10.1.m1.2.2.2.5">500</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.10.10.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=500}</annotation></semantics></math>)</th>
<td id="A5.T7.10.10.2" class="ltx_td ltx_align_center">34.99</td>
<td id="A5.T7.10.10.3" class="ltx_td ltx_align_center">13.65</td>
<td id="A5.T7.10.10.4" class="ltx_td ltx_align_center">28.19</td>
<td id="A5.T7.10.10.5" class="ltx_td ltx_align_center">31.0</td>
<td id="A5.T7.10.10.6" class="ltx_td ltx_align_center">13.0</td>
<td id="A5.T7.10.10.7" class="ltx_td ltx_align_center">76.2</td>
</tr>
<tr id="A5.T7.11.11" class="ltx_tr">
<th id="A5.T7.11.11.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_r">
<span id="A5.T7.11.11.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (<math id="A5.T7.11.11.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=1000}" display="inline"><semantics id="A5.T7.11.11.1.m1.2a"><msub id="A5.T7.11.11.1.m1.2.3" xref="A5.T7.11.11.1.m1.2.3.cmml"><mi id="A5.T7.11.11.1.m1.2.3.2" xref="A5.T7.11.11.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.11.11.1.m1.2.2.2" xref="A5.T7.11.11.1.m1.2.2.2.cmml"><mrow id="A5.T7.11.11.1.m1.2.2.2.4.2" xref="A5.T7.11.11.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.11.11.1.m1.1.1.1.1" xref="A5.T7.11.11.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.11.11.1.m1.2.2.2.4.2.1" xref="A5.T7.11.11.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.11.11.1.m1.2.2.2.2" xref="A5.T7.11.11.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.11.11.1.m1.2.2.2.3" xref="A5.T7.11.11.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.11.11.1.m1.2.2.2.5" xref="A5.T7.11.11.1.m1.2.2.2.5.cmml">1000</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.11.11.1.m1.2b"><apply id="A5.T7.11.11.1.m1.2.3.cmml" xref="A5.T7.11.11.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.11.11.1.m1.2.3.1.cmml" xref="A5.T7.11.11.1.m1.2.3">subscript</csymbol><ci id="A5.T7.11.11.1.m1.2.3.2.cmml" xref="A5.T7.11.11.1.m1.2.3.2">Focus</ci><apply id="A5.T7.11.11.1.m1.2.2.2.cmml" xref="A5.T7.11.11.1.m1.2.2.2"><eq id="A5.T7.11.11.1.m1.2.2.2.3.cmml" xref="A5.T7.11.11.1.m1.2.2.2.3"></eq><list id="A5.T7.11.11.1.m1.2.2.2.4.1.cmml" xref="A5.T7.11.11.1.m1.2.2.2.4.2"><ci id="A5.T7.11.11.1.m1.1.1.1.1.cmml" xref="A5.T7.11.11.1.m1.1.1.1.1">top</ci><ci id="A5.T7.11.11.1.m1.2.2.2.2.cmml" xref="A5.T7.11.11.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.11.11.1.m1.2.2.2.5.cmml" xref="A5.T7.11.11.1.m1.2.2.2.5">1000</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.11.11.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=1000}</annotation></semantics></math>)</th>
<td id="A5.T7.11.11.2" class="ltx_td ltx_align_center">37.40</td>
<td id="A5.T7.11.11.3" class="ltx_td ltx_align_center">15.30</td>
<td id="A5.T7.11.11.4" class="ltx_td ltx_align_center">30.16</td>
<td id="A5.T7.11.11.5" class="ltx_td ltx_align_center">33.6</td>
<td id="A5.T7.11.11.6" class="ltx_td ltx_align_center">14.9</td>
<td id="A5.T7.11.11.7" class="ltx_td ltx_align_center">75.9</td>
</tr>
<tr id="A5.T7.12.12" class="ltx_tr">
<th id="A5.T7.12.12.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_bb ltx_border_r">
<span id="A5.T7.12.12.1.1" class="ltx_text ltx_font_smallcaps">PegFame</span> (<math id="A5.T7.12.12.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=10000}" display="inline"><semantics id="A5.T7.12.12.1.m1.2a"><msub id="A5.T7.12.12.1.m1.2.3" xref="A5.T7.12.12.1.m1.2.3.cmml"><mi id="A5.T7.12.12.1.m1.2.3.2" xref="A5.T7.12.12.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.12.12.1.m1.2.2.2" xref="A5.T7.12.12.1.m1.2.2.2.cmml"><mrow id="A5.T7.12.12.1.m1.2.2.2.4.2" xref="A5.T7.12.12.1.m1.2.2.2.4.1.cmml"><mi id="A5.T7.12.12.1.m1.1.1.1.1" xref="A5.T7.12.12.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.12.12.1.m1.2.2.2.4.2.1" xref="A5.T7.12.12.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.T7.12.12.1.m1.2.2.2.2" xref="A5.T7.12.12.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.T7.12.12.1.m1.2.2.2.3" xref="A5.T7.12.12.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.T7.12.12.1.m1.2.2.2.5" xref="A5.T7.12.12.1.m1.2.2.2.5.cmml">10000</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.12.12.1.m1.2b"><apply id="A5.T7.12.12.1.m1.2.3.cmml" xref="A5.T7.12.12.1.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.12.12.1.m1.2.3.1.cmml" xref="A5.T7.12.12.1.m1.2.3">subscript</csymbol><ci id="A5.T7.12.12.1.m1.2.3.2.cmml" xref="A5.T7.12.12.1.m1.2.3.2">Focus</ci><apply id="A5.T7.12.12.1.m1.2.2.2.cmml" xref="A5.T7.12.12.1.m1.2.2.2"><eq id="A5.T7.12.12.1.m1.2.2.2.3.cmml" xref="A5.T7.12.12.1.m1.2.2.2.3"></eq><list id="A5.T7.12.12.1.m1.2.2.2.4.1.cmml" xref="A5.T7.12.12.1.m1.2.2.2.4.2"><ci id="A5.T7.12.12.1.m1.1.1.1.1.cmml" xref="A5.T7.12.12.1.m1.1.1.1.1">top</ci><ci id="A5.T7.12.12.1.m1.2.2.2.2.cmml" xref="A5.T7.12.12.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.T7.12.12.1.m1.2.2.2.5.cmml" xref="A5.T7.12.12.1.m1.2.2.2.5">10000</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.12.12.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=10000}</annotation></semantics></math>)</th>
<td id="A5.T7.12.12.2" class="ltx_td ltx_align_center ltx_border_bb">42.76</td>
<td id="A5.T7.12.12.3" class="ltx_td ltx_align_center ltx_border_bb">19.89</td>
<td id="A5.T7.12.12.4" class="ltx_td ltx_align_center ltx_border_bb">34.97</td>
<td id="A5.T7.12.12.5" class="ltx_td ltx_align_center ltx_border_bb">40.2</td>
<td id="A5.T7.12.12.6" class="ltx_td ltx_align_center ltx_border_bb">20.1</td>
<td id="A5.T7.12.12.7" class="ltx_td ltx_align_center ltx_border_bb">80.5</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>Assessment of controlled summary generation with focus sampling <math id="A5.T7.17.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k}" display="inline"><semantics id="A5.T7.17.m1.2b"><msub id="A5.T7.17.m1.2.3" xref="A5.T7.17.m1.2.3.cmml"><mi id="A5.T7.17.m1.2.3.2" xref="A5.T7.17.m1.2.3.2.cmml">Focus</mi><mrow id="A5.T7.17.m1.2.2.2.4" xref="A5.T7.17.m1.2.2.2.3.cmml"><mi id="A5.T7.17.m1.1.1.1.1" xref="A5.T7.17.m1.1.1.1.1.cmml">top</mi><mo id="A5.T7.17.m1.2.2.2.4.1" xref="A5.T7.17.m1.2.2.2.3.cmml">,</mo><mi id="A5.T7.17.m1.2.2.2.2" xref="A5.T7.17.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.T7.17.m1.2c"><apply id="A5.T7.17.m1.2.3.cmml" xref="A5.T7.17.m1.2.3"><csymbol cd="ambiguous" id="A5.T7.17.m1.2.3.1.cmml" xref="A5.T7.17.m1.2.3">subscript</csymbol><ci id="A5.T7.17.m1.2.3.2.cmml" xref="A5.T7.17.m1.2.3.2">Focus</ci><list id="A5.T7.17.m1.2.2.2.3.cmml" xref="A5.T7.17.m1.2.2.2.4"><ci id="A5.T7.17.m1.1.1.1.1.cmml" xref="A5.T7.17.m1.1.1.1.1">top</ci><ci id="A5.T7.17.m1.2.2.2.2.cmml" xref="A5.T7.17.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.17.m1.2d">\mathrm{Focus}_{\mathrm{top},k}</annotation></semantics></math> on the <span id="A5.T7.30.1" class="ltx_text ltx_font_smallcaps">XSum</span> test set.
We experiment with limiting <span id="A5.T7.31.2" class="ltx_text ltx_font_smallcaps">Fame</span> models to different sizes of vocabulary <math id="A5.T7.18.m2.1" class="ltx_Math" alttext="V_{k}" display="inline"><semantics id="A5.T7.18.m2.1b"><msub id="A5.T7.18.m2.1.1" xref="A5.T7.18.m2.1.1.cmml"><mi id="A5.T7.18.m2.1.1.2" xref="A5.T7.18.m2.1.1.2.cmml">V</mi><mi id="A5.T7.18.m2.1.1.3" xref="A5.T7.18.m2.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="A5.T7.18.m2.1c"><apply id="A5.T7.18.m2.1.1.cmml" xref="A5.T7.18.m2.1.1"><csymbol cd="ambiguous" id="A5.T7.18.m2.1.1.1.cmml" xref="A5.T7.18.m2.1.1">subscript</csymbol><ci id="A5.T7.18.m2.1.1.2.cmml" xref="A5.T7.18.m2.1.1.2">𝑉</ci><ci id="A5.T7.18.m2.1.1.3.cmml" xref="A5.T7.18.m2.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.18.m2.1d">V_{k}</annotation></semantics></math> using the topic distribution <math id="A5.T7.19.m3.1" class="ltx_Math" alttext="t_{X}" display="inline"><semantics id="A5.T7.19.m3.1b"><msub id="A5.T7.19.m3.1.1" xref="A5.T7.19.m3.1.1.cmml"><mi id="A5.T7.19.m3.1.1.2" xref="A5.T7.19.m3.1.1.2.cmml">t</mi><mi id="A5.T7.19.m3.1.1.3" xref="A5.T7.19.m3.1.1.3.cmml">X</mi></msub><annotation-xml encoding="MathML-Content" id="A5.T7.19.m3.1c"><apply id="A5.T7.19.m3.1.1.cmml" xref="A5.T7.19.m3.1.1"><csymbol cd="ambiguous" id="A5.T7.19.m3.1.1.1.cmml" xref="A5.T7.19.m3.1.1">subscript</csymbol><ci id="A5.T7.19.m3.1.1.2.cmml" xref="A5.T7.19.m3.1.1.2">𝑡</ci><ci id="A5.T7.19.m3.1.1.3.cmml" xref="A5.T7.19.m3.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.19.m3.1d">t_{X}</annotation></semantics></math>; in particular, we experiment with <math id="A5.T7.20.m4.6" class="ltx_Math" alttext="k=\{50,100,200,500,1000,10000\}" display="inline"><semantics id="A5.T7.20.m4.6b"><mrow id="A5.T7.20.m4.6.7" xref="A5.T7.20.m4.6.7.cmml"><mi id="A5.T7.20.m4.6.7.2" xref="A5.T7.20.m4.6.7.2.cmml">k</mi><mo id="A5.T7.20.m4.6.7.1" xref="A5.T7.20.m4.6.7.1.cmml">=</mo><mrow id="A5.T7.20.m4.6.7.3.2" xref="A5.T7.20.m4.6.7.3.1.cmml"><mo stretchy="false" id="A5.T7.20.m4.6.7.3.2.1" xref="A5.T7.20.m4.6.7.3.1.cmml">{</mo><mn id="A5.T7.20.m4.1.1" xref="A5.T7.20.m4.1.1.cmml">50</mn><mo id="A5.T7.20.m4.6.7.3.2.2" xref="A5.T7.20.m4.6.7.3.1.cmml">,</mo><mn id="A5.T7.20.m4.2.2" xref="A5.T7.20.m4.2.2.cmml">100</mn><mo id="A5.T7.20.m4.6.7.3.2.3" xref="A5.T7.20.m4.6.7.3.1.cmml">,</mo><mn id="A5.T7.20.m4.3.3" xref="A5.T7.20.m4.3.3.cmml">200</mn><mo id="A5.T7.20.m4.6.7.3.2.4" xref="A5.T7.20.m4.6.7.3.1.cmml">,</mo><mn id="A5.T7.20.m4.4.4" xref="A5.T7.20.m4.4.4.cmml">500</mn><mo id="A5.T7.20.m4.6.7.3.2.5" xref="A5.T7.20.m4.6.7.3.1.cmml">,</mo><mn id="A5.T7.20.m4.5.5" xref="A5.T7.20.m4.5.5.cmml">1000</mn><mo id="A5.T7.20.m4.6.7.3.2.6" xref="A5.T7.20.m4.6.7.3.1.cmml">,</mo><mn id="A5.T7.20.m4.6.6" xref="A5.T7.20.m4.6.6.cmml">10000</mn><mo stretchy="false" id="A5.T7.20.m4.6.7.3.2.7" xref="A5.T7.20.m4.6.7.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.T7.20.m4.6c"><apply id="A5.T7.20.m4.6.7.cmml" xref="A5.T7.20.m4.6.7"><eq id="A5.T7.20.m4.6.7.1.cmml" xref="A5.T7.20.m4.6.7.1"></eq><ci id="A5.T7.20.m4.6.7.2.cmml" xref="A5.T7.20.m4.6.7.2">𝑘</ci><set id="A5.T7.20.m4.6.7.3.1.cmml" xref="A5.T7.20.m4.6.7.3.2"><cn type="integer" id="A5.T7.20.m4.1.1.cmml" xref="A5.T7.20.m4.1.1">50</cn><cn type="integer" id="A5.T7.20.m4.2.2.cmml" xref="A5.T7.20.m4.2.2">100</cn><cn type="integer" id="A5.T7.20.m4.3.3.cmml" xref="A5.T7.20.m4.3.3">200</cn><cn type="integer" id="A5.T7.20.m4.4.4.cmml" xref="A5.T7.20.m4.4.4">500</cn><cn type="integer" id="A5.T7.20.m4.5.5.cmml" xref="A5.T7.20.m4.5.5">1000</cn><cn type="integer" id="A5.T7.20.m4.6.6.cmml" xref="A5.T7.20.m4.6.6">10000</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.20.m4.6d">k=\{50,100,200,500,1000,10000\}</annotation></semantics></math>. We also report numbers for <span id="A5.T7.32.3" class="ltx_text ltx_font_smallcaps">RobertaS2S</span>, <span id="A5.T7.33.4" class="ltx_text ltx_font_smallcaps">RobFame</span>, <span id="A5.T7.34.5" class="ltx_text ltx_font_smallcaps">Pegasus</span> and <span id="A5.T7.35.6" class="ltx_text ltx_font_smallcaps">PegFame</span>, using the whole vocabulary of size 50k.
The <span id="A5.T7.36.7" class="ltx_text ltx_font_bold">bold</span> results in each block are the best performing <span id="A5.T7.37.8" class="ltx_text ltx_font_smallcaps">RobertaS2S</span>-based and <span id="A5.T7.38.9" class="ltx_text ltx_font_smallcaps">Pegasus</span>-based models.</figcaption>
</figure>
<figure id="A5.F5" class="ltx_figure">
<table id="A5.F5.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A5.F5.1.1.1" class="ltx_tr">
<th id="A5.F5.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="A5.F5.1.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Gold</span></th>
<th id="A5.F5.1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_tt">
<span id="A5.F5.1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.1.1.2.1.1" class="ltx_p" style="width:341.4pt;">Australia has expelled an Israeli diplomat saying Israel was behind the forging of Australian passports linked to the murder of a Hamas operative in Dubai.</span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A5.F5.1.2.1" class="ltx_tr">
<th id="A5.F5.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="A5.F5.1.2.1.1.1" class="ltx_text ltx_font_bold">Article</span></th>
<td id="A5.F5.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F5.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.2.1.2.1.1" class="ltx_p" style="width:341.4pt;">Australia’s foreign minister said these were “not the actions of a friend”.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.3.2" class="ltx_tr">
<th id="A5.F5.1.3.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A5.F5.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.3.2.2.1.1" class="ltx_p" style="width:341.4pt;">The UK took similar action in March, after concluding that Israel was responsible for the use of forged UK passports in the plot.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.4.3" class="ltx_tr">
<th id="A5.F5.1.4.3.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A5.F5.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.4.3.2.1.1" class="ltx_p" style="width:341.4pt;">The Israeli foreign ministry said Australia’s decision was disappointing.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.5.4" class="ltx_tr">
<th id="A5.F5.1.5.4.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A5.F5.1.5.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.5.4.2.1.1" class="ltx_p" style="width:341.4pt;">Ministry spokesman Yigal Palmor said it was “not in line with the importance and the quality of the relationship between our countries”.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.6.5" class="ltx_tr">
<th id="A5.F5.1.6.5.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A5.F5.1.6.5.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.6.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.6.5.2.1.1" class="ltx_p" style="width:341.4pt;">’Sorrow not anger’</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.7.6" class="ltx_tr">
<th id="A5.F5.1.7.6.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A5.F5.1.7.6.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.7.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.7.6.2.1.1" class="ltx_p" style="width:341.4pt;">At least four forged Australian passports were used in the killing of Mahmoud al-Mabhouh in Dubai in January. The originals belonged to Australians living in Israel.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.8.7" class="ltx_tr">
<th id="A5.F5.1.8.7.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A5.F5.1.8.7.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.8.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.8.7.2.1.1" class="ltx_p" style="width:341.4pt;">The Australian government said a police investigation had left it in no doubt that the Israeli authorities were behind “the abuse and counterfeiting of the passports”.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.9.8" class="ltx_tr">
<th id="A5.F5.1.9.8.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A5.F5.1.9.8.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.9.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.9.8.2.1.1" class="ltx_p" style="width:341.4pt;">As a result Foreign Minister Stephen Smith asked Israel to withdraw a diplomat, whom he did not identify.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.10.9" class="ltx_tr">
<th id="A5.F5.1.10.9.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A5.F5.1.10.9.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.10.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.10.9.2.1.1" class="ltx_p" style="width:341.4pt;">“The decision to ask Israel to remove from Australia one of its officers at the Israeli embassy in Canberra is not something which fills the Australian government with any joy,” he said.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.11.10" class="ltx_tr">
<th id="A5.F5.1.11.10.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A5.F5.1.11.10.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.11.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.11.10.2.1.1" class="ltx_p" style="width:341.4pt;">“On the contrary, the decision was made much more in sorrow than in anger.”</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.12.11" class="ltx_tr">
<th id="A5.F5.1.12.11.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A5.F5.1.12.11.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.12.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.12.11.2.1.1" class="ltx_p" style="width:341.4pt;">Passports from France, Ireland, Germany and Britain were used in the operation, and in March, the British government expelled an Israeli diplomat from London.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.13.12" class="ltx_tr">
<th id="A5.F5.1.13.12.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A5.F5.1.13.12.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.13.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.13.12.2.1.1" class="ltx_p" style="width:341.4pt;">The Israeli government has said there is no proof that it was behind the killing, although Dubai officials have said they are 99.9% sure that agents from Mossad were responsible.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.14.13" class="ltx_tr">
<th id="A5.F5.1.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="A5.F5.1.14.13.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobertaS2S</span></th>
<td id="A5.F5.1.14.13.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F5.1.14.13.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.14.13.2.1.1" class="ltx_p" style="width:341.4pt;">Australia has asked <span id="A5.F5.1.14.13.2.1.1.1" class="ltx_text" style="color:#E24C00;">Australia</span> to withdraw an Israeli diplomat from its embassy in Canberra after an alleged plot to kill a <span id="A5.F5.1.14.13.2.1.1.2" class="ltx_text" style="color:#E24C00;">Abu Dhabi militant</span> in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.15.14" class="ltx_tr">
<th id="A5.F5.1.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.F5.1.15.14.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame</span></th>
<td id="A5.F5.1.15.14.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.15.14.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.15.14.2.1.1" class="ltx_p" style="width:341.4pt;">Australia has asked Israel to withdraw one of its diplomats from its embassy in Canberra after <span id="A5.F5.1.15.14.2.1.1.1" class="ltx_text" style="color:#E24C00;">it admitted</span> it used forged passports.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.16.15" class="ltx_tr">
<th id="A5.F5.1.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.F5.1.16.15.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Pegasus</span></th>
<td id="A5.F5.1.16.15.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F5.1.16.15.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.16.15.2.1.1" class="ltx_p" style="width:341.4pt;">Australia has expelled an Israeli diplomat after concluding that forged Australian passports used in the killing of a Hamas militant in Dubai were issued by Israel.</span>
</span>
</td>
</tr>
<tr id="A5.F5.1.17.16" class="ltx_tr">
<th id="A5.F5.1.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="A5.F5.1.17.16.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame</span></th>
<td id="A5.F5.1.17.16.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.F5.1.17.16.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F5.1.17.16.2.1.1" class="ltx_p" style="width:341.4pt;">The Australian government has expelled an Israeli diplomat over the use of forged Australian passports in the killing of a Hamas militant in Dubai.</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>A 2010 BBC article from the <span id="A5.F5.7.1" class="ltx_text ltx_font_smallcaps">XSum</span> testset, its human written summary and model predictions from <span id="A5.F5.8.2" class="ltx_text ltx_font_smallcaps">RobertaS2S</span>, and <span id="A5.F5.9.3" class="ltx_text ltx_font_smallcaps">Pegasus</span>, with and without <span id="A5.F5.10.4" class="ltx_text ltx_font_smallcaps">Fame</span>. The text in <span id="A5.F5.11.5" class="ltx_text" style="color:#E24C00;">orange</span> is not supported by the input article.</figcaption>
</figure>
<figure id="A5.F6" class="ltx_figure">
<table id="A5.F6.12" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.F6.1.1" class="ltx_tr">
<th id="A5.F6.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt"><span id="A5.F6.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame<span id="A5.F6.1.1.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.1.1.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=50}" display="inline"><semantics id="A5.F6.1.1.1.1.1.m1.2a"><msub id="A5.F6.1.1.1.1.1.m1.2.3" xref="A5.F6.1.1.1.1.1.m1.2.3.cmml"><mi id="A5.F6.1.1.1.1.1.m1.2.3.2" xref="A5.F6.1.1.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.1.1.1.1.1.m1.2.2.2" xref="A5.F6.1.1.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.1.1.1.1.1.m1.2.2.2.4.2" xref="A5.F6.1.1.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.1.1.1.1.1.m1.1.1.1.1" xref="A5.F6.1.1.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.1.1.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.1.1.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.1.1.1.1.1.m1.2.2.2.2" xref="A5.F6.1.1.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.1.1.1.1.1.m1.2.2.2.3" xref="A5.F6.1.1.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.1.1.1.1.1.m1.2.2.2.5" xref="A5.F6.1.1.1.1.1.m1.2.2.2.5.cmml">50</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.1.1.1.1.1.m1.2b"><apply id="A5.F6.1.1.1.1.1.m1.2.3.cmml" xref="A5.F6.1.1.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.1.1.1.1.1.m1.2.3.1.cmml" xref="A5.F6.1.1.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.1.1.1.1.1.m1.2.3.2.cmml" xref="A5.F6.1.1.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.1.1.1.1.1.m1.2.2.2.cmml" xref="A5.F6.1.1.1.1.1.m1.2.2.2"><eq id="A5.F6.1.1.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.1.1.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.1.1.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.1.1.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.1.1.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.1.1.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.1.1.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.1.1.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.1.1.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.1.1.1.1.1.m1.2.2.2.5">50</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.1.1.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=50}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.F6.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.1.1.2.1.1" class="ltx_p" style="width:313.0pt;">Australia has said <span id="A5.F6.1.1.2.1.1.1" class="ltx_text" style="color:#E24C00;">it will not be expelled an ambassador from Australia following the alleged s agent for the so-called Arab Arab State.</span></span>
</span>
</td>
</tr>
<tr id="A5.F6.2.2" class="ltx_tr">
<th id="A5.F6.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.F6.2.2.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame<span id="A5.F6.2.2.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.2.2.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=100}" display="inline"><semantics id="A5.F6.2.2.1.1.1.m1.2a"><msub id="A5.F6.2.2.1.1.1.m1.2.3" xref="A5.F6.2.2.1.1.1.m1.2.3.cmml"><mi id="A5.F6.2.2.1.1.1.m1.2.3.2" xref="A5.F6.2.2.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.2.2.1.1.1.m1.2.2.2" xref="A5.F6.2.2.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.2.2.1.1.1.m1.2.2.2.4.2" xref="A5.F6.2.2.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.2.2.1.1.1.m1.1.1.1.1" xref="A5.F6.2.2.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.2.2.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.2.2.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.2.2.1.1.1.m1.2.2.2.2" xref="A5.F6.2.2.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.2.2.1.1.1.m1.2.2.2.3" xref="A5.F6.2.2.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.2.2.1.1.1.m1.2.2.2.5" xref="A5.F6.2.2.1.1.1.m1.2.2.2.5.cmml">100</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.2.2.1.1.1.m1.2b"><apply id="A5.F6.2.2.1.1.1.m1.2.3.cmml" xref="A5.F6.2.2.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.2.2.1.1.1.m1.2.3.1.cmml" xref="A5.F6.2.2.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.2.2.1.1.1.m1.2.3.2.cmml" xref="A5.F6.2.2.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.2.2.1.1.1.m1.2.2.2.cmml" xref="A5.F6.2.2.1.1.1.m1.2.2.2"><eq id="A5.F6.2.2.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.2.2.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.2.2.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.2.2.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.2.2.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.2.2.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.2.2.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.2.2.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.2.2.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.2.2.1.1.1.m1.2.2.2.5">100</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.2.2.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=100}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F6.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.2.2.2.1.1" class="ltx_p" style="width:313.0pt;">Australia has said <span id="A5.F6.2.2.2.1.1.1" class="ltx_text" style="color:#E24C00;">it will not be expelled an ambassador from Australia</span> following the killing of a terror agent in the Arab world.</span>
</span>
</td>
</tr>
<tr id="A5.F6.3.3" class="ltx_tr">
<th id="A5.F6.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.F6.3.3.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame<span id="A5.F6.3.3.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.3.3.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=200}" display="inline"><semantics id="A5.F6.3.3.1.1.1.m1.2a"><msub id="A5.F6.3.3.1.1.1.m1.2.3" xref="A5.F6.3.3.1.1.1.m1.2.3.cmml"><mi id="A5.F6.3.3.1.1.1.m1.2.3.2" xref="A5.F6.3.3.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.3.3.1.1.1.m1.2.2.2" xref="A5.F6.3.3.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.3.3.1.1.1.m1.2.2.2.4.2" xref="A5.F6.3.3.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.3.3.1.1.1.m1.1.1.1.1" xref="A5.F6.3.3.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.3.3.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.3.3.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.3.3.1.1.1.m1.2.2.2.2" xref="A5.F6.3.3.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.3.3.1.1.1.m1.2.2.2.3" xref="A5.F6.3.3.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.3.3.1.1.1.m1.2.2.2.5" xref="A5.F6.3.3.1.1.1.m1.2.2.2.5.cmml">200</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.3.3.1.1.1.m1.2b"><apply id="A5.F6.3.3.1.1.1.m1.2.3.cmml" xref="A5.F6.3.3.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.3.3.1.1.1.m1.2.3.1.cmml" xref="A5.F6.3.3.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.3.3.1.1.1.m1.2.3.2.cmml" xref="A5.F6.3.3.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.3.3.1.1.1.m1.2.2.2.cmml" xref="A5.F6.3.3.1.1.1.m1.2.2.2"><eq id="A5.F6.3.3.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.3.3.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.3.3.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.3.3.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.3.3.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.3.3.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.3.3.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.3.3.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.3.3.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.3.3.1.1.1.m1.2.2.2.5">200</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.3.3.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=200}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.3.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F6.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.3.3.2.1.1" class="ltx_p" style="width:313.0pt;">Australia has said <span id="A5.F6.3.3.2.1.1.1" class="ltx_text" style="color:#E24C00;">it will not be expelled an ambassador from Australia following the killing of an Australian terror suspect</span> in the Arab world.</span>
</span>
</td>
</tr>
<tr id="A5.F6.4.4" class="ltx_tr">
<th id="A5.F6.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.F6.4.4.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame<span id="A5.F6.4.4.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.4.4.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=500}" display="inline"><semantics id="A5.F6.4.4.1.1.1.m1.2a"><msub id="A5.F6.4.4.1.1.1.m1.2.3" xref="A5.F6.4.4.1.1.1.m1.2.3.cmml"><mi id="A5.F6.4.4.1.1.1.m1.2.3.2" xref="A5.F6.4.4.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.4.4.1.1.1.m1.2.2.2" xref="A5.F6.4.4.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.4.4.1.1.1.m1.2.2.2.4.2" xref="A5.F6.4.4.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.4.4.1.1.1.m1.1.1.1.1" xref="A5.F6.4.4.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.4.4.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.4.4.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.4.4.1.1.1.m1.2.2.2.2" xref="A5.F6.4.4.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.4.4.1.1.1.m1.2.2.2.3" xref="A5.F6.4.4.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.4.4.1.1.1.m1.2.2.2.5" xref="A5.F6.4.4.1.1.1.m1.2.2.2.5.cmml">500</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.4.4.1.1.1.m1.2b"><apply id="A5.F6.4.4.1.1.1.m1.2.3.cmml" xref="A5.F6.4.4.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.4.4.1.1.1.m1.2.3.1.cmml" xref="A5.F6.4.4.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.4.4.1.1.1.m1.2.3.2.cmml" xref="A5.F6.4.4.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.4.4.1.1.1.m1.2.2.2.cmml" xref="A5.F6.4.4.1.1.1.m1.2.2.2"><eq id="A5.F6.4.4.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.4.4.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.4.4.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.4.4.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.4.4.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.4.4.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.4.4.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.4.4.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.4.4.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.4.4.1.1.1.m1.2.2.2.5">500</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.4.4.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=500}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.4.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F6.4.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.4.4.2.1.1" class="ltx_p" style="width:313.0pt;">Australia has asked Israel <span id="A5.F6.4.4.2.1.1.1" class="ltx_text" style="color:#E24C00;">to end its diplomatic investigation into an alleged plot to murder an Australian terror suspect</span>.</span>
</span>
</td>
</tr>
<tr id="A5.F6.5.5" class="ltx_tr">
<th id="A5.F6.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.F6.5.5.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame<span id="A5.F6.5.5.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.5.5.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=1000}" display="inline"><semantics id="A5.F6.5.5.1.1.1.m1.2a"><msub id="A5.F6.5.5.1.1.1.m1.2.3" xref="A5.F6.5.5.1.1.1.m1.2.3.cmml"><mi id="A5.F6.5.5.1.1.1.m1.2.3.2" xref="A5.F6.5.5.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.5.5.1.1.1.m1.2.2.2" xref="A5.F6.5.5.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.5.5.1.1.1.m1.2.2.2.4.2" xref="A5.F6.5.5.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.5.5.1.1.1.m1.1.1.1.1" xref="A5.F6.5.5.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.5.5.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.5.5.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.5.5.1.1.1.m1.2.2.2.2" xref="A5.F6.5.5.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.5.5.1.1.1.m1.2.2.2.3" xref="A5.F6.5.5.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.5.5.1.1.1.m1.2.2.2.5" xref="A5.F6.5.5.1.1.1.m1.2.2.2.5.cmml">1000</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.5.5.1.1.1.m1.2b"><apply id="A5.F6.5.5.1.1.1.m1.2.3.cmml" xref="A5.F6.5.5.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.5.5.1.1.1.m1.2.3.1.cmml" xref="A5.F6.5.5.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.5.5.1.1.1.m1.2.3.2.cmml" xref="A5.F6.5.5.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.5.5.1.1.1.m1.2.2.2.cmml" xref="A5.F6.5.5.1.1.1.m1.2.2.2"><eq id="A5.F6.5.5.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.5.5.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.5.5.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.5.5.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.5.5.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.5.5.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.5.5.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.5.5.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.5.5.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.5.5.1.1.1.m1.2.2.2.5">1000</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.5.5.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=1000}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.5.5.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F6.5.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.5.5.2.1.1" class="ltx_p" style="width:313.0pt;">Australia has asked Israel to strip an ambassador from its embassy following the death of an Arab man in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F6.6.6" class="ltx_tr">
<th id="A5.F6.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.F6.6.6.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame<span id="A5.F6.6.6.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.6.6.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=10000}" display="inline"><semantics id="A5.F6.6.6.1.1.1.m1.2a"><msub id="A5.F6.6.6.1.1.1.m1.2.3" xref="A5.F6.6.6.1.1.1.m1.2.3.cmml"><mi id="A5.F6.6.6.1.1.1.m1.2.3.2" xref="A5.F6.6.6.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.6.6.1.1.1.m1.2.2.2" xref="A5.F6.6.6.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.6.6.1.1.1.m1.2.2.2.4.2" xref="A5.F6.6.6.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.6.6.1.1.1.m1.1.1.1.1" xref="A5.F6.6.6.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.6.6.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.6.6.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.6.6.1.1.1.m1.2.2.2.2" xref="A5.F6.6.6.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.6.6.1.1.1.m1.2.2.2.3" xref="A5.F6.6.6.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.6.6.1.1.1.m1.2.2.2.5" xref="A5.F6.6.6.1.1.1.m1.2.2.2.5.cmml">10000</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.6.6.1.1.1.m1.2b"><apply id="A5.F6.6.6.1.1.1.m1.2.3.cmml" xref="A5.F6.6.6.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.6.6.1.1.1.m1.2.3.1.cmml" xref="A5.F6.6.6.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.6.6.1.1.1.m1.2.3.2.cmml" xref="A5.F6.6.6.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.6.6.1.1.1.m1.2.2.2.cmml" xref="A5.F6.6.6.1.1.1.m1.2.2.2"><eq id="A5.F6.6.6.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.6.6.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.6.6.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.6.6.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.6.6.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.6.6.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.6.6.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.6.6.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.6.6.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.6.6.1.1.1.m1.2.2.2.5">10000</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.6.6.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=10000}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.6.6.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F6.6.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.6.6.2.1.1" class="ltx_p" style="width:313.0pt;">Australia has asked Israel to withdraw one of its diplomats from its embassy in Canberra following the death of a terror suspect.</span>
</span>
</td>
</tr>
<tr id="A5.F6.7.7" class="ltx_tr">
<th id="A5.F6.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="A5.F6.7.7.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame<span id="A5.F6.7.7.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.7.7.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=50}" display="inline"><semantics id="A5.F6.7.7.1.1.1.m1.2a"><msub id="A5.F6.7.7.1.1.1.m1.2.3" xref="A5.F6.7.7.1.1.1.m1.2.3.cmml"><mi id="A5.F6.7.7.1.1.1.m1.2.3.2" xref="A5.F6.7.7.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.7.7.1.1.1.m1.2.2.2" xref="A5.F6.7.7.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.7.7.1.1.1.m1.2.2.2.4.2" xref="A5.F6.7.7.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.7.7.1.1.1.m1.1.1.1.1" xref="A5.F6.7.7.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.7.7.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.7.7.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.7.7.1.1.1.m1.2.2.2.2" xref="A5.F6.7.7.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.7.7.1.1.1.m1.2.2.2.3" xref="A5.F6.7.7.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.7.7.1.1.1.m1.2.2.2.5" xref="A5.F6.7.7.1.1.1.m1.2.2.2.5.cmml">50</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.7.7.1.1.1.m1.2b"><apply id="A5.F6.7.7.1.1.1.m1.2.3.cmml" xref="A5.F6.7.7.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.7.7.1.1.1.m1.2.3.1.cmml" xref="A5.F6.7.7.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.7.7.1.1.1.m1.2.3.2.cmml" xref="A5.F6.7.7.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.7.7.1.1.1.m1.2.2.2.cmml" xref="A5.F6.7.7.1.1.1.m1.2.2.2"><eq id="A5.F6.7.7.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.7.7.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.7.7.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.7.7.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.7.7.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.7.7.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.7.7.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.7.7.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.7.7.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.7.7.1.1.1.m1.2.2.2.5">50</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.7.7.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=50}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.7.7.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F6.7.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.7.7.2.1.1" class="ltx_p" style="width:313.0pt;"><span id="A5.F6.7.7.2.1.1.1" class="ltx_text" style="color:#E24C00;">The Israeli government has been expelled</span> from the country after it was found that the country’s security agency, the Israeli intelligence agency, was to be <span id="A5.F6.7.7.2.1.1.2" class="ltx_text" style="color:#E24C00;">to be found to have used a number of the country’s out-of-country p when it was used in the Emirates car-j best.</span></span>
</span>
</td>
</tr>
<tr id="A5.F6.8.8" class="ltx_tr">
<th id="A5.F6.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.F6.8.8.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame<span id="A5.F6.8.8.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.8.8.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=100}" display="inline"><semantics id="A5.F6.8.8.1.1.1.m1.2a"><msub id="A5.F6.8.8.1.1.1.m1.2.3" xref="A5.F6.8.8.1.1.1.m1.2.3.cmml"><mi id="A5.F6.8.8.1.1.1.m1.2.3.2" xref="A5.F6.8.8.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.8.8.1.1.1.m1.2.2.2" xref="A5.F6.8.8.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.8.8.1.1.1.m1.2.2.2.4.2" xref="A5.F6.8.8.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.8.8.1.1.1.m1.1.1.1.1" xref="A5.F6.8.8.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.8.8.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.8.8.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.8.8.1.1.1.m1.2.2.2.2" xref="A5.F6.8.8.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.8.8.1.1.1.m1.2.2.2.3" xref="A5.F6.8.8.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.8.8.1.1.1.m1.2.2.2.5" xref="A5.F6.8.8.1.1.1.m1.2.2.2.5.cmml">100</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.8.8.1.1.1.m1.2b"><apply id="A5.F6.8.8.1.1.1.m1.2.3.cmml" xref="A5.F6.8.8.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.8.8.1.1.1.m1.2.3.1.cmml" xref="A5.F6.8.8.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.8.8.1.1.1.m1.2.3.2.cmml" xref="A5.F6.8.8.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.8.8.1.1.1.m1.2.2.2.cmml" xref="A5.F6.8.8.1.1.1.m1.2.2.2"><eq id="A5.F6.8.8.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.8.8.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.8.8.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.8.8.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.8.8.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.8.8.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.8.8.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.8.8.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.8.8.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.8.8.1.1.1.m1.2.2.2.5">100</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.8.8.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=100}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.8.8.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F6.8.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.8.8.2.1.1" class="ltx_p" style="width:313.0pt;"><span id="A5.F6.8.8.2.1.1.1" class="ltx_text" style="color:#E24C00;">The Israeli government has been expelled</span> from the country after it was found that the country’s security agency, the Israeli intelligence agency, had used the country’s visas in the <span id="A5.F6.8.8.2.1.1.2" class="ltx_text" style="color:#E24C00;">Emirates terror</span>.</span>
</span>
</td>
</tr>
<tr id="A5.F6.9.9" class="ltx_tr">
<th id="A5.F6.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.F6.9.9.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame<span id="A5.F6.9.9.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.9.9.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=200}" display="inline"><semantics id="A5.F6.9.9.1.1.1.m1.2a"><msub id="A5.F6.9.9.1.1.1.m1.2.3" xref="A5.F6.9.9.1.1.1.m1.2.3.cmml"><mi id="A5.F6.9.9.1.1.1.m1.2.3.2" xref="A5.F6.9.9.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.9.9.1.1.1.m1.2.2.2" xref="A5.F6.9.9.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.9.9.1.1.1.m1.2.2.2.4.2" xref="A5.F6.9.9.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.9.9.1.1.1.m1.1.1.1.1" xref="A5.F6.9.9.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.9.9.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.9.9.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.9.9.1.1.1.m1.2.2.2.2" xref="A5.F6.9.9.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.9.9.1.1.1.m1.2.2.2.3" xref="A5.F6.9.9.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.9.9.1.1.1.m1.2.2.2.5" xref="A5.F6.9.9.1.1.1.m1.2.2.2.5.cmml">200</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.9.9.1.1.1.m1.2b"><apply id="A5.F6.9.9.1.1.1.m1.2.3.cmml" xref="A5.F6.9.9.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.9.9.1.1.1.m1.2.3.1.cmml" xref="A5.F6.9.9.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.9.9.1.1.1.m1.2.3.2.cmml" xref="A5.F6.9.9.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.9.9.1.1.1.m1.2.2.2.cmml" xref="A5.F6.9.9.1.1.1.m1.2.2.2"><eq id="A5.F6.9.9.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.9.9.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.9.9.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.9.9.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.9.9.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.9.9.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.9.9.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.9.9.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.9.9.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.9.9.1.1.1.m1.2.2.2.5">200</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.9.9.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=200}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.9.9.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F6.9.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.9.9.2.1.1" class="ltx_p" style="width:313.0pt;">The Australian government has expelled an Israeli diplomats after it found that the country’s security agency, the Israeli intelligence agency, had used the country’s visas in the <span id="A5.F6.9.9.2.1.1.1" class="ltx_text" style="color:#E24C00;">Emirates terror attack</span>.</span>
</span>
</td>
</tr>
<tr id="A5.F6.10.10" class="ltx_tr">
<th id="A5.F6.10.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.F6.10.10.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame<span id="A5.F6.10.10.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.10.10.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=500}" display="inline"><semantics id="A5.F6.10.10.1.1.1.m1.2a"><msub id="A5.F6.10.10.1.1.1.m1.2.3" xref="A5.F6.10.10.1.1.1.m1.2.3.cmml"><mi id="A5.F6.10.10.1.1.1.m1.2.3.2" xref="A5.F6.10.10.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.10.10.1.1.1.m1.2.2.2" xref="A5.F6.10.10.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.10.10.1.1.1.m1.2.2.2.4.2" xref="A5.F6.10.10.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.10.10.1.1.1.m1.1.1.1.1" xref="A5.F6.10.10.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.10.10.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.10.10.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.10.10.1.1.1.m1.2.2.2.2" xref="A5.F6.10.10.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.10.10.1.1.1.m1.2.2.2.3" xref="A5.F6.10.10.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.10.10.1.1.1.m1.2.2.2.5" xref="A5.F6.10.10.1.1.1.m1.2.2.2.5.cmml">500</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.10.10.1.1.1.m1.2b"><apply id="A5.F6.10.10.1.1.1.m1.2.3.cmml" xref="A5.F6.10.10.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.10.10.1.1.1.m1.2.3.1.cmml" xref="A5.F6.10.10.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.10.10.1.1.1.m1.2.3.2.cmml" xref="A5.F6.10.10.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.10.10.1.1.1.m1.2.2.2.cmml" xref="A5.F6.10.10.1.1.1.m1.2.2.2"><eq id="A5.F6.10.10.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.10.10.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.10.10.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.10.10.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.10.10.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.10.10.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.10.10.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.10.10.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.10.10.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.10.10.1.1.1.m1.2.2.2.5">500</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.10.10.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=500}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.10.10.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F6.10.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.10.10.2.1.1" class="ltx_p" style="width:313.0pt;">The Australian government has expelled an Israeli diplomatic staff after accusing the country’s security agency, the Israeli intelligence agency, of using a number of Australian visas in the <span id="A5.F6.10.10.2.1.1.1" class="ltx_text" style="color:#E24C00;">Emirates terror attack</span>.</span>
</span>
</td>
</tr>
<tr id="A5.F6.11.11" class="ltx_tr">
<th id="A5.F6.11.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.F6.11.11.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame<span id="A5.F6.11.11.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.11.11.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=1000}" display="inline"><semantics id="A5.F6.11.11.1.1.1.m1.2a"><msub id="A5.F6.11.11.1.1.1.m1.2.3" xref="A5.F6.11.11.1.1.1.m1.2.3.cmml"><mi id="A5.F6.11.11.1.1.1.m1.2.3.2" xref="A5.F6.11.11.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.11.11.1.1.1.m1.2.2.2" xref="A5.F6.11.11.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.11.11.1.1.1.m1.2.2.2.4.2" xref="A5.F6.11.11.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.11.11.1.1.1.m1.1.1.1.1" xref="A5.F6.11.11.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.11.11.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.11.11.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.11.11.1.1.1.m1.2.2.2.2" xref="A5.F6.11.11.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.11.11.1.1.1.m1.2.2.2.3" xref="A5.F6.11.11.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.11.11.1.1.1.m1.2.2.2.5" xref="A5.F6.11.11.1.1.1.m1.2.2.2.5.cmml">1000</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.11.11.1.1.1.m1.2b"><apply id="A5.F6.11.11.1.1.1.m1.2.3.cmml" xref="A5.F6.11.11.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.11.11.1.1.1.m1.2.3.1.cmml" xref="A5.F6.11.11.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.11.11.1.1.1.m1.2.3.2.cmml" xref="A5.F6.11.11.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.11.11.1.1.1.m1.2.2.2.cmml" xref="A5.F6.11.11.1.1.1.m1.2.2.2"><eq id="A5.F6.11.11.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.11.11.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.11.11.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.11.11.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.11.11.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.11.11.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.11.11.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.11.11.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.11.11.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.11.11.1.1.1.m1.2.2.2.5">1000</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.11.11.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=1000}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.11.11.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F6.11.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.11.11.2.1.1" class="ltx_p" style="width:313.0pt;">Australia has expelled an Israeli diplomatic staff after accusing the country’s security agency, the Israeli military’s intelligence agency, of being responsible for the use of Australian visas used in the killing of a Palestinian.</span>
</span>
</td>
</tr>
<tr id="A5.F6.12.12" class="ltx_tr">
<th id="A5.F6.12.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="A5.F6.12.12.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame<span id="A5.F6.12.12.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F6.12.12.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k=10000}" display="inline"><semantics id="A5.F6.12.12.1.1.1.m1.2a"><msub id="A5.F6.12.12.1.1.1.m1.2.3" xref="A5.F6.12.12.1.1.1.m1.2.3.cmml"><mi id="A5.F6.12.12.1.1.1.m1.2.3.2" xref="A5.F6.12.12.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.12.12.1.1.1.m1.2.2.2" xref="A5.F6.12.12.1.1.1.m1.2.2.2.cmml"><mrow id="A5.F6.12.12.1.1.1.m1.2.2.2.4.2" xref="A5.F6.12.12.1.1.1.m1.2.2.2.4.1.cmml"><mi id="A5.F6.12.12.1.1.1.m1.1.1.1.1" xref="A5.F6.12.12.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.12.12.1.1.1.m1.2.2.2.4.2.1" xref="A5.F6.12.12.1.1.1.m1.2.2.2.4.1.cmml">,</mo><mi id="A5.F6.12.12.1.1.1.m1.2.2.2.2" xref="A5.F6.12.12.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow><mo id="A5.F6.12.12.1.1.1.m1.2.2.2.3" xref="A5.F6.12.12.1.1.1.m1.2.2.2.3.cmml">=</mo><mn id="A5.F6.12.12.1.1.1.m1.2.2.2.5" xref="A5.F6.12.12.1.1.1.m1.2.2.2.5.cmml">10000</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.12.12.1.1.1.m1.2b"><apply id="A5.F6.12.12.1.1.1.m1.2.3.cmml" xref="A5.F6.12.12.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.12.12.1.1.1.m1.2.3.1.cmml" xref="A5.F6.12.12.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F6.12.12.1.1.1.m1.2.3.2.cmml" xref="A5.F6.12.12.1.1.1.m1.2.3.2">Focus</ci><apply id="A5.F6.12.12.1.1.1.m1.2.2.2.cmml" xref="A5.F6.12.12.1.1.1.m1.2.2.2"><eq id="A5.F6.12.12.1.1.1.m1.2.2.2.3.cmml" xref="A5.F6.12.12.1.1.1.m1.2.2.2.3"></eq><list id="A5.F6.12.12.1.1.1.m1.2.2.2.4.1.cmml" xref="A5.F6.12.12.1.1.1.m1.2.2.2.4.2"><ci id="A5.F6.12.12.1.1.1.m1.1.1.1.1.cmml" xref="A5.F6.12.12.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F6.12.12.1.1.1.m1.2.2.2.2.cmml" xref="A5.F6.12.12.1.1.1.m1.2.2.2.2">𝑘</ci></list><cn type="integer" id="A5.F6.12.12.1.1.1.m1.2.2.2.5.cmml" xref="A5.F6.12.12.1.1.1.m1.2.2.2.5">10000</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.12.12.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{top},k=10000}</annotation></semantics></math>)</span></span></th>
<td id="A5.F6.12.12.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.F6.12.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F6.12.12.2.1.1" class="ltx_p" style="width:313.0pt;">Australia has expelled an Israeli diplomat over the use of forged Australian passports in the killing of a Hamas militant in Dubai.</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Model predictions with focus sampling <math id="A5.F6.15.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{top},k}" display="inline"><semantics id="A5.F6.15.m1.2b"><msub id="A5.F6.15.m1.2.3" xref="A5.F6.15.m1.2.3.cmml"><mi id="A5.F6.15.m1.2.3.2" xref="A5.F6.15.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F6.15.m1.2.2.2.4" xref="A5.F6.15.m1.2.2.2.3.cmml"><mi id="A5.F6.15.m1.1.1.1.1" xref="A5.F6.15.m1.1.1.1.1.cmml">top</mi><mo id="A5.F6.15.m1.2.2.2.4.1" xref="A5.F6.15.m1.2.2.2.3.cmml">,</mo><mi id="A5.F6.15.m1.2.2.2.2" xref="A5.F6.15.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F6.15.m1.2c"><apply id="A5.F6.15.m1.2.3.cmml" xref="A5.F6.15.m1.2.3"><csymbol cd="ambiguous" id="A5.F6.15.m1.2.3.1.cmml" xref="A5.F6.15.m1.2.3">subscript</csymbol><ci id="A5.F6.15.m1.2.3.2.cmml" xref="A5.F6.15.m1.2.3.2">Focus</ci><list id="A5.F6.15.m1.2.2.2.3.cmml" xref="A5.F6.15.m1.2.2.2.4"><ci id="A5.F6.15.m1.1.1.1.1.cmml" xref="A5.F6.15.m1.1.1.1.1">top</ci><ci id="A5.F6.15.m1.2.2.2.2.cmml" xref="A5.F6.15.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.15.m1.2d">\mathrm{Focus}_{\mathrm{top},k}</annotation></semantics></math>, a controlled generation setting. The text in <span id="A5.F6.20.1" class="ltx_text" style="color:#E24C00;">orange</span> is not supported by the input article. We note that with smaller values of <math id="A5.F6.16.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.F6.16.m2.1b"><mi id="A5.F6.16.m2.1.1" xref="A5.F6.16.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.F6.16.m2.1c"><ci id="A5.F6.16.m2.1.1.cmml" xref="A5.F6.16.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.F6.16.m2.1d">k</annotation></semantics></math>, both <span id="A5.F6.21.2" class="ltx_text ltx_font_smallcaps">RobertaS2S</span>-based and <span id="A5.F6.22.3" class="ltx_text ltx_font_smallcaps">Pegasus</span>-based models tend to hallucinate more often.</figcaption>
</figure>
<figure id="A5.F7" class="ltx_figure">
<table id="A5.F7.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.F7.1.1" class="ltx_tr">
<td id="A5.F7.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.F7.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F7.1.1.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A5.F7.1.1.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame<span id="A5.F7.1.1.1.1.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F7.1.1.1.1.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="A5.F7.1.1.1.1.1.1.1.m1.2a"><msub id="A5.F7.1.1.1.1.1.1.1.m1.2.3" xref="A5.F7.1.1.1.1.1.1.1.m1.2.3.cmml"><mi id="A5.F7.1.1.1.1.1.1.1.m1.2.3.2" xref="A5.F7.1.1.1.1.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F7.1.1.1.1.1.1.1.m1.2.2.2.4" xref="A5.F7.1.1.1.1.1.1.1.m1.2.2.2.3.cmml"><mi id="A5.F7.1.1.1.1.1.1.1.m1.1.1.1.1" xref="A5.F7.1.1.1.1.1.1.1.m1.1.1.1.1.cmml">sample</mi><mo id="A5.F7.1.1.1.1.1.1.1.m1.2.2.2.4.1" xref="A5.F7.1.1.1.1.1.1.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.F7.1.1.1.1.1.1.1.m1.2.2.2.2" xref="A5.F7.1.1.1.1.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F7.1.1.1.1.1.1.1.m1.2b"><apply id="A5.F7.1.1.1.1.1.1.1.m1.2.3.cmml" xref="A5.F7.1.1.1.1.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F7.1.1.1.1.1.1.1.m1.2.3.1.cmml" xref="A5.F7.1.1.1.1.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F7.1.1.1.1.1.1.1.m1.2.3.2.cmml" xref="A5.F7.1.1.1.1.1.1.1.m1.2.3.2">Focus</ci><list id="A5.F7.1.1.1.1.1.1.1.m1.2.2.2.3.cmml" xref="A5.F7.1.1.1.1.1.1.1.m1.2.2.2.4"><ci id="A5.F7.1.1.1.1.1.1.1.m1.1.1.1.1.cmml" xref="A5.F7.1.1.1.1.1.1.1.m1.1.1.1.1">sample</ci><ci id="A5.F7.1.1.1.1.1.1.1.m1.2.2.2.2.cmml" xref="A5.F7.1.1.1.1.1.1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F7.1.1.1.1.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F7.2.3.1" class="ltx_tr">
<td id="A5.F7.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F7.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F7.2.3.1.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to strip one of its diplomats from its embassy following the death of an Arab man in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F7.2.4.2" class="ltx_tr">
<td id="A5.F7.2.4.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F7.2.4.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F7.2.4.2.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has <span id="A5.F7.2.4.2.1.1.1.1" class="ltx_text" style="color:#E24C00;">asked Israel to end its diplomatic investigation</span> into an alleged plot to murder an Australian terror suspect.</span>
</span>
</td>
</tr>
<tr id="A5.F7.2.5.3" class="ltx_tr">
<td id="A5.F7.2.5.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F7.2.5.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F7.2.5.3.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to strip one of its diplomats from its embassy in Australia over the death of a terror suspect.</span>
</span>
</td>
</tr>
<tr id="A5.F7.2.2" class="ltx_tr">
<td id="A5.F7.2.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F7.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F7.2.2.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A5.F7.2.2.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame<span id="A5.F7.2.2.1.1.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F7.2.2.1.1.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="A5.F7.2.2.1.1.1.1.1.m1.2a"><msub id="A5.F7.2.2.1.1.1.1.1.m1.2.3" xref="A5.F7.2.2.1.1.1.1.1.m1.2.3.cmml"><mi id="A5.F7.2.2.1.1.1.1.1.m1.2.3.2" xref="A5.F7.2.2.1.1.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F7.2.2.1.1.1.1.1.m1.2.2.2.4" xref="A5.F7.2.2.1.1.1.1.1.m1.2.2.2.3.cmml"><mi id="A5.F7.2.2.1.1.1.1.1.m1.1.1.1.1" xref="A5.F7.2.2.1.1.1.1.1.m1.1.1.1.1.cmml">sample</mi><mo id="A5.F7.2.2.1.1.1.1.1.m1.2.2.2.4.1" xref="A5.F7.2.2.1.1.1.1.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.F7.2.2.1.1.1.1.1.m1.2.2.2.2" xref="A5.F7.2.2.1.1.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F7.2.2.1.1.1.1.1.m1.2b"><apply id="A5.F7.2.2.1.1.1.1.1.m1.2.3.cmml" xref="A5.F7.2.2.1.1.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F7.2.2.1.1.1.1.1.m1.2.3.1.cmml" xref="A5.F7.2.2.1.1.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F7.2.2.1.1.1.1.1.m1.2.3.2.cmml" xref="A5.F7.2.2.1.1.1.1.1.m1.2.3.2">Focus</ci><list id="A5.F7.2.2.1.1.1.1.1.m1.2.2.2.3.cmml" xref="A5.F7.2.2.1.1.1.1.1.m1.2.2.2.4"><ci id="A5.F7.2.2.1.1.1.1.1.m1.1.1.1.1.cmml" xref="A5.F7.2.2.1.1.1.1.1.m1.1.1.1.1">sample</ci><ci id="A5.F7.2.2.1.1.1.1.1.m1.2.2.2.2.cmml" xref="A5.F7.2.2.1.1.1.1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F7.2.2.1.1.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F7.2.6.4" class="ltx_tr">
<td id="A5.F7.2.6.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F7.2.6.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F7.2.6.4.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has expelled an Israeli diplomatic staff after accusing it of using a number of Australian visas in the killing of a Palestinian <span id="A5.F7.2.6.4.1.1.1.1" class="ltx_text" style="color:#E24C00;">in a car bombing</span>.</span>
</span>
</td>
</tr>
<tr id="A5.F7.2.7.5" class="ltx_tr">
<td id="A5.F7.2.7.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F7.2.7.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F7.2.7.5.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has expelled an Israeli diplomatic staff after it said the country was responsible for the use of Australian visas used in the killing of a Palestinian <span id="A5.F7.2.7.5.1.1.1.1" class="ltx_text" style="color:#E24C00;">in a car bombing</span>.</span>
</span>
</td>
</tr>
<tr id="A5.F7.2.8.6" class="ltx_tr">
<td id="A5.F7.2.8.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F7.2.8.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F7.2.8.6.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomatic staff after accusing the country’s security agency, the Israeli military’s intelligence agency, of being responsible for the use of Australian visas used in the killing of a Palestinian.</span>
</span>
</td>
</tr>
<tr id="A5.F7.2.9.7" class="ltx_tr">
<td id="A5.F7.2.9.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F7.2.9.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F7.2.9.7.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomatic mission after accusing the country’s security agency, the Israeli military’s intelligence agency, of being responsible for the use of Australian visas used in the killing of a Palestinian in the Arab city of Emirates.</span>
</span>
</td>
</tr>
<tr id="A5.F7.2.10.8" class="ltx_tr">
<td id="A5.F7.2.10.8.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.F7.2.10.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F7.2.10.8.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has expelled an Israeli diplomatic staff after it said the country was responsible for the use of Australian visas used in the killing of a Palestinian in the Middle East.</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span><span id="A5.F7.9.1" class="ltx_text ltx_font_smallcaps">Fame</span> model predictions with <math id="A5.F7.5.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="A5.F7.5.m1.2b"><msub id="A5.F7.5.m1.2.3" xref="A5.F7.5.m1.2.3.cmml"><mi id="A5.F7.5.m1.2.3.2" xref="A5.F7.5.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F7.5.m1.2.2.2.4" xref="A5.F7.5.m1.2.2.2.3.cmml"><mi id="A5.F7.5.m1.1.1.1.1" xref="A5.F7.5.m1.1.1.1.1.cmml">sample</mi><mo id="A5.F7.5.m1.2.2.2.4.1" xref="A5.F7.5.m1.2.2.2.3.cmml">,</mo><mi id="A5.F7.5.m1.2.2.2.2" xref="A5.F7.5.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F7.5.m1.2c"><apply id="A5.F7.5.m1.2.3.cmml" xref="A5.F7.5.m1.2.3"><csymbol cd="ambiguous" id="A5.F7.5.m1.2.3.1.cmml" xref="A5.F7.5.m1.2.3">subscript</csymbol><ci id="A5.F7.5.m1.2.3.2.cmml" xref="A5.F7.5.m1.2.3.2">Focus</ci><list id="A5.F7.5.m1.2.2.2.3.cmml" xref="A5.F7.5.m1.2.2.2.4"><ci id="A5.F7.5.m1.1.1.1.1.cmml" xref="A5.F7.5.m1.1.1.1.1">sample</ci><ci id="A5.F7.5.m1.2.2.2.2.cmml" xref="A5.F7.5.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F7.5.m1.2d">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math> (<math id="A5.F7.6.m2.1" class="ltx_Math" alttext="k=10000" display="inline"><semantics id="A5.F7.6.m2.1b"><mrow id="A5.F7.6.m2.1.1" xref="A5.F7.6.m2.1.1.cmml"><mi id="A5.F7.6.m2.1.1.2" xref="A5.F7.6.m2.1.1.2.cmml">k</mi><mo id="A5.F7.6.m2.1.1.1" xref="A5.F7.6.m2.1.1.1.cmml">=</mo><mn id="A5.F7.6.m2.1.1.3" xref="A5.F7.6.m2.1.1.3.cmml">10000</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.F7.6.m2.1c"><apply id="A5.F7.6.m2.1.1.cmml" xref="A5.F7.6.m2.1.1"><eq id="A5.F7.6.m2.1.1.1.cmml" xref="A5.F7.6.m2.1.1.1"></eq><ci id="A5.F7.6.m2.1.1.2.cmml" xref="A5.F7.6.m2.1.1.2">𝑘</ci><cn type="integer" id="A5.F7.6.m2.1.1.3.cmml" xref="A5.F7.6.m2.1.1.3">10000</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F7.6.m2.1d">k=10000</annotation></semantics></math>). The text in <span id="A5.F7.10.2" class="ltx_text" style="color:#E24C00;">orange</span> is not supported by the input article.</figcaption>
</figure>
<figure id="A5.F8" class="ltx_figure">
<table id="A5.F8.4" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.F8.1.1" class="ltx_tr">
<td id="A5.F8.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.F8.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.1.1.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A5.F8.1.1.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobertaS2S<span id="A5.F8.1.1.1.1.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F8.1.1.1.1.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="A5.F8.1.1.1.1.1.1.1.m1.2a"><msub id="A5.F8.1.1.1.1.1.1.1.m1.2.3" xref="A5.F8.1.1.1.1.1.1.1.m1.2.3.cmml"><mi id="A5.F8.1.1.1.1.1.1.1.m1.2.3.2" xref="A5.F8.1.1.1.1.1.1.1.m1.2.3.2.cmml">Div</mi><mrow id="A5.F8.1.1.1.1.1.1.1.m1.2.2.2.4" xref="A5.F8.1.1.1.1.1.1.1.m1.2.2.2.3.cmml"><mi id="A5.F8.1.1.1.1.1.1.1.m1.1.1.1.1" xref="A5.F8.1.1.1.1.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F8.1.1.1.1.1.1.1.m1.2.2.2.4.1" xref="A5.F8.1.1.1.1.1.1.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.F8.1.1.1.1.1.1.1.m1.2.2.2.2" xref="A5.F8.1.1.1.1.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F8.1.1.1.1.1.1.1.m1.2b"><apply id="A5.F8.1.1.1.1.1.1.1.m1.2.3.cmml" xref="A5.F8.1.1.1.1.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F8.1.1.1.1.1.1.1.m1.2.3.1.cmml" xref="A5.F8.1.1.1.1.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F8.1.1.1.1.1.1.1.m1.2.3.2.cmml" xref="A5.F8.1.1.1.1.1.1.1.m1.2.3.2">Div</ci><list id="A5.F8.1.1.1.1.1.1.1.m1.2.2.2.3.cmml" xref="A5.F8.1.1.1.1.1.1.1.m1.2.2.2.4"><ci id="A5.F8.1.1.1.1.1.1.1.m1.1.1.1.1.cmml" xref="A5.F8.1.1.1.1.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F8.1.1.1.1.1.1.1.m1.2.2.2.2.cmml" xref="A5.F8.1.1.1.1.1.1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F8.1.1.1.1.1.1.1.m1.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F8.4.5.1" class="ltx_tr">
<td id="A5.F8.4.5.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.5.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.5.1.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked for an Ivan “shivers” officer to be asked to leave Australia after the performance of an Israeli flag was alleged to have been used as terrorism suspects in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.6.2" class="ltx_tr">
<td id="A5.F8.4.6.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.6.2.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked an Israeli ambassador to Sydney over an alleged implicated Australian diplomat alleging the murder of a Australian national in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.7.3" class="ltx_tr">
<td id="A5.F8.4.7.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.7.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.7.3.1.1.1" class="ltx_p" style="width:426.8pt;">Israel has asked Israel to withdraw an Israeli ambassador from Canberra amid claims that the alleged invasion of its territory by a foreign agent was behind the murder of a terror suspect in Abuabad.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.8.4" class="ltx_tr">
<td id="A5.F8.4.8.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.8.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.8.4.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to withdraw a diplomat Izzy Kanhuh, an Israeli diplomat involved in solving tensions over the sale of imported shotguns for the Dubai Abu Dhabiuddin bombing.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.9.5" class="ltx_tr">
<td id="A5.F8.4.9.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.9.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.9.5.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Australia to withdraw an ambassador from the country, amid a growing row over the alleged role of an Israel-based Abu Abu Malak director of agents.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.10.6" class="ltx_tr">
<td id="A5.F8.4.10.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.10.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.10.6.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to replace its ambassador over a fatal stabbing in Sydney last week.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.11.7" class="ltx_tr">
<td id="A5.F8.4.11.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.11.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.11.7.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to withdraw an Egyptian diplomat following the suicide of a suspected Abu Abu Mabhulas in the Australian capital, Canberra.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.12.8" class="ltx_tr">
<td id="A5.F8.4.12.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.12.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.12.8.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Australia for an official withdrawal from its embassy in Sydney after the death of a Palestinian diplomat in a Dublin diplomatic fanbase earlier this month.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.13.9" class="ltx_tr">
<td id="A5.F8.4.13.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.13.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.13.9.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to withdraw an Israeli diplomat as part of a probe into the alleged involvement in the murder of a Abu Abuab militant.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.14.10" class="ltx_tr">
<td id="A5.F8.4.14.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.14.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.14.10.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked an Israeli diplomat to be withdrawn from the country over the Diamondad bombing of a Abu Waduh as part of an investigation into its 2002 murders of a Abu Abu Baye bomber.</span>
</span>
</td>
</tr>
<tr id="A5.F8.2.2" class="ltx_tr">
<td id="A5.F8.2.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F8.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.2.2.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A5.F8.2.2.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame<span id="A5.F8.2.2.1.1.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F8.2.2.1.1.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="A5.F8.2.2.1.1.1.1.1.m1.2a"><msub id="A5.F8.2.2.1.1.1.1.1.m1.2.3" xref="A5.F8.2.2.1.1.1.1.1.m1.2.3.cmml"><mi id="A5.F8.2.2.1.1.1.1.1.m1.2.3.2" xref="A5.F8.2.2.1.1.1.1.1.m1.2.3.2.cmml">Div</mi><mrow id="A5.F8.2.2.1.1.1.1.1.m1.2.2.2.4" xref="A5.F8.2.2.1.1.1.1.1.m1.2.2.2.3.cmml"><mi id="A5.F8.2.2.1.1.1.1.1.m1.1.1.1.1" xref="A5.F8.2.2.1.1.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F8.2.2.1.1.1.1.1.m1.2.2.2.4.1" xref="A5.F8.2.2.1.1.1.1.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.F8.2.2.1.1.1.1.1.m1.2.2.2.2" xref="A5.F8.2.2.1.1.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F8.2.2.1.1.1.1.1.m1.2b"><apply id="A5.F8.2.2.1.1.1.1.1.m1.2.3.cmml" xref="A5.F8.2.2.1.1.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F8.2.2.1.1.1.1.1.m1.2.3.1.cmml" xref="A5.F8.2.2.1.1.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F8.2.2.1.1.1.1.1.m1.2.3.2.cmml" xref="A5.F8.2.2.1.1.1.1.1.m1.2.3.2">Div</ci><list id="A5.F8.2.2.1.1.1.1.1.m1.2.2.2.3.cmml" xref="A5.F8.2.2.1.1.1.1.1.m1.2.2.2.4"><ci id="A5.F8.2.2.1.1.1.1.1.m1.1.1.1.1.cmml" xref="A5.F8.2.2.1.1.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F8.2.2.1.1.1.1.1.m1.2.2.2.2.cmml" xref="A5.F8.2.2.1.1.1.1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F8.2.2.1.1.1.1.1.m1.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F8.4.15.11" class="ltx_tr">
<td id="A5.F8.4.15.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.15.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.15.11.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has played down claims its state ambassador was involved in finding out why the Mossad spy agent was behind the Rio stabbing.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.16.12" class="ltx_tr">
<td id="A5.F8.4.16.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.16.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.16.12.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to withdraw one of its diplomats after it confessed the so-called Mossad agent agent had used a fake Melbourne funery.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.17.13" class="ltx_tr">
<td id="A5.F8.4.17.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.17.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.17.13.1.1.1" class="ltx_p" style="width:426.8pt;">Australia says it will withdraw an envoy after the Israel spy agent accused of involvement in the murder of an Arab smuggler was suspended.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.18.14" class="ltx_tr">
<td id="A5.F8.4.18.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.18.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.18.14.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to expel one of its citizens after the country leaked the state agent that led later a deadly mafia murder in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.19.15" class="ltx_tr">
<td id="A5.F8.4.19.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.19.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.19.15.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to withdraw its consulate at Canberra because from its embassy after it claimed it used the Falcon fuelling plan for a suicide bomb.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.20.16" class="ltx_tr">
<td id="A5.F8.4.20.16.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.20.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.20.16.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to withdraw its support for Europe’s embassy for its arrest of an Edinburgh diplomat over the death of a heroin smuggling gang.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.21.17" class="ltx_tr">
<td id="A5.F8.4.21.17.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.21.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.21.17.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to remove an ambassador from its embassy over the shooting dead of an Australian man on a Dubai delivery scheme.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.22.18" class="ltx_tr">
<td id="A5.F8.4.22.18.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.22.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.22.18.1.1.1" class="ltx_p" style="width:426.8pt;">Australia is to withdraw a diplomat from its embassy in Canberra over allegations it worked on the mastermind for an alleged spying plot for the Mossad operation.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.23.19" class="ltx_tr">
<td id="A5.F8.4.23.19.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.23.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.23.19.1.1.1" class="ltx_p" style="width:426.8pt;">Australiachas asked Israel to withdraw an anonymous diplomat from its embassy following investigation into the passage of a Falcon recruiting device.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.24.20" class="ltx_tr">
<td id="A5.F8.4.24.20.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.24.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.24.20.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked the Israeli embassy to pull out of its alleged response to the murder of a British terror suspect, accusing it of responsibility.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.4" class="ltx_tr">
<td id="A5.F8.4.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F8.4.4.2.2" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.4.2.2.2" class="ltx_p" style="width:426.8pt;"><span id="A5.F8.4.4.2.2.2.2" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame<span id="A5.F8.4.4.2.2.2.2.2" class="ltx_text ltx_font_upright"> (<math id="A5.F8.3.3.1.1.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="A5.F8.3.3.1.1.1.1.1.m1.2a"><msub id="A5.F8.3.3.1.1.1.1.1.m1.2.3" xref="A5.F8.3.3.1.1.1.1.1.m1.2.3.cmml"><mi id="A5.F8.3.3.1.1.1.1.1.m1.2.3.2" xref="A5.F8.3.3.1.1.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F8.3.3.1.1.1.1.1.m1.2.2.2.4" xref="A5.F8.3.3.1.1.1.1.1.m1.2.2.2.3.cmml"><mi id="A5.F8.3.3.1.1.1.1.1.m1.1.1.1.1" xref="A5.F8.3.3.1.1.1.1.1.m1.1.1.1.1.cmml">sample</mi><mo id="A5.F8.3.3.1.1.1.1.1.m1.2.2.2.4.1" xref="A5.F8.3.3.1.1.1.1.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.F8.3.3.1.1.1.1.1.m1.2.2.2.2" xref="A5.F8.3.3.1.1.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F8.3.3.1.1.1.1.1.m1.2b"><apply id="A5.F8.3.3.1.1.1.1.1.m1.2.3.cmml" xref="A5.F8.3.3.1.1.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F8.3.3.1.1.1.1.1.m1.2.3.1.cmml" xref="A5.F8.3.3.1.1.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F8.3.3.1.1.1.1.1.m1.2.3.2.cmml" xref="A5.F8.3.3.1.1.1.1.1.m1.2.3.2">Focus</ci><list id="A5.F8.3.3.1.1.1.1.1.m1.2.2.2.3.cmml" xref="A5.F8.3.3.1.1.1.1.1.m1.2.2.2.4"><ci id="A5.F8.3.3.1.1.1.1.1.m1.1.1.1.1.cmml" xref="A5.F8.3.3.1.1.1.1.1.m1.1.1.1.1">sample</ci><ci id="A5.F8.3.3.1.1.1.1.1.m1.2.2.2.2.cmml" xref="A5.F8.3.3.1.1.1.1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F8.3.3.1.1.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, <math id="A5.F8.4.4.2.2.2.2.2.m2.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="A5.F8.4.4.2.2.2.2.2.m2.2a"><msub id="A5.F8.4.4.2.2.2.2.2.m2.2.3" xref="A5.F8.4.4.2.2.2.2.2.m2.2.3.cmml"><mi id="A5.F8.4.4.2.2.2.2.2.m2.2.3.2" xref="A5.F8.4.4.2.2.2.2.2.m2.2.3.2.cmml">Div</mi><mrow id="A5.F8.4.4.2.2.2.2.2.m2.2.2.2.4" xref="A5.F8.4.4.2.2.2.2.2.m2.2.2.2.3.cmml"><mi id="A5.F8.4.4.2.2.2.2.2.m2.1.1.1.1" xref="A5.F8.4.4.2.2.2.2.2.m2.1.1.1.1.cmml">top</mi><mo id="A5.F8.4.4.2.2.2.2.2.m2.2.2.2.4.1" xref="A5.F8.4.4.2.2.2.2.2.m2.2.2.2.3.cmml">,</mo><mi id="A5.F8.4.4.2.2.2.2.2.m2.2.2.2.2" xref="A5.F8.4.4.2.2.2.2.2.m2.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F8.4.4.2.2.2.2.2.m2.2b"><apply id="A5.F8.4.4.2.2.2.2.2.m2.2.3.cmml" xref="A5.F8.4.4.2.2.2.2.2.m2.2.3"><csymbol cd="ambiguous" id="A5.F8.4.4.2.2.2.2.2.m2.2.3.1.cmml" xref="A5.F8.4.4.2.2.2.2.2.m2.2.3">subscript</csymbol><ci id="A5.F8.4.4.2.2.2.2.2.m2.2.3.2.cmml" xref="A5.F8.4.4.2.2.2.2.2.m2.2.3.2">Div</ci><list id="A5.F8.4.4.2.2.2.2.2.m2.2.2.2.3.cmml" xref="A5.F8.4.4.2.2.2.2.2.m2.2.2.2.4"><ci id="A5.F8.4.4.2.2.2.2.2.m2.1.1.1.1.cmml" xref="A5.F8.4.4.2.2.2.2.2.m2.1.1.1.1">top</ci><ci id="A5.F8.4.4.2.2.2.2.2.m2.2.2.2.2.cmml" xref="A5.F8.4.4.2.2.2.2.2.m2.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F8.4.4.2.2.2.2.2.m2.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F8.4.25.21" class="ltx_tr">
<td id="A5.F8.4.25.21.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.25.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.25.21.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to answer the decision to honour its state ambassador following the alleged involvement in the killing of a Dubai terror suspect.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.26.22" class="ltx_tr">
<td id="A5.F8.4.26.22.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.26.22.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.26.22.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel for a second diplomat to be expelled from Australia after an alleged plot to murder a man in a bomb plot linked to Mossad.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.27.23" class="ltx_tr">
<td id="A5.F8.4.27.23.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.27.23.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.27.23.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to make a state diplomat its top diplomat after an alleged plot to bomb an Arab Emirates terror operation was blamed on a terror agent.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.28.24" class="ltx_tr">
<td id="A5.F8.4.28.24.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.28.24.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.28.24.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Egypt to end its diplomatic at-top diplomatic response to the murder of a top Arab diplomat in the Arab world.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.29.25" class="ltx_tr">
<td id="A5.F8.4.29.25.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.29.25.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.29.25.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to be expelled from the embassy in Australia following the death of a Sydney spy in a spy investigation.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.30.26" class="ltx_tr">
<td id="A5.F8.4.30.26.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.30.26.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.30.26.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to to strip an diplomat of its consulate from its embassy since a deadly operation against the Mossad spy agent at a terror squad in Australia last month.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.31.27" class="ltx_tr">
<td id="A5.F8.4.31.27.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.31.27.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.31.27.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked the Israel embassy to withdrawing its diplomats following the death of an Arab man by Mossad agents.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.32.28" class="ltx_tr">
<td id="A5.F8.4.32.28.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.32.28.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.32.28.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to end the original accusations that a diplomat is responsible for the killing of an agent from Mossad.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.33.29" class="ltx_tr">
<td id="A5.F8.4.33.29.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F8.4.33.29.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.33.29.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to answer the investigation that admitted its diplomats used his agent as a suicide bomb in a Dubai plot.</span>
</span>
</td>
</tr>
<tr id="A5.F8.4.34.30" class="ltx_tr">
<td id="A5.F8.4.34.30.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.F8.4.34.30.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F8.4.34.30.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to support its ambassador after it admitted being involved in the murder of a suspect in the deadly one-off terror killing in a Melbourne bomb attack.</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>Diverse summaries predicted using <span id="A5.F8.9.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> and <span id="A5.F8.10.2" class="ltx_text ltx_font_smallcaps">RobFame</span> models with <math id="A5.F8.6.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="A5.F8.6.m1.2b"><msub id="A5.F8.6.m1.2.3" xref="A5.F8.6.m1.2.3.cmml"><mi id="A5.F8.6.m1.2.3.2" xref="A5.F8.6.m1.2.3.2.cmml">Div</mi><mrow id="A5.F8.6.m1.2.2.2.4" xref="A5.F8.6.m1.2.2.2.3.cmml"><mi id="A5.F8.6.m1.1.1.1.1" xref="A5.F8.6.m1.1.1.1.1.cmml">top</mi><mo id="A5.F8.6.m1.2.2.2.4.1" xref="A5.F8.6.m1.2.2.2.3.cmml">,</mo><mi id="A5.F8.6.m1.2.2.2.2" xref="A5.F8.6.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F8.6.m1.2c"><apply id="A5.F8.6.m1.2.3.cmml" xref="A5.F8.6.m1.2.3"><csymbol cd="ambiguous" id="A5.F8.6.m1.2.3.1.cmml" xref="A5.F8.6.m1.2.3">subscript</csymbol><ci id="A5.F8.6.m1.2.3.2.cmml" xref="A5.F8.6.m1.2.3.2">Div</ci><list id="A5.F8.6.m1.2.2.2.3.cmml" xref="A5.F8.6.m1.2.2.2.4"><ci id="A5.F8.6.m1.1.1.1.1.cmml" xref="A5.F8.6.m1.1.1.1.1">top</ci><ci id="A5.F8.6.m1.2.2.2.2.cmml" xref="A5.F8.6.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F8.6.m1.2d">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>.</figcaption>
</figure>
<figure id="A5.F9" class="ltx_figure">
<table id="A5.F9.4" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.F9.1.1" class="ltx_tr">
<td id="A5.F9.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.F9.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.1.1.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A5.F9.1.1.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobertaS2S<span id="A5.F9.1.1.1.1.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F9.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="A5.F9.1.1.1.1.1.1.1.m1.1a"><msub id="A5.F9.1.1.1.1.1.1.1.m1.1.1" xref="A5.F9.1.1.1.1.1.1.1.m1.1.1.cmml"><mi id="A5.F9.1.1.1.1.1.1.1.m1.1.1.2" xref="A5.F9.1.1.1.1.1.1.1.m1.1.1.2.cmml">Div</mi><mi id="A5.F9.1.1.1.1.1.1.1.m1.1.1.3" xref="A5.F9.1.1.1.1.1.1.1.m1.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="A5.F9.1.1.1.1.1.1.1.m1.1b"><apply id="A5.F9.1.1.1.1.1.1.1.m1.1.1.cmml" xref="A5.F9.1.1.1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.F9.1.1.1.1.1.1.1.m1.1.1.1.cmml" xref="A5.F9.1.1.1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="A5.F9.1.1.1.1.1.1.1.m1.1.1.2.cmml" xref="A5.F9.1.1.1.1.1.1.1.m1.1.1.2">Div</ci><ci id="A5.F9.1.1.1.1.1.1.1.m1.1.1.3.cmml" xref="A5.F9.1.1.1.1.1.1.1.m1.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F9.1.1.1.1.1.1.1.m1.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F9.4.5.1" class="ltx_tr">
<td id="A5.F9.4.5.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.5.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.5.1.1.1.1" class="ltx_p" style="width:426.8pt;">Australia says man hasenzelled an Israeli envoy following the arrest of one of its diplomats in Dubai from the countries’ deepest-running terrorism resistant group. badly documented.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.6.2" class="ltx_tr">
<td id="A5.F9.4.6.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.6.2.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked for Israel to out retrieving an Israeli diplomat who was expelled from the country after Australia accused the FBI of involvement in a 2013 murder in rogue Myersad drug smuggling operation.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.7.3" class="ltx_tr">
<td id="A5.F9.4.7.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.7.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.7.3.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to remove an envoy from its embassy in Sydney in an escalating row over the killing of a Yazad Bin Ab alcohol dealer in the United Arab Emirates.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.8.4" class="ltx_tr">
<td id="A5.F9.4.8.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.8.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.8.4.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to clarify its response to a data breach cull from rendition with a relapse of a suspected Abu Abuabuded jihadist.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.9.5" class="ltx_tr">
<td id="A5.F9.4.9.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.9.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.9.5.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Australia to pull out of Israel after an Israeli diplomat was accused of having used sreleased Australian agent Abuadab in the murder of an Abu Dhabi carrier.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.10.6" class="ltx_tr">
<td id="A5.F9.4.10.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.10.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.10.6.1.1.1" class="ltx_p" style="width:426.8pt;">Australia Herb Allen has led Australia’s ambassadorsaints over an investigation into what was allegedly led by one of its diplomats at Nessadab consultancy in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.11.7" class="ltx_tr">
<td id="A5.F9.4.11.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.11.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.11.7.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has urged Israel to withdraw an ambassador pshorze over alleged links to the murder of a Sydney binnington.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.12.8" class="ltx_tr">
<td id="A5.F9.4.12.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.12.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.12.8.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked the Israeli ambassador to Australia over an inferno at a Sydney diplomatic consulate for a senior recruiter which printers had wanted a Willis bin Laden agent to be charged.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.13.9" class="ltx_tr">
<td id="A5.F9.4.13.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.13.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.13.9.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Australia for the withdrawal of an Israeli ambassador after an investigation into it was linked to a Vietnam-based gang in which a young dungeonsad spy was killed.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.14.10" class="ltx_tr">
<td id="A5.F9.4.14.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.14.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.14.10.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel for an emotional withdrawal from its embassy in Canberra, accusing an Israeli diplomat of involvement in a feuding plot to kill a terror suspect.</span>
</span>
</td>
</tr>
<tr id="A5.F9.2.2" class="ltx_tr">
<td id="A5.F9.2.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F9.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.2.2.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A5.F9.2.2.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame<span id="A5.F9.2.2.1.1.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F9.2.2.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="A5.F9.2.2.1.1.1.1.1.m1.1a"><msub id="A5.F9.2.2.1.1.1.1.1.m1.1.1" xref="A5.F9.2.2.1.1.1.1.1.m1.1.1.cmml"><mi id="A5.F9.2.2.1.1.1.1.1.m1.1.1.2" xref="A5.F9.2.2.1.1.1.1.1.m1.1.1.2.cmml">Div</mi><mi id="A5.F9.2.2.1.1.1.1.1.m1.1.1.3" xref="A5.F9.2.2.1.1.1.1.1.m1.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="A5.F9.2.2.1.1.1.1.1.m1.1b"><apply id="A5.F9.2.2.1.1.1.1.1.m1.1.1.cmml" xref="A5.F9.2.2.1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.F9.2.2.1.1.1.1.1.m1.1.1.1.cmml" xref="A5.F9.2.2.1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="A5.F9.2.2.1.1.1.1.1.m1.1.1.2.cmml" xref="A5.F9.2.2.1.1.1.1.1.m1.1.1.2">Div</ci><ci id="A5.F9.2.2.1.1.1.1.1.m1.1.1.3.cmml" xref="A5.F9.2.2.1.1.1.1.1.m1.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F9.2.2.1.1.1.1.1.m1.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F9.4.15.11" class="ltx_tr">
<td id="A5.F9.4.15.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.15.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.15.11.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to withdraw an Israeli official over a Team Mossad bomb plot that left one of its suspects in the Dubai Arab desert.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.16.12" class="ltx_tr">
<td id="A5.F9.4.16.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.16.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.16.12.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked all Israeli diplomats to leave Canberra after the living place of an alleged Russian special forces agent was identified at the email bug held bymacadad.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.17.13" class="ltx_tr">
<td id="A5.F9.4.17.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.17.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.17.13.1.1.1" class="ltx_p" style="width:426.8pt;">Australia is to withdraw an official sensitivity inquiry from its foreign ministers after Israel was accused of involvement in a plot to kill a Dubai terror suspect.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.18.14" class="ltx_tr">
<td id="A5.F9.4.18.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.18.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.18.14.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel deep back into allegations it carried out a wanted plot Cunning deaths in a Dubai plot by Mossad agents.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.19.15" class="ltx_tr">
<td id="A5.F9.4.19.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.19.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.19.15.1.1.1" class="ltx_p" style="width:426.8pt;">AustraliaplayedAX has asked the Israeli government to withdraw an official language envoy from its embassies following the killing of a murdered cons consulate officer.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.20.16" class="ltx_tr">
<td id="A5.F9.4.20.16.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.20.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.20.16.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked an Israeli official to withdraw an official ambassador after it made a murder in a deadly shooting Presumably by Mossad.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.21.17" class="ltx_tr">
<td id="A5.F9.4.21.17.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.21.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.21.17.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel over allegations that an agent used forged passports to plot the Woolstroken murder by agentsbased in Pakistan.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.22.18" class="ltx_tr">
<td id="A5.F9.4.22.18.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.22.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.22.18.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israeli authorities to withdraw an official diplomat from Australia after the mafia was accused by the Israel embassy of contributing to its alleged failed murder of an Alquer Arab Shia terrorist.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.23.19" class="ltx_tr">
<td id="A5.F9.4.23.19.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.23.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.23.19.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked an Israeli embassy to withdraw a diplomat from Australia following the Jewlands’ murder of an unnamed man.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.24.20" class="ltx_tr">
<td id="A5.F9.4.24.20.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.24.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.24.20.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has annexed its embassy up tolishes at the start of the year after Israel confirmed it assessed the role of an undercover officer during the Dubai heroinmer plot.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.4" class="ltx_tr">
<td id="A5.F9.4.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F9.4.4.2.2" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.4.2.2.2" class="ltx_p" style="width:426.8pt;"><span id="A5.F9.4.4.2.2.2.2" class="ltx_text ltx_font_bold ltx_font_smallcaps">RobFame<span id="A5.F9.4.4.2.2.2.2.2" class="ltx_text ltx_font_upright"> (<math id="A5.F9.3.3.1.1.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="A5.F9.3.3.1.1.1.1.1.m1.2a"><msub id="A5.F9.3.3.1.1.1.1.1.m1.2.3" xref="A5.F9.3.3.1.1.1.1.1.m1.2.3.cmml"><mi id="A5.F9.3.3.1.1.1.1.1.m1.2.3.2" xref="A5.F9.3.3.1.1.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F9.3.3.1.1.1.1.1.m1.2.2.2.4" xref="A5.F9.3.3.1.1.1.1.1.m1.2.2.2.3.cmml"><mi id="A5.F9.3.3.1.1.1.1.1.m1.1.1.1.1" xref="A5.F9.3.3.1.1.1.1.1.m1.1.1.1.1.cmml">sample</mi><mo id="A5.F9.3.3.1.1.1.1.1.m1.2.2.2.4.1" xref="A5.F9.3.3.1.1.1.1.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.F9.3.3.1.1.1.1.1.m1.2.2.2.2" xref="A5.F9.3.3.1.1.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F9.3.3.1.1.1.1.1.m1.2b"><apply id="A5.F9.3.3.1.1.1.1.1.m1.2.3.cmml" xref="A5.F9.3.3.1.1.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F9.3.3.1.1.1.1.1.m1.2.3.1.cmml" xref="A5.F9.3.3.1.1.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F9.3.3.1.1.1.1.1.m1.2.3.2.cmml" xref="A5.F9.3.3.1.1.1.1.1.m1.2.3.2">Focus</ci><list id="A5.F9.3.3.1.1.1.1.1.m1.2.2.2.3.cmml" xref="A5.F9.3.3.1.1.1.1.1.m1.2.2.2.4"><ci id="A5.F9.3.3.1.1.1.1.1.m1.1.1.1.1.cmml" xref="A5.F9.3.3.1.1.1.1.1.m1.1.1.1.1">sample</ci><ci id="A5.F9.3.3.1.1.1.1.1.m1.2.2.2.2.cmml" xref="A5.F9.3.3.1.1.1.1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F9.3.3.1.1.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, <math id="A5.F9.4.4.2.2.2.2.2.m2.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="A5.F9.4.4.2.2.2.2.2.m2.1a"><msub id="A5.F9.4.4.2.2.2.2.2.m2.1.1" xref="A5.F9.4.4.2.2.2.2.2.m2.1.1.cmml"><mi id="A5.F9.4.4.2.2.2.2.2.m2.1.1.2" xref="A5.F9.4.4.2.2.2.2.2.m2.1.1.2.cmml">Div</mi><mi id="A5.F9.4.4.2.2.2.2.2.m2.1.1.3" xref="A5.F9.4.4.2.2.2.2.2.m2.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="A5.F9.4.4.2.2.2.2.2.m2.1b"><apply id="A5.F9.4.4.2.2.2.2.2.m2.1.1.cmml" xref="A5.F9.4.4.2.2.2.2.2.m2.1.1"><csymbol cd="ambiguous" id="A5.F9.4.4.2.2.2.2.2.m2.1.1.1.cmml" xref="A5.F9.4.4.2.2.2.2.2.m2.1.1">subscript</csymbol><ci id="A5.F9.4.4.2.2.2.2.2.m2.1.1.2.cmml" xref="A5.F9.4.4.2.2.2.2.2.m2.1.1.2">Div</ci><ci id="A5.F9.4.4.2.2.2.2.2.m2.1.1.3.cmml" xref="A5.F9.4.4.2.2.2.2.2.m2.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F9.4.4.2.2.2.2.2.m2.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F9.4.25.21" class="ltx_tr">
<td id="A5.F9.4.25.21.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.25.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.25.21.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to expelled an embassy diplomat over a deadly Sydney plot to spy on the Mossad operation.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.26.22" class="ltx_tr">
<td id="A5.F9.4.26.22.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.26.22.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.26.22.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to end its diplomatic inruru from Australia after it accused its diplomatic staff of involvement in last year’s deadly attack on a Melbourne terror attack.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.27.23" class="ltx_tr">
<td id="A5.F9.4.27.23.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.27.23.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.27.23.1.1.1" class="ltx_p" style="width:426.8pt;">Israel has asked Israel to make an embassy ambassador over a deadly email killing of a man in a terror plot.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.28.24" class="ltx_tr">
<td id="A5.F9.4.28.24.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.28.24.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.28.24.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to strip its diplomatic staff of its passport following an alleged plot to murder a Dubai terror suspect.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.29.25" class="ltx_tr">
<td id="A5.F9.4.29.25.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.29.25.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.29.25.1.1.1" class="ltx_p" style="width:426.8pt;">Israel has asked Israel to expelled one of its diplomats after the Mossad agent accused a Melbourne man of being the agent for the Mossad spy agent for his role in an alleged plot to murder a man.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.30.26" class="ltx_tr">
<td id="A5.F9.4.30.26.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.30.26.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.30.26.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to strip a top envoy from his embassy following its investigation into the killing of an alleged spy in a Melbourne email plot.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.31.27" class="ltx_tr">
<td id="A5.F9.4.31.27.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.31.27.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.31.27.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to expelled one diplomat following allegations it used a military agent to spy for Mossad.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.32.28" class="ltx_tr">
<td id="A5.F9.4.32.28.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.32.28.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.32.28.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked the Israel embassy to be expelled from Australia after an Australian diplomat was found guilty of his role in the murder of an Australian terror suspect.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.33.29" class="ltx_tr">
<td id="A5.F9.4.33.29.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F9.4.33.29.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.33.29.1.1.1" class="ltx_p" style="width:426.8pt;">Australia is to expelled its top diplomat from Australia after his country was accused by the UN of being responsible for an alleged plot to murder a Melbourne-Arab m intelligence agent.</span>
</span>
</td>
</tr>
<tr id="A5.F9.4.34.30" class="ltx_tr">
<td id="A5.F9.4.34.30.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.F9.4.34.30.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F9.4.34.30.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to strip an ambassador from its embassy, in response to the death of a Sydney-from-agent for the so-called “Mossad, was responsible”.</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>Diverse summaries predicted using <span id="A5.F9.9.1" class="ltx_text ltx_font_smallcaps">RobertaS2S</span> and <span id="A5.F9.10.2" class="ltx_text ltx_font_smallcaps">RobFame</span> models with <math id="A5.F9.6.m1.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="A5.F9.6.m1.1b"><msub id="A5.F9.6.m1.1.1" xref="A5.F9.6.m1.1.1.cmml"><mi id="A5.F9.6.m1.1.1.2" xref="A5.F9.6.m1.1.1.2.cmml">Div</mi><mi id="A5.F9.6.m1.1.1.3" xref="A5.F9.6.m1.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="A5.F9.6.m1.1c"><apply id="A5.F9.6.m1.1.1.cmml" xref="A5.F9.6.m1.1.1"><csymbol cd="ambiguous" id="A5.F9.6.m1.1.1.1.cmml" xref="A5.F9.6.m1.1.1">subscript</csymbol><ci id="A5.F9.6.m1.1.1.2.cmml" xref="A5.F9.6.m1.1.1.2">Div</ci><ci id="A5.F9.6.m1.1.1.3.cmml" xref="A5.F9.6.m1.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F9.6.m1.1d">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>.</figcaption>
</figure>
<figure id="A5.F10" class="ltx_figure">
<table id="A5.F10.4" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.F10.1.1" class="ltx_tr">
<td id="A5.F10.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.F10.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.1.1.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A5.F10.1.1.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Pegasus<span id="A5.F10.1.1.1.1.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F10.1.1.1.1.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="A5.F10.1.1.1.1.1.1.1.m1.2a"><msub id="A5.F10.1.1.1.1.1.1.1.m1.2.3" xref="A5.F10.1.1.1.1.1.1.1.m1.2.3.cmml"><mi id="A5.F10.1.1.1.1.1.1.1.m1.2.3.2" xref="A5.F10.1.1.1.1.1.1.1.m1.2.3.2.cmml">Div</mi><mrow id="A5.F10.1.1.1.1.1.1.1.m1.2.2.2.4" xref="A5.F10.1.1.1.1.1.1.1.m1.2.2.2.3.cmml"><mi id="A5.F10.1.1.1.1.1.1.1.m1.1.1.1.1" xref="A5.F10.1.1.1.1.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F10.1.1.1.1.1.1.1.m1.2.2.2.4.1" xref="A5.F10.1.1.1.1.1.1.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.F10.1.1.1.1.1.1.1.m1.2.2.2.2" xref="A5.F10.1.1.1.1.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F10.1.1.1.1.1.1.1.m1.2b"><apply id="A5.F10.1.1.1.1.1.1.1.m1.2.3.cmml" xref="A5.F10.1.1.1.1.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F10.1.1.1.1.1.1.1.m1.2.3.1.cmml" xref="A5.F10.1.1.1.1.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F10.1.1.1.1.1.1.1.m1.2.3.2.cmml" xref="A5.F10.1.1.1.1.1.1.1.m1.2.3.2">Div</ci><list id="A5.F10.1.1.1.1.1.1.1.m1.2.2.2.3.cmml" xref="A5.F10.1.1.1.1.1.1.1.m1.2.2.2.4"><ci id="A5.F10.1.1.1.1.1.1.1.m1.1.1.1.1.cmml" xref="A5.F10.1.1.1.1.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F10.1.1.1.1.1.1.1.m1.2.2.2.2.cmml" xref="A5.F10.1.1.1.1.1.1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F10.1.1.1.1.1.1.1.m1.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F10.4.5.1" class="ltx_tr">
<td id="A5.F10.4.5.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.5.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.5.1.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomat over the use of forged Australian passports in the killing of Hamas detainee Mahmoud al-Mabhouh in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.6.2" class="ltx_tr">
<td id="A5.F10.4.6.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.6.2.1.1.1" class="ltx_p" style="width:426.8pt;">Israel has summoned the Australian ambassador to complain after the Australian government said forged passports used in the killing of a Hamas operative in Dubai belonged to Netanyahu’s foreign ministry.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.7.3" class="ltx_tr">
<td id="A5.F10.4.7.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.7.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.7.3.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has ordered Israel to withdraw an officer over the use of forged Australian passports used by the 2013 murder of a Lebanese opposition figure in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.8.4" class="ltx_tr">
<td id="A5.F10.4.8.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.8.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.8.4.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has expelled an Israeli diplomat over allegations that fake Australian passports were used to kill a Lebanese militant in Egypt two years ago.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.9.5" class="ltx_tr">
<td id="A5.F10.4.9.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.9.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.9.5.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to withdraw a diplomat over the use of forged Australian passports to kill a Hamas operative in January.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.10.6" class="ltx_tr">
<td id="A5.F10.4.10.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.10.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.10.6.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomat in a row over the authenticated use of forged Australian passports in last year’s killing of a Hamas figure in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.11.7" class="ltx_tr">
<td id="A5.F10.4.11.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.11.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.11.7.1.1.1" class="ltx_p" style="width:426.8pt;">Australia says it is expulsion an Israeli diplomat in protest over Israel’s alleged role in the killing of a Hamas militant in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.12.8" class="ltx_tr">
<td id="A5.F10.4.12.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.12.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.12.8.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has recalled a diplomat from Israel after accusing Berlin of fabricating false passports used in the assassination of a Hamas operative in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.13.9" class="ltx_tr">
<td id="A5.F10.4.13.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.13.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.13.9.1.1.1" class="ltx_p" style="width:426.8pt;">Israel has been asked to withdraw an official from Australia, accusing it of complicity in the falsification of Australian passports used in the killing of a Hamas operative in Dubai in January.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.14.10" class="ltx_tr">
<td id="A5.F10.4.14.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.14.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.14.10.1.1.1" class="ltx_p" style="width:426.8pt;">Israel has withdrawn one of its diplomats after Canberra said it concluded that Passport Bureau agents participated in an internal Mossad plot to kill a Hamas operative in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F10.2.2" class="ltx_tr">
<td id="A5.F10.2.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F10.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.2.2.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A5.F10.2.2.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame<span id="A5.F10.2.2.1.1.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F10.2.2.1.1.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="A5.F10.2.2.1.1.1.1.1.m1.2a"><msub id="A5.F10.2.2.1.1.1.1.1.m1.2.3" xref="A5.F10.2.2.1.1.1.1.1.m1.2.3.cmml"><mi id="A5.F10.2.2.1.1.1.1.1.m1.2.3.2" xref="A5.F10.2.2.1.1.1.1.1.m1.2.3.2.cmml">Div</mi><mrow id="A5.F10.2.2.1.1.1.1.1.m1.2.2.2.4" xref="A5.F10.2.2.1.1.1.1.1.m1.2.2.2.3.cmml"><mi id="A5.F10.2.2.1.1.1.1.1.m1.1.1.1.1" xref="A5.F10.2.2.1.1.1.1.1.m1.1.1.1.1.cmml">top</mi><mo id="A5.F10.2.2.1.1.1.1.1.m1.2.2.2.4.1" xref="A5.F10.2.2.1.1.1.1.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.F10.2.2.1.1.1.1.1.m1.2.2.2.2" xref="A5.F10.2.2.1.1.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F10.2.2.1.1.1.1.1.m1.2b"><apply id="A5.F10.2.2.1.1.1.1.1.m1.2.3.cmml" xref="A5.F10.2.2.1.1.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F10.2.2.1.1.1.1.1.m1.2.3.1.cmml" xref="A5.F10.2.2.1.1.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F10.2.2.1.1.1.1.1.m1.2.3.2.cmml" xref="A5.F10.2.2.1.1.1.1.1.m1.2.3.2">Div</ci><list id="A5.F10.2.2.1.1.1.1.1.m1.2.2.2.3.cmml" xref="A5.F10.2.2.1.1.1.1.1.m1.2.2.2.4"><ci id="A5.F10.2.2.1.1.1.1.1.m1.1.1.1.1.cmml" xref="A5.F10.2.2.1.1.1.1.1.m1.1.1.1.1">top</ci><ci id="A5.F10.2.2.1.1.1.1.1.m1.2.2.2.2.cmml" xref="A5.F10.2.2.1.1.1.1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F10.2.2.1.1.1.1.1.m1.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F10.4.15.11" class="ltx_tr">
<td id="A5.F10.4.15.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.15.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.15.11.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomat after it concluded somebody close to Israel’s security agency, Mossad, owned forged passports which were used to abduct a Hamas rocket maker.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.16.12" class="ltx_tr">
<td id="A5.F10.4.16.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.16.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.16.12.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomat over allegations that its intelligence agency Mossad was behind the use of forged passports in the killing of a suspected Palestinian militant.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.17.13" class="ltx_tr">
<td id="A5.F10.4.17.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.17.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.17.13.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomat amid accusations Israel-run Mossad used forged Australian passports in the killing of a Hamas militant.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.18.14" class="ltx_tr">
<td id="A5.F10.4.18.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.18.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.18.14.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomat in a dispute over the use of stolen Australian passports for a hit in the Dubai killing of a Lebanese militant earlier this year.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.19.15" class="ltx_tr">
<td id="A5.F10.4.19.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.19.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.19.15.1.1.1" class="ltx_p" style="width:426.8pt;">An Israeli diplomat has been expelled from Australia after a Sydney police team concluded that agents from the country’s security agency Mossad took part in the poisoning of Egypt’s president.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.20.16" class="ltx_tr">
<td id="A5.F10.4.20.16.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.20.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.20.16.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has expelled an Israeli diplomat, after it concluded that his desk was responsible for the issuance of forged Australian passports used in the killing of a Hamas militant.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.21.17" class="ltx_tr">
<td id="A5.F10.4.21.17.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.21.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.21.17.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has recalled her envoy from Israel, after finding that an Israeli diplomat was responsible for the counterfeiting of passports used by the Unesco agency director who was killed in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.22.18" class="ltx_tr">
<td id="A5.F10.4.22.18.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.22.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.22.18.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has asked Israel to withdraw from its Embassy in Melbourne after accusing it of using forged Australian passports to fund the killing of a Palestinian militant.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.23.19" class="ltx_tr">
<td id="A5.F10.4.23.19.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.23.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.23.19.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has asked Israel to withdraw its ambassador for failing to acknowledge its role in the use of forged Australian passports in the killing of a British businessman.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.24.20" class="ltx_tr">
<td id="A5.F10.4.24.20.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.24.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.24.20.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has formally demanded the removal of anIsrael diplomat in response to a decision to accuse the Jewish organisation Mossad of use of forged Australian passports mentioned in a Dubai bombing plot.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.4" class="ltx_tr">
<td id="A5.F10.4.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F10.4.4.2.2" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.4.2.2.2" class="ltx_p" style="width:426.8pt;"><span id="A5.F10.4.4.2.2.2.2" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame<span id="A5.F10.4.4.2.2.2.2.2" class="ltx_text ltx_font_upright"> (<math id="A5.F10.3.3.1.1.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="A5.F10.3.3.1.1.1.1.1.m1.2a"><msub id="A5.F10.3.3.1.1.1.1.1.m1.2.3" xref="A5.F10.3.3.1.1.1.1.1.m1.2.3.cmml"><mi id="A5.F10.3.3.1.1.1.1.1.m1.2.3.2" xref="A5.F10.3.3.1.1.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F10.3.3.1.1.1.1.1.m1.2.2.2.4" xref="A5.F10.3.3.1.1.1.1.1.m1.2.2.2.3.cmml"><mi id="A5.F10.3.3.1.1.1.1.1.m1.1.1.1.1" xref="A5.F10.3.3.1.1.1.1.1.m1.1.1.1.1.cmml">sample</mi><mo id="A5.F10.3.3.1.1.1.1.1.m1.2.2.2.4.1" xref="A5.F10.3.3.1.1.1.1.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.F10.3.3.1.1.1.1.1.m1.2.2.2.2" xref="A5.F10.3.3.1.1.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F10.3.3.1.1.1.1.1.m1.2b"><apply id="A5.F10.3.3.1.1.1.1.1.m1.2.3.cmml" xref="A5.F10.3.3.1.1.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F10.3.3.1.1.1.1.1.m1.2.3.1.cmml" xref="A5.F10.3.3.1.1.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F10.3.3.1.1.1.1.1.m1.2.3.2.cmml" xref="A5.F10.3.3.1.1.1.1.1.m1.2.3.2">Focus</ci><list id="A5.F10.3.3.1.1.1.1.1.m1.2.2.2.3.cmml" xref="A5.F10.3.3.1.1.1.1.1.m1.2.2.2.4"><ci id="A5.F10.3.3.1.1.1.1.1.m1.1.1.1.1.cmml" xref="A5.F10.3.3.1.1.1.1.1.m1.1.1.1.1">sample</ci><ci id="A5.F10.3.3.1.1.1.1.1.m1.2.2.2.2.cmml" xref="A5.F10.3.3.1.1.1.1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F10.3.3.1.1.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, <math id="A5.F10.4.4.2.2.2.2.2.m2.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="A5.F10.4.4.2.2.2.2.2.m2.2a"><msub id="A5.F10.4.4.2.2.2.2.2.m2.2.3" xref="A5.F10.4.4.2.2.2.2.2.m2.2.3.cmml"><mi id="A5.F10.4.4.2.2.2.2.2.m2.2.3.2" xref="A5.F10.4.4.2.2.2.2.2.m2.2.3.2.cmml">Div</mi><mrow id="A5.F10.4.4.2.2.2.2.2.m2.2.2.2.4" xref="A5.F10.4.4.2.2.2.2.2.m2.2.2.2.3.cmml"><mi id="A5.F10.4.4.2.2.2.2.2.m2.1.1.1.1" xref="A5.F10.4.4.2.2.2.2.2.m2.1.1.1.1.cmml">top</mi><mo id="A5.F10.4.4.2.2.2.2.2.m2.2.2.2.4.1" xref="A5.F10.4.4.2.2.2.2.2.m2.2.2.2.3.cmml">,</mo><mi id="A5.F10.4.4.2.2.2.2.2.m2.2.2.2.2" xref="A5.F10.4.4.2.2.2.2.2.m2.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F10.4.4.2.2.2.2.2.m2.2b"><apply id="A5.F10.4.4.2.2.2.2.2.m2.2.3.cmml" xref="A5.F10.4.4.2.2.2.2.2.m2.2.3"><csymbol cd="ambiguous" id="A5.F10.4.4.2.2.2.2.2.m2.2.3.1.cmml" xref="A5.F10.4.4.2.2.2.2.2.m2.2.3">subscript</csymbol><ci id="A5.F10.4.4.2.2.2.2.2.m2.2.3.2.cmml" xref="A5.F10.4.4.2.2.2.2.2.m2.2.3.2">Div</ci><list id="A5.F10.4.4.2.2.2.2.2.m2.2.2.2.3.cmml" xref="A5.F10.4.4.2.2.2.2.2.m2.2.2.2.4"><ci id="A5.F10.4.4.2.2.2.2.2.m2.1.1.1.1.cmml" xref="A5.F10.4.4.2.2.2.2.2.m2.1.1.1.1">top</ci><ci id="A5.F10.4.4.2.2.2.2.2.m2.2.2.2.2.cmml" xref="A5.F10.4.4.2.2.2.2.2.m2.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F10.4.4.2.2.2.2.2.m2.2c">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F10.4.25.21" class="ltx_tr">
<td id="A5.F10.4.25.21.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.25.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.25.21.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli because “its anti-espionage agents” used visas from other nations to issue a Palestinian agent’s body £2.3m (£“2.3,1) car and land lift to Hezbollah in the killing of a senior Palestinian in the city:</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.26.22" class="ltx_tr">
<td id="A5.F10.4.26.22.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.26.22.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.26.22.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has ejected an Israeli at its embassy over the use of Australia’s visas in the killing of a terror attack in the city of D’scale.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.27.23" class="ltx_tr">
<td id="A5.F10.4.27.23.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.27.23.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.27.23.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomats following an investigation into the use of the country’s travel services as cards used in a terror attack.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.28.24" class="ltx_tr">
<td id="A5.F10.4.28.24.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.28.24.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.28.24.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli embassy transport staff after a police investigation found the country’s intelligence agency, the Israeli intelligence agency, was at responsible.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.29.25" class="ltx_tr">
<td id="A5.F10.4.29.25.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.29.25.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.29.25.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli embassy gathering after it said the country was responsible for the use of the use of Australia’s Australian emails in an emailed attack on a former Australian consulate in the Arab world.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.30.26" class="ltx_tr">
<td id="A5.F10.4.30.26.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.30.26.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.30.26.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomatic side after accusing it of using at first issued Australian DNA test cards to produce the Irish agent in the stepped-up Emirates bombing.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.31.27" class="ltx_tr">
<td id="A5.F10.4.31.27.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.31.27.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.31.27.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled Israeli diplomats after its foreign minister said the country had been to be responsible for the use of staged Australian emails by Israeli intelligence.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.32.28" class="ltx_tr">
<td id="A5.F10.4.32.28.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F10.4.32.28.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.32.28.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has told Israel to withdraw a diplomatic mission from its country - after it said it was “in no Petroleum to Finish” the killing of a Palestinian in the killing of a terror the network by the Israeli security agency, enzymes.</span>
</span>
</td>
</tr>
<tr id="A5.F10.4.33.29" class="ltx_tr">
<td id="A5.F10.4.33.29.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.F10.4.33.29.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F10.4.33.29.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has demanded Israel withdraw a diplomats following the Israel Security Service’s use of Israeli-issued Australian travel visas to help one of its agents commit a terror attack.</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>Diverse summaries predicted using <span id="A5.F10.9.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> and <span id="A5.F10.10.2" class="ltx_text ltx_font_smallcaps">PegFame</span> models with <math id="A5.F10.6.m1.2" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{top},k}" display="inline"><semantics id="A5.F10.6.m1.2b"><msub id="A5.F10.6.m1.2.3" xref="A5.F10.6.m1.2.3.cmml"><mi id="A5.F10.6.m1.2.3.2" xref="A5.F10.6.m1.2.3.2.cmml">Div</mi><mrow id="A5.F10.6.m1.2.2.2.4" xref="A5.F10.6.m1.2.2.2.3.cmml"><mi id="A5.F10.6.m1.1.1.1.1" xref="A5.F10.6.m1.1.1.1.1.cmml">top</mi><mo id="A5.F10.6.m1.2.2.2.4.1" xref="A5.F10.6.m1.2.2.2.3.cmml">,</mo><mi id="A5.F10.6.m1.2.2.2.2" xref="A5.F10.6.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F10.6.m1.2c"><apply id="A5.F10.6.m1.2.3.cmml" xref="A5.F10.6.m1.2.3"><csymbol cd="ambiguous" id="A5.F10.6.m1.2.3.1.cmml" xref="A5.F10.6.m1.2.3">subscript</csymbol><ci id="A5.F10.6.m1.2.3.2.cmml" xref="A5.F10.6.m1.2.3.2">Div</ci><list id="A5.F10.6.m1.2.2.2.3.cmml" xref="A5.F10.6.m1.2.2.2.4"><ci id="A5.F10.6.m1.1.1.1.1.cmml" xref="A5.F10.6.m1.1.1.1.1">top</ci><ci id="A5.F10.6.m1.2.2.2.2.cmml" xref="A5.F10.6.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F10.6.m1.2d">\mathrm{Div}_{\mathrm{top},k}</annotation></semantics></math>.</figcaption>
</figure>
<figure id="A5.F11" class="ltx_figure">
<table id="A5.F11.4" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.F11.1.1" class="ltx_tr">
<td id="A5.F11.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.F11.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.1.1.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A5.F11.1.1.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Pegasus<span id="A5.F11.1.1.1.1.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F11.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="A5.F11.1.1.1.1.1.1.1.m1.1a"><msub id="A5.F11.1.1.1.1.1.1.1.m1.1.1" xref="A5.F11.1.1.1.1.1.1.1.m1.1.1.cmml"><mi id="A5.F11.1.1.1.1.1.1.1.m1.1.1.2" xref="A5.F11.1.1.1.1.1.1.1.m1.1.1.2.cmml">Div</mi><mi id="A5.F11.1.1.1.1.1.1.1.m1.1.1.3" xref="A5.F11.1.1.1.1.1.1.1.m1.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="A5.F11.1.1.1.1.1.1.1.m1.1b"><apply id="A5.F11.1.1.1.1.1.1.1.m1.1.1.cmml" xref="A5.F11.1.1.1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.F11.1.1.1.1.1.1.1.m1.1.1.1.cmml" xref="A5.F11.1.1.1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="A5.F11.1.1.1.1.1.1.1.m1.1.1.2.cmml" xref="A5.F11.1.1.1.1.1.1.1.m1.1.1.2">Div</ci><ci id="A5.F11.1.1.1.1.1.1.1.m1.1.1.3.cmml" xref="A5.F11.1.1.1.1.1.1.1.m1.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F11.1.1.1.1.1.1.1.m1.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F11.4.5.1" class="ltx_tr">
<td id="A5.F11.4.5.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.5.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.5.1.1.1.1" class="ltx_p" style="width:426.8pt;">Israel hasracuse withdrawn an envoy after the Australian government said it concluded that Israeli agents used forged passports used to kill a Dubai Bendigo businessman.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.6.2" class="ltx_tr">
<td id="A5.F11.4.6.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.6.2.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has demanded the withdrawal of an Israeli diplomat, saying his arrival in Canberra was only necessary to deal with a spillover from the killing of a Hamas militant in Dubai in January.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.7.3" class="ltx_tr">
<td id="A5.F11.4.7.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.7.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.7.3.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has recalled an Israeli diplomat over accusation that fake Australian passports used 436 kilometres (300 miles) from Canberra in the death of a Hamas militant were stolen by Israeli agents.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.8.4" class="ltx_tr">
<td id="A5.F11.4.8.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.8.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.8.4.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has recalled aLatis from Israel for having strong evidence that their embassy was used to counterfeite passports used in the killing of a bidder in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.9.5" class="ltx_tr">
<td id="A5.F11.4.9.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.9.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.9.5.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomat in a row about the use429 Australian passportsocally used in the killing of an intends in Dubai. slew of passports were used cosmetics in the killing.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.10.6" class="ltx_tr">
<td id="A5.F11.4.10.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.10.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.10.6.1.1.1" class="ltx_p" style="width:426.8pt;">Australia is seeking to expel an Israeli envoyrolet over the use of forged Australian passports in the murder of a militant in Dubaiselection.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.11.7" class="ltx_tr">
<td id="A5.F11.4.11.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.11.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.11.7.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomat after saying it was “certain” eagle-eyed undercover agents were Rhys Shapiro and Glenn Clift, who used forged Australian passports to kill an Soros geneticist in Dubai in 2015.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.12.8" class="ltx_tr">
<td id="A5.F11.4.12.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.12.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.12.8.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has removed the Israeli ambassador following a decision to conclude that forged Australian passports used in the death of a Palestinian Deals in the DesertDex were collaborated from Israel.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.13.9" class="ltx_tr">
<td id="A5.F11.4.13.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.13.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.13.9.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has recalled an Israeli diplomat, accusing Tel Aviv of “engaging in a pattern of alarming behaviour”, after concluding that forged Australian passports were used in the killing of a Hamas operative in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.14.10" class="ltx_tr">
<td id="A5.F11.4.14.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.14.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.14.10.1.1.1" class="ltx_p" style="width:426.8pt;">Israel has expelled one of its diplomats because of allegations that it helped Isabel al-Mabhouh, a British-based PalestinianORED to be kill in January, by using forged Australian passports.</span>
</span>
</td>
</tr>
<tr id="A5.F11.2.2" class="ltx_tr">
<td id="A5.F11.2.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F11.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.2.2.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A5.F11.2.2.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame<span id="A5.F11.2.2.1.1.1.1.1" class="ltx_text ltx_font_upright"> (<math id="A5.F11.2.2.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="A5.F11.2.2.1.1.1.1.1.m1.1a"><msub id="A5.F11.2.2.1.1.1.1.1.m1.1.1" xref="A5.F11.2.2.1.1.1.1.1.m1.1.1.cmml"><mi id="A5.F11.2.2.1.1.1.1.1.m1.1.1.2" xref="A5.F11.2.2.1.1.1.1.1.m1.1.1.2.cmml">Div</mi><mi id="A5.F11.2.2.1.1.1.1.1.m1.1.1.3" xref="A5.F11.2.2.1.1.1.1.1.m1.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="A5.F11.2.2.1.1.1.1.1.m1.1b"><apply id="A5.F11.2.2.1.1.1.1.1.m1.1.1.cmml" xref="A5.F11.2.2.1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.F11.2.2.1.1.1.1.1.m1.1.1.1.cmml" xref="A5.F11.2.2.1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="A5.F11.2.2.1.1.1.1.1.m1.1.1.2.cmml" xref="A5.F11.2.2.1.1.1.1.1.m1.1.1.2">Div</ci><ci id="A5.F11.2.2.1.1.1.1.1.m1.1.1.3.cmml" xref="A5.F11.2.2.1.1.1.1.1.m1.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F11.2.2.1.1.1.1.1.m1.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F11.4.15.11" class="ltx_tr">
<td id="A5.F11.4.15.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.15.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.15.11.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has summoned Idair Kernatic, a Jerusalem consulate official, inv summoned after the extraction of a document touting the use of forged passports for a deadly bomb plot.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.16.12" class="ltx_tr">
<td id="A5.F11.4.16.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.16.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.16.12.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has recalled a diplomat from Israel, claiming Israel stole the original identities of passports used to kill a Hamas operative.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.17.13" class="ltx_tr">
<td id="A5.F11.4.17.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.17.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.17.13.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel Fever to withdraw a diplomat after New Zealand said Israeli agents used the fake local passports used to identify a keyoine Killer.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.18.14" class="ltx_tr">
<td id="A5.F11.4.18.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.18.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.18.14.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has asked Israel to withdraw a diplomat after claiming the Jewish terror group Mossad used forged Australian passports in a plot to murder a Dubai imam.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.19.15" class="ltx_tr">
<td id="A5.F11.4.19.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.19.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.19.15.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomat after confirming fake Australian passports were used to help the killing ofmagazine boss Mahmoud al-Mabhouh in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.20.16" class="ltx_tr">
<td id="A5.F11.4.20.16.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.20.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.20.16.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has withdrawn a military characteristic of Israel after alleging its officials were behind the use of stolen Australian passportsxiety in a Dubai cash-in-transit plot.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.21.17" class="ltx_tr">
<td id="A5.F11.4.21.17.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.21.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.21.17.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expel an Israeli diplomat over allegations that the country’s Mossad spy was behind at least Jong-Bam’s Becket murder.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.22.18" class="ltx_tr">
<td id="A5.F11.4.22.18.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.22.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.22.18.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has withdrawn an Israeli diplomat halves its embassy in Canberra over accusations the country’s security service, Mossad, was responsible for issuing forged Australian passports.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.23.19" class="ltx_tr">
<td id="A5.F11.4.23.19.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.23.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.23.19.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to withdraw one of its diplomats from Canberra after finding that phony Australian passports were used to kill an Egyptian cleric.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.24.20" class="ltx_tr">
<td id="A5.F11.4.24.20.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.24.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.24.20.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has asked Israel to withdraw a diplomat after it said Israel was behinduse parts of forged Australian passports used in the bombing of a kayaker in Dubai.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.4" class="ltx_tr">
<td id="A5.F11.4.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A5.F11.4.4.2.2" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.4.2.2.2" class="ltx_p" style="width:426.8pt;"><span id="A5.F11.4.4.2.2.2.2" class="ltx_text ltx_font_bold ltx_font_smallcaps">PegFame<span id="A5.F11.4.4.2.2.2.2.2" class="ltx_text ltx_font_upright"> (<math id="A5.F11.3.3.1.1.1.1.1.m1.2" class="ltx_Math" alttext="\mathrm{Focus}_{\mathrm{sample},k}" display="inline"><semantics id="A5.F11.3.3.1.1.1.1.1.m1.2a"><msub id="A5.F11.3.3.1.1.1.1.1.m1.2.3" xref="A5.F11.3.3.1.1.1.1.1.m1.2.3.cmml"><mi id="A5.F11.3.3.1.1.1.1.1.m1.2.3.2" xref="A5.F11.3.3.1.1.1.1.1.m1.2.3.2.cmml">Focus</mi><mrow id="A5.F11.3.3.1.1.1.1.1.m1.2.2.2.4" xref="A5.F11.3.3.1.1.1.1.1.m1.2.2.2.3.cmml"><mi id="A5.F11.3.3.1.1.1.1.1.m1.1.1.1.1" xref="A5.F11.3.3.1.1.1.1.1.m1.1.1.1.1.cmml">sample</mi><mo id="A5.F11.3.3.1.1.1.1.1.m1.2.2.2.4.1" xref="A5.F11.3.3.1.1.1.1.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.F11.3.3.1.1.1.1.1.m1.2.2.2.2" xref="A5.F11.3.3.1.1.1.1.1.m1.2.2.2.2.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F11.3.3.1.1.1.1.1.m1.2b"><apply id="A5.F11.3.3.1.1.1.1.1.m1.2.3.cmml" xref="A5.F11.3.3.1.1.1.1.1.m1.2.3"><csymbol cd="ambiguous" id="A5.F11.3.3.1.1.1.1.1.m1.2.3.1.cmml" xref="A5.F11.3.3.1.1.1.1.1.m1.2.3">subscript</csymbol><ci id="A5.F11.3.3.1.1.1.1.1.m1.2.3.2.cmml" xref="A5.F11.3.3.1.1.1.1.1.m1.2.3.2">Focus</ci><list id="A5.F11.3.3.1.1.1.1.1.m1.2.2.2.3.cmml" xref="A5.F11.3.3.1.1.1.1.1.m1.2.2.2.4"><ci id="A5.F11.3.3.1.1.1.1.1.m1.1.1.1.1.cmml" xref="A5.F11.3.3.1.1.1.1.1.m1.1.1.1.1">sample</ci><ci id="A5.F11.3.3.1.1.1.1.1.m1.2.2.2.2.cmml" xref="A5.F11.3.3.1.1.1.1.1.m1.2.2.2.2">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F11.3.3.1.1.1.1.1.m1.2c">\mathrm{Focus}_{\mathrm{sample},k}</annotation></semantics></math>, <math id="A5.F11.4.4.2.2.2.2.2.m2.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="A5.F11.4.4.2.2.2.2.2.m2.1a"><msub id="A5.F11.4.4.2.2.2.2.2.m2.1.1" xref="A5.F11.4.4.2.2.2.2.2.m2.1.1.cmml"><mi id="A5.F11.4.4.2.2.2.2.2.m2.1.1.2" xref="A5.F11.4.4.2.2.2.2.2.m2.1.1.2.cmml">Div</mi><mi id="A5.F11.4.4.2.2.2.2.2.m2.1.1.3" xref="A5.F11.4.4.2.2.2.2.2.m2.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="A5.F11.4.4.2.2.2.2.2.m2.1b"><apply id="A5.F11.4.4.2.2.2.2.2.m2.1.1.cmml" xref="A5.F11.4.4.2.2.2.2.2.m2.1.1"><csymbol cd="ambiguous" id="A5.F11.4.4.2.2.2.2.2.m2.1.1.1.cmml" xref="A5.F11.4.4.2.2.2.2.2.m2.1.1">subscript</csymbol><ci id="A5.F11.4.4.2.2.2.2.2.m2.1.1.2.cmml" xref="A5.F11.4.4.2.2.2.2.2.m2.1.1.2">Div</ci><ci id="A5.F11.4.4.2.2.2.2.2.m2.1.1.3.cmml" xref="A5.F11.4.4.2.2.2.2.2.m2.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F11.4.4.2.2.2.2.2.m2.1c">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>)</span></span></span>
</span>
</td>
</tr>
<tr id="A5.F11.4.25.21" class="ltx_tr">
<td id="A5.F11.4.25.21.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.25.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.25.21.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli government agent after accusing it of using the use of Australian travel visas to help Israel’s intelligence agency, arrest a Palestinian in a drug operation.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.26.22" class="ltx_tr">
<td id="A5.F11.4.26.22.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.26.22.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.26.22.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli pulled over the use of Australian espionage proteins in a terror attack.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.27.23" class="ltx_tr">
<td id="A5.F11.4.27.23.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.27.23.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.27.23.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government says Israel should withdraw a senior police mission from its embassy following an investigation into the use of Australian gel-making equipment in the killing of a Palestinian in a car bombing in a Emirates airport.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.28.24" class="ltx_tr">
<td id="A5.F11.4.28.24.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.28.24.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.28.24.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomats for its support for a Palestinian that was used to hack the email messages of the former head of the intelligence agency, reasoning that the expulsion was “in the best security” of the two countries.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.29.25" class="ltx_tr">
<td id="A5.F11.4.29.25.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.29.25.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.29.25.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled Israel’s second in service special operations, after accusing the country’s intelligence agency, theahl, of a “poisoning”.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.30.26" class="ltx_tr">
<td id="A5.F11.4.30.26.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.30.26.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.30.26.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli government in protest at “the use” of an Australian denied diplomatic entry in a diplomatic killing in the Arab city of controversives.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.31.27" class="ltx_tr">
<td id="A5.F11.4.31.27.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.31.27.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.31.27.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli posting at its embassy in a “diplomatic action”, after it was found that Israeli agents had used issued Australian visas in the killing of an Egyptian man.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.32.28" class="ltx_tr">
<td id="A5.F11.4.32.28.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.32.28.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.32.28.1.1.1" class="ltx_p" style="width:426.8pt;">Australia has expelled an Israeli diplomatic, accusing it of using a home-grown Palestinian with a crime on the plane he was using to be arrested in the West.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.33.29" class="ltx_tr">
<td id="A5.F11.4.33.29.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.F11.4.33.29.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.33.29.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has expel an Israeli diplomatic team, following a public investigation into the use of Australian visas to help the killing of a Palestinian in the Emirates.</span>
</span>
</td>
</tr>
<tr id="A5.F11.4.34.30" class="ltx_tr">
<td id="A5.F11.4.34.30.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.F11.4.34.30.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.F11.4.34.30.1.1.1" class="ltx_p" style="width:426.8pt;">The Australian government has expelled an Israeli embassy consulate in Australia after saying it was “left in no suggestions” it was responsible for the use of Australian terror attack credentials in the killing of a Palestinian in the desert.</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>Diverse summaries predicted using <span id="A5.F11.9.1" class="ltx_text ltx_font_smallcaps">Pegasus</span> and <span id="A5.F11.10.2" class="ltx_text ltx_font_smallcaps">PegFame</span> models with <math id="A5.F11.6.m1.1" class="ltx_Math" alttext="\mathrm{Div}_{\mathrm{nucleus}}" display="inline"><semantics id="A5.F11.6.m1.1b"><msub id="A5.F11.6.m1.1.1" xref="A5.F11.6.m1.1.1.cmml"><mi id="A5.F11.6.m1.1.1.2" xref="A5.F11.6.m1.1.1.2.cmml">Div</mi><mi id="A5.F11.6.m1.1.1.3" xref="A5.F11.6.m1.1.1.3.cmml">nucleus</mi></msub><annotation-xml encoding="MathML-Content" id="A5.F11.6.m1.1c"><apply id="A5.F11.6.m1.1.1.cmml" xref="A5.F11.6.m1.1.1"><csymbol cd="ambiguous" id="A5.F11.6.m1.1.1.1.cmml" xref="A5.F11.6.m1.1.1">subscript</csymbol><ci id="A5.F11.6.m1.1.1.2.cmml" xref="A5.F11.6.m1.1.1.2">Div</ci><ci id="A5.F11.6.m1.1.1.3.cmml" xref="A5.F11.6.m1.1.1.3">nucleus</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F11.6.m1.1d">\mathrm{Div}_{\mathrm{nucleus}}</annotation></semantics></math>.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2105.11920" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2105.11921" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2105.11921">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2105.11921" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2105.11922" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Thu Mar  7 00:22:55 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2105.11921
Professional summaries are written with document-level information, such as the theme of the document, in mind. This is in contrast with most seq2seq decoders which simultaneously learn to focus on salient content, while deciding what to generate, at each decoding step. With the motivation to narrow this gap, we introduce Focus Attention Mechanism, a simple yet effective method to encourage decoders to proactively generate tokens that are similar or topical to the input document. Further, we propose a Focus Sampling method to enable generation of diverse summaries, an area currently understudied in summarization. When evaluated on the BBC extreme summarization task, two state-of-the-art models augmented with Focus Attention generate summaries that are closer to the target and more faithful to their input documents,
outperforming their vanilla counterparts onrougeand multiple faithfulness measures.
We also empirically demonstrate that Focus Sampling is more effective in generating diverse and faithful summaries than top-or nucleus sampling-based decoding methods.
abstract
Focus Attention: Promoting Faithfulness and Diversity in Summarization
Abstract
2105.11921
1IntroductionDocument summarization — producing the shorter version of a document while preserving salient information— is challenging even for humans. Today, systems can generate summaries with a high level of fluency and coherence. This is due to recent advances such as sequence-to-sequence architectures (seq2seq)  with attention and copy mechanism, fully attention-based Transformer architectures, and large pretrained language models.AGold: Australia has expelled an Israeli diplomat saying Israel was behind the forging of Australian passports linked to the murder of a Hamas operative in Dubai.Pegasus: Australia has expelled an Israeli diplomat after concluding that forged Australian passports used in the killing of a Hamas militant in Dubai were issued by Israel.OurPegFamemodel: The Australian government has expelled an Israeli diplomat over the use of forged Australian passports in the killing of a Hamas militant in Dubai.BPegasuswith Top-SamplingIsrael has summoned the Australian ambassador to complainafter the Australian government said forged passports used in the killing of a Hamas operative in Dubai belonged to Netanyahu’s foreign ministry.The Australian government has ordered Israel to withdraw an officer over the use of forged Australian passports usedby the 2013 murder of a Lebanese opposition figure in Dubai.Pegasuswith Nucleus SamplingIsrael hasracuse withdrawn an envoy afterthe Australian government said it concluded that Israeli agents used forged passports used to kill a DubaiBendigo businessman.The Australian government has recalled an Israeli diplomat over accusation that fake Australian passportsused 436 kilometres (300 miles)  from Canberrain the death of a Hamas militantwere stolen by Israeli agents.COurPegFamemodel with novel Focus SamplingAustralia has expelled an Israeli diplomatic staff after accusing the country’s security agency, the Israeli military’s intelligence agency, of being responsible for the use of Australian visas used in the killing of a Palestinian.The Australian government has expelled an Israeli diplomatic staff after it said the country was responsible for the use of Australian visas used in the killing of a Palestinian in the Middle East.Figure 1:Block A shows the best predictions fromPegasusand ourPegFame(PegasuswithFame)  model, along with theGoldsummary for anXSumarticle. Block B presents diverse summaries generated fromPegasususing top-and nucleus sampling. Block C shows diverse summaries generated using ourPegFamemodel with Focus sampling. The text inorangeis not supported by the input article.However, in terms of summary quality, many challenges remain. For example, generating summaries that are faithful to the input is an unsolved problem.
Furthermore, there can be multiple equally good summaries per source document. Neural generation models fail to account for this and
tend to generate outputs with low diversity due to standard likelihood training, approximate decoding objectives, and lack of high quality multi-reference datasets. Not much attention has been given to generation of diverse, yet faithful summaries – two goals are often challenging to achieve simultaneously; a model can produce diverse outputs through sampling, but at the cost of quality.In this paper we introduce a Focus Attention MEchanism (orFame)  to transformer-based seq2seq architectures.Fameis inspired by how humans write summaries. Specifically,Fameaims to perform source-side planning to focus the summary on supported and topical content.Fameachieves this through a novel technique which augments standard contextual representations with a dynamic source-conditioned vocabulary biasing layer.
We present the following experimental findings:Famepromotes summaries faithful to the sourceWhen evaluated on the BBC extreme summarization task (XSum;,) , experiments with two state-of-the-art summarizers –RobertaS2SandPegasus–
show that both models generate summaries that are more faithful to their input documents when augmented withFame, in comparison with their vanilla counterparts.111In the paper we focus on assessingFameonXSum. But other summarization and text editing results can be found in AppendixBandC.Faithfulness is measured through a variety of previously proposed metrics. In addition, we leverage the manually annotated document-summary pairs for faithfulness fromand train a scorer which serves as an efficient proxy for expensive human evaluations. We call this metricBERTFaithful.Fameenables diverse summariesFame, by design, supportsFocus Sampling– a technique that is more effective in sampling topically relevant tokens to generate diverse, yet topically consistent and faithful outputs, than other sampling methods. Figure1illustrates how focus sampling generates better summaries than other sampling methods. We demonstrate the effectiveness of our new Focus Sampling technique using a variety of existing diversity and faithfulness measures. Empirically, we find that optimizing for high diversity often comes at the cost of faithfulness. ThusFameprovides a mechanism for trading-off high faithfulness with better diversity in summarization.
introduction
Focus Attention: Promoting Faithfulness and Diversity in Summarization
1Introduction
2105.11921
2Related WorkTask-Specific Architectural PriorsSeveral works enhance seq2seq architectures with task-specific priors. Pointer-generator style modelscan accurately generate mostly extractive summaries by copying words from the source text via pointing. Text editing modelscast text generation as a sequence tagging problem with carefully selected edit operations required for the task. Others focus on improving content selection to better constrain the model to likely input phrasesor by improving the representation of relevant input tokens. Instead of directly modeling such priors,Famelearns the theme of the document through dynamic vocabulary biasing. Thus,Famecan be seen as a generalization of Pointer-generator or text-editing models via soft vocabulary learning. In fact, ourFamemodels achieve state-of-the-art on text-editing tasks (AppendixC) .Topic-Aware Generation ModelsThe idea of capturing document-level semantic information has been widely explored in the summarization community.use WordNetto model a text’s content relative to a topic based on lexical chains.propose to learn topic signatures for summarizing documents. Recently, document-level topic information has been used for improving neural language models, neural response generators, and not surprisingly, neural summarizers. Both,and, use a pretrained Latent Dirichlet Allocation (LDA;,)  model, whereas,use Poisson factor analysis, to synthesize topic vectors for the input. Instead, we dynamically learn a target-induced topic distribution for the input under the assumption that the human-written summary is a good proxy for the input document.Faithful Generation Modelsforce faithful generation by conditioning on both source text and extracted fact descriptions from the source text.propose to jointly generate a sentence and its syntactic dependency parse to induce grammaticality and faithfulness.learn a confidence score to ensure that the model attends to the source whenever necessary.introduce new input-output matching and embedding similarity losses to alleviate hallucination issues.
Yet, the task of generating text that is consistent with the input remains an open problem.Diverse Generation ModelsThere has been a surge of interest in making language models generate more diverse and human-like outputs.anddiversify beam search, using a task-specific scoring function, or constrain beam hypotheses to be sufficiently different. Others avoid text degeneration by truncating the unreliable tail of the probability distribution at each decoding step, either by sampling from the top-tokens (Top-Sampling;,)  or by sampling from a dynamic nucleus of tokens with the bulk of the probability mass (Nucleus Sampling;,) . Others modify the training objective to make the distribution sparseor assign lower probability to unlikely generations.For conditional text generation, most work focuses on generating diverse questionsor paraphrases. Following,use a mixture of experts to sample different binary masks on the source sequence for diverse content selection for summarization.Our focus sampling is similar to top-and nucleus sampling methods; in that it truncates the tail of the probability distribution. However, instead of truncating it at each decoding step, it biases the decoder proactively to generate output from a set of tokens which are topically-relevant to the input.Figure 2:A Transformer-based encoder-decoder architecture withFame.
body
Focus Attention: Promoting Faithfulness and Diversity in Summarization
2Related Work
2105.11921
3Summarization with Focus AttentionGiven an input document, we aim to generate its summary, whereandare input and output sequence lengths. We address this problem using seq2seq architectures with Transformer encoder and decoder, augmented withFame, as depicted in Figure2.Famelearns a distributionfor each input tokenover the vocabulary, measuring similarity of(in context)  to the tokens in the vocabulary. The vocabulary distributions,, for allare combined to form a dynamic vocabulary bias that is added to the decoder logits. This mechanism enhances the conditioning on the input source and encourages the decoder to generate tokens that are topically similar to the input.Transformer-based seq2seq ModelThe encoder uses BERT Transformer layers with multi-headed self-attention to encodeto a vector sequence, with, whereis the size of hidden representation. The decoder uses an identical architecture, except that
at decoding step, layeradds a conditional representationfor the tokenby attending to the output representationgenerated so far through self-attention and by attending to the input contextual representationthrough encoder-decoder attention. The probability of predicting the next tokenfrom a vocabularyis:(1) where,is the representation from the final decoder layer,the embedding matrix andthe model parameters. Parameters are trained by minimizing cross-entropy at each decoding step:(2) where,is the human-written summary.Focus Attention MEchansim (Fame) It is challenging for a decoder to obtain all relevant information from the conditional representationto learn the vocabulary output logits such that predictionsare consistent with the input.
Other modeling factors, specifically the decoder language model, can overwhelm model predictions.Fame(Figure2)  addresses this by introducing a short-circuit from the source to the vocabulary output logits via a source-conditioned bias on vocabulary items.We take the encoder representationand learn aToken-level Vocabulary Distribution, for each tokenin the input sequence.measures the contextual similarity of the input tokento the tokens in the vocabulary;andare parameters of newly introduced dense layers,is the intermediate filter size. We define aSource-conditioned Vocabulary Distributionasas an average of token-level vocabulary distributions for tokens present in the input sequence, capturing the similarity ofto the tokens in the vocabulary.Letbe the encoder-decoder attention distribution over the source tokens for the output tokenand the final decoder layer. We useto produce a weighted sum of the token-level vocabulary distributions to compute a dynamic vocabulary bias, orFocus Biasat decoding step. We modify the probability of predicting the next tokenfrom a vocabularyas:(3) We call thisFocused Probability Distribution, and it modifies the output logits dynamically to put more focus on those tokens in the vocabulary which are similar to the attended tokens in. The focus bias introduces a human-inspired control to the model where we do not generate the output in a fully abstractive manner (as in Eq. equation1) , but we proactively generate output tokens that are similar to the input tokens (as in Eq. equation3) .Summary-induced Topic Focused DistributionWe aim to guide our focus biasto be a better representative of the topical content relevant for the task. We achieve this by using the human-written summaryas a proxy for the topical content of the input and impose the following prior on the source-conditioned vocabulary distribution:(4) (5) We further refine Eq. equation5by replacingwith, whereis a set ofmost frequent tokens in the vocabulary,222which are usually articles or other function words.to improve focus on
content words. Our final loss function is then(6) where,is an hyper parameter.333is set to 0.5 for all experiments.By enforcingto be a topic distribution for the input, we encourage the focus biasto promote topically relevant tokens, and subsequently generate topically consistent outputs. Importantly, our focus bias with target-induced topic distribution is task-agnostic and less vulnerable to reference divergence issues, and can learn any property embodied in the target relevant for the task. For example, depending on the task,can learn to favour input tokens (e.g., for mostly extractive summaries)  or new tokens (e.g., for mostly abstractive summaries) . This is in sharp contrast to models that introduce task-specific priors, e.g., the pointer-generator networkthat can copy words from the source text, but does not do well on extreme summarization which is highly abstractive in nature.Focus Sampling: Promoting Diversity in Faithful GenerationWe introduceFocus SamplingwithFameto construct a subsetby
samplingtokens from the topic distribution() .
Then, we modify Eq. equation3as(7) (8) For document summarization, the subsetwill capture topically salient tokens necessary to generate a summary;is always added toto ensure that the model has access to function words. By tuning the parameters of sampling, we can enforce the model to control the faithfulness or diversity of the outputs.Focus sampling has similarities to top-(;,)  and nucleus sampling (;,) ; in that they all aim to promote diversity. At each decoding step, the top-sampling diversifies the generation process by sampling a token from the toptokens in the final output distribution. Similarly, nucleus sampling samples from a dynamic nucleus of tokens containing the vast majority (with a cumulative probability)  of the probability distribution. Both top-and nucleus sampling shorten the tail of the output distribution at each decoding step, whereas focus sampling constrains the decoder to use a fixed and topically relevant vocabulary. Unlike the other two techniques,can also benefit from standard beam search decoding, leading to superior generation that is not only diverse, but also consistent with the input document.
body
Focus Attention: Promoting Faithfulness and Diversity in Summarization
3Summarization with Focus Attention
2105.11921
4Experimental SetupIn this section we present our experimental setup to assess the ability of ourFamemodels to generate faithful summaries and to demonstrate that focus sampling is more effective in generating diverse and faithful summaries than other sampling-based decoding methods.4.1Extreme SummarizationWe evaluateFamemodels on extreme document summarization (XSum;,) . TheXSumsummaries, are extreme in that the documents are summarized into single-sentence summaries. These summaries demonstrate a high level of abstractiveness, and generating them automatically requires document-level inference, abstraction, and paraphrasing. Due to their extreme nature,XSumsummaries are ideal to evaluateFamemodels’ ability to capture the theme of the document.444We further experiment with long-form story highlight generation (Cnn/Dm;,)  and two text editing tasks: Sentence Fusionand Sentence Splitting. Their results can be found in AppendixBandC. OurFamemodels achieve SOTA on both text-editing tasks.We use on the original cased version consisting of
204,045/11,332/11,334 training/validation/test document-summary pairs. During training, the input documents are truncated to 512 tokens. The length of the summaries
are limited to 64.4.2Pretrained Models withFameWe introduceFameto two popular seq2seq architectures: RoBERTa initialized seq2seq (RobertaS2S,,)  andPegasus. We referRobertaS2Smodels withFameasRobFameandPegasuswithFamewithPegFame.We experiment withRobertaS2S-Large with shared encoder and decoder; it has 24 layers, a hidden size of 1024, filter size of 4096, 16 attention heads, and a vocabulary with 50K sentence pieces.RobertaS2Shas around 455M parameters andRobFamehas an additional 8M parameters.The best-performingPegasusmodel fromis not directly comparable withRobertaS2S. It does not share the encoder and decoder, it only has 16 layers, a hidden size of 1024, filter size of 4096, 16 attention heads, with a total of 568M parameters, and it also uses a much larger vocabulary with 91K sentence pieces. Hence, we trained our ownPegasusmodel. We use the same architecture asRobertaS2Sand pretrain it on a mixture of C4and HugeNewsdatasets with the original objective of generating salient GAP-sentences.Our experiments focus on this newly trainedPegasusmodel which has same number of parameters and vocabulary asRobertaS2S. But in contrast toRobertaS2S, the encoder-decoder attention inPegasusis pretrained. This allows us to analyse how focus attention affects pretrained (Pegasus)  vs randomly-initialized (RobertaS2S)  encoder-decoder attentions.555See AppendixAfor implementation details and hyperparameter settings.ModelsLexical Overlap (w/ ref) Sem. Sim.Faithfulnessothersent.FeqaBERTFaithfulLen.Rep.() R1(P%) R1R2RLBERTSc.%conf.With doc.RobertaS2S41.4518.7933.9080.639.119.821.50.21621.224.271.1RobFame42.1519.6834.8180.841.321.222.70.22620.820.772.5Pegasus44.8522.2637.0381.743.624.527.00.26321.16.073.8PegFame45.3122.7537.4681.944.824.827.30.26920.85.374.3Table 1:Abstractive Summarization results onXSumtest set comparingFamemodels with their baselines. For all our models, we use standard beam decoding with a beam size of 4 to generate the single best summary for a document. Focus sampling is not used here. See Section4.3for details on the evaluation metrics reported. Best number for each metric isboldfaced.4.3Evaluation MetricsLexical OverlapWe reportrougeF1 scoresagainst reference summaries; in particular, we report onrouge-1androuge-2for informativeness androuge-lfor fluency.666We lowercased candidate and reference summaries and usedpyrougewith parameters “-a -c 95 -m -n 4 -w 1.2.”Semantic SimilarityWe reportBERTScorewhich computes the contextual similarity between a candidate and its reference summary.Faithfulnessrougeand BERTScore do not correlate well with faithfulness of the generated summaries. Human evaluation is traditionally considered as the gold standard for measuring faithfulness. But recent research has shown that even human evaluation has shortcomings. Moreover, it is prohibitively expensive. This has led to the proposal of meta-evaluation metrics for various generation tasks.We evaluateFamemodels on semantic inference metrics such as textual entailmentand question answering. In particular, we report the probability of a summary entailing (ent.)  its input documentand QA-basedFeqascores. For ent. scores, we train an entailment classifier by fine-tuning a BERT-Large pretrained modelon the Multi-NLI dataset. For Feqa, we use a fine-tuned BARTlanguage model for question generation to generate questions from the summaries, and a BERT-base model fine-tuned on SQuADto answer the generated questions with input document as context.777We used the Feqa code available here:https://github.com/esdurmus/feqa/.In addition toent.andFeqa, we train a scorer leveraging manually annotated document-summary pairs for faithfulness, as a surrogate for human evaluation and call this metricBERTFaithful.888A very similar scorer was used in the GEM benchmarkto identify and extract the subset with faithful reference summaries from the XSum dataset.In particular, we finetune a BERT-Base classifier on 500 manually annotated document and gold summary pairs for the XSum dataset fromto predict whether a summary is faithful to the input document or not.999Out of 500, 90% of the document-summary pairs were used for training and the rest 50 document-summary pairs were used for validation. We used the validation set to estimate Spearman’s correlation coefficients of different metrics with the human assessment for faithfulness. We found that both entailment scores (ent.)  andBERTFaithfulare moderately correlated with faithfulness with correlation coefficients of 0.4387 and 0.3889, respectively. As such, we believe that BERTFaithful works as an efficient proxy for expensive human evaluation for faithfulness for XSum summaries. More work is needed to understand if BERTFaithful generalizes to other datasets.We report the percentage of summaries that were faithful ()  and the model’s confidence to generate faithful summaries () ;is the total number of examples in the test set.DiversityWe report the number of times (out of) , a model is able to generate a completely new summary (Unique) , andDistinct-N, measuring the lexical diversity in the generated summaries. Distinct-N is estimated as the number of distinct-grams of orderdivided by the total number of-grams of the same order, in all generated summaries.Finally, we also report the average length of summaries (Len.) , repetition errors (Rep., estimated as the percentage of summaries with at least one repetition of rare or content words) , androuge-1precision against the input document (R1, P%) , to better understand their quality.
body
Focus Attention: Promoting Faithfulness and Diversity in Summarization
4Experimental Setup
2105.11921
5ResultsMetricsUniqueDist.-Nrougeent.BERTSc.123R1R2RLRobertaS2S() 9.982.525.057.733.612.026.521.876.9RobertaS2S() 9.994.130.162.232.411.425.619.775.7RobFame() 9.992.325.058.132.711.325.720.376.6RobFame() 9.994.130.763.231.310.624.718.075.4RobFame() 1.613.522.443.938.015.731.034.378.6RobFame(,) 9.992.120.351.831.810.224.724.375.4RobFame(,) 9.981.918.448.232.911.125.825.976.1Pegasus() 9.981.923.255.336.614.328.827.778.4Pegasus() 9.993.830.563.134.112.826.922.776.5PegFame() 9.981.923.255.536.714.529.028.578.5PegFame() 9.993.830.463.134.212.827.023.276.6PegFame() 2.772.416.534.237.515.430.333.677.9PegFame(,) 8.992.823.054.731.510.324.422.874.7PegFame(,) 9.982.620.850.932.511.025.324.875.3Table 2:Assessment of diversity, relevance and faithfulness with focus sampling on theXSumtest set.FameSummaries are More Fluent, Informative and Faithful.Table1presents results comparing ourFamemodels,RobFameandPegFame, against their counterpartsRobertaS2SandPegasus, respectively. BothFamemodels clearly outperform their vanilla counterparts in terms of generating summaries that are more fluent (see RL and Rep.) , more informative (see R1, R2 and BERTSc.)  and more faithful (see ent., Feqa and BERTFaithful) . Among all four models,PegFamesummaries are most fluent, informative and faithful.We further did pairwise comparisons for all measures in Table1and found that all differences are statistically significant except for BERTScore and faithfulness measures betweenPegasusandPegFame.101010All significance tests in this work are pairwise comparisons (one-way ANOVA with posthoc Tukey HSD tests;) .These assessments demonstrate thatFamemodels aid bothRobertaS2SandPegasusin generating fluent, faithful and relevant summaries, but are more effective inRobertaS2Sthan inPegasusfor extreme summarization.Generating Diverse and Faithful Summaries with Focus Sampling.Table2presents results assessing focus sampling () , top-sampling ()  and nucleus sampling () , for their abilities to generate diverse and faithful summaries. For, we choose. We followand chooseand the nucleus probability, forand, respectively. For, we decode with a beam size of 4. We also reportwithandto assess if they can benefit one-another.
In each setting we sample 10 summaries for each input document.
For all metrics, we report the average over all 10 samples.111111Feqa and BERTFaithful scores are dropped due to time constraints.Bothandalmost always generate a new summary. In comparisongenerates 1.61 and 2.77 unique summaries usingRobFameandPegFamemodels, respectively.tends to generate the most distinct unigrams, bigrams, and trigrams. Interestingly,summaries have a more diverse collection of unigrams than insummaries (3.5% vs 2.3% forRobFameand 2.4% vs 1.9% forPegFame) .The high diversity inandcomes at the cost of faithfulness; summaries generated with these sampling techniques have poor entailment scores., on the other hand, generates summaries which entail documents the most. It also has the highestrougescores across the board. Some of the generated examples can be seen in Figure1. More predictions from other models can be found in AppendixE. Augmentingandwithis not desirable because, though it increases diversity in terms of uniqueness and Distinct-scores, faithfulness suffers again.Comparing results in Table2to the results in Table1, it is clear that diversity comes at the cost of quality (e.g., RL/ent. scores forRobFameandRobFame-are 34.81/41.3 and 31.0/34.3, respectively) . However,is superior to bothandin generating better quality summaries.Figure 3:Top 40 sentence pieces and their logits from topic distributioninRobFameandPegFamefor theXSumarticle discussed in Figure1.Figure 4:rouge-1F1 scores ofRobFameandPegFamemodels with different top-vocabularies (Eq. equation8)  on theXSumtest set. Similar patters are observed forrouge-2androuge-lscores.Focus Attention and Sampling Work Differently inRobFameandPegFame.Since both encoder-decoder and focus attention parameters ofRobFameare randomly initialized, they learn to compliment each other and learn a peaky topic distribution. On the other hand, sincePegFame’s encoder-decoder attention is pre-trained, there is a push-pull effect between it and focus attention. This results in a smoother topic distribution, as seen in Figure3.121212This difference in topic distributions is consistent across the whole test set. We compute the peakiness score of a topic distribution as the slope of the line connecting logits of the top-1st token to the top-100th token. The average peakiness scores across theXSumtestset forRobFameandPegFameare 1.25 (51∘)  and 0.45 (24.3∘) , respectively.Although we see that both models’ token sets capture the target intent well, the peaky distribution ofRobFameenables more accurate predictions than that ofPegFame, in a controlled generation setting. A comparison is presented in Figure4where we show howrouge-1scores vary when we use only top-tokens fromfor generation.131313Additional results and model predictions for these experiments can be found in AppendixD.We observe thatRobFameconsistently outperformsPegFamewith the lower values of.ModelsR1R2RLLead16.301.6111.95PtGen29.709.2123.24ConvS2S31.8911.5425.75MMN32.0012.1026.00MASS39.7517.2431.95BART45.1422.2737.25Pegasus47.2124.5639.25RobertaS2S41.4518.7933.90RobFame(w/o Eq. equation5) 41.2718.8633.90RobFame42.1519.6834.81Oracle72.2242.2253.89Pegasus(ours) 44.8522.2637.03PegFame(w/o Eq. equation5) 44.5422.0036.83PegFame45.3122.7537.46Oracle82.3960.6169.19Table 3:Ablations and SOTA comparisons onXSumdataset. Theunderlined boldresults are from the best performing models from literature and theboldresults are the best performingFamemodels.Further, we observe thatRobFamegenerates fewer unique summaries (1.61 vs 2.77)  but has higher Distinct-N scores (3.5/22.4/43.9 vs 2.4/16.5/34.2)  thanPegFame, within Table2. This can be again be attributed to howFameworks differently inRobFameandPegFame. Whenis sampled fromRobFame’s peaky distribution, the beam search decoding often tends to generate similar summaries (leading to a lower Uniqueness score)  as the sampleds do not diverge by much from each other. But when it does diverge, the decoder tends to generate completely new summaries (leading to higher Distinct-N scores) .Currently, we setfor our focus sampling experiments following our observations in Figure4. Future work will focus on how to better leverage trade-off between diversity and faithfulness by
controlling the peakiness of the topic distribution.Ablations and SOTA ComparisonsWe emphasize thatFameor focus sampling does not aim to improve on state-of-the-results in terms ofrouge, but to generate more faithful or diverse summaries while maintaining their quality. For completeness, we compare ourRobFameandPegFamemodels to their ablations and other state-of-the-art models onXSumin Table3.We reportrougescores forFamein the ideal scenario (Oracle)  where it focuses on all the correct tokens in the input, i.e., the topic distributionis identical to the distribution observed in the reference summary.
These models generate summaries with very highrougescores when the model is given the correct tokens to focus on. The gap between theOracleandFamescores suggests that there is still a lot of work to be done in this space.
Focus attention without any topical supervision (models w/o Eq. equation5)  is not significantly better than the baselines. ButRobFameandPegFame(trained with joint supervision in Eq. equation6)  significantly outperformRobertaS2SandPegasus, respectively.Our best modelPegFameperforms better than PtGen,
ConvS2S, MMN,
MASSand BART, but worse when the originalPegasus. This can be expected as the number of parameters inPegFameis far less than that in the originalPegasus.
body
Focus Attention: Promoting Faithfulness and Diversity in Summarization
5Results
2105.11921
6ConclusionWe introducedFame, a new attention mechanism which dynamically biases the decoder to proactively generate tokens that are topically similar to the input.Fameenhances the faithfulness of existing state-of-the-art abstract summarization models while improving their overallrougescores. Finally, our newly introduced focus sampling technique is a better alternative to top-or nucleus sampling to generate diverse set of faithful summaries.
body
Focus Attention: Promoting Faithfulness and Diversity in Summarization
6Conclusion
2105.11921
AcknowledgementsWe thank Sebastian Gehrmann, Slav Petrov, the reviewers, and the action editor for their invaluable feedback.
body
Focus Attention: Promoting Faithfulness and Diversity in Summarization
Acknowledgements
2105.11921
Ethical ConsiderationsThe nature of text generation leads to multiple ethical considerations when applied to applications. The main failure mode is that the model can learn to mimic target properties in the training data that are not desirable.Faithfulness and FactualitySince models create new text, there is the danger that they may neither be faithful to the source material nor factual. This can be exacerbated when the data itself has highly abstractive targets, which require the model to generate words not seen in the source material during training. This often leads the model to generate content inconsistent with the source material.Trustworthy DataIf the data itself is not trustworthy (comes from suspect or malicious sources)  the model itself will naturally become untrustworthy as it will ultimately learn the language and topics of the training data. For instance, if the training data is about Obama birther conspiracies, and the model is asked to generate information about the early life of Obama, there is a risk that such false claims will be predicted by the model.Bias in DataSimilarly, biases in the data around gender, race, etc., risk being propagated in the model predictions, which is common for most NLP tasks. This is especially true when the models are trained from non-contemporary data that do not represent current norms and practices.The above considerations are non-malicious, in that the model is merely learning to behave as its underlying source material. If users of such models are not aware of these issues and do not account for them, e.g., with better data selection, evaluation, etc., then the generated text can be damaging.Generation models can also be misused in malicious ways. These include generating fake news, spam, and other text meant to mislead large parts of the general population.
conclusion
Focus Attention: Promoting Faithfulness and Diversity in Summarization
Ethical Considerations