<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[1704.04368] Get To The Point: Summarization with Pointer-Generator Networks</title><meta property="og:description" content="Neural sequence-to-sequence models have provided a viable new approach
for abstractive text summarization (meaning they are not restricted to simply selecting and rearranging passages from the original text).
However, …">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Get To The Point: Summarization with Pointer-Generator Networks">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Get To The Point: Summarization with Pointer-Generator Networks">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/1704.04368">

<!--Generated on Fri Mar  8 21:28:27 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Get To The Point: Summarization with Pointer-Generator Networks</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Abigail See 
<br class="ltx_break">Stanford University 
<br class="ltx_break"><span id="id1.1.id1" class="ltx_text ltx_font_typewriter">abisee@stanford.edu</span> 
<br class="ltx_break">&amp;Peter J. Liu 
<br class="ltx_break">Google Brain 
<br class="ltx_break"><span id="id2.2.id2" class="ltx_text ltx_font_typewriter">peterjliu@google.com</span> 
<br class="ltx_break">&amp;Christopher D. Manning
<br class="ltx_break">Stanford University 
<br class="ltx_break"><span id="id3.3.id3" class="ltx_text ltx_font_typewriter">manning@stanford.edu</span> 
<br class="ltx_break">
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id4.id1" class="ltx_p">Neural sequence-to-sequence models have provided a viable new approach
for <span id="id4.id1.1" class="ltx_text ltx_font_italic">abstractive</span> text summarization (meaning they are not restricted to simply selecting and rearranging passages from the original text).
However, these models have two shortcomings: they are liable to reproduce factual details inaccurately, and they tend to repeat themselves.
In this work we propose a novel architecture that augments the standard sequence-to-sequence attentional model in two orthogonal ways.
First, we use a hybrid pointer-generator network that can copy words from the source text via <span id="id4.id1.2" class="ltx_text ltx_font_italic">pointing</span>, which aids accurate reproduction of information, while retaining the ability to produce novel words through the <span id="id4.id1.3" class="ltx_text ltx_font_italic">generator</span>.
Second, we use <span id="id4.id1.4" class="ltx_text ltx_font_italic">coverage</span> to keep track of what has been summarized, which discourages repetition.
We apply our model to the <span id="id4.id1.5" class="ltx_text ltx_font_italic">CNN / Daily Mail</span> summarization
task, outperforming the current abstractive state-of-the-art
by at least 2 ROUGE points.</p>
</div>
<figure id="S0.F1" class="ltx_figure">
<p id="S0.F1.1" class="ltx_p">
<span id="S0.F1.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:426.8pt;">
<span id="S0.F1.1.1.1" class="ltx_p"><span id="S0.F1.1.1.1.1" class="ltx_text ltx_font_bold">Original Text (truncated):</span>
lagos, nigeria (cnn) a day after winning nigeria’s presidency, <span id="S0.F1.1.1.1.2" class="ltx_text ltx_font_bold ltx_font_italic" style="color:#4285F4;">muhammadu buhari</span> told cnn’s christiane amanpour that <span id="S0.F1.1.1.1.3" class="ltx_text ltx_font_bold" style="color:#4285F4;">he plans to aggressively fight corruption that has long plagued nigeria</span> and go after the root of the nation’s unrest. <span id="S0.F1.1.1.1.4" class="ltx_text ltx_font_italic">buhari</span> said he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring nations chad, cameroon and niger, <span id="S0.F1.1.1.1.5" class="ltx_text ltx_font_bold" style="color:#4285F4;">he said his administration is confident it will be able to thwart criminals</span> and others contributing to nigeria’s instability. for the first time in nigeria’s history, the opposition defeated the ruling party in democratic elections. <span id="S0.F1.1.1.1.6" class="ltx_text ltx_font_italic">buhari</span> defeated incumbent goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral commission. <span id="S0.F1.1.1.1.7" class="ltx_text ltx_font_bold" style="color:#4285F4;">the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s most populous nation.</span>

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:1.2pt;background:black;display:inline-block;"> </span>
<span id="S0.F1.1.1.1.8" class="ltx_text ltx_font_bold">Baseline Seq2Seq + Attention:</span>
<span id="S0.F1.1.1.1.9" class="ltx_text ltx_font_bold" style="color:#DB4437;">UNK UNK</span> says his administration is confident it will be able to <span id="S0.F1.1.1.1.10" class="ltx_text ltx_font_bold" style="color:#DB4437;">destabilize nigeria’s economy</span>. <span id="S0.F1.1.1.1.11" class="ltx_text ltx_font_bold" style="color:#DB4437;">UNK</span> says his administration is confident it will be able to thwart criminals and other <span id="S0.F1.1.1.1.12" class="ltx_text ltx_font_bold" style="color:#DB4437;">nigerians</span>. <span id="S0.F1.1.1.1.13" class="ltx_text ltx_font_bold" style="color:#DB4437;">he says the country has long nigeria and nigeria’s economy.</span>

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="S0.F1.1.1.1.14" class="ltx_text ltx_font_bold">Pointer-Gen:</span>
<span id="S0.F1.1.1.1.15" class="ltx_text ltx_font_italic">muhammadu buhari</span> says he plans to aggressively fight corruption <span id="S0.F1.1.1.1.16" class="ltx_text ltx_font_bold" style="color:#0F9D58;">in the northeast part of nigeria</span>. he says he’ll “rapidly give attention” to curbing violence <span id="S0.F1.1.1.1.17" class="ltx_text ltx_font_bold" style="color:#0F9D58;">in the northeast part of nigeria</span>.
he says his administration is confident it will be able to thwart criminals.

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="S0.F1.1.1.1.18" class="ltx_text ltx_font_bold">Pointer-Gen + Coverage:</span>
<span id="S0.F1.1.1.1.19" class="ltx_text ltx_font_italic">muhammadu buhari</span> says he plans to aggressively fight corruption that has long plagued nigeria. he says his administration is confident it will be able to thwart criminals. the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s most populous nation.</span>
</span></p>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>
Comparison of output of 3 abstractive summarization models on a news article.
The baseline model makes <span id="S0.F1.8.1" class="ltx_text ltx_font_bold" style="color:#DB4437;">factual errors</span>, a <span id="S0.F1.9.2" class="ltx_text ltx_font_bold" style="color:#DB4437;">nonsensical sentence</span> and struggles with OOV words <span id="S0.F1.10.3" class="ltx_text ltx_font_italic">muhammadu buhari</span>.
The pointer-generator model is accurate but <span id="S0.F1.11.4" class="ltx_text ltx_font_bold" style="color:#0F9D58;">repeats</span> <span id="S0.F1.12.5" class="ltx_text ltx_font_bold" style="color:#0F9D58;">itself</span>.
Coverage eliminates repetition.
The final summary is composed from <span id="S0.F1.13.6" class="ltx_text ltx_font_bold" style="color:#4285F4;">several fragments</span>.
</figcaption>
</figure>
<figure id="S0.F2" class="ltx_figure"><img src="/html/1704.04368/assets/x1.png" id="S0.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="313" height="125" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Baseline sequence-to-sequence model with attention. The model may attend to relevant words in the source text to generate novel words, e.g., to produce the novel word <span id="S0.F2.5.1" class="ltx_text ltx_font_italic">beat</span> in the abstractive summary <span id="S0.F2.6.2" class="ltx_text ltx_font_italic">Germany <span id="S0.F2.6.2.1" class="ltx_text ltx_font_bold">beat</span> Argentina 2-0</span> the model may attend to the words <span id="S0.F2.7.3" class="ltx_text ltx_font_italic">victorious</span> and <span id="S0.F2.8.4" class="ltx_text ltx_font_italic">win</span> in the source text.</figcaption>
</figure>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Summarization is the task of condensing a piece of text to a shorter version that contains the main information from the original.
There are two broad approaches to summarization: <em id="S1.p1.1.1" class="ltx_emph ltx_font_italic">extractive</em> and <em id="S1.p1.1.2" class="ltx_emph ltx_font_italic">abstractive</em>.
<span id="S1.p1.1.3" class="ltx_text ltx_font_italic">Extractive methods</span> assemble summaries exclusively from
passages (usually whole sentences) taken directly from the source
text, while <span id="S1.p1.1.4" class="ltx_text ltx_font_italic">abstractive methods</span> may generate novel words and
phrases not featured in the source text – as a human-written
abstract usually does.
The extractive approach is easier, because copying large chunks of text from the source document ensures baseline levels of grammaticality and accuracy.
On the other hand, sophisticated abilities that are crucial to high-quality summarization, such as paraphrasing, generalization, or the incorporation of real-world knowledge, are possible only in an abstractive framework (see Figure <a href="#S7.F5" title="Figure 5 ‣ 7 Discussion ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>).</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Due to the difficulty of abstractive summarization, the great majority of past work has been extractive <cite class="ltx_cite ltx_citemacro_cite">Kupiec et al. (<a href="#bib.bib12" title="" class="ltx_ref">1995</a>); Paice (<a href="#bib.bib20" title="" class="ltx_ref">1990</a>); Saggion and Poibeau (<a href="#bib.bib23" title="" class="ltx_ref">2013</a>)</cite>.
However, the recent success of <span id="S1.p2.1.1" class="ltx_text ltx_font_italic">sequence-to-sequence</span> models <cite class="ltx_cite ltx_citemacro_cite">Sutskever et al. (<a href="#bib.bib25" title="" class="ltx_ref">2014</a>)</cite>, in which recurrent neural networks (RNNs) both read and freely generate text, has made abstractive summarization viable <cite class="ltx_cite ltx_citemacro_cite">Chopra et al. (<a href="#bib.bib4" title="" class="ltx_ref">2016</a>); Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>); Rush et al. (<a href="#bib.bib22" title="" class="ltx_ref">2015</a>); Zeng et al. (<a href="#bib.bib31" title="" class="ltx_ref">2016</a>)</cite>.
Though these systems are promising, they exhibit undesirable behavior such as inaccurately reproducing factual details, an inability to deal with out-of-vocabulary (OOV) words, and repeating themselves (see Figure <a href="#S0.F1" title="Figure 1 ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this paper we present an architecture that addresses these three
issues in the context of multi-sentence summaries.
While most recent abstractive work has focused on headline generation
tasks (reducing one or two sentences to a single headline), we believe
that longer-text summarization is both more challenging (requiring
higher levels of abstraction while avoiding repetition) and ultimately more useful.
Therefore we apply our model to the recently-introduced
<span id="S1.p3.1.1" class="ltx_text ltx_font_italic">CNN/Daily Mail</span> dataset <cite class="ltx_cite ltx_citemacro_cite">Hermann et al. (<a href="#bib.bib9" title="" class="ltx_ref">2015</a>); Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite>, which contains news articles (39 sentences on average) paired with multi-sentence summaries, and show that we outperform the state-of-the-art abstractive system by at least 2 ROUGE points.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Our hybrid <span id="S1.p4.1.1" class="ltx_text ltx_font_italic">pointer-generator</span> network facilitates copying
words from the source text via <span id="S1.p4.1.2" class="ltx_text ltx_font_italic">pointing</span> <cite class="ltx_cite ltx_citemacro_cite">Vinyals et al. (<a href="#bib.bib29" title="" class="ltx_ref">2015</a>)</cite>, which improves accuracy and handling of OOV words, while retaining the ability to <span id="S1.p4.1.3" class="ltx_text ltx_font_italic">generate</span> new words.
The network, which can be viewed as a balance between extractive and abstractive approaches, is similar to <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib7" title="" class="ltx_ref">Gu et al.</a></cite>’s <cite class="ltx_cite ltx_citemacro_cite">Gu et al. (<a href="#bib.bib7" title="" class="ltx_ref">2016</a>)</cite> CopyNet and <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib17" title="" class="ltx_ref">Miao and Blunsom</a></cite>’s <cite class="ltx_cite ltx_citemacro_cite">Miao and Blunsom (<a href="#bib.bib17" title="" class="ltx_ref">2016</a>)</cite> Forced-Attention Sentence Compression, that were applied to short-text summarization.
We propose a novel variant of the <span id="S1.p4.1.4" class="ltx_text ltx_font_italic">coverage vector</span> <cite class="ltx_cite ltx_citemacro_cite">Tu et al. (<a href="#bib.bib28" title="" class="ltx_ref">2016</a>)</cite> from Neural Machine Translation, which we use to track and control coverage of the source document.
We show that coverage is remarkably effective for eliminating repetition.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Our Models</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">In this section we describe (1) our baseline sequence-to-sequence model, (2) our pointer-generator model, and (3) our coverage mechanism that can be added to either of the first two models. The code for our models is available online.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span><a href="www.github.com/abisee/pointer-generator" title="" class="ltx_ref ltx_url ltx_font_typewriter">www.github.com/abisee/pointer-generator</a></span></span></span></p>
</div>
<figure id="S2.F3" class="ltx_figure"><img src="/html/1704.04368/assets/x2.png" id="S2.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="313" height="171" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Pointer-generator model.
For each decoder timestep a generation probability <math id="S2.F3.2.m1.2" class="ltx_Math" alttext="p_{\text{gen}}\in[0,1]" display="inline"><semantics id="S2.F3.2.m1.2b"><mrow id="S2.F3.2.m1.2.3" xref="S2.F3.2.m1.2.3.cmml"><msub id="S2.F3.2.m1.2.3.2" xref="S2.F3.2.m1.2.3.2.cmml"><mi id="S2.F3.2.m1.2.3.2.2" xref="S2.F3.2.m1.2.3.2.2.cmml">p</mi><mtext id="S2.F3.2.m1.2.3.2.3" xref="S2.F3.2.m1.2.3.2.3a.cmml">gen</mtext></msub><mo id="S2.F3.2.m1.2.3.1" xref="S2.F3.2.m1.2.3.1.cmml">∈</mo><mrow id="S2.F3.2.m1.2.3.3.2" xref="S2.F3.2.m1.2.3.3.1.cmml"><mo stretchy="false" id="S2.F3.2.m1.2.3.3.2.1" xref="S2.F3.2.m1.2.3.3.1.cmml">[</mo><mn id="S2.F3.2.m1.1.1" xref="S2.F3.2.m1.1.1.cmml">0</mn><mo id="S2.F3.2.m1.2.3.3.2.2" xref="S2.F3.2.m1.2.3.3.1.cmml">,</mo><mn id="S2.F3.2.m1.2.2" xref="S2.F3.2.m1.2.2.cmml">1</mn><mo stretchy="false" id="S2.F3.2.m1.2.3.3.2.3" xref="S2.F3.2.m1.2.3.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.F3.2.m1.2c"><apply id="S2.F3.2.m1.2.3.cmml" xref="S2.F3.2.m1.2.3"><in id="S2.F3.2.m1.2.3.1.cmml" xref="S2.F3.2.m1.2.3.1"></in><apply id="S2.F3.2.m1.2.3.2.cmml" xref="S2.F3.2.m1.2.3.2"><csymbol cd="ambiguous" id="S2.F3.2.m1.2.3.2.1.cmml" xref="S2.F3.2.m1.2.3.2">subscript</csymbol><ci id="S2.F3.2.m1.2.3.2.2.cmml" xref="S2.F3.2.m1.2.3.2.2">𝑝</ci><ci id="S2.F3.2.m1.2.3.2.3a.cmml" xref="S2.F3.2.m1.2.3.2.3"><mtext mathsize="70%" id="S2.F3.2.m1.2.3.2.3.cmml" xref="S2.F3.2.m1.2.3.2.3">gen</mtext></ci></apply><interval closure="closed" id="S2.F3.2.m1.2.3.3.1.cmml" xref="S2.F3.2.m1.2.3.3.2"><cn type="integer" id="S2.F3.2.m1.1.1.cmml" xref="S2.F3.2.m1.1.1">0</cn><cn type="integer" id="S2.F3.2.m1.2.2.cmml" xref="S2.F3.2.m1.2.2">1</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F3.2.m1.2d">p_{\text{gen}}\in[0,1]</annotation></semantics></math> is calculated, which weights the probability of <span id="S2.F3.6.1" class="ltx_text ltx_font_italic">generating</span> words from the vocabulary, versus <span id="S2.F3.7.2" class="ltx_text ltx_font_italic">copying</span> words from the source text.
The vocabulary distribution and the attention distribution are weighted and summed to obtain the final distribution, from which we make our prediction.
Note that out-of-vocabulary article words such as <span id="S2.F3.8.3" class="ltx_text ltx_font_italic">2-0</span> are included in the final distribution.
Best viewed in color.
</figcaption>
</figure>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Sequence-to-sequence attentional model</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.5" class="ltx_p">Our baseline model is similar to that of <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib19" title="" class="ltx_ref">Nallapati et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite>, and is depicted in Figure <a href="#S0.F2" title="Figure 2 ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
The tokens of the article <math id="S2.SS1.p1.1.m1.1" class="ltx_Math" alttext="w_{i}" display="inline"><semantics id="S2.SS1.p1.1.m1.1a"><msub id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml"><mi id="S2.SS1.p1.1.m1.1.1.2" xref="S2.SS1.p1.1.m1.1.1.2.cmml">w</mi><mi id="S2.SS1.p1.1.m1.1.1.3" xref="S2.SS1.p1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.1b"><apply id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.1.m1.1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1">subscript</csymbol><ci id="S2.SS1.p1.1.m1.1.1.2.cmml" xref="S2.SS1.p1.1.m1.1.1.2">𝑤</ci><ci id="S2.SS1.p1.1.m1.1.1.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.1c">w_{i}</annotation></semantics></math> are fed one-by-one into the encoder (a single-layer bidirectional LSTM),
producing a sequence of <span id="S2.SS1.p1.5.1" class="ltx_text ltx_font_italic">encoder hidden states</span> <math id="S2.SS1.p1.2.m2.1" class="ltx_Math" alttext="h_{i}" display="inline"><semantics id="S2.SS1.p1.2.m2.1a"><msub id="S2.SS1.p1.2.m2.1.1" xref="S2.SS1.p1.2.m2.1.1.cmml"><mi id="S2.SS1.p1.2.m2.1.1.2" xref="S2.SS1.p1.2.m2.1.1.2.cmml">h</mi><mi id="S2.SS1.p1.2.m2.1.1.3" xref="S2.SS1.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.2.m2.1b"><apply id="S2.SS1.p1.2.m2.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.2.m2.1.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1">subscript</csymbol><ci id="S2.SS1.p1.2.m2.1.1.2.cmml" xref="S2.SS1.p1.2.m2.1.1.2">ℎ</ci><ci id="S2.SS1.p1.2.m2.1.1.3.cmml" xref="S2.SS1.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.2.m2.1c">h_{i}</annotation></semantics></math>.
On each step <math id="S2.SS1.p1.3.m3.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.SS1.p1.3.m3.1a"><mi id="S2.SS1.p1.3.m3.1.1" xref="S2.SS1.p1.3.m3.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.3.m3.1b"><ci id="S2.SS1.p1.3.m3.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.3.m3.1c">t</annotation></semantics></math>, the decoder (a single-layer unidirectional LSTM) receives the word embedding of the previous word (while training, this is the previous word of the reference summary; at test time it is the previous word emitted by the decoder), and has <span id="S2.SS1.p1.5.2" class="ltx_text ltx_font_italic">decoder state</span> <math id="S2.SS1.p1.4.m4.1" class="ltx_Math" alttext="s_{t}" display="inline"><semantics id="S2.SS1.p1.4.m4.1a"><msub id="S2.SS1.p1.4.m4.1.1" xref="S2.SS1.p1.4.m4.1.1.cmml"><mi id="S2.SS1.p1.4.m4.1.1.2" xref="S2.SS1.p1.4.m4.1.1.2.cmml">s</mi><mi id="S2.SS1.p1.4.m4.1.1.3" xref="S2.SS1.p1.4.m4.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.4.m4.1b"><apply id="S2.SS1.p1.4.m4.1.1.cmml" xref="S2.SS1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.4.m4.1.1.1.cmml" xref="S2.SS1.p1.4.m4.1.1">subscript</csymbol><ci id="S2.SS1.p1.4.m4.1.1.2.cmml" xref="S2.SS1.p1.4.m4.1.1.2">𝑠</ci><ci id="S2.SS1.p1.4.m4.1.1.3.cmml" xref="S2.SS1.p1.4.m4.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.4.m4.1c">s_{t}</annotation></semantics></math>.
The <span id="S2.SS1.p1.5.3" class="ltx_text ltx_font_italic">attention distribution</span> <math id="S2.SS1.p1.5.m5.1" class="ltx_Math" alttext="a^{t}" display="inline"><semantics id="S2.SS1.p1.5.m5.1a"><msup id="S2.SS1.p1.5.m5.1.1" xref="S2.SS1.p1.5.m5.1.1.cmml"><mi id="S2.SS1.p1.5.m5.1.1.2" xref="S2.SS1.p1.5.m5.1.1.2.cmml">a</mi><mi id="S2.SS1.p1.5.m5.1.1.3" xref="S2.SS1.p1.5.m5.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.5.m5.1b"><apply id="S2.SS1.p1.5.m5.1.1.cmml" xref="S2.SS1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.5.m5.1.1.1.cmml" xref="S2.SS1.p1.5.m5.1.1">superscript</csymbol><ci id="S2.SS1.p1.5.m5.1.1.2.cmml" xref="S2.SS1.p1.5.m5.1.1.2">𝑎</ci><ci id="S2.SS1.p1.5.m5.1.1.3.cmml" xref="S2.SS1.p1.5.m5.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.5.m5.1c">a^{t}</annotation></semantics></math> is calculated as in <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib1" title="" class="ltx_ref">Bahdanau et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Bahdanau et al. (<a href="#bib.bib1" title="" class="ltx_ref">2015</a>)</cite>:</p>
<table id="Ax1.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E1.m1.1" class="ltx_Math" alttext="\displaystyle e_{i}^{t}" display="inline"><semantics id="S2.E1.m1.1a"><msubsup id="S2.E1.m1.1.1" xref="S2.E1.m1.1.1.cmml"><mi id="S2.E1.m1.1.1.2.2" xref="S2.E1.m1.1.1.2.2.cmml">e</mi><mi id="S2.E1.m1.1.1.2.3" xref="S2.E1.m1.1.1.2.3.cmml">i</mi><mi id="S2.E1.m1.1.1.3" xref="S2.E1.m1.1.1.3.cmml">t</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.E1.m1.1b"><apply id="S2.E1.m1.1.1.cmml" xref="S2.E1.m1.1.1"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.cmml" xref="S2.E1.m1.1.1">superscript</csymbol><apply id="S2.E1.m1.1.1.2.cmml" xref="S2.E1.m1.1.1"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.2.1.cmml" xref="S2.E1.m1.1.1">subscript</csymbol><ci id="S2.E1.m1.1.1.2.2.cmml" xref="S2.E1.m1.1.1.2.2">𝑒</ci><ci id="S2.E1.m1.1.1.2.3.cmml" xref="S2.E1.m1.1.1.2.3">𝑖</ci></apply><ci id="S2.E1.m1.1.1.3.cmml" xref="S2.E1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.1c">\displaystyle e_{i}^{t}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E1.m2.2" class="ltx_Math" alttext="\displaystyle=v^{T}\tanh(W_{h}h_{i}+W_{s}s_{t}+b_{\text{attn}})" display="inline"><semantics id="S2.E1.m2.2a"><mrow id="S2.E1.m2.2.2" xref="S2.E1.m2.2.2.cmml"><mi id="S2.E1.m2.2.2.3" xref="S2.E1.m2.2.2.3.cmml"></mi><mo id="S2.E1.m2.2.2.2" xref="S2.E1.m2.2.2.2.cmml">=</mo><mrow id="S2.E1.m2.2.2.1" xref="S2.E1.m2.2.2.1.cmml"><msup id="S2.E1.m2.2.2.1.3" xref="S2.E1.m2.2.2.1.3.cmml"><mi id="S2.E1.m2.2.2.1.3.2" xref="S2.E1.m2.2.2.1.3.2.cmml">v</mi><mi id="S2.E1.m2.2.2.1.3.3" xref="S2.E1.m2.2.2.1.3.3.cmml">T</mi></msup><mo lspace="0.167em" rspace="0em" id="S2.E1.m2.2.2.1.2" xref="S2.E1.m2.2.2.1.2.cmml">​</mo><mrow id="S2.E1.m2.2.2.1.1.1" xref="S2.E1.m2.2.2.1.1.2.cmml"><mi id="S2.E1.m2.1.1" xref="S2.E1.m2.1.1.cmml">tanh</mi><mo id="S2.E1.m2.2.2.1.1.1a" xref="S2.E1.m2.2.2.1.1.2.cmml">⁡</mo><mrow id="S2.E1.m2.2.2.1.1.1.1" xref="S2.E1.m2.2.2.1.1.2.cmml"><mo stretchy="false" id="S2.E1.m2.2.2.1.1.1.1.2" xref="S2.E1.m2.2.2.1.1.2.cmml">(</mo><mrow id="S2.E1.m2.2.2.1.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.1.cmml"><mrow id="S2.E1.m2.2.2.1.1.1.1.1.2" xref="S2.E1.m2.2.2.1.1.1.1.1.2.cmml"><msub id="S2.E1.m2.2.2.1.1.1.1.1.2.2" xref="S2.E1.m2.2.2.1.1.1.1.1.2.2.cmml"><mi id="S2.E1.m2.2.2.1.1.1.1.1.2.2.2" xref="S2.E1.m2.2.2.1.1.1.1.1.2.2.2.cmml">W</mi><mi id="S2.E1.m2.2.2.1.1.1.1.1.2.2.3" xref="S2.E1.m2.2.2.1.1.1.1.1.2.2.3.cmml">h</mi></msub><mo lspace="0em" rspace="0em" id="S2.E1.m2.2.2.1.1.1.1.1.2.1" xref="S2.E1.m2.2.2.1.1.1.1.1.2.1.cmml">​</mo><msub id="S2.E1.m2.2.2.1.1.1.1.1.2.3" xref="S2.E1.m2.2.2.1.1.1.1.1.2.3.cmml"><mi id="S2.E1.m2.2.2.1.1.1.1.1.2.3.2" xref="S2.E1.m2.2.2.1.1.1.1.1.2.3.2.cmml">h</mi><mi id="S2.E1.m2.2.2.1.1.1.1.1.2.3.3" xref="S2.E1.m2.2.2.1.1.1.1.1.2.3.3.cmml">i</mi></msub></mrow><mo id="S2.E1.m2.2.2.1.1.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.1.1.cmml">+</mo><mrow id="S2.E1.m2.2.2.1.1.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.1.1.3.cmml"><msub id="S2.E1.m2.2.2.1.1.1.1.1.3.2" xref="S2.E1.m2.2.2.1.1.1.1.1.3.2.cmml"><mi id="S2.E1.m2.2.2.1.1.1.1.1.3.2.2" xref="S2.E1.m2.2.2.1.1.1.1.1.3.2.2.cmml">W</mi><mi id="S2.E1.m2.2.2.1.1.1.1.1.3.2.3" xref="S2.E1.m2.2.2.1.1.1.1.1.3.2.3.cmml">s</mi></msub><mo lspace="0em" rspace="0em" id="S2.E1.m2.2.2.1.1.1.1.1.3.1" xref="S2.E1.m2.2.2.1.1.1.1.1.3.1.cmml">​</mo><msub id="S2.E1.m2.2.2.1.1.1.1.1.3.3" xref="S2.E1.m2.2.2.1.1.1.1.1.3.3.cmml"><mi id="S2.E1.m2.2.2.1.1.1.1.1.3.3.2" xref="S2.E1.m2.2.2.1.1.1.1.1.3.3.2.cmml">s</mi><mi id="S2.E1.m2.2.2.1.1.1.1.1.3.3.3" xref="S2.E1.m2.2.2.1.1.1.1.1.3.3.3.cmml">t</mi></msub></mrow><mo id="S2.E1.m2.2.2.1.1.1.1.1.1a" xref="S2.E1.m2.2.2.1.1.1.1.1.1.cmml">+</mo><msub id="S2.E1.m2.2.2.1.1.1.1.1.4" xref="S2.E1.m2.2.2.1.1.1.1.1.4.cmml"><mi id="S2.E1.m2.2.2.1.1.1.1.1.4.2" xref="S2.E1.m2.2.2.1.1.1.1.1.4.2.cmml">b</mi><mtext id="S2.E1.m2.2.2.1.1.1.1.1.4.3" xref="S2.E1.m2.2.2.1.1.1.1.1.4.3a.cmml">attn</mtext></msub></mrow><mo stretchy="false" id="S2.E1.m2.2.2.1.1.1.1.3" xref="S2.E1.m2.2.2.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m2.2b"><apply id="S2.E1.m2.2.2.cmml" xref="S2.E1.m2.2.2"><eq id="S2.E1.m2.2.2.2.cmml" xref="S2.E1.m2.2.2.2"></eq><csymbol cd="latexml" id="S2.E1.m2.2.2.3.cmml" xref="S2.E1.m2.2.2.3">absent</csymbol><apply id="S2.E1.m2.2.2.1.cmml" xref="S2.E1.m2.2.2.1"><times id="S2.E1.m2.2.2.1.2.cmml" xref="S2.E1.m2.2.2.1.2"></times><apply id="S2.E1.m2.2.2.1.3.cmml" xref="S2.E1.m2.2.2.1.3"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.3.1.cmml" xref="S2.E1.m2.2.2.1.3">superscript</csymbol><ci id="S2.E1.m2.2.2.1.3.2.cmml" xref="S2.E1.m2.2.2.1.3.2">𝑣</ci><ci id="S2.E1.m2.2.2.1.3.3.cmml" xref="S2.E1.m2.2.2.1.3.3">𝑇</ci></apply><apply id="S2.E1.m2.2.2.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1"><tanh id="S2.E1.m2.1.1.cmml" xref="S2.E1.m2.1.1"></tanh><apply id="S2.E1.m2.2.2.1.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1"><plus id="S2.E1.m2.2.2.1.1.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1"></plus><apply id="S2.E1.m2.2.2.1.1.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.2"><times id="S2.E1.m2.2.2.1.1.1.1.1.2.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.2.1"></times><apply id="S2.E1.m2.2.2.1.1.1.1.1.2.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.1.1.1.2.2.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.2.2">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.1.1.1.2.2.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.2.2.2">𝑊</ci><ci id="S2.E1.m2.2.2.1.1.1.1.1.2.2.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.2.2.3">ℎ</ci></apply><apply id="S2.E1.m2.2.2.1.1.1.1.1.2.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.1.1.1.2.3.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.2.3">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.1.1.1.2.3.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.2.3.2">ℎ</ci><ci id="S2.E1.m2.2.2.1.1.1.1.1.2.3.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.2.3.3">𝑖</ci></apply></apply><apply id="S2.E1.m2.2.2.1.1.1.1.1.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3"><times id="S2.E1.m2.2.2.1.1.1.1.1.3.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.1"></times><apply id="S2.E1.m2.2.2.1.1.1.1.1.3.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.1.1.1.3.2.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.2">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.1.1.1.3.2.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.2.2">𝑊</ci><ci id="S2.E1.m2.2.2.1.1.1.1.1.3.2.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.2.3">𝑠</ci></apply><apply id="S2.E1.m2.2.2.1.1.1.1.1.3.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.1.1.1.3.3.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.3">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.1.1.1.3.3.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.3.2">𝑠</ci><ci id="S2.E1.m2.2.2.1.1.1.1.1.3.3.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply><apply id="S2.E1.m2.2.2.1.1.1.1.1.4.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.1.1.1.4.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.4">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.1.1.1.4.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.4.2">𝑏</ci><ci id="S2.E1.m2.2.2.1.1.1.1.1.4.3a.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.4.3"><mtext mathsize="70%" id="S2.E1.m2.2.2.1.1.1.1.1.4.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.4.3">attn</mtext></ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m2.2c">\displaystyle=v^{T}\tanh(W_{h}h_{i}+W_{s}s_{t}+b_{\text{attn}})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
<tbody id="S2.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E2.m1.1" class="ltx_Math" alttext="\displaystyle a^{t}" display="inline"><semantics id="S2.E2.m1.1a"><msup id="S2.E2.m1.1.1" xref="S2.E2.m1.1.1.cmml"><mi id="S2.E2.m1.1.1.2" xref="S2.E2.m1.1.1.2.cmml">a</mi><mi id="S2.E2.m1.1.1.3" xref="S2.E2.m1.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="S2.E2.m1.1b"><apply id="S2.E2.m1.1.1.cmml" xref="S2.E2.m1.1.1"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.1.cmml" xref="S2.E2.m1.1.1">superscript</csymbol><ci id="S2.E2.m1.1.1.2.cmml" xref="S2.E2.m1.1.1.2">𝑎</ci><ci id="S2.E2.m1.1.1.3.cmml" xref="S2.E2.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.1c">\displaystyle a^{t}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E2.m2.1" class="ltx_Math" alttext="\displaystyle=\text{softmax}(e^{t})" display="inline"><semantics id="S2.E2.m2.1a"><mrow id="S2.E2.m2.1.1" xref="S2.E2.m2.1.1.cmml"><mi id="S2.E2.m2.1.1.3" xref="S2.E2.m2.1.1.3.cmml"></mi><mo id="S2.E2.m2.1.1.2" xref="S2.E2.m2.1.1.2.cmml">=</mo><mrow id="S2.E2.m2.1.1.1" xref="S2.E2.m2.1.1.1.cmml"><mtext id="S2.E2.m2.1.1.1.3" xref="S2.E2.m2.1.1.1.3a.cmml">softmax</mtext><mo lspace="0em" rspace="0em" id="S2.E2.m2.1.1.1.2" xref="S2.E2.m2.1.1.1.2.cmml">​</mo><mrow id="S2.E2.m2.1.1.1.1.1" xref="S2.E2.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E2.m2.1.1.1.1.1.2" xref="S2.E2.m2.1.1.1.1.1.1.cmml">(</mo><msup id="S2.E2.m2.1.1.1.1.1.1" xref="S2.E2.m2.1.1.1.1.1.1.cmml"><mi id="S2.E2.m2.1.1.1.1.1.1.2" xref="S2.E2.m2.1.1.1.1.1.1.2.cmml">e</mi><mi id="S2.E2.m2.1.1.1.1.1.1.3" xref="S2.E2.m2.1.1.1.1.1.1.3.cmml">t</mi></msup><mo stretchy="false" id="S2.E2.m2.1.1.1.1.1.3" xref="S2.E2.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m2.1b"><apply id="S2.E2.m2.1.1.cmml" xref="S2.E2.m2.1.1"><eq id="S2.E2.m2.1.1.2.cmml" xref="S2.E2.m2.1.1.2"></eq><csymbol cd="latexml" id="S2.E2.m2.1.1.3.cmml" xref="S2.E2.m2.1.1.3">absent</csymbol><apply id="S2.E2.m2.1.1.1.cmml" xref="S2.E2.m2.1.1.1"><times id="S2.E2.m2.1.1.1.2.cmml" xref="S2.E2.m2.1.1.1.2"></times><ci id="S2.E2.m2.1.1.1.3a.cmml" xref="S2.E2.m2.1.1.1.3"><mtext id="S2.E2.m2.1.1.1.3.cmml" xref="S2.E2.m2.1.1.1.3">softmax</mtext></ci><apply id="S2.E2.m2.1.1.1.1.1.1.cmml" xref="S2.E2.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m2.1.1.1.1.1.1.1.cmml" xref="S2.E2.m2.1.1.1.1.1">superscript</csymbol><ci id="S2.E2.m2.1.1.1.1.1.1.2.cmml" xref="S2.E2.m2.1.1.1.1.1.1.2">𝑒</ci><ci id="S2.E2.m2.1.1.1.1.1.1.3.cmml" xref="S2.E2.m2.1.1.1.1.1.1.3">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m2.1c">\displaystyle=\text{softmax}(e^{t})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S2.SS1.p1.10" class="ltx_p">where <math id="S2.SS1.p1.6.m1.1" class="ltx_Math" alttext="v" display="inline"><semantics id="S2.SS1.p1.6.m1.1a"><mi id="S2.SS1.p1.6.m1.1.1" xref="S2.SS1.p1.6.m1.1.1.cmml">v</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.6.m1.1b"><ci id="S2.SS1.p1.6.m1.1.1.cmml" xref="S2.SS1.p1.6.m1.1.1">𝑣</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.6.m1.1c">v</annotation></semantics></math>, <math id="S2.SS1.p1.7.m2.1" class="ltx_Math" alttext="W_{h}" display="inline"><semantics id="S2.SS1.p1.7.m2.1a"><msub id="S2.SS1.p1.7.m2.1.1" xref="S2.SS1.p1.7.m2.1.1.cmml"><mi id="S2.SS1.p1.7.m2.1.1.2" xref="S2.SS1.p1.7.m2.1.1.2.cmml">W</mi><mi id="S2.SS1.p1.7.m2.1.1.3" xref="S2.SS1.p1.7.m2.1.1.3.cmml">h</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.7.m2.1b"><apply id="S2.SS1.p1.7.m2.1.1.cmml" xref="S2.SS1.p1.7.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.7.m2.1.1.1.cmml" xref="S2.SS1.p1.7.m2.1.1">subscript</csymbol><ci id="S2.SS1.p1.7.m2.1.1.2.cmml" xref="S2.SS1.p1.7.m2.1.1.2">𝑊</ci><ci id="S2.SS1.p1.7.m2.1.1.3.cmml" xref="S2.SS1.p1.7.m2.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.7.m2.1c">W_{h}</annotation></semantics></math>, <math id="S2.SS1.p1.8.m3.1" class="ltx_Math" alttext="W_{s}" display="inline"><semantics id="S2.SS1.p1.8.m3.1a"><msub id="S2.SS1.p1.8.m3.1.1" xref="S2.SS1.p1.8.m3.1.1.cmml"><mi id="S2.SS1.p1.8.m3.1.1.2" xref="S2.SS1.p1.8.m3.1.1.2.cmml">W</mi><mi id="S2.SS1.p1.8.m3.1.1.3" xref="S2.SS1.p1.8.m3.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.8.m3.1b"><apply id="S2.SS1.p1.8.m3.1.1.cmml" xref="S2.SS1.p1.8.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.8.m3.1.1.1.cmml" xref="S2.SS1.p1.8.m3.1.1">subscript</csymbol><ci id="S2.SS1.p1.8.m3.1.1.2.cmml" xref="S2.SS1.p1.8.m3.1.1.2">𝑊</ci><ci id="S2.SS1.p1.8.m3.1.1.3.cmml" xref="S2.SS1.p1.8.m3.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.8.m3.1c">W_{s}</annotation></semantics></math> and <math id="S2.SS1.p1.9.m4.1" class="ltx_Math" alttext="b_{\text{attn}}" display="inline"><semantics id="S2.SS1.p1.9.m4.1a"><msub id="S2.SS1.p1.9.m4.1.1" xref="S2.SS1.p1.9.m4.1.1.cmml"><mi id="S2.SS1.p1.9.m4.1.1.2" xref="S2.SS1.p1.9.m4.1.1.2.cmml">b</mi><mtext id="S2.SS1.p1.9.m4.1.1.3" xref="S2.SS1.p1.9.m4.1.1.3a.cmml">attn</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.9.m4.1b"><apply id="S2.SS1.p1.9.m4.1.1.cmml" xref="S2.SS1.p1.9.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.9.m4.1.1.1.cmml" xref="S2.SS1.p1.9.m4.1.1">subscript</csymbol><ci id="S2.SS1.p1.9.m4.1.1.2.cmml" xref="S2.SS1.p1.9.m4.1.1.2">𝑏</ci><ci id="S2.SS1.p1.9.m4.1.1.3a.cmml" xref="S2.SS1.p1.9.m4.1.1.3"><mtext mathsize="70%" id="S2.SS1.p1.9.m4.1.1.3.cmml" xref="S2.SS1.p1.9.m4.1.1.3">attn</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.9.m4.1c">b_{\text{attn}}</annotation></semantics></math> are learnable parameters.
The attention distribution can be viewed as a probability distribution over the source words, that tells the decoder where to look to produce the next word.
Next, the attention distribution is used to produce a weighted sum of the encoder hidden states, known as the <span id="S2.SS1.p1.10.1" class="ltx_text ltx_font_italic">context vector</span> <math id="S2.SS1.p1.10.m5.1" class="ltx_Math" alttext="h^{*}_{t}" display="inline"><semantics id="S2.SS1.p1.10.m5.1a"><msubsup id="S2.SS1.p1.10.m5.1.1" xref="S2.SS1.p1.10.m5.1.1.cmml"><mi id="S2.SS1.p1.10.m5.1.1.2.2" xref="S2.SS1.p1.10.m5.1.1.2.2.cmml">h</mi><mi id="S2.SS1.p1.10.m5.1.1.3" xref="S2.SS1.p1.10.m5.1.1.3.cmml">t</mi><mo id="S2.SS1.p1.10.m5.1.1.2.3" xref="S2.SS1.p1.10.m5.1.1.2.3.cmml">∗</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.10.m5.1b"><apply id="S2.SS1.p1.10.m5.1.1.cmml" xref="S2.SS1.p1.10.m5.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.10.m5.1.1.1.cmml" xref="S2.SS1.p1.10.m5.1.1">subscript</csymbol><apply id="S2.SS1.p1.10.m5.1.1.2.cmml" xref="S2.SS1.p1.10.m5.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.10.m5.1.1.2.1.cmml" xref="S2.SS1.p1.10.m5.1.1">superscript</csymbol><ci id="S2.SS1.p1.10.m5.1.1.2.2.cmml" xref="S2.SS1.p1.10.m5.1.1.2.2">ℎ</ci><times id="S2.SS1.p1.10.m5.1.1.2.3.cmml" xref="S2.SS1.p1.10.m5.1.1.2.3"></times></apply><ci id="S2.SS1.p1.10.m5.1.1.3.cmml" xref="S2.SS1.p1.10.m5.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.10.m5.1c">h^{*}_{t}</annotation></semantics></math>:</p>
<table id="S2.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E3.m1.1" class="ltx_Math" alttext="h^{*}_{t}=\sum\nolimits_{i}a_{i}^{t}h_{i}" display="block"><semantics id="S2.E3.m1.1a"><mrow id="S2.E3.m1.1.1" xref="S2.E3.m1.1.1.cmml"><msubsup id="S2.E3.m1.1.1.2" xref="S2.E3.m1.1.1.2.cmml"><mi id="S2.E3.m1.1.1.2.2.2" xref="S2.E3.m1.1.1.2.2.2.cmml">h</mi><mi id="S2.E3.m1.1.1.2.3" xref="S2.E3.m1.1.1.2.3.cmml">t</mi><mo id="S2.E3.m1.1.1.2.2.3" xref="S2.E3.m1.1.1.2.2.3.cmml">∗</mo></msubsup><mo rspace="0.111em" id="S2.E3.m1.1.1.1" xref="S2.E3.m1.1.1.1.cmml">=</mo><mrow id="S2.E3.m1.1.1.3" xref="S2.E3.m1.1.1.3.cmml"><msub id="S2.E3.m1.1.1.3.1" xref="S2.E3.m1.1.1.3.1.cmml"><mo id="S2.E3.m1.1.1.3.1.2" xref="S2.E3.m1.1.1.3.1.2.cmml">∑</mo><mi id="S2.E3.m1.1.1.3.1.3" xref="S2.E3.m1.1.1.3.1.3.cmml">i</mi></msub><mrow id="S2.E3.m1.1.1.3.2" xref="S2.E3.m1.1.1.3.2.cmml"><msubsup id="S2.E3.m1.1.1.3.2.2" xref="S2.E3.m1.1.1.3.2.2.cmml"><mi id="S2.E3.m1.1.1.3.2.2.2.2" xref="S2.E3.m1.1.1.3.2.2.2.2.cmml">a</mi><mi id="S2.E3.m1.1.1.3.2.2.2.3" xref="S2.E3.m1.1.1.3.2.2.2.3.cmml">i</mi><mi id="S2.E3.m1.1.1.3.2.2.3" xref="S2.E3.m1.1.1.3.2.2.3.cmml">t</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.E3.m1.1.1.3.2.1" xref="S2.E3.m1.1.1.3.2.1.cmml">​</mo><msub id="S2.E3.m1.1.1.3.2.3" xref="S2.E3.m1.1.1.3.2.3.cmml"><mi id="S2.E3.m1.1.1.3.2.3.2" xref="S2.E3.m1.1.1.3.2.3.2.cmml">h</mi><mi id="S2.E3.m1.1.1.3.2.3.3" xref="S2.E3.m1.1.1.3.2.3.3.cmml">i</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E3.m1.1b"><apply id="S2.E3.m1.1.1.cmml" xref="S2.E3.m1.1.1"><eq id="S2.E3.m1.1.1.1.cmml" xref="S2.E3.m1.1.1.1"></eq><apply id="S2.E3.m1.1.1.2.cmml" xref="S2.E3.m1.1.1.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.2.1.cmml" xref="S2.E3.m1.1.1.2">subscript</csymbol><apply id="S2.E3.m1.1.1.2.2.cmml" xref="S2.E3.m1.1.1.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.2.2.1.cmml" xref="S2.E3.m1.1.1.2">superscript</csymbol><ci id="S2.E3.m1.1.1.2.2.2.cmml" xref="S2.E3.m1.1.1.2.2.2">ℎ</ci><times id="S2.E3.m1.1.1.2.2.3.cmml" xref="S2.E3.m1.1.1.2.2.3"></times></apply><ci id="S2.E3.m1.1.1.2.3.cmml" xref="S2.E3.m1.1.1.2.3">𝑡</ci></apply><apply id="S2.E3.m1.1.1.3.cmml" xref="S2.E3.m1.1.1.3"><apply id="S2.E3.m1.1.1.3.1.cmml" xref="S2.E3.m1.1.1.3.1"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.3.1.1.cmml" xref="S2.E3.m1.1.1.3.1">subscript</csymbol><sum id="S2.E3.m1.1.1.3.1.2.cmml" xref="S2.E3.m1.1.1.3.1.2"></sum><ci id="S2.E3.m1.1.1.3.1.3.cmml" xref="S2.E3.m1.1.1.3.1.3">𝑖</ci></apply><apply id="S2.E3.m1.1.1.3.2.cmml" xref="S2.E3.m1.1.1.3.2"><times id="S2.E3.m1.1.1.3.2.1.cmml" xref="S2.E3.m1.1.1.3.2.1"></times><apply id="S2.E3.m1.1.1.3.2.2.cmml" xref="S2.E3.m1.1.1.3.2.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.3.2.2.1.cmml" xref="S2.E3.m1.1.1.3.2.2">superscript</csymbol><apply id="S2.E3.m1.1.1.3.2.2.2.cmml" xref="S2.E3.m1.1.1.3.2.2"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.3.2.2.2.1.cmml" xref="S2.E3.m1.1.1.3.2.2">subscript</csymbol><ci id="S2.E3.m1.1.1.3.2.2.2.2.cmml" xref="S2.E3.m1.1.1.3.2.2.2.2">𝑎</ci><ci id="S2.E3.m1.1.1.3.2.2.2.3.cmml" xref="S2.E3.m1.1.1.3.2.2.2.3">𝑖</ci></apply><ci id="S2.E3.m1.1.1.3.2.2.3.cmml" xref="S2.E3.m1.1.1.3.2.2.3">𝑡</ci></apply><apply id="S2.E3.m1.1.1.3.2.3.cmml" xref="S2.E3.m1.1.1.3.2.3"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.3.2.3.1.cmml" xref="S2.E3.m1.1.1.3.2.3">subscript</csymbol><ci id="S2.E3.m1.1.1.3.2.3.2.cmml" xref="S2.E3.m1.1.1.3.2.3.2">ℎ</ci><ci id="S2.E3.m1.1.1.3.2.3.3.cmml" xref="S2.E3.m1.1.1.3.2.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3.m1.1c">h^{*}_{t}=\sum\nolimits_{i}a_{i}^{t}h_{i}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S2.SS1.p1.12" class="ltx_p">The context vector, which can be seen as a fixed-size representation of what has been read from the source for this step, is concatenated with the decoder state <math id="S2.SS1.p1.11.m1.1" class="ltx_Math" alttext="s_{t}" display="inline"><semantics id="S2.SS1.p1.11.m1.1a"><msub id="S2.SS1.p1.11.m1.1.1" xref="S2.SS1.p1.11.m1.1.1.cmml"><mi id="S2.SS1.p1.11.m1.1.1.2" xref="S2.SS1.p1.11.m1.1.1.2.cmml">s</mi><mi id="S2.SS1.p1.11.m1.1.1.3" xref="S2.SS1.p1.11.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.11.m1.1b"><apply id="S2.SS1.p1.11.m1.1.1.cmml" xref="S2.SS1.p1.11.m1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.11.m1.1.1.1.cmml" xref="S2.SS1.p1.11.m1.1.1">subscript</csymbol><ci id="S2.SS1.p1.11.m1.1.1.2.cmml" xref="S2.SS1.p1.11.m1.1.1.2">𝑠</ci><ci id="S2.SS1.p1.11.m1.1.1.3.cmml" xref="S2.SS1.p1.11.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.11.m1.1c">s_{t}</annotation></semantics></math> and fed through two linear layers to produce the vocabulary distribution <math id="S2.SS1.p1.12.m2.1" class="ltx_Math" alttext="P_{\text{vocab}}" display="inline"><semantics id="S2.SS1.p1.12.m2.1a"><msub id="S2.SS1.p1.12.m2.1.1" xref="S2.SS1.p1.12.m2.1.1.cmml"><mi id="S2.SS1.p1.12.m2.1.1.2" xref="S2.SS1.p1.12.m2.1.1.2.cmml">P</mi><mtext id="S2.SS1.p1.12.m2.1.1.3" xref="S2.SS1.p1.12.m2.1.1.3a.cmml">vocab</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.12.m2.1b"><apply id="S2.SS1.p1.12.m2.1.1.cmml" xref="S2.SS1.p1.12.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.12.m2.1.1.1.cmml" xref="S2.SS1.p1.12.m2.1.1">subscript</csymbol><ci id="S2.SS1.p1.12.m2.1.1.2.cmml" xref="S2.SS1.p1.12.m2.1.1.2">𝑃</ci><ci id="S2.SS1.p1.12.m2.1.1.3a.cmml" xref="S2.SS1.p1.12.m2.1.1.3"><mtext mathsize="70%" id="S2.SS1.p1.12.m2.1.1.3.cmml" xref="S2.SS1.p1.12.m2.1.1.3">vocab</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.12.m2.1c">P_{\text{vocab}}</annotation></semantics></math>:</p>
<table id="Ax1.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.E4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E4.m1.1" class="ltx_Math" alttext="\displaystyle P_{\text{vocab}}" display="inline"><semantics id="S2.E4.m1.1a"><msub id="S2.E4.m1.1.1" xref="S2.E4.m1.1.1.cmml"><mi id="S2.E4.m1.1.1.2" xref="S2.E4.m1.1.1.2.cmml">P</mi><mtext id="S2.E4.m1.1.1.3" xref="S2.E4.m1.1.1.3a.cmml">vocab</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.E4.m1.1b"><apply id="S2.E4.m1.1.1.cmml" xref="S2.E4.m1.1.1"><csymbol cd="ambiguous" id="S2.E4.m1.1.1.1.cmml" xref="S2.E4.m1.1.1">subscript</csymbol><ci id="S2.E4.m1.1.1.2.cmml" xref="S2.E4.m1.1.1.2">𝑃</ci><ci id="S2.E4.m1.1.1.3a.cmml" xref="S2.E4.m1.1.1.3"><mtext mathsize="70%" id="S2.E4.m1.1.1.3.cmml" xref="S2.E4.m1.1.1.3">vocab</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E4.m1.1c">\displaystyle P_{\text{vocab}}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E4.m2.1" class="ltx_Math" alttext="\displaystyle=\text{softmax}(V^{\prime}(V[s_{t},h^{*}_{t}]+b)+b^{\prime})" display="inline"><semantics id="S2.E4.m2.1a"><mrow id="S2.E4.m2.1.1" xref="S2.E4.m2.1.1.cmml"><mi id="S2.E4.m2.1.1.3" xref="S2.E4.m2.1.1.3.cmml"></mi><mo id="S2.E4.m2.1.1.2" xref="S2.E4.m2.1.1.2.cmml">=</mo><mrow id="S2.E4.m2.1.1.1" xref="S2.E4.m2.1.1.1.cmml"><mtext id="S2.E4.m2.1.1.1.3" xref="S2.E4.m2.1.1.1.3a.cmml">softmax</mtext><mo lspace="0em" rspace="0em" id="S2.E4.m2.1.1.1.2" xref="S2.E4.m2.1.1.1.2.cmml">​</mo><mrow id="S2.E4.m2.1.1.1.1.1" xref="S2.E4.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E4.m2.1.1.1.1.1.2" xref="S2.E4.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E4.m2.1.1.1.1.1.1" xref="S2.E4.m2.1.1.1.1.1.1.cmml"><mrow id="S2.E4.m2.1.1.1.1.1.1.1" xref="S2.E4.m2.1.1.1.1.1.1.1.cmml"><msup id="S2.E4.m2.1.1.1.1.1.1.1.3" xref="S2.E4.m2.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E4.m2.1.1.1.1.1.1.1.3.2" xref="S2.E4.m2.1.1.1.1.1.1.1.3.2.cmml">V</mi><mo id="S2.E4.m2.1.1.1.1.1.1.1.3.3" xref="S2.E4.m2.1.1.1.1.1.1.1.3.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="S2.E4.m2.1.1.1.1.1.1.1.2" xref="S2.E4.m2.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.E4.m2.1.1.1.1.1.1.1.1.1" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E4.m2.1.1.1.1.1.1.1.1.1.2" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.4" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.4.cmml">V</mi><mo lspace="0em" rspace="0em" id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.3" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.3.cmml">​</mo><mrow id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.3" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">[</mo><msub id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">s</mi><mi id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">t</mi></msub><mo id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.4" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">,</mo><msubsup id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml"><mi id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.2" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.cmml">h</mi><mi id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.3" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.3.cmml">t</mi><mo id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.3" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.3.cmml">∗</mo></msubsup><mo stretchy="false" id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.5" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">]</mo></mrow></mrow><mo id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.3" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.3.cmml">+</mo><mi id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.4" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.4.cmml">b</mi></mrow><mo stretchy="false" id="S2.E4.m2.1.1.1.1.1.1.1.1.1.3" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.E4.m2.1.1.1.1.1.1.2" xref="S2.E4.m2.1.1.1.1.1.1.2.cmml">+</mo><msup id="S2.E4.m2.1.1.1.1.1.1.3" xref="S2.E4.m2.1.1.1.1.1.1.3.cmml"><mi id="S2.E4.m2.1.1.1.1.1.1.3.2" xref="S2.E4.m2.1.1.1.1.1.1.3.2.cmml">b</mi><mo id="S2.E4.m2.1.1.1.1.1.1.3.3" xref="S2.E4.m2.1.1.1.1.1.1.3.3.cmml">′</mo></msup></mrow><mo stretchy="false" id="S2.E4.m2.1.1.1.1.1.3" xref="S2.E4.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E4.m2.1b"><apply id="S2.E4.m2.1.1.cmml" xref="S2.E4.m2.1.1"><eq id="S2.E4.m2.1.1.2.cmml" xref="S2.E4.m2.1.1.2"></eq><csymbol cd="latexml" id="S2.E4.m2.1.1.3.cmml" xref="S2.E4.m2.1.1.3">absent</csymbol><apply id="S2.E4.m2.1.1.1.cmml" xref="S2.E4.m2.1.1.1"><times id="S2.E4.m2.1.1.1.2.cmml" xref="S2.E4.m2.1.1.1.2"></times><ci id="S2.E4.m2.1.1.1.3a.cmml" xref="S2.E4.m2.1.1.1.3"><mtext id="S2.E4.m2.1.1.1.3.cmml" xref="S2.E4.m2.1.1.1.3">softmax</mtext></ci><apply id="S2.E4.m2.1.1.1.1.1.1.cmml" xref="S2.E4.m2.1.1.1.1.1"><plus id="S2.E4.m2.1.1.1.1.1.1.2.cmml" xref="S2.E4.m2.1.1.1.1.1.1.2"></plus><apply id="S2.E4.m2.1.1.1.1.1.1.1.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1"><times id="S2.E4.m2.1.1.1.1.1.1.1.2.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.2"></times><apply id="S2.E4.m2.1.1.1.1.1.1.1.3.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E4.m2.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.E4.m2.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.3.2">𝑉</ci><ci id="S2.E4.m2.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.3.3">′</ci></apply><apply id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1"><plus id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.3"></plus><apply id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2"><times id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.3"></times><ci id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.4.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.4">𝑉</ci><interval closure="closed" id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2"><apply id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2">𝑠</ci><ci id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.1.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2">subscript</csymbol><apply id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.1.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2">superscript</csymbol><ci id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.2">ℎ</ci><times id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.3.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.3"></times></apply><ci id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.3.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.2.2.2.2.3">𝑡</ci></apply></interval></apply><ci id="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.4.cmml" xref="S2.E4.m2.1.1.1.1.1.1.1.1.1.1.4">𝑏</ci></apply></apply><apply id="S2.E4.m2.1.1.1.1.1.1.3.cmml" xref="S2.E4.m2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E4.m2.1.1.1.1.1.1.3.1.cmml" xref="S2.E4.m2.1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.E4.m2.1.1.1.1.1.1.3.2.cmml" xref="S2.E4.m2.1.1.1.1.1.1.3.2">𝑏</ci><ci id="S2.E4.m2.1.1.1.1.1.1.3.3.cmml" xref="S2.E4.m2.1.1.1.1.1.1.3.3">′</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E4.m2.1c">\displaystyle=\text{softmax}(V^{\prime}(V[s_{t},h^{*}_{t}]+b)+b^{\prime})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S2.SS1.p1.18" class="ltx_p">where <math id="S2.SS1.p1.13.m1.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S2.SS1.p1.13.m1.1a"><mi id="S2.SS1.p1.13.m1.1.1" xref="S2.SS1.p1.13.m1.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.13.m1.1b"><ci id="S2.SS1.p1.13.m1.1.1.cmml" xref="S2.SS1.p1.13.m1.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.13.m1.1c">V</annotation></semantics></math>, <math id="S2.SS1.p1.14.m2.1" class="ltx_Math" alttext="V^{\prime}" display="inline"><semantics id="S2.SS1.p1.14.m2.1a"><msup id="S2.SS1.p1.14.m2.1.1" xref="S2.SS1.p1.14.m2.1.1.cmml"><mi id="S2.SS1.p1.14.m2.1.1.2" xref="S2.SS1.p1.14.m2.1.1.2.cmml">V</mi><mo id="S2.SS1.p1.14.m2.1.1.3" xref="S2.SS1.p1.14.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.14.m2.1b"><apply id="S2.SS1.p1.14.m2.1.1.cmml" xref="S2.SS1.p1.14.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.14.m2.1.1.1.cmml" xref="S2.SS1.p1.14.m2.1.1">superscript</csymbol><ci id="S2.SS1.p1.14.m2.1.1.2.cmml" xref="S2.SS1.p1.14.m2.1.1.2">𝑉</ci><ci id="S2.SS1.p1.14.m2.1.1.3.cmml" xref="S2.SS1.p1.14.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.14.m2.1c">V^{\prime}</annotation></semantics></math>, <math id="S2.SS1.p1.15.m3.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.SS1.p1.15.m3.1a"><mi id="S2.SS1.p1.15.m3.1.1" xref="S2.SS1.p1.15.m3.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.15.m3.1b"><ci id="S2.SS1.p1.15.m3.1.1.cmml" xref="S2.SS1.p1.15.m3.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.15.m3.1c">b</annotation></semantics></math> and <math id="S2.SS1.p1.16.m4.1" class="ltx_Math" alttext="b^{\prime}" display="inline"><semantics id="S2.SS1.p1.16.m4.1a"><msup id="S2.SS1.p1.16.m4.1.1" xref="S2.SS1.p1.16.m4.1.1.cmml"><mi id="S2.SS1.p1.16.m4.1.1.2" xref="S2.SS1.p1.16.m4.1.1.2.cmml">b</mi><mo id="S2.SS1.p1.16.m4.1.1.3" xref="S2.SS1.p1.16.m4.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.16.m4.1b"><apply id="S2.SS1.p1.16.m4.1.1.cmml" xref="S2.SS1.p1.16.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.16.m4.1.1.1.cmml" xref="S2.SS1.p1.16.m4.1.1">superscript</csymbol><ci id="S2.SS1.p1.16.m4.1.1.2.cmml" xref="S2.SS1.p1.16.m4.1.1.2">𝑏</ci><ci id="S2.SS1.p1.16.m4.1.1.3.cmml" xref="S2.SS1.p1.16.m4.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.16.m4.1c">b^{\prime}</annotation></semantics></math> are learnable parameters.
<math id="S2.SS1.p1.17.m5.1" class="ltx_Math" alttext="P_{\text{vocab}}" display="inline"><semantics id="S2.SS1.p1.17.m5.1a"><msub id="S2.SS1.p1.17.m5.1.1" xref="S2.SS1.p1.17.m5.1.1.cmml"><mi id="S2.SS1.p1.17.m5.1.1.2" xref="S2.SS1.p1.17.m5.1.1.2.cmml">P</mi><mtext id="S2.SS1.p1.17.m5.1.1.3" xref="S2.SS1.p1.17.m5.1.1.3a.cmml">vocab</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.17.m5.1b"><apply id="S2.SS1.p1.17.m5.1.1.cmml" xref="S2.SS1.p1.17.m5.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.17.m5.1.1.1.cmml" xref="S2.SS1.p1.17.m5.1.1">subscript</csymbol><ci id="S2.SS1.p1.17.m5.1.1.2.cmml" xref="S2.SS1.p1.17.m5.1.1.2">𝑃</ci><ci id="S2.SS1.p1.17.m5.1.1.3a.cmml" xref="S2.SS1.p1.17.m5.1.1.3"><mtext mathsize="70%" id="S2.SS1.p1.17.m5.1.1.3.cmml" xref="S2.SS1.p1.17.m5.1.1.3">vocab</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.17.m5.1c">P_{\text{vocab}}</annotation></semantics></math> is a probability distribution over all words in the vocabulary, and provides us with our final distribution from which to predict words <math id="S2.SS1.p1.18.m6.1" class="ltx_Math" alttext="w" display="inline"><semantics id="S2.SS1.p1.18.m6.1a"><mi id="S2.SS1.p1.18.m6.1.1" xref="S2.SS1.p1.18.m6.1.1.cmml">w</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.18.m6.1b"><ci id="S2.SS1.p1.18.m6.1.1.cmml" xref="S2.SS1.p1.18.m6.1.1">𝑤</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.18.m6.1c">w</annotation></semantics></math>:</p>
<table id="S2.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E5.m1.2" class="ltx_Math" alttext="P(w)=P_{\text{vocab}}(w)" display="block"><semantics id="S2.E5.m1.2a"><mrow id="S2.E5.m1.2.3" xref="S2.E5.m1.2.3.cmml"><mrow id="S2.E5.m1.2.3.2" xref="S2.E5.m1.2.3.2.cmml"><mi id="S2.E5.m1.2.3.2.2" xref="S2.E5.m1.2.3.2.2.cmml">P</mi><mo lspace="0em" rspace="0em" id="S2.E5.m1.2.3.2.1" xref="S2.E5.m1.2.3.2.1.cmml">​</mo><mrow id="S2.E5.m1.2.3.2.3.2" xref="S2.E5.m1.2.3.2.cmml"><mo stretchy="false" id="S2.E5.m1.2.3.2.3.2.1" xref="S2.E5.m1.2.3.2.cmml">(</mo><mi id="S2.E5.m1.1.1" xref="S2.E5.m1.1.1.cmml">w</mi><mo stretchy="false" id="S2.E5.m1.2.3.2.3.2.2" xref="S2.E5.m1.2.3.2.cmml">)</mo></mrow></mrow><mo id="S2.E5.m1.2.3.1" xref="S2.E5.m1.2.3.1.cmml">=</mo><mrow id="S2.E5.m1.2.3.3" xref="S2.E5.m1.2.3.3.cmml"><msub id="S2.E5.m1.2.3.3.2" xref="S2.E5.m1.2.3.3.2.cmml"><mi id="S2.E5.m1.2.3.3.2.2" xref="S2.E5.m1.2.3.3.2.2.cmml">P</mi><mtext id="S2.E5.m1.2.3.3.2.3" xref="S2.E5.m1.2.3.3.2.3a.cmml">vocab</mtext></msub><mo lspace="0em" rspace="0em" id="S2.E5.m1.2.3.3.1" xref="S2.E5.m1.2.3.3.1.cmml">​</mo><mrow id="S2.E5.m1.2.3.3.3.2" xref="S2.E5.m1.2.3.3.cmml"><mo stretchy="false" id="S2.E5.m1.2.3.3.3.2.1" xref="S2.E5.m1.2.3.3.cmml">(</mo><mi id="S2.E5.m1.2.2" xref="S2.E5.m1.2.2.cmml">w</mi><mo stretchy="false" id="S2.E5.m1.2.3.3.3.2.2" xref="S2.E5.m1.2.3.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E5.m1.2b"><apply id="S2.E5.m1.2.3.cmml" xref="S2.E5.m1.2.3"><eq id="S2.E5.m1.2.3.1.cmml" xref="S2.E5.m1.2.3.1"></eq><apply id="S2.E5.m1.2.3.2.cmml" xref="S2.E5.m1.2.3.2"><times id="S2.E5.m1.2.3.2.1.cmml" xref="S2.E5.m1.2.3.2.1"></times><ci id="S2.E5.m1.2.3.2.2.cmml" xref="S2.E5.m1.2.3.2.2">𝑃</ci><ci id="S2.E5.m1.1.1.cmml" xref="S2.E5.m1.1.1">𝑤</ci></apply><apply id="S2.E5.m1.2.3.3.cmml" xref="S2.E5.m1.2.3.3"><times id="S2.E5.m1.2.3.3.1.cmml" xref="S2.E5.m1.2.3.3.1"></times><apply id="S2.E5.m1.2.3.3.2.cmml" xref="S2.E5.m1.2.3.3.2"><csymbol cd="ambiguous" id="S2.E5.m1.2.3.3.2.1.cmml" xref="S2.E5.m1.2.3.3.2">subscript</csymbol><ci id="S2.E5.m1.2.3.3.2.2.cmml" xref="S2.E5.m1.2.3.3.2.2">𝑃</ci><ci id="S2.E5.m1.2.3.3.2.3a.cmml" xref="S2.E5.m1.2.3.3.2.3"><mtext mathsize="70%" id="S2.E5.m1.2.3.3.2.3.cmml" xref="S2.E5.m1.2.3.3.2.3">vocab</mtext></ci></apply><ci id="S2.E5.m1.2.2.cmml" xref="S2.E5.m1.2.2">𝑤</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E5.m1.2c">P(w)=P_{\text{vocab}}(w)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="S2.SS1.p1.20" class="ltx_p">During training, the loss for timestep <math id="S2.SS1.p1.19.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.SS1.p1.19.m1.1a"><mi id="S2.SS1.p1.19.m1.1.1" xref="S2.SS1.p1.19.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.19.m1.1b"><ci id="S2.SS1.p1.19.m1.1.1.cmml" xref="S2.SS1.p1.19.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.19.m1.1c">t</annotation></semantics></math> is the negative log likelihood of the target word <math id="S2.SS1.p1.20.m2.1" class="ltx_Math" alttext="w^{*}_{t}" display="inline"><semantics id="S2.SS1.p1.20.m2.1a"><msubsup id="S2.SS1.p1.20.m2.1.1" xref="S2.SS1.p1.20.m2.1.1.cmml"><mi id="S2.SS1.p1.20.m2.1.1.2.2" xref="S2.SS1.p1.20.m2.1.1.2.2.cmml">w</mi><mi id="S2.SS1.p1.20.m2.1.1.3" xref="S2.SS1.p1.20.m2.1.1.3.cmml">t</mi><mo id="S2.SS1.p1.20.m2.1.1.2.3" xref="S2.SS1.p1.20.m2.1.1.2.3.cmml">∗</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.20.m2.1b"><apply id="S2.SS1.p1.20.m2.1.1.cmml" xref="S2.SS1.p1.20.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.20.m2.1.1.1.cmml" xref="S2.SS1.p1.20.m2.1.1">subscript</csymbol><apply id="S2.SS1.p1.20.m2.1.1.2.cmml" xref="S2.SS1.p1.20.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.20.m2.1.1.2.1.cmml" xref="S2.SS1.p1.20.m2.1.1">superscript</csymbol><ci id="S2.SS1.p1.20.m2.1.1.2.2.cmml" xref="S2.SS1.p1.20.m2.1.1.2.2">𝑤</ci><times id="S2.SS1.p1.20.m2.1.1.2.3.cmml" xref="S2.SS1.p1.20.m2.1.1.2.3"></times></apply><ci id="S2.SS1.p1.20.m2.1.1.3.cmml" xref="S2.SS1.p1.20.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.20.m2.1c">w^{*}_{t}</annotation></semantics></math> for that timestep:</p>
<table id="S2.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E6.m1.1" class="ltx_Math" alttext="\text{loss}_{t}=-\log P(w^{*}_{t})" display="block"><semantics id="S2.E6.m1.1a"><mrow id="S2.E6.m1.1.1" xref="S2.E6.m1.1.1.cmml"><msub id="S2.E6.m1.1.1.3" xref="S2.E6.m1.1.1.3.cmml"><mtext id="S2.E6.m1.1.1.3.2" xref="S2.E6.m1.1.1.3.2a.cmml">loss</mtext><mi id="S2.E6.m1.1.1.3.3" xref="S2.E6.m1.1.1.3.3.cmml">t</mi></msub><mo id="S2.E6.m1.1.1.2" xref="S2.E6.m1.1.1.2.cmml">=</mo><mrow id="S2.E6.m1.1.1.1" xref="S2.E6.m1.1.1.1.cmml"><mo rspace="0.167em" id="S2.E6.m1.1.1.1a" xref="S2.E6.m1.1.1.1.cmml">−</mo><mrow id="S2.E6.m1.1.1.1.1" xref="S2.E6.m1.1.1.1.1.cmml"><mrow id="S2.E6.m1.1.1.1.1.3" xref="S2.E6.m1.1.1.1.1.3.cmml"><mi id="S2.E6.m1.1.1.1.1.3.1" xref="S2.E6.m1.1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S2.E6.m1.1.1.1.1.3a" xref="S2.E6.m1.1.1.1.1.3.cmml">⁡</mo><mi id="S2.E6.m1.1.1.1.1.3.2" xref="S2.E6.m1.1.1.1.1.3.2.cmml">P</mi></mrow><mo lspace="0em" rspace="0em" id="S2.E6.m1.1.1.1.1.2" xref="S2.E6.m1.1.1.1.1.2.cmml">​</mo><mrow id="S2.E6.m1.1.1.1.1.1.1" xref="S2.E6.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E6.m1.1.1.1.1.1.1.2" xref="S2.E6.m1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.E6.m1.1.1.1.1.1.1.1" xref="S2.E6.m1.1.1.1.1.1.1.1.cmml"><mi id="S2.E6.m1.1.1.1.1.1.1.1.2.2" xref="S2.E6.m1.1.1.1.1.1.1.1.2.2.cmml">w</mi><mi id="S2.E6.m1.1.1.1.1.1.1.1.3" xref="S2.E6.m1.1.1.1.1.1.1.1.3.cmml">t</mi><mo id="S2.E6.m1.1.1.1.1.1.1.1.2.3" xref="S2.E6.m1.1.1.1.1.1.1.1.2.3.cmml">∗</mo></msubsup><mo stretchy="false" id="S2.E6.m1.1.1.1.1.1.1.3" xref="S2.E6.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E6.m1.1b"><apply id="S2.E6.m1.1.1.cmml" xref="S2.E6.m1.1.1"><eq id="S2.E6.m1.1.1.2.cmml" xref="S2.E6.m1.1.1.2"></eq><apply id="S2.E6.m1.1.1.3.cmml" xref="S2.E6.m1.1.1.3"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.3.1.cmml" xref="S2.E6.m1.1.1.3">subscript</csymbol><ci id="S2.E6.m1.1.1.3.2a.cmml" xref="S2.E6.m1.1.1.3.2"><mtext id="S2.E6.m1.1.1.3.2.cmml" xref="S2.E6.m1.1.1.3.2">loss</mtext></ci><ci id="S2.E6.m1.1.1.3.3.cmml" xref="S2.E6.m1.1.1.3.3">𝑡</ci></apply><apply id="S2.E6.m1.1.1.1.cmml" xref="S2.E6.m1.1.1.1"><minus id="S2.E6.m1.1.1.1.2.cmml" xref="S2.E6.m1.1.1.1"></minus><apply id="S2.E6.m1.1.1.1.1.cmml" xref="S2.E6.m1.1.1.1.1"><times id="S2.E6.m1.1.1.1.1.2.cmml" xref="S2.E6.m1.1.1.1.1.2"></times><apply id="S2.E6.m1.1.1.1.1.3.cmml" xref="S2.E6.m1.1.1.1.1.3"><log id="S2.E6.m1.1.1.1.1.3.1.cmml" xref="S2.E6.m1.1.1.1.1.3.1"></log><ci id="S2.E6.m1.1.1.1.1.3.2.cmml" xref="S2.E6.m1.1.1.1.1.3.2">𝑃</ci></apply><apply id="S2.E6.m1.1.1.1.1.1.1.1.cmml" xref="S2.E6.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.E6.m1.1.1.1.1.1.1">subscript</csymbol><apply id="S2.E6.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.E6.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E6.m1.1.1.1.1.1.1">superscript</csymbol><ci id="S2.E6.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.2.2">𝑤</ci><times id="S2.E6.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.2.3"></times></apply><ci id="S2.E6.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.E6.m1.1.1.1.1.1.1.1.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E6.m1.1c">\text{loss}_{t}=-\log P(w^{*}_{t})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="S2.SS1.p1.21" class="ltx_p">and the overall loss for the whole sequence is:</p>
<table id="S2.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E7.m1.1" class="ltx_Math" alttext="\text{loss}=\frac{1}{T}\sum\nolimits_{t=0}^{T}\text{loss}_{t}" display="block"><semantics id="S2.E7.m1.1a"><mrow id="S2.E7.m1.1.1" xref="S2.E7.m1.1.1.cmml"><mtext id="S2.E7.m1.1.1.2" xref="S2.E7.m1.1.1.2a.cmml">loss</mtext><mo id="S2.E7.m1.1.1.1" xref="S2.E7.m1.1.1.1.cmml">=</mo><mrow id="S2.E7.m1.1.1.3" xref="S2.E7.m1.1.1.3.cmml"><mfrac id="S2.E7.m1.1.1.3.2" xref="S2.E7.m1.1.1.3.2.cmml"><mn id="S2.E7.m1.1.1.3.2.2" xref="S2.E7.m1.1.1.3.2.2.cmml">1</mn><mi id="S2.E7.m1.1.1.3.2.3" xref="S2.E7.m1.1.1.3.2.3.cmml">T</mi></mfrac><mo lspace="0em" rspace="0em" id="S2.E7.m1.1.1.3.1" xref="S2.E7.m1.1.1.3.1.cmml">​</mo><mrow id="S2.E7.m1.1.1.3.3" xref="S2.E7.m1.1.1.3.3.cmml"><msubsup id="S2.E7.m1.1.1.3.3.1" xref="S2.E7.m1.1.1.3.3.1.cmml"><mo id="S2.E7.m1.1.1.3.3.1.2.2" xref="S2.E7.m1.1.1.3.3.1.2.2.cmml">∑</mo><mrow id="S2.E7.m1.1.1.3.3.1.2.3" xref="S2.E7.m1.1.1.3.3.1.2.3.cmml"><mi id="S2.E7.m1.1.1.3.3.1.2.3.2" xref="S2.E7.m1.1.1.3.3.1.2.3.2.cmml">t</mi><mo id="S2.E7.m1.1.1.3.3.1.2.3.1" xref="S2.E7.m1.1.1.3.3.1.2.3.1.cmml">=</mo><mn id="S2.E7.m1.1.1.3.3.1.2.3.3" xref="S2.E7.m1.1.1.3.3.1.2.3.3.cmml">0</mn></mrow><mi id="S2.E7.m1.1.1.3.3.1.3" xref="S2.E7.m1.1.1.3.3.1.3.cmml">T</mi></msubsup><msub id="S2.E7.m1.1.1.3.3.2" xref="S2.E7.m1.1.1.3.3.2.cmml"><mtext id="S2.E7.m1.1.1.3.3.2.2" xref="S2.E7.m1.1.1.3.3.2.2a.cmml">loss</mtext><mi id="S2.E7.m1.1.1.3.3.2.3" xref="S2.E7.m1.1.1.3.3.2.3.cmml">t</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E7.m1.1b"><apply id="S2.E7.m1.1.1.cmml" xref="S2.E7.m1.1.1"><eq id="S2.E7.m1.1.1.1.cmml" xref="S2.E7.m1.1.1.1"></eq><ci id="S2.E7.m1.1.1.2a.cmml" xref="S2.E7.m1.1.1.2"><mtext id="S2.E7.m1.1.1.2.cmml" xref="S2.E7.m1.1.1.2">loss</mtext></ci><apply id="S2.E7.m1.1.1.3.cmml" xref="S2.E7.m1.1.1.3"><times id="S2.E7.m1.1.1.3.1.cmml" xref="S2.E7.m1.1.1.3.1"></times><apply id="S2.E7.m1.1.1.3.2.cmml" xref="S2.E7.m1.1.1.3.2"><divide id="S2.E7.m1.1.1.3.2.1.cmml" xref="S2.E7.m1.1.1.3.2"></divide><cn type="integer" id="S2.E7.m1.1.1.3.2.2.cmml" xref="S2.E7.m1.1.1.3.2.2">1</cn><ci id="S2.E7.m1.1.1.3.2.3.cmml" xref="S2.E7.m1.1.1.3.2.3">𝑇</ci></apply><apply id="S2.E7.m1.1.1.3.3.cmml" xref="S2.E7.m1.1.1.3.3"><apply id="S2.E7.m1.1.1.3.3.1.cmml" xref="S2.E7.m1.1.1.3.3.1"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.3.3.1.1.cmml" xref="S2.E7.m1.1.1.3.3.1">superscript</csymbol><apply id="S2.E7.m1.1.1.3.3.1.2.cmml" xref="S2.E7.m1.1.1.3.3.1"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.3.3.1.2.1.cmml" xref="S2.E7.m1.1.1.3.3.1">subscript</csymbol><sum id="S2.E7.m1.1.1.3.3.1.2.2.cmml" xref="S2.E7.m1.1.1.3.3.1.2.2"></sum><apply id="S2.E7.m1.1.1.3.3.1.2.3.cmml" xref="S2.E7.m1.1.1.3.3.1.2.3"><eq id="S2.E7.m1.1.1.3.3.1.2.3.1.cmml" xref="S2.E7.m1.1.1.3.3.1.2.3.1"></eq><ci id="S2.E7.m1.1.1.3.3.1.2.3.2.cmml" xref="S2.E7.m1.1.1.3.3.1.2.3.2">𝑡</ci><cn type="integer" id="S2.E7.m1.1.1.3.3.1.2.3.3.cmml" xref="S2.E7.m1.1.1.3.3.1.2.3.3">0</cn></apply></apply><ci id="S2.E7.m1.1.1.3.3.1.3.cmml" xref="S2.E7.m1.1.1.3.3.1.3">𝑇</ci></apply><apply id="S2.E7.m1.1.1.3.3.2.cmml" xref="S2.E7.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.3.3.2.1.cmml" xref="S2.E7.m1.1.1.3.3.2">subscript</csymbol><ci id="S2.E7.m1.1.1.3.3.2.2a.cmml" xref="S2.E7.m1.1.1.3.3.2.2"><mtext id="S2.E7.m1.1.1.3.3.2.2.cmml" xref="S2.E7.m1.1.1.3.3.2.2">loss</mtext></ci><ci id="S2.E7.m1.1.1.3.3.2.3.cmml" xref="S2.E7.m1.1.1.3.3.2.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E7.m1.1c">\text{loss}=\frac{1}{T}\sum\nolimits_{t=0}^{T}\text{loss}_{t}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Pointer-generator network</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.7" class="ltx_p">Our pointer-generator network is a hybrid between our baseline and a pointer network <cite class="ltx_cite ltx_citemacro_cite">Vinyals et al. (<a href="#bib.bib29" title="" class="ltx_ref">2015</a>)</cite>, as it allows both copying words via pointing, and generating words from a fixed vocabulary.
In the pointer-generator model (depicted in Figure <a href="#S2.F3" title="Figure 3 ‣ 2 Our Models ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>) the attention distribution <math id="S2.SS2.p1.1.m1.1" class="ltx_Math" alttext="a^{t}" display="inline"><semantics id="S2.SS2.p1.1.m1.1a"><msup id="S2.SS2.p1.1.m1.1.1" xref="S2.SS2.p1.1.m1.1.1.cmml"><mi id="S2.SS2.p1.1.m1.1.1.2" xref="S2.SS2.p1.1.m1.1.1.2.cmml">a</mi><mi id="S2.SS2.p1.1.m1.1.1.3" xref="S2.SS2.p1.1.m1.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.1.m1.1b"><apply id="S2.SS2.p1.1.m1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.1.m1.1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1">superscript</csymbol><ci id="S2.SS2.p1.1.m1.1.1.2.cmml" xref="S2.SS2.p1.1.m1.1.1.2">𝑎</ci><ci id="S2.SS2.p1.1.m1.1.1.3.cmml" xref="S2.SS2.p1.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.1.m1.1c">a^{t}</annotation></semantics></math> and context vector <math id="S2.SS2.p1.2.m2.1" class="ltx_Math" alttext="h^{*}_{t}" display="inline"><semantics id="S2.SS2.p1.2.m2.1a"><msubsup id="S2.SS2.p1.2.m2.1.1" xref="S2.SS2.p1.2.m2.1.1.cmml"><mi id="S2.SS2.p1.2.m2.1.1.2.2" xref="S2.SS2.p1.2.m2.1.1.2.2.cmml">h</mi><mi id="S2.SS2.p1.2.m2.1.1.3" xref="S2.SS2.p1.2.m2.1.1.3.cmml">t</mi><mo id="S2.SS2.p1.2.m2.1.1.2.3" xref="S2.SS2.p1.2.m2.1.1.2.3.cmml">∗</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.2.m2.1b"><apply id="S2.SS2.p1.2.m2.1.1.cmml" xref="S2.SS2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.2.m2.1.1.1.cmml" xref="S2.SS2.p1.2.m2.1.1">subscript</csymbol><apply id="S2.SS2.p1.2.m2.1.1.2.cmml" xref="S2.SS2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.2.m2.1.1.2.1.cmml" xref="S2.SS2.p1.2.m2.1.1">superscript</csymbol><ci id="S2.SS2.p1.2.m2.1.1.2.2.cmml" xref="S2.SS2.p1.2.m2.1.1.2.2">ℎ</ci><times id="S2.SS2.p1.2.m2.1.1.2.3.cmml" xref="S2.SS2.p1.2.m2.1.1.2.3"></times></apply><ci id="S2.SS2.p1.2.m2.1.1.3.cmml" xref="S2.SS2.p1.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.2.m2.1c">h^{*}_{t}</annotation></semantics></math> are calculated as in section <a href="#S2.SS1" title="2.1 Sequence-to-sequence attentional model ‣ 2 Our Models ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>.
In addition, the <span id="S2.SS2.p1.7.1" class="ltx_text ltx_font_italic">generation probability</span> <math id="S2.SS2.p1.3.m3.2" class="ltx_Math" alttext="p_{\text{gen}}\in[0,1]" display="inline"><semantics id="S2.SS2.p1.3.m3.2a"><mrow id="S2.SS2.p1.3.m3.2.3" xref="S2.SS2.p1.3.m3.2.3.cmml"><msub id="S2.SS2.p1.3.m3.2.3.2" xref="S2.SS2.p1.3.m3.2.3.2.cmml"><mi id="S2.SS2.p1.3.m3.2.3.2.2" xref="S2.SS2.p1.3.m3.2.3.2.2.cmml">p</mi><mtext id="S2.SS2.p1.3.m3.2.3.2.3" xref="S2.SS2.p1.3.m3.2.3.2.3a.cmml">gen</mtext></msub><mo id="S2.SS2.p1.3.m3.2.3.1" xref="S2.SS2.p1.3.m3.2.3.1.cmml">∈</mo><mrow id="S2.SS2.p1.3.m3.2.3.3.2" xref="S2.SS2.p1.3.m3.2.3.3.1.cmml"><mo stretchy="false" id="S2.SS2.p1.3.m3.2.3.3.2.1" xref="S2.SS2.p1.3.m3.2.3.3.1.cmml">[</mo><mn id="S2.SS2.p1.3.m3.1.1" xref="S2.SS2.p1.3.m3.1.1.cmml">0</mn><mo id="S2.SS2.p1.3.m3.2.3.3.2.2" xref="S2.SS2.p1.3.m3.2.3.3.1.cmml">,</mo><mn id="S2.SS2.p1.3.m3.2.2" xref="S2.SS2.p1.3.m3.2.2.cmml">1</mn><mo stretchy="false" id="S2.SS2.p1.3.m3.2.3.3.2.3" xref="S2.SS2.p1.3.m3.2.3.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.3.m3.2b"><apply id="S2.SS2.p1.3.m3.2.3.cmml" xref="S2.SS2.p1.3.m3.2.3"><in id="S2.SS2.p1.3.m3.2.3.1.cmml" xref="S2.SS2.p1.3.m3.2.3.1"></in><apply id="S2.SS2.p1.3.m3.2.3.2.cmml" xref="S2.SS2.p1.3.m3.2.3.2"><csymbol cd="ambiguous" id="S2.SS2.p1.3.m3.2.3.2.1.cmml" xref="S2.SS2.p1.3.m3.2.3.2">subscript</csymbol><ci id="S2.SS2.p1.3.m3.2.3.2.2.cmml" xref="S2.SS2.p1.3.m3.2.3.2.2">𝑝</ci><ci id="S2.SS2.p1.3.m3.2.3.2.3a.cmml" xref="S2.SS2.p1.3.m3.2.3.2.3"><mtext mathsize="70%" id="S2.SS2.p1.3.m3.2.3.2.3.cmml" xref="S2.SS2.p1.3.m3.2.3.2.3">gen</mtext></ci></apply><interval closure="closed" id="S2.SS2.p1.3.m3.2.3.3.1.cmml" xref="S2.SS2.p1.3.m3.2.3.3.2"><cn type="integer" id="S2.SS2.p1.3.m3.1.1.cmml" xref="S2.SS2.p1.3.m3.1.1">0</cn><cn type="integer" id="S2.SS2.p1.3.m3.2.2.cmml" xref="S2.SS2.p1.3.m3.2.2">1</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.3.m3.2c">p_{\text{gen}}\in[0,1]</annotation></semantics></math> for timestep <math id="S2.SS2.p1.4.m4.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.SS2.p1.4.m4.1a"><mi id="S2.SS2.p1.4.m4.1.1" xref="S2.SS2.p1.4.m4.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.4.m4.1b"><ci id="S2.SS2.p1.4.m4.1.1.cmml" xref="S2.SS2.p1.4.m4.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.4.m4.1c">t</annotation></semantics></math> is calculated from the context vector <math id="S2.SS2.p1.5.m5.1" class="ltx_Math" alttext="h^{*}_{t}" display="inline"><semantics id="S2.SS2.p1.5.m5.1a"><msubsup id="S2.SS2.p1.5.m5.1.1" xref="S2.SS2.p1.5.m5.1.1.cmml"><mi id="S2.SS2.p1.5.m5.1.1.2.2" xref="S2.SS2.p1.5.m5.1.1.2.2.cmml">h</mi><mi id="S2.SS2.p1.5.m5.1.1.3" xref="S2.SS2.p1.5.m5.1.1.3.cmml">t</mi><mo id="S2.SS2.p1.5.m5.1.1.2.3" xref="S2.SS2.p1.5.m5.1.1.2.3.cmml">∗</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.5.m5.1b"><apply id="S2.SS2.p1.5.m5.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.5.m5.1.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1">subscript</csymbol><apply id="S2.SS2.p1.5.m5.1.1.2.cmml" xref="S2.SS2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.5.m5.1.1.2.1.cmml" xref="S2.SS2.p1.5.m5.1.1">superscript</csymbol><ci id="S2.SS2.p1.5.m5.1.1.2.2.cmml" xref="S2.SS2.p1.5.m5.1.1.2.2">ℎ</ci><times id="S2.SS2.p1.5.m5.1.1.2.3.cmml" xref="S2.SS2.p1.5.m5.1.1.2.3"></times></apply><ci id="S2.SS2.p1.5.m5.1.1.3.cmml" xref="S2.SS2.p1.5.m5.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.5.m5.1c">h^{*}_{t}</annotation></semantics></math>, the decoder state <math id="S2.SS2.p1.6.m6.1" class="ltx_Math" alttext="s_{t}" display="inline"><semantics id="S2.SS2.p1.6.m6.1a"><msub id="S2.SS2.p1.6.m6.1.1" xref="S2.SS2.p1.6.m6.1.1.cmml"><mi id="S2.SS2.p1.6.m6.1.1.2" xref="S2.SS2.p1.6.m6.1.1.2.cmml">s</mi><mi id="S2.SS2.p1.6.m6.1.1.3" xref="S2.SS2.p1.6.m6.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.6.m6.1b"><apply id="S2.SS2.p1.6.m6.1.1.cmml" xref="S2.SS2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.6.m6.1.1.1.cmml" xref="S2.SS2.p1.6.m6.1.1">subscript</csymbol><ci id="S2.SS2.p1.6.m6.1.1.2.cmml" xref="S2.SS2.p1.6.m6.1.1.2">𝑠</ci><ci id="S2.SS2.p1.6.m6.1.1.3.cmml" xref="S2.SS2.p1.6.m6.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.6.m6.1c">s_{t}</annotation></semantics></math> and the decoder input <math id="S2.SS2.p1.7.m7.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS2.p1.7.m7.1a"><msub id="S2.SS2.p1.7.m7.1.1" xref="S2.SS2.p1.7.m7.1.1.cmml"><mi id="S2.SS2.p1.7.m7.1.1.2" xref="S2.SS2.p1.7.m7.1.1.2.cmml">x</mi><mi id="S2.SS2.p1.7.m7.1.1.3" xref="S2.SS2.p1.7.m7.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.7.m7.1b"><apply id="S2.SS2.p1.7.m7.1.1.cmml" xref="S2.SS2.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.7.m7.1.1.1.cmml" xref="S2.SS2.p1.7.m7.1.1">subscript</csymbol><ci id="S2.SS2.p1.7.m7.1.1.2.cmml" xref="S2.SS2.p1.7.m7.1.1.2">𝑥</ci><ci id="S2.SS2.p1.7.m7.1.1.3.cmml" xref="S2.SS2.p1.7.m7.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.7.m7.1c">x_{t}</annotation></semantics></math>:</p>
<table id="S2.E8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E8.m1.1" class="ltx_Math" alttext="p_{\text{gen}}=\sigma(w_{h^{*}}^{T}h^{*}_{t}+w_{s}^{T}s_{t}+w_{x}^{T}x_{t}+b_{\text{ptr}})" display="block"><semantics id="S2.E8.m1.1a"><mrow id="S2.E8.m1.1.1" xref="S2.E8.m1.1.1.cmml"><msub id="S2.E8.m1.1.1.3" xref="S2.E8.m1.1.1.3.cmml"><mi id="S2.E8.m1.1.1.3.2" xref="S2.E8.m1.1.1.3.2.cmml">p</mi><mtext id="S2.E8.m1.1.1.3.3" xref="S2.E8.m1.1.1.3.3a.cmml">gen</mtext></msub><mo id="S2.E8.m1.1.1.2" xref="S2.E8.m1.1.1.2.cmml">=</mo><mrow id="S2.E8.m1.1.1.1" xref="S2.E8.m1.1.1.1.cmml"><mi id="S2.E8.m1.1.1.1.3" xref="S2.E8.m1.1.1.1.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S2.E8.m1.1.1.1.2" xref="S2.E8.m1.1.1.1.2.cmml">​</mo><mrow id="S2.E8.m1.1.1.1.1.1" xref="S2.E8.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E8.m1.1.1.1.1.1.2" xref="S2.E8.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E8.m1.1.1.1.1.1.1" xref="S2.E8.m1.1.1.1.1.1.1.cmml"><mrow id="S2.E8.m1.1.1.1.1.1.1.2" xref="S2.E8.m1.1.1.1.1.1.1.2.cmml"><msubsup id="S2.E8.m1.1.1.1.1.1.1.2.2" xref="S2.E8.m1.1.1.1.1.1.1.2.2.cmml"><mi id="S2.E8.m1.1.1.1.1.1.1.2.2.2.2" xref="S2.E8.m1.1.1.1.1.1.1.2.2.2.2.cmml">w</mi><msup id="S2.E8.m1.1.1.1.1.1.1.2.2.2.3" xref="S2.E8.m1.1.1.1.1.1.1.2.2.2.3.cmml"><mi id="S2.E8.m1.1.1.1.1.1.1.2.2.2.3.2" xref="S2.E8.m1.1.1.1.1.1.1.2.2.2.3.2.cmml">h</mi><mo id="S2.E8.m1.1.1.1.1.1.1.2.2.2.3.3" xref="S2.E8.m1.1.1.1.1.1.1.2.2.2.3.3.cmml">∗</mo></msup><mi id="S2.E8.m1.1.1.1.1.1.1.2.2.3" xref="S2.E8.m1.1.1.1.1.1.1.2.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.E8.m1.1.1.1.1.1.1.2.1" xref="S2.E8.m1.1.1.1.1.1.1.2.1.cmml">​</mo><msubsup id="S2.E8.m1.1.1.1.1.1.1.2.3" xref="S2.E8.m1.1.1.1.1.1.1.2.3.cmml"><mi id="S2.E8.m1.1.1.1.1.1.1.2.3.2.2" xref="S2.E8.m1.1.1.1.1.1.1.2.3.2.2.cmml">h</mi><mi id="S2.E8.m1.1.1.1.1.1.1.2.3.3" xref="S2.E8.m1.1.1.1.1.1.1.2.3.3.cmml">t</mi><mo id="S2.E8.m1.1.1.1.1.1.1.2.3.2.3" xref="S2.E8.m1.1.1.1.1.1.1.2.3.2.3.cmml">∗</mo></msubsup></mrow><mo id="S2.E8.m1.1.1.1.1.1.1.1" xref="S2.E8.m1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="S2.E8.m1.1.1.1.1.1.1.3" xref="S2.E8.m1.1.1.1.1.1.1.3.cmml"><msubsup id="S2.E8.m1.1.1.1.1.1.1.3.2" xref="S2.E8.m1.1.1.1.1.1.1.3.2.cmml"><mi id="S2.E8.m1.1.1.1.1.1.1.3.2.2.2" xref="S2.E8.m1.1.1.1.1.1.1.3.2.2.2.cmml">w</mi><mi id="S2.E8.m1.1.1.1.1.1.1.3.2.2.3" xref="S2.E8.m1.1.1.1.1.1.1.3.2.2.3.cmml">s</mi><mi id="S2.E8.m1.1.1.1.1.1.1.3.2.3" xref="S2.E8.m1.1.1.1.1.1.1.3.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.E8.m1.1.1.1.1.1.1.3.1" xref="S2.E8.m1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S2.E8.m1.1.1.1.1.1.1.3.3" xref="S2.E8.m1.1.1.1.1.1.1.3.3.cmml"><mi id="S2.E8.m1.1.1.1.1.1.1.3.3.2" xref="S2.E8.m1.1.1.1.1.1.1.3.3.2.cmml">s</mi><mi id="S2.E8.m1.1.1.1.1.1.1.3.3.3" xref="S2.E8.m1.1.1.1.1.1.1.3.3.3.cmml">t</mi></msub></mrow><mo id="S2.E8.m1.1.1.1.1.1.1.1a" xref="S2.E8.m1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="S2.E8.m1.1.1.1.1.1.1.4" xref="S2.E8.m1.1.1.1.1.1.1.4.cmml"><msubsup id="S2.E8.m1.1.1.1.1.1.1.4.2" xref="S2.E8.m1.1.1.1.1.1.1.4.2.cmml"><mi id="S2.E8.m1.1.1.1.1.1.1.4.2.2.2" xref="S2.E8.m1.1.1.1.1.1.1.4.2.2.2.cmml">w</mi><mi id="S2.E8.m1.1.1.1.1.1.1.4.2.2.3" xref="S2.E8.m1.1.1.1.1.1.1.4.2.2.3.cmml">x</mi><mi id="S2.E8.m1.1.1.1.1.1.1.4.2.3" xref="S2.E8.m1.1.1.1.1.1.1.4.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.E8.m1.1.1.1.1.1.1.4.1" xref="S2.E8.m1.1.1.1.1.1.1.4.1.cmml">​</mo><msub id="S2.E8.m1.1.1.1.1.1.1.4.3" xref="S2.E8.m1.1.1.1.1.1.1.4.3.cmml"><mi id="S2.E8.m1.1.1.1.1.1.1.4.3.2" xref="S2.E8.m1.1.1.1.1.1.1.4.3.2.cmml">x</mi><mi id="S2.E8.m1.1.1.1.1.1.1.4.3.3" xref="S2.E8.m1.1.1.1.1.1.1.4.3.3.cmml">t</mi></msub></mrow><mo id="S2.E8.m1.1.1.1.1.1.1.1b" xref="S2.E8.m1.1.1.1.1.1.1.1.cmml">+</mo><msub id="S2.E8.m1.1.1.1.1.1.1.5" xref="S2.E8.m1.1.1.1.1.1.1.5.cmml"><mi id="S2.E8.m1.1.1.1.1.1.1.5.2" xref="S2.E8.m1.1.1.1.1.1.1.5.2.cmml">b</mi><mtext id="S2.E8.m1.1.1.1.1.1.1.5.3" xref="S2.E8.m1.1.1.1.1.1.1.5.3a.cmml">ptr</mtext></msub></mrow><mo stretchy="false" id="S2.E8.m1.1.1.1.1.1.3" xref="S2.E8.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E8.m1.1b"><apply id="S2.E8.m1.1.1.cmml" xref="S2.E8.m1.1.1"><eq id="S2.E8.m1.1.1.2.cmml" xref="S2.E8.m1.1.1.2"></eq><apply id="S2.E8.m1.1.1.3.cmml" xref="S2.E8.m1.1.1.3"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.3.1.cmml" xref="S2.E8.m1.1.1.3">subscript</csymbol><ci id="S2.E8.m1.1.1.3.2.cmml" xref="S2.E8.m1.1.1.3.2">𝑝</ci><ci id="S2.E8.m1.1.1.3.3a.cmml" xref="S2.E8.m1.1.1.3.3"><mtext mathsize="70%" id="S2.E8.m1.1.1.3.3.cmml" xref="S2.E8.m1.1.1.3.3">gen</mtext></ci></apply><apply id="S2.E8.m1.1.1.1.cmml" xref="S2.E8.m1.1.1.1"><times id="S2.E8.m1.1.1.1.2.cmml" xref="S2.E8.m1.1.1.1.2"></times><ci id="S2.E8.m1.1.1.1.3.cmml" xref="S2.E8.m1.1.1.1.3">𝜎</ci><apply id="S2.E8.m1.1.1.1.1.1.1.cmml" xref="S2.E8.m1.1.1.1.1.1"><plus id="S2.E8.m1.1.1.1.1.1.1.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.1"></plus><apply id="S2.E8.m1.1.1.1.1.1.1.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2"><times id="S2.E8.m1.1.1.1.1.1.1.2.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.1"></times><apply id="S2.E8.m1.1.1.1.1.1.1.2.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.2.2.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.2">superscript</csymbol><apply id="S2.E8.m1.1.1.1.1.1.1.2.2.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.2.2.2.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="S2.E8.m1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.2.2.2">𝑤</ci><apply id="S2.E8.m1.1.1.1.1.1.1.2.2.2.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.2.2.3"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.2.2.2.3.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.2.2.3">superscript</csymbol><ci id="S2.E8.m1.1.1.1.1.1.1.2.2.2.3.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.2.2.3.2">ℎ</ci><times id="S2.E8.m1.1.1.1.1.1.1.2.2.2.3.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.2.2.3.3"></times></apply></apply><ci id="S2.E8.m1.1.1.1.1.1.1.2.2.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.2.3">𝑇</ci></apply><apply id="S2.E8.m1.1.1.1.1.1.1.2.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.2.3.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.3">subscript</csymbol><apply id="S2.E8.m1.1.1.1.1.1.1.2.3.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.2.3.2.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.3">superscript</csymbol><ci id="S2.E8.m1.1.1.1.1.1.1.2.3.2.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.3.2.2">ℎ</ci><times id="S2.E8.m1.1.1.1.1.1.1.2.3.2.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.3.2.3"></times></apply><ci id="S2.E8.m1.1.1.1.1.1.1.2.3.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.2.3.3">𝑡</ci></apply></apply><apply id="S2.E8.m1.1.1.1.1.1.1.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3"><times id="S2.E8.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.1"></times><apply id="S2.E8.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.3.2.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.2">superscript</csymbol><apply id="S2.E8.m1.1.1.1.1.1.1.3.2.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.3.2.2.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S2.E8.m1.1.1.1.1.1.1.3.2.2.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.2.2.2">𝑤</ci><ci id="S2.E8.m1.1.1.1.1.1.1.3.2.2.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.2.2.3">𝑠</ci></apply><ci id="S2.E8.m1.1.1.1.1.1.1.3.2.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.2.3">𝑇</ci></apply><apply id="S2.E8.m1.1.1.1.1.1.1.3.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.3.3.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S2.E8.m1.1.1.1.1.1.1.3.3.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.3.2">𝑠</ci><ci id="S2.E8.m1.1.1.1.1.1.1.3.3.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply><apply id="S2.E8.m1.1.1.1.1.1.1.4.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4"><times id="S2.E8.m1.1.1.1.1.1.1.4.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.1"></times><apply id="S2.E8.m1.1.1.1.1.1.1.4.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.2"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.4.2.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.2">superscript</csymbol><apply id="S2.E8.m1.1.1.1.1.1.1.4.2.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.2"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.4.2.2.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.2">subscript</csymbol><ci id="S2.E8.m1.1.1.1.1.1.1.4.2.2.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.2.2.2">𝑤</ci><ci id="S2.E8.m1.1.1.1.1.1.1.4.2.2.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.2.2.3">𝑥</ci></apply><ci id="S2.E8.m1.1.1.1.1.1.1.4.2.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.2.3">𝑇</ci></apply><apply id="S2.E8.m1.1.1.1.1.1.1.4.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.3"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.4.3.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.3">subscript</csymbol><ci id="S2.E8.m1.1.1.1.1.1.1.4.3.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.3.2">𝑥</ci><ci id="S2.E8.m1.1.1.1.1.1.1.4.3.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.4.3.3">𝑡</ci></apply></apply><apply id="S2.E8.m1.1.1.1.1.1.1.5.cmml" xref="S2.E8.m1.1.1.1.1.1.1.5"><csymbol cd="ambiguous" id="S2.E8.m1.1.1.1.1.1.1.5.1.cmml" xref="S2.E8.m1.1.1.1.1.1.1.5">subscript</csymbol><ci id="S2.E8.m1.1.1.1.1.1.1.5.2.cmml" xref="S2.E8.m1.1.1.1.1.1.1.5.2">𝑏</ci><ci id="S2.E8.m1.1.1.1.1.1.1.5.3a.cmml" xref="S2.E8.m1.1.1.1.1.1.1.5.3"><mtext mathsize="70%" id="S2.E8.m1.1.1.1.1.1.1.5.3.cmml" xref="S2.E8.m1.1.1.1.1.1.1.5.3">ptr</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E8.m1.1c">p_{\text{gen}}=\sigma(w_{h^{*}}^{T}h^{*}_{t}+w_{s}^{T}s_{t}+w_{x}^{T}x_{t}+b_{\text{ptr}})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<p id="S2.SS2.p1.15" class="ltx_p">where vectors <math id="S2.SS2.p1.8.m1.1" class="ltx_Math" alttext="w_{h^{*}}" display="inline"><semantics id="S2.SS2.p1.8.m1.1a"><msub id="S2.SS2.p1.8.m1.1.1" xref="S2.SS2.p1.8.m1.1.1.cmml"><mi id="S2.SS2.p1.8.m1.1.1.2" xref="S2.SS2.p1.8.m1.1.1.2.cmml">w</mi><msup id="S2.SS2.p1.8.m1.1.1.3" xref="S2.SS2.p1.8.m1.1.1.3.cmml"><mi id="S2.SS2.p1.8.m1.1.1.3.2" xref="S2.SS2.p1.8.m1.1.1.3.2.cmml">h</mi><mo id="S2.SS2.p1.8.m1.1.1.3.3" xref="S2.SS2.p1.8.m1.1.1.3.3.cmml">∗</mo></msup></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.8.m1.1b"><apply id="S2.SS2.p1.8.m1.1.1.cmml" xref="S2.SS2.p1.8.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.8.m1.1.1.1.cmml" xref="S2.SS2.p1.8.m1.1.1">subscript</csymbol><ci id="S2.SS2.p1.8.m1.1.1.2.cmml" xref="S2.SS2.p1.8.m1.1.1.2">𝑤</ci><apply id="S2.SS2.p1.8.m1.1.1.3.cmml" xref="S2.SS2.p1.8.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p1.8.m1.1.1.3.1.cmml" xref="S2.SS2.p1.8.m1.1.1.3">superscript</csymbol><ci id="S2.SS2.p1.8.m1.1.1.3.2.cmml" xref="S2.SS2.p1.8.m1.1.1.3.2">ℎ</ci><times id="S2.SS2.p1.8.m1.1.1.3.3.cmml" xref="S2.SS2.p1.8.m1.1.1.3.3"></times></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.8.m1.1c">w_{h^{*}}</annotation></semantics></math>, <math id="S2.SS2.p1.9.m2.1" class="ltx_Math" alttext="w_{s}" display="inline"><semantics id="S2.SS2.p1.9.m2.1a"><msub id="S2.SS2.p1.9.m2.1.1" xref="S2.SS2.p1.9.m2.1.1.cmml"><mi id="S2.SS2.p1.9.m2.1.1.2" xref="S2.SS2.p1.9.m2.1.1.2.cmml">w</mi><mi id="S2.SS2.p1.9.m2.1.1.3" xref="S2.SS2.p1.9.m2.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.9.m2.1b"><apply id="S2.SS2.p1.9.m2.1.1.cmml" xref="S2.SS2.p1.9.m2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.9.m2.1.1.1.cmml" xref="S2.SS2.p1.9.m2.1.1">subscript</csymbol><ci id="S2.SS2.p1.9.m2.1.1.2.cmml" xref="S2.SS2.p1.9.m2.1.1.2">𝑤</ci><ci id="S2.SS2.p1.9.m2.1.1.3.cmml" xref="S2.SS2.p1.9.m2.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.9.m2.1c">w_{s}</annotation></semantics></math>, <math id="S2.SS2.p1.10.m3.1" class="ltx_Math" alttext="w_{x}" display="inline"><semantics id="S2.SS2.p1.10.m3.1a"><msub id="S2.SS2.p1.10.m3.1.1" xref="S2.SS2.p1.10.m3.1.1.cmml"><mi id="S2.SS2.p1.10.m3.1.1.2" xref="S2.SS2.p1.10.m3.1.1.2.cmml">w</mi><mi id="S2.SS2.p1.10.m3.1.1.3" xref="S2.SS2.p1.10.m3.1.1.3.cmml">x</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.10.m3.1b"><apply id="S2.SS2.p1.10.m3.1.1.cmml" xref="S2.SS2.p1.10.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.10.m3.1.1.1.cmml" xref="S2.SS2.p1.10.m3.1.1">subscript</csymbol><ci id="S2.SS2.p1.10.m3.1.1.2.cmml" xref="S2.SS2.p1.10.m3.1.1.2">𝑤</ci><ci id="S2.SS2.p1.10.m3.1.1.3.cmml" xref="S2.SS2.p1.10.m3.1.1.3">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.10.m3.1c">w_{x}</annotation></semantics></math> and scalar <math id="S2.SS2.p1.11.m4.1" class="ltx_Math" alttext="b_{\text{ptr}}" display="inline"><semantics id="S2.SS2.p1.11.m4.1a"><msub id="S2.SS2.p1.11.m4.1.1" xref="S2.SS2.p1.11.m4.1.1.cmml"><mi id="S2.SS2.p1.11.m4.1.1.2" xref="S2.SS2.p1.11.m4.1.1.2.cmml">b</mi><mtext id="S2.SS2.p1.11.m4.1.1.3" xref="S2.SS2.p1.11.m4.1.1.3a.cmml">ptr</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.11.m4.1b"><apply id="S2.SS2.p1.11.m4.1.1.cmml" xref="S2.SS2.p1.11.m4.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.11.m4.1.1.1.cmml" xref="S2.SS2.p1.11.m4.1.1">subscript</csymbol><ci id="S2.SS2.p1.11.m4.1.1.2.cmml" xref="S2.SS2.p1.11.m4.1.1.2">𝑏</ci><ci id="S2.SS2.p1.11.m4.1.1.3a.cmml" xref="S2.SS2.p1.11.m4.1.1.3"><mtext mathsize="70%" id="S2.SS2.p1.11.m4.1.1.3.cmml" xref="S2.SS2.p1.11.m4.1.1.3">ptr</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.11.m4.1c">b_{\text{ptr}}</annotation></semantics></math> are learnable parameters and <math id="S2.SS2.p1.12.m5.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="S2.SS2.p1.12.m5.1a"><mi id="S2.SS2.p1.12.m5.1.1" xref="S2.SS2.p1.12.m5.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.12.m5.1b"><ci id="S2.SS2.p1.12.m5.1.1.cmml" xref="S2.SS2.p1.12.m5.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.12.m5.1c">\sigma</annotation></semantics></math> is the sigmoid function.
Next, <math id="S2.SS2.p1.13.m6.1" class="ltx_Math" alttext="p_{\text{gen}}" display="inline"><semantics id="S2.SS2.p1.13.m6.1a"><msub id="S2.SS2.p1.13.m6.1.1" xref="S2.SS2.p1.13.m6.1.1.cmml"><mi id="S2.SS2.p1.13.m6.1.1.2" xref="S2.SS2.p1.13.m6.1.1.2.cmml">p</mi><mtext id="S2.SS2.p1.13.m6.1.1.3" xref="S2.SS2.p1.13.m6.1.1.3a.cmml">gen</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.13.m6.1b"><apply id="S2.SS2.p1.13.m6.1.1.cmml" xref="S2.SS2.p1.13.m6.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.13.m6.1.1.1.cmml" xref="S2.SS2.p1.13.m6.1.1">subscript</csymbol><ci id="S2.SS2.p1.13.m6.1.1.2.cmml" xref="S2.SS2.p1.13.m6.1.1.2">𝑝</ci><ci id="S2.SS2.p1.13.m6.1.1.3a.cmml" xref="S2.SS2.p1.13.m6.1.1.3"><mtext mathsize="70%" id="S2.SS2.p1.13.m6.1.1.3.cmml" xref="S2.SS2.p1.13.m6.1.1.3">gen</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.13.m6.1c">p_{\text{gen}}</annotation></semantics></math> is used as a soft switch to choose between <span id="S2.SS2.p1.15.1" class="ltx_text ltx_font_italic">generating</span> a word from the vocabulary by sampling from <math id="S2.SS2.p1.14.m7.1" class="ltx_Math" alttext="P_{\text{vocab}}" display="inline"><semantics id="S2.SS2.p1.14.m7.1a"><msub id="S2.SS2.p1.14.m7.1.1" xref="S2.SS2.p1.14.m7.1.1.cmml"><mi id="S2.SS2.p1.14.m7.1.1.2" xref="S2.SS2.p1.14.m7.1.1.2.cmml">P</mi><mtext id="S2.SS2.p1.14.m7.1.1.3" xref="S2.SS2.p1.14.m7.1.1.3a.cmml">vocab</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.14.m7.1b"><apply id="S2.SS2.p1.14.m7.1.1.cmml" xref="S2.SS2.p1.14.m7.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.14.m7.1.1.1.cmml" xref="S2.SS2.p1.14.m7.1.1">subscript</csymbol><ci id="S2.SS2.p1.14.m7.1.1.2.cmml" xref="S2.SS2.p1.14.m7.1.1.2">𝑃</ci><ci id="S2.SS2.p1.14.m7.1.1.3a.cmml" xref="S2.SS2.p1.14.m7.1.1.3"><mtext mathsize="70%" id="S2.SS2.p1.14.m7.1.1.3.cmml" xref="S2.SS2.p1.14.m7.1.1.3">vocab</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.14.m7.1c">P_{\text{vocab}}</annotation></semantics></math>, or <span id="S2.SS2.p1.15.2" class="ltx_text ltx_font_italic">copying</span> a word from the input sequence by sampling from the attention distribution <math id="S2.SS2.p1.15.m8.1" class="ltx_Math" alttext="a^{t}" display="inline"><semantics id="S2.SS2.p1.15.m8.1a"><msup id="S2.SS2.p1.15.m8.1.1" xref="S2.SS2.p1.15.m8.1.1.cmml"><mi id="S2.SS2.p1.15.m8.1.1.2" xref="S2.SS2.p1.15.m8.1.1.2.cmml">a</mi><mi id="S2.SS2.p1.15.m8.1.1.3" xref="S2.SS2.p1.15.m8.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.15.m8.1b"><apply id="S2.SS2.p1.15.m8.1.1.cmml" xref="S2.SS2.p1.15.m8.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.15.m8.1.1.1.cmml" xref="S2.SS2.p1.15.m8.1.1">superscript</csymbol><ci id="S2.SS2.p1.15.m8.1.1.2.cmml" xref="S2.SS2.p1.15.m8.1.1.2">𝑎</ci><ci id="S2.SS2.p1.15.m8.1.1.3.cmml" xref="S2.SS2.p1.15.m8.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.15.m8.1c">a^{t}</annotation></semantics></math>.
For each document let the <span id="S2.SS2.p1.15.3" class="ltx_text ltx_font_italic">extended vocabulary</span> denote the union of the vocabulary, and all words appearing in the source document.
We obtain the following probability distribution over the extended vocabulary:</p>
<table id="S2.E9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E9.m1.3" class="ltx_Math" alttext="P(w)=p_{\text{gen}}P_{\text{vocab}}(w)+(1-p_{\text{gen}})\sum\nolimits_{i:w_{i}=w}a^{t}_{i}" display="block"><semantics id="S2.E9.m1.3a"><mrow id="S2.E9.m1.3.3" xref="S2.E9.m1.3.3.cmml"><mrow id="S2.E9.m1.3.3.3" xref="S2.E9.m1.3.3.3.cmml"><mi id="S2.E9.m1.3.3.3.2" xref="S2.E9.m1.3.3.3.2.cmml">P</mi><mo lspace="0em" rspace="0em" id="S2.E9.m1.3.3.3.1" xref="S2.E9.m1.3.3.3.1.cmml">​</mo><mrow id="S2.E9.m1.3.3.3.3.2" xref="S2.E9.m1.3.3.3.cmml"><mo stretchy="false" id="S2.E9.m1.3.3.3.3.2.1" xref="S2.E9.m1.3.3.3.cmml">(</mo><mi id="S2.E9.m1.1.1" xref="S2.E9.m1.1.1.cmml">w</mi><mo stretchy="false" id="S2.E9.m1.3.3.3.3.2.2" xref="S2.E9.m1.3.3.3.cmml">)</mo></mrow></mrow><mo id="S2.E9.m1.3.3.2" xref="S2.E9.m1.3.3.2.cmml">=</mo><mrow id="S2.E9.m1.3.3.1" xref="S2.E9.m1.3.3.1.cmml"><mrow id="S2.E9.m1.3.3.1.3" xref="S2.E9.m1.3.3.1.3.cmml"><msub id="S2.E9.m1.3.3.1.3.2" xref="S2.E9.m1.3.3.1.3.2.cmml"><mi id="S2.E9.m1.3.3.1.3.2.2" xref="S2.E9.m1.3.3.1.3.2.2.cmml">p</mi><mtext id="S2.E9.m1.3.3.1.3.2.3" xref="S2.E9.m1.3.3.1.3.2.3a.cmml">gen</mtext></msub><mo lspace="0em" rspace="0em" id="S2.E9.m1.3.3.1.3.1" xref="S2.E9.m1.3.3.1.3.1.cmml">​</mo><msub id="S2.E9.m1.3.3.1.3.3" xref="S2.E9.m1.3.3.1.3.3.cmml"><mi id="S2.E9.m1.3.3.1.3.3.2" xref="S2.E9.m1.3.3.1.3.3.2.cmml">P</mi><mtext id="S2.E9.m1.3.3.1.3.3.3" xref="S2.E9.m1.3.3.1.3.3.3a.cmml">vocab</mtext></msub><mo lspace="0em" rspace="0em" id="S2.E9.m1.3.3.1.3.1a" xref="S2.E9.m1.3.3.1.3.1.cmml">​</mo><mrow id="S2.E9.m1.3.3.1.3.4.2" xref="S2.E9.m1.3.3.1.3.cmml"><mo stretchy="false" id="S2.E9.m1.3.3.1.3.4.2.1" xref="S2.E9.m1.3.3.1.3.cmml">(</mo><mi id="S2.E9.m1.2.2" xref="S2.E9.m1.2.2.cmml">w</mi><mo stretchy="false" id="S2.E9.m1.3.3.1.3.4.2.2" xref="S2.E9.m1.3.3.1.3.cmml">)</mo></mrow></mrow><mo id="S2.E9.m1.3.3.1.2" xref="S2.E9.m1.3.3.1.2.cmml">+</mo><mrow id="S2.E9.m1.3.3.1.1" xref="S2.E9.m1.3.3.1.1.cmml"><mrow id="S2.E9.m1.3.3.1.1.1.1" xref="S2.E9.m1.3.3.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E9.m1.3.3.1.1.1.1.2" xref="S2.E9.m1.3.3.1.1.1.1.1.cmml">(</mo><mrow id="S2.E9.m1.3.3.1.1.1.1.1" xref="S2.E9.m1.3.3.1.1.1.1.1.cmml"><mn id="S2.E9.m1.3.3.1.1.1.1.1.2" xref="S2.E9.m1.3.3.1.1.1.1.1.2.cmml">1</mn><mo id="S2.E9.m1.3.3.1.1.1.1.1.1" xref="S2.E9.m1.3.3.1.1.1.1.1.1.cmml">−</mo><msub id="S2.E9.m1.3.3.1.1.1.1.1.3" xref="S2.E9.m1.3.3.1.1.1.1.1.3.cmml"><mi id="S2.E9.m1.3.3.1.1.1.1.1.3.2" xref="S2.E9.m1.3.3.1.1.1.1.1.3.2.cmml">p</mi><mtext id="S2.E9.m1.3.3.1.1.1.1.1.3.3" xref="S2.E9.m1.3.3.1.1.1.1.1.3.3a.cmml">gen</mtext></msub></mrow><mo stretchy="false" id="S2.E9.m1.3.3.1.1.1.1.3" xref="S2.E9.m1.3.3.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.E9.m1.3.3.1.1.2" xref="S2.E9.m1.3.3.1.1.2.cmml">​</mo><mrow id="S2.E9.m1.3.3.1.1.3" xref="S2.E9.m1.3.3.1.1.3.cmml"><msub id="S2.E9.m1.3.3.1.1.3.1" xref="S2.E9.m1.3.3.1.1.3.1.cmml"><mo id="S2.E9.m1.3.3.1.1.3.1.2" xref="S2.E9.m1.3.3.1.1.3.1.2.cmml">∑</mo><mrow id="S2.E9.m1.3.3.1.1.3.1.3" xref="S2.E9.m1.3.3.1.1.3.1.3.cmml"><mi id="S2.E9.m1.3.3.1.1.3.1.3.2" xref="S2.E9.m1.3.3.1.1.3.1.3.2.cmml">i</mi><mo lspace="0.278em" rspace="0.278em" id="S2.E9.m1.3.3.1.1.3.1.3.1" xref="S2.E9.m1.3.3.1.1.3.1.3.1.cmml">:</mo><mrow id="S2.E9.m1.3.3.1.1.3.1.3.3" xref="S2.E9.m1.3.3.1.1.3.1.3.3.cmml"><msub id="S2.E9.m1.3.3.1.1.3.1.3.3.2" xref="S2.E9.m1.3.3.1.1.3.1.3.3.2.cmml"><mi id="S2.E9.m1.3.3.1.1.3.1.3.3.2.2" xref="S2.E9.m1.3.3.1.1.3.1.3.3.2.2.cmml">w</mi><mi id="S2.E9.m1.3.3.1.1.3.1.3.3.2.3" xref="S2.E9.m1.3.3.1.1.3.1.3.3.2.3.cmml">i</mi></msub><mo id="S2.E9.m1.3.3.1.1.3.1.3.3.1" xref="S2.E9.m1.3.3.1.1.3.1.3.3.1.cmml">=</mo><mi id="S2.E9.m1.3.3.1.1.3.1.3.3.3" xref="S2.E9.m1.3.3.1.1.3.1.3.3.3.cmml">w</mi></mrow></mrow></msub><msubsup id="S2.E9.m1.3.3.1.1.3.2" xref="S2.E9.m1.3.3.1.1.3.2.cmml"><mi id="S2.E9.m1.3.3.1.1.3.2.2.2" xref="S2.E9.m1.3.3.1.1.3.2.2.2.cmml">a</mi><mi id="S2.E9.m1.3.3.1.1.3.2.3" xref="S2.E9.m1.3.3.1.1.3.2.3.cmml">i</mi><mi id="S2.E9.m1.3.3.1.1.3.2.2.3" xref="S2.E9.m1.3.3.1.1.3.2.2.3.cmml">t</mi></msubsup></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E9.m1.3b"><apply id="S2.E9.m1.3.3.cmml" xref="S2.E9.m1.3.3"><eq id="S2.E9.m1.3.3.2.cmml" xref="S2.E9.m1.3.3.2"></eq><apply id="S2.E9.m1.3.3.3.cmml" xref="S2.E9.m1.3.3.3"><times id="S2.E9.m1.3.3.3.1.cmml" xref="S2.E9.m1.3.3.3.1"></times><ci id="S2.E9.m1.3.3.3.2.cmml" xref="S2.E9.m1.3.3.3.2">𝑃</ci><ci id="S2.E9.m1.1.1.cmml" xref="S2.E9.m1.1.1">𝑤</ci></apply><apply id="S2.E9.m1.3.3.1.cmml" xref="S2.E9.m1.3.3.1"><plus id="S2.E9.m1.3.3.1.2.cmml" xref="S2.E9.m1.3.3.1.2"></plus><apply id="S2.E9.m1.3.3.1.3.cmml" xref="S2.E9.m1.3.3.1.3"><times id="S2.E9.m1.3.3.1.3.1.cmml" xref="S2.E9.m1.3.3.1.3.1"></times><apply id="S2.E9.m1.3.3.1.3.2.cmml" xref="S2.E9.m1.3.3.1.3.2"><csymbol cd="ambiguous" id="S2.E9.m1.3.3.1.3.2.1.cmml" xref="S2.E9.m1.3.3.1.3.2">subscript</csymbol><ci id="S2.E9.m1.3.3.1.3.2.2.cmml" xref="S2.E9.m1.3.3.1.3.2.2">𝑝</ci><ci id="S2.E9.m1.3.3.1.3.2.3a.cmml" xref="S2.E9.m1.3.3.1.3.2.3"><mtext mathsize="70%" id="S2.E9.m1.3.3.1.3.2.3.cmml" xref="S2.E9.m1.3.3.1.3.2.3">gen</mtext></ci></apply><apply id="S2.E9.m1.3.3.1.3.3.cmml" xref="S2.E9.m1.3.3.1.3.3"><csymbol cd="ambiguous" id="S2.E9.m1.3.3.1.3.3.1.cmml" xref="S2.E9.m1.3.3.1.3.3">subscript</csymbol><ci id="S2.E9.m1.3.3.1.3.3.2.cmml" xref="S2.E9.m1.3.3.1.3.3.2">𝑃</ci><ci id="S2.E9.m1.3.3.1.3.3.3a.cmml" xref="S2.E9.m1.3.3.1.3.3.3"><mtext mathsize="70%" id="S2.E9.m1.3.3.1.3.3.3.cmml" xref="S2.E9.m1.3.3.1.3.3.3">vocab</mtext></ci></apply><ci id="S2.E9.m1.2.2.cmml" xref="S2.E9.m1.2.2">𝑤</ci></apply><apply id="S2.E9.m1.3.3.1.1.cmml" xref="S2.E9.m1.3.3.1.1"><times id="S2.E9.m1.3.3.1.1.2.cmml" xref="S2.E9.m1.3.3.1.1.2"></times><apply id="S2.E9.m1.3.3.1.1.1.1.1.cmml" xref="S2.E9.m1.3.3.1.1.1.1"><minus id="S2.E9.m1.3.3.1.1.1.1.1.1.cmml" xref="S2.E9.m1.3.3.1.1.1.1.1.1"></minus><cn type="integer" id="S2.E9.m1.3.3.1.1.1.1.1.2.cmml" xref="S2.E9.m1.3.3.1.1.1.1.1.2">1</cn><apply id="S2.E9.m1.3.3.1.1.1.1.1.3.cmml" xref="S2.E9.m1.3.3.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E9.m1.3.3.1.1.1.1.1.3.1.cmml" xref="S2.E9.m1.3.3.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E9.m1.3.3.1.1.1.1.1.3.2.cmml" xref="S2.E9.m1.3.3.1.1.1.1.1.3.2">𝑝</ci><ci id="S2.E9.m1.3.3.1.1.1.1.1.3.3a.cmml" xref="S2.E9.m1.3.3.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S2.E9.m1.3.3.1.1.1.1.1.3.3.cmml" xref="S2.E9.m1.3.3.1.1.1.1.1.3.3">gen</mtext></ci></apply></apply><apply id="S2.E9.m1.3.3.1.1.3.cmml" xref="S2.E9.m1.3.3.1.1.3"><apply id="S2.E9.m1.3.3.1.1.3.1.cmml" xref="S2.E9.m1.3.3.1.1.3.1"><csymbol cd="ambiguous" id="S2.E9.m1.3.3.1.1.3.1.1.cmml" xref="S2.E9.m1.3.3.1.1.3.1">subscript</csymbol><sum id="S2.E9.m1.3.3.1.1.3.1.2.cmml" xref="S2.E9.m1.3.3.1.1.3.1.2"></sum><apply id="S2.E9.m1.3.3.1.1.3.1.3.cmml" xref="S2.E9.m1.3.3.1.1.3.1.3"><ci id="S2.E9.m1.3.3.1.1.3.1.3.1.cmml" xref="S2.E9.m1.3.3.1.1.3.1.3.1">:</ci><ci id="S2.E9.m1.3.3.1.1.3.1.3.2.cmml" xref="S2.E9.m1.3.3.1.1.3.1.3.2">𝑖</ci><apply id="S2.E9.m1.3.3.1.1.3.1.3.3.cmml" xref="S2.E9.m1.3.3.1.1.3.1.3.3"><eq id="S2.E9.m1.3.3.1.1.3.1.3.3.1.cmml" xref="S2.E9.m1.3.3.1.1.3.1.3.3.1"></eq><apply id="S2.E9.m1.3.3.1.1.3.1.3.3.2.cmml" xref="S2.E9.m1.3.3.1.1.3.1.3.3.2"><csymbol cd="ambiguous" id="S2.E9.m1.3.3.1.1.3.1.3.3.2.1.cmml" xref="S2.E9.m1.3.3.1.1.3.1.3.3.2">subscript</csymbol><ci id="S2.E9.m1.3.3.1.1.3.1.3.3.2.2.cmml" xref="S2.E9.m1.3.3.1.1.3.1.3.3.2.2">𝑤</ci><ci id="S2.E9.m1.3.3.1.1.3.1.3.3.2.3.cmml" xref="S2.E9.m1.3.3.1.1.3.1.3.3.2.3">𝑖</ci></apply><ci id="S2.E9.m1.3.3.1.1.3.1.3.3.3.cmml" xref="S2.E9.m1.3.3.1.1.3.1.3.3.3">𝑤</ci></apply></apply></apply><apply id="S2.E9.m1.3.3.1.1.3.2.cmml" xref="S2.E9.m1.3.3.1.1.3.2"><csymbol cd="ambiguous" id="S2.E9.m1.3.3.1.1.3.2.1.cmml" xref="S2.E9.m1.3.3.1.1.3.2">subscript</csymbol><apply id="S2.E9.m1.3.3.1.1.3.2.2.cmml" xref="S2.E9.m1.3.3.1.1.3.2"><csymbol cd="ambiguous" id="S2.E9.m1.3.3.1.1.3.2.2.1.cmml" xref="S2.E9.m1.3.3.1.1.3.2">superscript</csymbol><ci id="S2.E9.m1.3.3.1.1.3.2.2.2.cmml" xref="S2.E9.m1.3.3.1.1.3.2.2.2">𝑎</ci><ci id="S2.E9.m1.3.3.1.1.3.2.2.3.cmml" xref="S2.E9.m1.3.3.1.1.3.2.2.3">𝑡</ci></apply><ci id="S2.E9.m1.3.3.1.1.3.2.3.cmml" xref="S2.E9.m1.3.3.1.1.3.2.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E9.m1.3c">P(w)=p_{\text{gen}}P_{\text{vocab}}(w)+(1-p_{\text{gen}})\sum\nolimits_{i:w_{i}=w}a^{t}_{i}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
</table>
<p id="S2.SS2.p1.19" class="ltx_p">Note that if <math id="S2.SS2.p1.16.m1.1" class="ltx_Math" alttext="w" display="inline"><semantics id="S2.SS2.p1.16.m1.1a"><mi id="S2.SS2.p1.16.m1.1.1" xref="S2.SS2.p1.16.m1.1.1.cmml">w</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.16.m1.1b"><ci id="S2.SS2.p1.16.m1.1.1.cmml" xref="S2.SS2.p1.16.m1.1.1">𝑤</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.16.m1.1c">w</annotation></semantics></math> is an out-of-vocabulary (OOV) word, then <math id="S2.SS2.p1.17.m2.1" class="ltx_Math" alttext="P_{\text{vocab}}(w)" display="inline"><semantics id="S2.SS2.p1.17.m2.1a"><mrow id="S2.SS2.p1.17.m2.1.2" xref="S2.SS2.p1.17.m2.1.2.cmml"><msub id="S2.SS2.p1.17.m2.1.2.2" xref="S2.SS2.p1.17.m2.1.2.2.cmml"><mi id="S2.SS2.p1.17.m2.1.2.2.2" xref="S2.SS2.p1.17.m2.1.2.2.2.cmml">P</mi><mtext id="S2.SS2.p1.17.m2.1.2.2.3" xref="S2.SS2.p1.17.m2.1.2.2.3a.cmml">vocab</mtext></msub><mo lspace="0em" rspace="0em" id="S2.SS2.p1.17.m2.1.2.1" xref="S2.SS2.p1.17.m2.1.2.1.cmml">​</mo><mrow id="S2.SS2.p1.17.m2.1.2.3.2" xref="S2.SS2.p1.17.m2.1.2.cmml"><mo stretchy="false" id="S2.SS2.p1.17.m2.1.2.3.2.1" xref="S2.SS2.p1.17.m2.1.2.cmml">(</mo><mi id="S2.SS2.p1.17.m2.1.1" xref="S2.SS2.p1.17.m2.1.1.cmml">w</mi><mo stretchy="false" id="S2.SS2.p1.17.m2.1.2.3.2.2" xref="S2.SS2.p1.17.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.17.m2.1b"><apply id="S2.SS2.p1.17.m2.1.2.cmml" xref="S2.SS2.p1.17.m2.1.2"><times id="S2.SS2.p1.17.m2.1.2.1.cmml" xref="S2.SS2.p1.17.m2.1.2.1"></times><apply id="S2.SS2.p1.17.m2.1.2.2.cmml" xref="S2.SS2.p1.17.m2.1.2.2"><csymbol cd="ambiguous" id="S2.SS2.p1.17.m2.1.2.2.1.cmml" xref="S2.SS2.p1.17.m2.1.2.2">subscript</csymbol><ci id="S2.SS2.p1.17.m2.1.2.2.2.cmml" xref="S2.SS2.p1.17.m2.1.2.2.2">𝑃</ci><ci id="S2.SS2.p1.17.m2.1.2.2.3a.cmml" xref="S2.SS2.p1.17.m2.1.2.2.3"><mtext mathsize="70%" id="S2.SS2.p1.17.m2.1.2.2.3.cmml" xref="S2.SS2.p1.17.m2.1.2.2.3">vocab</mtext></ci></apply><ci id="S2.SS2.p1.17.m2.1.1.cmml" xref="S2.SS2.p1.17.m2.1.1">𝑤</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.17.m2.1c">P_{\text{vocab}}(w)</annotation></semantics></math> is zero;
similarly if <math id="S2.SS2.p1.18.m3.1" class="ltx_Math" alttext="w" display="inline"><semantics id="S2.SS2.p1.18.m3.1a"><mi id="S2.SS2.p1.18.m3.1.1" xref="S2.SS2.p1.18.m3.1.1.cmml">w</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.18.m3.1b"><ci id="S2.SS2.p1.18.m3.1.1.cmml" xref="S2.SS2.p1.18.m3.1.1">𝑤</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.18.m3.1c">w</annotation></semantics></math> does not appear in the source document, then <math id="S2.SS2.p1.19.m4.1" class="ltx_Math" alttext="\sum_{i:w_{i}=w}a^{t}_{i}" display="inline"><semantics id="S2.SS2.p1.19.m4.1a"><mrow id="S2.SS2.p1.19.m4.1.1" xref="S2.SS2.p1.19.m4.1.1.cmml"><msub id="S2.SS2.p1.19.m4.1.1.1" xref="S2.SS2.p1.19.m4.1.1.1.cmml"><mo id="S2.SS2.p1.19.m4.1.1.1.2" xref="S2.SS2.p1.19.m4.1.1.1.2.cmml">∑</mo><mrow id="S2.SS2.p1.19.m4.1.1.1.3" xref="S2.SS2.p1.19.m4.1.1.1.3.cmml"><mi id="S2.SS2.p1.19.m4.1.1.1.3.2" xref="S2.SS2.p1.19.m4.1.1.1.3.2.cmml">i</mi><mo lspace="0.278em" rspace="0.278em" id="S2.SS2.p1.19.m4.1.1.1.3.1" xref="S2.SS2.p1.19.m4.1.1.1.3.1.cmml">:</mo><mrow id="S2.SS2.p1.19.m4.1.1.1.3.3" xref="S2.SS2.p1.19.m4.1.1.1.3.3.cmml"><msub id="S2.SS2.p1.19.m4.1.1.1.3.3.2" xref="S2.SS2.p1.19.m4.1.1.1.3.3.2.cmml"><mi id="S2.SS2.p1.19.m4.1.1.1.3.3.2.2" xref="S2.SS2.p1.19.m4.1.1.1.3.3.2.2.cmml">w</mi><mi id="S2.SS2.p1.19.m4.1.1.1.3.3.2.3" xref="S2.SS2.p1.19.m4.1.1.1.3.3.2.3.cmml">i</mi></msub><mo id="S2.SS2.p1.19.m4.1.1.1.3.3.1" xref="S2.SS2.p1.19.m4.1.1.1.3.3.1.cmml">=</mo><mi id="S2.SS2.p1.19.m4.1.1.1.3.3.3" xref="S2.SS2.p1.19.m4.1.1.1.3.3.3.cmml">w</mi></mrow></mrow></msub><msubsup id="S2.SS2.p1.19.m4.1.1.2" xref="S2.SS2.p1.19.m4.1.1.2.cmml"><mi id="S2.SS2.p1.19.m4.1.1.2.2.2" xref="S2.SS2.p1.19.m4.1.1.2.2.2.cmml">a</mi><mi id="S2.SS2.p1.19.m4.1.1.2.3" xref="S2.SS2.p1.19.m4.1.1.2.3.cmml">i</mi><mi id="S2.SS2.p1.19.m4.1.1.2.2.3" xref="S2.SS2.p1.19.m4.1.1.2.2.3.cmml">t</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.19.m4.1b"><apply id="S2.SS2.p1.19.m4.1.1.cmml" xref="S2.SS2.p1.19.m4.1.1"><apply id="S2.SS2.p1.19.m4.1.1.1.cmml" xref="S2.SS2.p1.19.m4.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.19.m4.1.1.1.1.cmml" xref="S2.SS2.p1.19.m4.1.1.1">subscript</csymbol><sum id="S2.SS2.p1.19.m4.1.1.1.2.cmml" xref="S2.SS2.p1.19.m4.1.1.1.2"></sum><apply id="S2.SS2.p1.19.m4.1.1.1.3.cmml" xref="S2.SS2.p1.19.m4.1.1.1.3"><ci id="S2.SS2.p1.19.m4.1.1.1.3.1.cmml" xref="S2.SS2.p1.19.m4.1.1.1.3.1">:</ci><ci id="S2.SS2.p1.19.m4.1.1.1.3.2.cmml" xref="S2.SS2.p1.19.m4.1.1.1.3.2">𝑖</ci><apply id="S2.SS2.p1.19.m4.1.1.1.3.3.cmml" xref="S2.SS2.p1.19.m4.1.1.1.3.3"><eq id="S2.SS2.p1.19.m4.1.1.1.3.3.1.cmml" xref="S2.SS2.p1.19.m4.1.1.1.3.3.1"></eq><apply id="S2.SS2.p1.19.m4.1.1.1.3.3.2.cmml" xref="S2.SS2.p1.19.m4.1.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.SS2.p1.19.m4.1.1.1.3.3.2.1.cmml" xref="S2.SS2.p1.19.m4.1.1.1.3.3.2">subscript</csymbol><ci id="S2.SS2.p1.19.m4.1.1.1.3.3.2.2.cmml" xref="S2.SS2.p1.19.m4.1.1.1.3.3.2.2">𝑤</ci><ci id="S2.SS2.p1.19.m4.1.1.1.3.3.2.3.cmml" xref="S2.SS2.p1.19.m4.1.1.1.3.3.2.3">𝑖</ci></apply><ci id="S2.SS2.p1.19.m4.1.1.1.3.3.3.cmml" xref="S2.SS2.p1.19.m4.1.1.1.3.3.3">𝑤</ci></apply></apply></apply><apply id="S2.SS2.p1.19.m4.1.1.2.cmml" xref="S2.SS2.p1.19.m4.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.p1.19.m4.1.1.2.1.cmml" xref="S2.SS2.p1.19.m4.1.1.2">subscript</csymbol><apply id="S2.SS2.p1.19.m4.1.1.2.2.cmml" xref="S2.SS2.p1.19.m4.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.p1.19.m4.1.1.2.2.1.cmml" xref="S2.SS2.p1.19.m4.1.1.2">superscript</csymbol><ci id="S2.SS2.p1.19.m4.1.1.2.2.2.cmml" xref="S2.SS2.p1.19.m4.1.1.2.2.2">𝑎</ci><ci id="S2.SS2.p1.19.m4.1.1.2.2.3.cmml" xref="S2.SS2.p1.19.m4.1.1.2.2.3">𝑡</ci></apply><ci id="S2.SS2.p1.19.m4.1.1.2.3.cmml" xref="S2.SS2.p1.19.m4.1.1.2.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.19.m4.1c">\sum_{i:w_{i}=w}a^{t}_{i}</annotation></semantics></math> is zero.
The ability to produce OOV words is one of the primary advantages of pointer-generator models; by contrast models such as our baseline are restricted to their pre-set vocabulary.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.1" class="ltx_p">The loss function is as described in equations (<a href="#S2.E6" title="In 2.1 Sequence-to-sequence attentional model ‣ 2 Our Models ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>) and (<a href="#S2.E7" title="In 2.1 Sequence-to-sequence attentional model ‣ 2 Our Models ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>), but with respect to our modified probability distribution <math id="S2.SS2.p2.1.m1.1" class="ltx_Math" alttext="P(w)" display="inline"><semantics id="S2.SS2.p2.1.m1.1a"><mrow id="S2.SS2.p2.1.m1.1.2" xref="S2.SS2.p2.1.m1.1.2.cmml"><mi id="S2.SS2.p2.1.m1.1.2.2" xref="S2.SS2.p2.1.m1.1.2.2.cmml">P</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p2.1.m1.1.2.1" xref="S2.SS2.p2.1.m1.1.2.1.cmml">​</mo><mrow id="S2.SS2.p2.1.m1.1.2.3.2" xref="S2.SS2.p2.1.m1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p2.1.m1.1.2.3.2.1" xref="S2.SS2.p2.1.m1.1.2.cmml">(</mo><mi id="S2.SS2.p2.1.m1.1.1" xref="S2.SS2.p2.1.m1.1.1.cmml">w</mi><mo stretchy="false" id="S2.SS2.p2.1.m1.1.2.3.2.2" xref="S2.SS2.p2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.1.m1.1b"><apply id="S2.SS2.p2.1.m1.1.2.cmml" xref="S2.SS2.p2.1.m1.1.2"><times id="S2.SS2.p2.1.m1.1.2.1.cmml" xref="S2.SS2.p2.1.m1.1.2.1"></times><ci id="S2.SS2.p2.1.m1.1.2.2.cmml" xref="S2.SS2.p2.1.m1.1.2.2">𝑃</ci><ci id="S2.SS2.p2.1.m1.1.1.cmml" xref="S2.SS2.p2.1.m1.1.1">𝑤</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.1.m1.1c">P(w)</annotation></semantics></math> given in equation (<a href="#S2.E9" title="In 2.2 Pointer-generator network ‣ 2 Our Models ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>).</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Coverage mechanism</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">Repetition is a common problem for sequence-to-sequence models <cite class="ltx_cite ltx_citemacro_cite">Tu et al. (<a href="#bib.bib28" title="" class="ltx_ref">2016</a>); Mi et al. (<a href="#bib.bib16" title="" class="ltx_ref">2016</a>); Sankaran et al. (<a href="#bib.bib24" title="" class="ltx_ref">2016</a>); Suzuki and Nagata (<a href="#bib.bib26" title="" class="ltx_ref">2016</a>)</cite>, and is especially pronounced when generating multi-sentence text (see Figure <a href="#S0.F1" title="Figure 1 ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).
We adapt the <span id="S2.SS3.p1.1.1" class="ltx_text ltx_font_italic">coverage model</span> of <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib28" title="" class="ltx_ref">Tu et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Tu et al. (<a href="#bib.bib28" title="" class="ltx_ref">2016</a>)</cite> to solve the problem.
In our coverage model, we maintain a <span id="S2.SS3.p1.1.2" class="ltx_text ltx_font_italic">coverage vector</span> <math id="S2.SS3.p1.1.m1.1" class="ltx_Math" alttext="c^{t}" display="inline"><semantics id="S2.SS3.p1.1.m1.1a"><msup id="S2.SS3.p1.1.m1.1.1" xref="S2.SS3.p1.1.m1.1.1.cmml"><mi id="S2.SS3.p1.1.m1.1.1.2" xref="S2.SS3.p1.1.m1.1.1.2.cmml">c</mi><mi id="S2.SS3.p1.1.m1.1.1.3" xref="S2.SS3.p1.1.m1.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.1.m1.1b"><apply id="S2.SS3.p1.1.m1.1.1.cmml" xref="S2.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p1.1.m1.1.1.1.cmml" xref="S2.SS3.p1.1.m1.1.1">superscript</csymbol><ci id="S2.SS3.p1.1.m1.1.1.2.cmml" xref="S2.SS3.p1.1.m1.1.1.2">𝑐</ci><ci id="S2.SS3.p1.1.m1.1.1.3.cmml" xref="S2.SS3.p1.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.1.m1.1c">c^{t}</annotation></semantics></math>, which is the sum of attention distributions over all previous decoder timesteps:</p>
<table id="S2.E10" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E10.m1.1" class="ltx_Math" alttext="c^{t}=\sum\nolimits_{t^{\prime}=0}^{t-1}a^{t^{\prime}}" display="block"><semantics id="S2.E10.m1.1a"><mrow id="S2.E10.m1.1.1" xref="S2.E10.m1.1.1.cmml"><msup id="S2.E10.m1.1.1.2" xref="S2.E10.m1.1.1.2.cmml"><mi id="S2.E10.m1.1.1.2.2" xref="S2.E10.m1.1.1.2.2.cmml">c</mi><mi id="S2.E10.m1.1.1.2.3" xref="S2.E10.m1.1.1.2.3.cmml">t</mi></msup><mo rspace="0.111em" id="S2.E10.m1.1.1.1" xref="S2.E10.m1.1.1.1.cmml">=</mo><mrow id="S2.E10.m1.1.1.3" xref="S2.E10.m1.1.1.3.cmml"><msubsup id="S2.E10.m1.1.1.3.1" xref="S2.E10.m1.1.1.3.1.cmml"><mo id="S2.E10.m1.1.1.3.1.2.2" xref="S2.E10.m1.1.1.3.1.2.2.cmml">∑</mo><mrow id="S2.E10.m1.1.1.3.1.2.3" xref="S2.E10.m1.1.1.3.1.2.3.cmml"><msup id="S2.E10.m1.1.1.3.1.2.3.2" xref="S2.E10.m1.1.1.3.1.2.3.2.cmml"><mi id="S2.E10.m1.1.1.3.1.2.3.2.2" xref="S2.E10.m1.1.1.3.1.2.3.2.2.cmml">t</mi><mo id="S2.E10.m1.1.1.3.1.2.3.2.3" xref="S2.E10.m1.1.1.3.1.2.3.2.3.cmml">′</mo></msup><mo id="S2.E10.m1.1.1.3.1.2.3.1" xref="S2.E10.m1.1.1.3.1.2.3.1.cmml">=</mo><mn id="S2.E10.m1.1.1.3.1.2.3.3" xref="S2.E10.m1.1.1.3.1.2.3.3.cmml">0</mn></mrow><mrow id="S2.E10.m1.1.1.3.1.3" xref="S2.E10.m1.1.1.3.1.3.cmml"><mi id="S2.E10.m1.1.1.3.1.3.2" xref="S2.E10.m1.1.1.3.1.3.2.cmml">t</mi><mo id="S2.E10.m1.1.1.3.1.3.1" xref="S2.E10.m1.1.1.3.1.3.1.cmml">−</mo><mn id="S2.E10.m1.1.1.3.1.3.3" xref="S2.E10.m1.1.1.3.1.3.3.cmml">1</mn></mrow></msubsup><msup id="S2.E10.m1.1.1.3.2" xref="S2.E10.m1.1.1.3.2.cmml"><mi id="S2.E10.m1.1.1.3.2.2" xref="S2.E10.m1.1.1.3.2.2.cmml">a</mi><msup id="S2.E10.m1.1.1.3.2.3" xref="S2.E10.m1.1.1.3.2.3.cmml"><mi id="S2.E10.m1.1.1.3.2.3.2" xref="S2.E10.m1.1.1.3.2.3.2.cmml">t</mi><mo id="S2.E10.m1.1.1.3.2.3.3" xref="S2.E10.m1.1.1.3.2.3.3.cmml">′</mo></msup></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E10.m1.1b"><apply id="S2.E10.m1.1.1.cmml" xref="S2.E10.m1.1.1"><eq id="S2.E10.m1.1.1.1.cmml" xref="S2.E10.m1.1.1.1"></eq><apply id="S2.E10.m1.1.1.2.cmml" xref="S2.E10.m1.1.1.2"><csymbol cd="ambiguous" id="S2.E10.m1.1.1.2.1.cmml" xref="S2.E10.m1.1.1.2">superscript</csymbol><ci id="S2.E10.m1.1.1.2.2.cmml" xref="S2.E10.m1.1.1.2.2">𝑐</ci><ci id="S2.E10.m1.1.1.2.3.cmml" xref="S2.E10.m1.1.1.2.3">𝑡</ci></apply><apply id="S2.E10.m1.1.1.3.cmml" xref="S2.E10.m1.1.1.3"><apply id="S2.E10.m1.1.1.3.1.cmml" xref="S2.E10.m1.1.1.3.1"><csymbol cd="ambiguous" id="S2.E10.m1.1.1.3.1.1.cmml" xref="S2.E10.m1.1.1.3.1">superscript</csymbol><apply id="S2.E10.m1.1.1.3.1.2.cmml" xref="S2.E10.m1.1.1.3.1"><csymbol cd="ambiguous" id="S2.E10.m1.1.1.3.1.2.1.cmml" xref="S2.E10.m1.1.1.3.1">subscript</csymbol><sum id="S2.E10.m1.1.1.3.1.2.2.cmml" xref="S2.E10.m1.1.1.3.1.2.2"></sum><apply id="S2.E10.m1.1.1.3.1.2.3.cmml" xref="S2.E10.m1.1.1.3.1.2.3"><eq id="S2.E10.m1.1.1.3.1.2.3.1.cmml" xref="S2.E10.m1.1.1.3.1.2.3.1"></eq><apply id="S2.E10.m1.1.1.3.1.2.3.2.cmml" xref="S2.E10.m1.1.1.3.1.2.3.2"><csymbol cd="ambiguous" id="S2.E10.m1.1.1.3.1.2.3.2.1.cmml" xref="S2.E10.m1.1.1.3.1.2.3.2">superscript</csymbol><ci id="S2.E10.m1.1.1.3.1.2.3.2.2.cmml" xref="S2.E10.m1.1.1.3.1.2.3.2.2">𝑡</ci><ci id="S2.E10.m1.1.1.3.1.2.3.2.3.cmml" xref="S2.E10.m1.1.1.3.1.2.3.2.3">′</ci></apply><cn type="integer" id="S2.E10.m1.1.1.3.1.2.3.3.cmml" xref="S2.E10.m1.1.1.3.1.2.3.3">0</cn></apply></apply><apply id="S2.E10.m1.1.1.3.1.3.cmml" xref="S2.E10.m1.1.1.3.1.3"><minus id="S2.E10.m1.1.1.3.1.3.1.cmml" xref="S2.E10.m1.1.1.3.1.3.1"></minus><ci id="S2.E10.m1.1.1.3.1.3.2.cmml" xref="S2.E10.m1.1.1.3.1.3.2">𝑡</ci><cn type="integer" id="S2.E10.m1.1.1.3.1.3.3.cmml" xref="S2.E10.m1.1.1.3.1.3.3">1</cn></apply></apply><apply id="S2.E10.m1.1.1.3.2.cmml" xref="S2.E10.m1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E10.m1.1.1.3.2.1.cmml" xref="S2.E10.m1.1.1.3.2">superscript</csymbol><ci id="S2.E10.m1.1.1.3.2.2.cmml" xref="S2.E10.m1.1.1.3.2.2">𝑎</ci><apply id="S2.E10.m1.1.1.3.2.3.cmml" xref="S2.E10.m1.1.1.3.2.3"><csymbol cd="ambiguous" id="S2.E10.m1.1.1.3.2.3.1.cmml" xref="S2.E10.m1.1.1.3.2.3">superscript</csymbol><ci id="S2.E10.m1.1.1.3.2.3.2.cmml" xref="S2.E10.m1.1.1.3.2.3.2">𝑡</ci><ci id="S2.E10.m1.1.1.3.2.3.3.cmml" xref="S2.E10.m1.1.1.3.2.3.3">′</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E10.m1.1c">c^{t}=\sum\nolimits_{t^{\prime}=0}^{t-1}a^{t^{\prime}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(10)</span></td>
</tr></tbody>
</table>
<p id="S2.SS3.p1.3" class="ltx_p">Intuitively, <math id="S2.SS3.p1.2.m1.1" class="ltx_Math" alttext="c^{t}" display="inline"><semantics id="S2.SS3.p1.2.m1.1a"><msup id="S2.SS3.p1.2.m1.1.1" xref="S2.SS3.p1.2.m1.1.1.cmml"><mi id="S2.SS3.p1.2.m1.1.1.2" xref="S2.SS3.p1.2.m1.1.1.2.cmml">c</mi><mi id="S2.SS3.p1.2.m1.1.1.3" xref="S2.SS3.p1.2.m1.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.2.m1.1b"><apply id="S2.SS3.p1.2.m1.1.1.cmml" xref="S2.SS3.p1.2.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p1.2.m1.1.1.1.cmml" xref="S2.SS3.p1.2.m1.1.1">superscript</csymbol><ci id="S2.SS3.p1.2.m1.1.1.2.cmml" xref="S2.SS3.p1.2.m1.1.1.2">𝑐</ci><ci id="S2.SS3.p1.2.m1.1.1.3.cmml" xref="S2.SS3.p1.2.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.2.m1.1c">c^{t}</annotation></semantics></math> is a (unnormalized) distribution over the source document words that represents the degree of coverage that those words have received from the attention mechanism so far.
Note that <math id="S2.SS3.p1.3.m2.1" class="ltx_Math" alttext="c^{0}" display="inline"><semantics id="S2.SS3.p1.3.m2.1a"><msup id="S2.SS3.p1.3.m2.1.1" xref="S2.SS3.p1.3.m2.1.1.cmml"><mi id="S2.SS3.p1.3.m2.1.1.2" xref="S2.SS3.p1.3.m2.1.1.2.cmml">c</mi><mn id="S2.SS3.p1.3.m2.1.1.3" xref="S2.SS3.p1.3.m2.1.1.3.cmml">0</mn></msup><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.3.m2.1b"><apply id="S2.SS3.p1.3.m2.1.1.cmml" xref="S2.SS3.p1.3.m2.1.1"><csymbol cd="ambiguous" id="S2.SS3.p1.3.m2.1.1.1.cmml" xref="S2.SS3.p1.3.m2.1.1">superscript</csymbol><ci id="S2.SS3.p1.3.m2.1.1.2.cmml" xref="S2.SS3.p1.3.m2.1.1.2">𝑐</ci><cn type="integer" id="S2.SS3.p1.3.m2.1.1.3.cmml" xref="S2.SS3.p1.3.m2.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.3.m2.1c">c^{0}</annotation></semantics></math> is a zero vector, because on the first timestep, none of the source document has been covered.</p>
</div>
<div id="S2.SS3.p2" class="ltx_para">
<p id="S2.SS3.p2.4" class="ltx_p">The coverage vector is used as extra input to the attention mechanism, changing equation (<a href="#S2.E1" title="In 2.1 Sequence-to-sequence attentional model ‣ 2 Our Models ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>) to:</p>
<table id="S2.E11" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E11.m1.2" class="ltx_Math" alttext="e_{i}^{t}=v^{T}\tanh(W_{h}h_{i}+W_{s}s_{t}+w_{c}c^{t}_{i}+b_{\text{attn}})" display="block"><semantics id="S2.E11.m1.2a"><mrow id="S2.E11.m1.2.2" xref="S2.E11.m1.2.2.cmml"><msubsup id="S2.E11.m1.2.2.3" xref="S2.E11.m1.2.2.3.cmml"><mi id="S2.E11.m1.2.2.3.2.2" xref="S2.E11.m1.2.2.3.2.2.cmml">e</mi><mi id="S2.E11.m1.2.2.3.2.3" xref="S2.E11.m1.2.2.3.2.3.cmml">i</mi><mi id="S2.E11.m1.2.2.3.3" xref="S2.E11.m1.2.2.3.3.cmml">t</mi></msubsup><mo id="S2.E11.m1.2.2.2" xref="S2.E11.m1.2.2.2.cmml">=</mo><mrow id="S2.E11.m1.2.2.1" xref="S2.E11.m1.2.2.1.cmml"><msup id="S2.E11.m1.2.2.1.3" xref="S2.E11.m1.2.2.1.3.cmml"><mi id="S2.E11.m1.2.2.1.3.2" xref="S2.E11.m1.2.2.1.3.2.cmml">v</mi><mi id="S2.E11.m1.2.2.1.3.3" xref="S2.E11.m1.2.2.1.3.3.cmml">T</mi></msup><mo lspace="0.167em" rspace="0em" id="S2.E11.m1.2.2.1.2" xref="S2.E11.m1.2.2.1.2.cmml">​</mo><mrow id="S2.E11.m1.2.2.1.1.1" xref="S2.E11.m1.2.2.1.1.2.cmml"><mi id="S2.E11.m1.1.1" xref="S2.E11.m1.1.1.cmml">tanh</mi><mo id="S2.E11.m1.2.2.1.1.1a" xref="S2.E11.m1.2.2.1.1.2.cmml">⁡</mo><mrow id="S2.E11.m1.2.2.1.1.1.1" xref="S2.E11.m1.2.2.1.1.2.cmml"><mo stretchy="false" id="S2.E11.m1.2.2.1.1.1.1.2" xref="S2.E11.m1.2.2.1.1.2.cmml">(</mo><mrow id="S2.E11.m1.2.2.1.1.1.1.1" xref="S2.E11.m1.2.2.1.1.1.1.1.cmml"><mrow id="S2.E11.m1.2.2.1.1.1.1.1.2" xref="S2.E11.m1.2.2.1.1.1.1.1.2.cmml"><msub id="S2.E11.m1.2.2.1.1.1.1.1.2.2" xref="S2.E11.m1.2.2.1.1.1.1.1.2.2.cmml"><mi id="S2.E11.m1.2.2.1.1.1.1.1.2.2.2" xref="S2.E11.m1.2.2.1.1.1.1.1.2.2.2.cmml">W</mi><mi id="S2.E11.m1.2.2.1.1.1.1.1.2.2.3" xref="S2.E11.m1.2.2.1.1.1.1.1.2.2.3.cmml">h</mi></msub><mo lspace="0em" rspace="0em" id="S2.E11.m1.2.2.1.1.1.1.1.2.1" xref="S2.E11.m1.2.2.1.1.1.1.1.2.1.cmml">​</mo><msub id="S2.E11.m1.2.2.1.1.1.1.1.2.3" xref="S2.E11.m1.2.2.1.1.1.1.1.2.3.cmml"><mi id="S2.E11.m1.2.2.1.1.1.1.1.2.3.2" xref="S2.E11.m1.2.2.1.1.1.1.1.2.3.2.cmml">h</mi><mi id="S2.E11.m1.2.2.1.1.1.1.1.2.3.3" xref="S2.E11.m1.2.2.1.1.1.1.1.2.3.3.cmml">i</mi></msub></mrow><mo id="S2.E11.m1.2.2.1.1.1.1.1.1" xref="S2.E11.m1.2.2.1.1.1.1.1.1.cmml">+</mo><mrow id="S2.E11.m1.2.2.1.1.1.1.1.3" xref="S2.E11.m1.2.2.1.1.1.1.1.3.cmml"><msub id="S2.E11.m1.2.2.1.1.1.1.1.3.2" xref="S2.E11.m1.2.2.1.1.1.1.1.3.2.cmml"><mi id="S2.E11.m1.2.2.1.1.1.1.1.3.2.2" xref="S2.E11.m1.2.2.1.1.1.1.1.3.2.2.cmml">W</mi><mi id="S2.E11.m1.2.2.1.1.1.1.1.3.2.3" xref="S2.E11.m1.2.2.1.1.1.1.1.3.2.3.cmml">s</mi></msub><mo lspace="0em" rspace="0em" id="S2.E11.m1.2.2.1.1.1.1.1.3.1" xref="S2.E11.m1.2.2.1.1.1.1.1.3.1.cmml">​</mo><msub id="S2.E11.m1.2.2.1.1.1.1.1.3.3" xref="S2.E11.m1.2.2.1.1.1.1.1.3.3.cmml"><mi id="S2.E11.m1.2.2.1.1.1.1.1.3.3.2" xref="S2.E11.m1.2.2.1.1.1.1.1.3.3.2.cmml">s</mi><mi id="S2.E11.m1.2.2.1.1.1.1.1.3.3.3" xref="S2.E11.m1.2.2.1.1.1.1.1.3.3.3.cmml">t</mi></msub></mrow><mo id="S2.E11.m1.2.2.1.1.1.1.1.1a" xref="S2.E11.m1.2.2.1.1.1.1.1.1.cmml">+</mo><mrow id="S2.E11.m1.2.2.1.1.1.1.1.4" xref="S2.E11.m1.2.2.1.1.1.1.1.4.cmml"><msub id="S2.E11.m1.2.2.1.1.1.1.1.4.2" xref="S2.E11.m1.2.2.1.1.1.1.1.4.2.cmml"><mi id="S2.E11.m1.2.2.1.1.1.1.1.4.2.2" xref="S2.E11.m1.2.2.1.1.1.1.1.4.2.2.cmml">w</mi><mi id="S2.E11.m1.2.2.1.1.1.1.1.4.2.3" xref="S2.E11.m1.2.2.1.1.1.1.1.4.2.3.cmml">c</mi></msub><mo lspace="0em" rspace="0em" id="S2.E11.m1.2.2.1.1.1.1.1.4.1" xref="S2.E11.m1.2.2.1.1.1.1.1.4.1.cmml">​</mo><msubsup id="S2.E11.m1.2.2.1.1.1.1.1.4.3" xref="S2.E11.m1.2.2.1.1.1.1.1.4.3.cmml"><mi id="S2.E11.m1.2.2.1.1.1.1.1.4.3.2.2" xref="S2.E11.m1.2.2.1.1.1.1.1.4.3.2.2.cmml">c</mi><mi id="S2.E11.m1.2.2.1.1.1.1.1.4.3.3" xref="S2.E11.m1.2.2.1.1.1.1.1.4.3.3.cmml">i</mi><mi id="S2.E11.m1.2.2.1.1.1.1.1.4.3.2.3" xref="S2.E11.m1.2.2.1.1.1.1.1.4.3.2.3.cmml">t</mi></msubsup></mrow><mo id="S2.E11.m1.2.2.1.1.1.1.1.1b" xref="S2.E11.m1.2.2.1.1.1.1.1.1.cmml">+</mo><msub id="S2.E11.m1.2.2.1.1.1.1.1.5" xref="S2.E11.m1.2.2.1.1.1.1.1.5.cmml"><mi id="S2.E11.m1.2.2.1.1.1.1.1.5.2" xref="S2.E11.m1.2.2.1.1.1.1.1.5.2.cmml">b</mi><mtext id="S2.E11.m1.2.2.1.1.1.1.1.5.3" xref="S2.E11.m1.2.2.1.1.1.1.1.5.3a.cmml">attn</mtext></msub></mrow><mo stretchy="false" id="S2.E11.m1.2.2.1.1.1.1.3" xref="S2.E11.m1.2.2.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E11.m1.2b"><apply id="S2.E11.m1.2.2.cmml" xref="S2.E11.m1.2.2"><eq id="S2.E11.m1.2.2.2.cmml" xref="S2.E11.m1.2.2.2"></eq><apply id="S2.E11.m1.2.2.3.cmml" xref="S2.E11.m1.2.2.3"><csymbol cd="ambiguous" id="S2.E11.m1.2.2.3.1.cmml" xref="S2.E11.m1.2.2.3">superscript</csymbol><apply id="S2.E11.m1.2.2.3.2.cmml" xref="S2.E11.m1.2.2.3"><csymbol cd="ambiguous" id="S2.E11.m1.2.2.3.2.1.cmml" xref="S2.E11.m1.2.2.3">subscript</csymbol><ci id="S2.E11.m1.2.2.3.2.2.cmml" xref="S2.E11.m1.2.2.3.2.2">𝑒</ci><ci id="S2.E11.m1.2.2.3.2.3.cmml" xref="S2.E11.m1.2.2.3.2.3">𝑖</ci></apply><ci id="S2.E11.m1.2.2.3.3.cmml" xref="S2.E11.m1.2.2.3.3">𝑡</ci></apply><apply id="S2.E11.m1.2.2.1.cmml" xref="S2.E11.m1.2.2.1"><times id="S2.E11.m1.2.2.1.2.cmml" xref="S2.E11.m1.2.2.1.2"></times><apply id="S2.E11.m1.2.2.1.3.cmml" xref="S2.E11.m1.2.2.1.3"><csymbol cd="ambiguous" id="S2.E11.m1.2.2.1.3.1.cmml" xref="S2.E11.m1.2.2.1.3">superscript</csymbol><ci id="S2.E11.m1.2.2.1.3.2.cmml" xref="S2.E11.m1.2.2.1.3.2">𝑣</ci><ci id="S2.E11.m1.2.2.1.3.3.cmml" xref="S2.E11.m1.2.2.1.3.3">𝑇</ci></apply><apply id="S2.E11.m1.2.2.1.1.2.cmml" xref="S2.E11.m1.2.2.1.1.1"><tanh id="S2.E11.m1.1.1.cmml" xref="S2.E11.m1.1.1"></tanh><apply id="S2.E11.m1.2.2.1.1.1.1.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1"><plus id="S2.E11.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.1"></plus><apply id="S2.E11.m1.2.2.1.1.1.1.1.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.2"><times id="S2.E11.m1.2.2.1.1.1.1.1.2.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.2.1"></times><apply id="S2.E11.m1.2.2.1.1.1.1.1.2.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.E11.m1.2.2.1.1.1.1.1.2.2.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.2.2">subscript</csymbol><ci id="S2.E11.m1.2.2.1.1.1.1.1.2.2.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.2.2.2">𝑊</ci><ci id="S2.E11.m1.2.2.1.1.1.1.1.2.2.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.2.2.3">ℎ</ci></apply><apply id="S2.E11.m1.2.2.1.1.1.1.1.2.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.E11.m1.2.2.1.1.1.1.1.2.3.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.2.3">subscript</csymbol><ci id="S2.E11.m1.2.2.1.1.1.1.1.2.3.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.2.3.2">ℎ</ci><ci id="S2.E11.m1.2.2.1.1.1.1.1.2.3.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.2.3.3">𝑖</ci></apply></apply><apply id="S2.E11.m1.2.2.1.1.1.1.1.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.3"><times id="S2.E11.m1.2.2.1.1.1.1.1.3.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.3.1"></times><apply id="S2.E11.m1.2.2.1.1.1.1.1.3.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E11.m1.2.2.1.1.1.1.1.3.2.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.3.2">subscript</csymbol><ci id="S2.E11.m1.2.2.1.1.1.1.1.3.2.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.3.2.2">𝑊</ci><ci id="S2.E11.m1.2.2.1.1.1.1.1.3.2.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.3.2.3">𝑠</ci></apply><apply id="S2.E11.m1.2.2.1.1.1.1.1.3.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E11.m1.2.2.1.1.1.1.1.3.3.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.3.3">subscript</csymbol><ci id="S2.E11.m1.2.2.1.1.1.1.1.3.3.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.3.3.2">𝑠</ci><ci id="S2.E11.m1.2.2.1.1.1.1.1.3.3.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply><apply id="S2.E11.m1.2.2.1.1.1.1.1.4.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4"><times id="S2.E11.m1.2.2.1.1.1.1.1.4.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.1"></times><apply id="S2.E11.m1.2.2.1.1.1.1.1.4.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.2"><csymbol cd="ambiguous" id="S2.E11.m1.2.2.1.1.1.1.1.4.2.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.2">subscript</csymbol><ci id="S2.E11.m1.2.2.1.1.1.1.1.4.2.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.2.2">𝑤</ci><ci id="S2.E11.m1.2.2.1.1.1.1.1.4.2.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.2.3">𝑐</ci></apply><apply id="S2.E11.m1.2.2.1.1.1.1.1.4.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.3"><csymbol cd="ambiguous" id="S2.E11.m1.2.2.1.1.1.1.1.4.3.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.3">subscript</csymbol><apply id="S2.E11.m1.2.2.1.1.1.1.1.4.3.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.3"><csymbol cd="ambiguous" id="S2.E11.m1.2.2.1.1.1.1.1.4.3.2.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.3">superscript</csymbol><ci id="S2.E11.m1.2.2.1.1.1.1.1.4.3.2.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.3.2.2">𝑐</ci><ci id="S2.E11.m1.2.2.1.1.1.1.1.4.3.2.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.3.2.3">𝑡</ci></apply><ci id="S2.E11.m1.2.2.1.1.1.1.1.4.3.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.4.3.3">𝑖</ci></apply></apply><apply id="S2.E11.m1.2.2.1.1.1.1.1.5.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.5"><csymbol cd="ambiguous" id="S2.E11.m1.2.2.1.1.1.1.1.5.1.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.5">subscript</csymbol><ci id="S2.E11.m1.2.2.1.1.1.1.1.5.2.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.5.2">𝑏</ci><ci id="S2.E11.m1.2.2.1.1.1.1.1.5.3a.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.5.3"><mtext mathsize="70%" id="S2.E11.m1.2.2.1.1.1.1.1.5.3.cmml" xref="S2.E11.m1.2.2.1.1.1.1.1.5.3">attn</mtext></ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E11.m1.2c">e_{i}^{t}=v^{T}\tanh(W_{h}h_{i}+W_{s}s_{t}+w_{c}c^{t}_{i}+b_{\text{attn}})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(11)</span></td>
</tr></tbody>
</table>
<p id="S2.SS3.p2.3" class="ltx_p">where <math id="S2.SS3.p2.1.m1.1" class="ltx_Math" alttext="w_{c}" display="inline"><semantics id="S2.SS3.p2.1.m1.1a"><msub id="S2.SS3.p2.1.m1.1.1" xref="S2.SS3.p2.1.m1.1.1.cmml"><mi id="S2.SS3.p2.1.m1.1.1.2" xref="S2.SS3.p2.1.m1.1.1.2.cmml">w</mi><mi id="S2.SS3.p2.1.m1.1.1.3" xref="S2.SS3.p2.1.m1.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.1.m1.1b"><apply id="S2.SS3.p2.1.m1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.1.m1.1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1">subscript</csymbol><ci id="S2.SS3.p2.1.m1.1.1.2.cmml" xref="S2.SS3.p2.1.m1.1.1.2">𝑤</ci><ci id="S2.SS3.p2.1.m1.1.1.3.cmml" xref="S2.SS3.p2.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.1.m1.1c">w_{c}</annotation></semantics></math> is a learnable parameter vector of same length as <math id="S2.SS3.p2.2.m2.1" class="ltx_Math" alttext="v" display="inline"><semantics id="S2.SS3.p2.2.m2.1a"><mi id="S2.SS3.p2.2.m2.1.1" xref="S2.SS3.p2.2.m2.1.1.cmml">v</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.2.m2.1b"><ci id="S2.SS3.p2.2.m2.1.1.cmml" xref="S2.SS3.p2.2.m2.1.1">𝑣</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.2.m2.1c">v</annotation></semantics></math>.
This ensures that the attention mechanism’s current decision (choosing where to attend next) is informed by a reminder of its previous decisions (summarized in <math id="S2.SS3.p2.3.m3.1" class="ltx_Math" alttext="c^{t}" display="inline"><semantics id="S2.SS3.p2.3.m3.1a"><msup id="S2.SS3.p2.3.m3.1.1" xref="S2.SS3.p2.3.m3.1.1.cmml"><mi id="S2.SS3.p2.3.m3.1.1.2" xref="S2.SS3.p2.3.m3.1.1.2.cmml">c</mi><mi id="S2.SS3.p2.3.m3.1.1.3" xref="S2.SS3.p2.3.m3.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.3.m3.1b"><apply id="S2.SS3.p2.3.m3.1.1.cmml" xref="S2.SS3.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.3.m3.1.1.1.cmml" xref="S2.SS3.p2.3.m3.1.1">superscript</csymbol><ci id="S2.SS3.p2.3.m3.1.1.2.cmml" xref="S2.SS3.p2.3.m3.1.1.2">𝑐</ci><ci id="S2.SS3.p2.3.m3.1.1.3.cmml" xref="S2.SS3.p2.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.3.m3.1c">c^{t}</annotation></semantics></math>).
This should make it easier for the attention mechanism to avoid repeatedly attending to the same locations, and thus avoid generating repetitive text.</p>
</div>
<div id="S2.SS3.p3" class="ltx_para">
<p id="S2.SS3.p3.3" class="ltx_p">We find it necessary (see section <a href="#S5" title="5 Experiments ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>) to additionally define a <span id="S2.SS3.p3.3.1" class="ltx_text ltx_font_italic">coverage loss</span> to penalize repeatedly attending to the same locations:</p>
<table id="S2.E12" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E12.m1.3" class="ltx_Math" alttext="\text{covloss}_{t}=\sum\nolimits_{i}\min(a_{i}^{t},c_{i}^{t})" display="block"><semantics id="S2.E12.m1.3a"><mrow id="S2.E12.m1.3.3" xref="S2.E12.m1.3.3.cmml"><msub id="S2.E12.m1.3.3.4" xref="S2.E12.m1.3.3.4.cmml"><mtext id="S2.E12.m1.3.3.4.2" xref="S2.E12.m1.3.3.4.2a.cmml">covloss</mtext><mi id="S2.E12.m1.3.3.4.3" xref="S2.E12.m1.3.3.4.3.cmml">t</mi></msub><mo rspace="0.111em" id="S2.E12.m1.3.3.3" xref="S2.E12.m1.3.3.3.cmml">=</mo><mrow id="S2.E12.m1.3.3.2" xref="S2.E12.m1.3.3.2.cmml"><msub id="S2.E12.m1.3.3.2.3" xref="S2.E12.m1.3.3.2.3.cmml"><mo id="S2.E12.m1.3.3.2.3.2" xref="S2.E12.m1.3.3.2.3.2.cmml">∑</mo><mi id="S2.E12.m1.3.3.2.3.3" xref="S2.E12.m1.3.3.2.3.3.cmml">i</mi></msub><mrow id="S2.E12.m1.3.3.2.2.2" xref="S2.E12.m1.3.3.2.2.3.cmml"><mi id="S2.E12.m1.1.1" xref="S2.E12.m1.1.1.cmml">min</mi><mo id="S2.E12.m1.3.3.2.2.2a" xref="S2.E12.m1.3.3.2.2.3.cmml">⁡</mo><mrow id="S2.E12.m1.3.3.2.2.2.2" xref="S2.E12.m1.3.3.2.2.3.cmml"><mo stretchy="false" id="S2.E12.m1.3.3.2.2.2.2.3" xref="S2.E12.m1.3.3.2.2.3.cmml">(</mo><msubsup id="S2.E12.m1.2.2.1.1.1.1.1" xref="S2.E12.m1.2.2.1.1.1.1.1.cmml"><mi id="S2.E12.m1.2.2.1.1.1.1.1.2.2" xref="S2.E12.m1.2.2.1.1.1.1.1.2.2.cmml">a</mi><mi id="S2.E12.m1.2.2.1.1.1.1.1.2.3" xref="S2.E12.m1.2.2.1.1.1.1.1.2.3.cmml">i</mi><mi id="S2.E12.m1.2.2.1.1.1.1.1.3" xref="S2.E12.m1.2.2.1.1.1.1.1.3.cmml">t</mi></msubsup><mo id="S2.E12.m1.3.3.2.2.2.2.4" xref="S2.E12.m1.3.3.2.2.3.cmml">,</mo><msubsup id="S2.E12.m1.3.3.2.2.2.2.2" xref="S2.E12.m1.3.3.2.2.2.2.2.cmml"><mi id="S2.E12.m1.3.3.2.2.2.2.2.2.2" xref="S2.E12.m1.3.3.2.2.2.2.2.2.2.cmml">c</mi><mi id="S2.E12.m1.3.3.2.2.2.2.2.2.3" xref="S2.E12.m1.3.3.2.2.2.2.2.2.3.cmml">i</mi><mi id="S2.E12.m1.3.3.2.2.2.2.2.3" xref="S2.E12.m1.3.3.2.2.2.2.2.3.cmml">t</mi></msubsup><mo stretchy="false" id="S2.E12.m1.3.3.2.2.2.2.5" xref="S2.E12.m1.3.3.2.2.3.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E12.m1.3b"><apply id="S2.E12.m1.3.3.cmml" xref="S2.E12.m1.3.3"><eq id="S2.E12.m1.3.3.3.cmml" xref="S2.E12.m1.3.3.3"></eq><apply id="S2.E12.m1.3.3.4.cmml" xref="S2.E12.m1.3.3.4"><csymbol cd="ambiguous" id="S2.E12.m1.3.3.4.1.cmml" xref="S2.E12.m1.3.3.4">subscript</csymbol><ci id="S2.E12.m1.3.3.4.2a.cmml" xref="S2.E12.m1.3.3.4.2"><mtext id="S2.E12.m1.3.3.4.2.cmml" xref="S2.E12.m1.3.3.4.2">covloss</mtext></ci><ci id="S2.E12.m1.3.3.4.3.cmml" xref="S2.E12.m1.3.3.4.3">𝑡</ci></apply><apply id="S2.E12.m1.3.3.2.cmml" xref="S2.E12.m1.3.3.2"><apply id="S2.E12.m1.3.3.2.3.cmml" xref="S2.E12.m1.3.3.2.3"><csymbol cd="ambiguous" id="S2.E12.m1.3.3.2.3.1.cmml" xref="S2.E12.m1.3.3.2.3">subscript</csymbol><sum id="S2.E12.m1.3.3.2.3.2.cmml" xref="S2.E12.m1.3.3.2.3.2"></sum><ci id="S2.E12.m1.3.3.2.3.3.cmml" xref="S2.E12.m1.3.3.2.3.3">𝑖</ci></apply><apply id="S2.E12.m1.3.3.2.2.3.cmml" xref="S2.E12.m1.3.3.2.2.2"><min id="S2.E12.m1.1.1.cmml" xref="S2.E12.m1.1.1"></min><apply id="S2.E12.m1.2.2.1.1.1.1.1.cmml" xref="S2.E12.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E12.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.E12.m1.2.2.1.1.1.1.1">superscript</csymbol><apply id="S2.E12.m1.2.2.1.1.1.1.1.2.cmml" xref="S2.E12.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E12.m1.2.2.1.1.1.1.1.2.1.cmml" xref="S2.E12.m1.2.2.1.1.1.1.1">subscript</csymbol><ci id="S2.E12.m1.2.2.1.1.1.1.1.2.2.cmml" xref="S2.E12.m1.2.2.1.1.1.1.1.2.2">𝑎</ci><ci id="S2.E12.m1.2.2.1.1.1.1.1.2.3.cmml" xref="S2.E12.m1.2.2.1.1.1.1.1.2.3">𝑖</ci></apply><ci id="S2.E12.m1.2.2.1.1.1.1.1.3.cmml" xref="S2.E12.m1.2.2.1.1.1.1.1.3">𝑡</ci></apply><apply id="S2.E12.m1.3.3.2.2.2.2.2.cmml" xref="S2.E12.m1.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E12.m1.3.3.2.2.2.2.2.1.cmml" xref="S2.E12.m1.3.3.2.2.2.2.2">superscript</csymbol><apply id="S2.E12.m1.3.3.2.2.2.2.2.2.cmml" xref="S2.E12.m1.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E12.m1.3.3.2.2.2.2.2.2.1.cmml" xref="S2.E12.m1.3.3.2.2.2.2.2">subscript</csymbol><ci id="S2.E12.m1.3.3.2.2.2.2.2.2.2.cmml" xref="S2.E12.m1.3.3.2.2.2.2.2.2.2">𝑐</ci><ci id="S2.E12.m1.3.3.2.2.2.2.2.2.3.cmml" xref="S2.E12.m1.3.3.2.2.2.2.2.2.3">𝑖</ci></apply><ci id="S2.E12.m1.3.3.2.2.2.2.2.3.cmml" xref="S2.E12.m1.3.3.2.2.2.2.2.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E12.m1.3c">\text{covloss}_{t}=\sum\nolimits_{i}\min(a_{i}^{t},c_{i}^{t})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(12)</span></td>
</tr></tbody>
</table>
<p id="S2.SS3.p3.2" class="ltx_p">Note that the coverage loss is bounded; in particular <math id="S2.SS3.p3.1.m1.1" class="ltx_Math" alttext="\text{covloss}_{t}\leq\sum_{i}a_{i}^{t}=1" display="inline"><semantics id="S2.SS3.p3.1.m1.1a"><mrow id="S2.SS3.p3.1.m1.1.1" xref="S2.SS3.p3.1.m1.1.1.cmml"><msub id="S2.SS3.p3.1.m1.1.1.2" xref="S2.SS3.p3.1.m1.1.1.2.cmml"><mtext id="S2.SS3.p3.1.m1.1.1.2.2" xref="S2.SS3.p3.1.m1.1.1.2.2a.cmml">covloss</mtext><mi id="S2.SS3.p3.1.m1.1.1.2.3" xref="S2.SS3.p3.1.m1.1.1.2.3.cmml">t</mi></msub><mo rspace="0.111em" id="S2.SS3.p3.1.m1.1.1.3" xref="S2.SS3.p3.1.m1.1.1.3.cmml">≤</mo><mrow id="S2.SS3.p3.1.m1.1.1.4" xref="S2.SS3.p3.1.m1.1.1.4.cmml"><msub id="S2.SS3.p3.1.m1.1.1.4.1" xref="S2.SS3.p3.1.m1.1.1.4.1.cmml"><mo id="S2.SS3.p3.1.m1.1.1.4.1.2" xref="S2.SS3.p3.1.m1.1.1.4.1.2.cmml">∑</mo><mi id="S2.SS3.p3.1.m1.1.1.4.1.3" xref="S2.SS3.p3.1.m1.1.1.4.1.3.cmml">i</mi></msub><msubsup id="S2.SS3.p3.1.m1.1.1.4.2" xref="S2.SS3.p3.1.m1.1.1.4.2.cmml"><mi id="S2.SS3.p3.1.m1.1.1.4.2.2.2" xref="S2.SS3.p3.1.m1.1.1.4.2.2.2.cmml">a</mi><mi id="S2.SS3.p3.1.m1.1.1.4.2.2.3" xref="S2.SS3.p3.1.m1.1.1.4.2.2.3.cmml">i</mi><mi id="S2.SS3.p3.1.m1.1.1.4.2.3" xref="S2.SS3.p3.1.m1.1.1.4.2.3.cmml">t</mi></msubsup></mrow><mo id="S2.SS3.p3.1.m1.1.1.5" xref="S2.SS3.p3.1.m1.1.1.5.cmml">=</mo><mn id="S2.SS3.p3.1.m1.1.1.6" xref="S2.SS3.p3.1.m1.1.1.6.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.1.m1.1b"><apply id="S2.SS3.p3.1.m1.1.1.cmml" xref="S2.SS3.p3.1.m1.1.1"><and id="S2.SS3.p3.1.m1.1.1a.cmml" xref="S2.SS3.p3.1.m1.1.1"></and><apply id="S2.SS3.p3.1.m1.1.1b.cmml" xref="S2.SS3.p3.1.m1.1.1"><leq id="S2.SS3.p3.1.m1.1.1.3.cmml" xref="S2.SS3.p3.1.m1.1.1.3"></leq><apply id="S2.SS3.p3.1.m1.1.1.2.cmml" xref="S2.SS3.p3.1.m1.1.1.2"><csymbol cd="ambiguous" id="S2.SS3.p3.1.m1.1.1.2.1.cmml" xref="S2.SS3.p3.1.m1.1.1.2">subscript</csymbol><ci id="S2.SS3.p3.1.m1.1.1.2.2a.cmml" xref="S2.SS3.p3.1.m1.1.1.2.2"><mtext id="S2.SS3.p3.1.m1.1.1.2.2.cmml" xref="S2.SS3.p3.1.m1.1.1.2.2">covloss</mtext></ci><ci id="S2.SS3.p3.1.m1.1.1.2.3.cmml" xref="S2.SS3.p3.1.m1.1.1.2.3">𝑡</ci></apply><apply id="S2.SS3.p3.1.m1.1.1.4.cmml" xref="S2.SS3.p3.1.m1.1.1.4"><apply id="S2.SS3.p3.1.m1.1.1.4.1.cmml" xref="S2.SS3.p3.1.m1.1.1.4.1"><csymbol cd="ambiguous" id="S2.SS3.p3.1.m1.1.1.4.1.1.cmml" xref="S2.SS3.p3.1.m1.1.1.4.1">subscript</csymbol><sum id="S2.SS3.p3.1.m1.1.1.4.1.2.cmml" xref="S2.SS3.p3.1.m1.1.1.4.1.2"></sum><ci id="S2.SS3.p3.1.m1.1.1.4.1.3.cmml" xref="S2.SS3.p3.1.m1.1.1.4.1.3">𝑖</ci></apply><apply id="S2.SS3.p3.1.m1.1.1.4.2.cmml" xref="S2.SS3.p3.1.m1.1.1.4.2"><csymbol cd="ambiguous" id="S2.SS3.p3.1.m1.1.1.4.2.1.cmml" xref="S2.SS3.p3.1.m1.1.1.4.2">superscript</csymbol><apply id="S2.SS3.p3.1.m1.1.1.4.2.2.cmml" xref="S2.SS3.p3.1.m1.1.1.4.2"><csymbol cd="ambiguous" id="S2.SS3.p3.1.m1.1.1.4.2.2.1.cmml" xref="S2.SS3.p3.1.m1.1.1.4.2">subscript</csymbol><ci id="S2.SS3.p3.1.m1.1.1.4.2.2.2.cmml" xref="S2.SS3.p3.1.m1.1.1.4.2.2.2">𝑎</ci><ci id="S2.SS3.p3.1.m1.1.1.4.2.2.3.cmml" xref="S2.SS3.p3.1.m1.1.1.4.2.2.3">𝑖</ci></apply><ci id="S2.SS3.p3.1.m1.1.1.4.2.3.cmml" xref="S2.SS3.p3.1.m1.1.1.4.2.3">𝑡</ci></apply></apply></apply><apply id="S2.SS3.p3.1.m1.1.1c.cmml" xref="S2.SS3.p3.1.m1.1.1"><eq id="S2.SS3.p3.1.m1.1.1.5.cmml" xref="S2.SS3.p3.1.m1.1.1.5"></eq><share href="#S2.SS3.p3.1.m1.1.1.4.cmml" id="S2.SS3.p3.1.m1.1.1d.cmml" xref="S2.SS3.p3.1.m1.1.1"></share><cn type="integer" id="S2.SS3.p3.1.m1.1.1.6.cmml" xref="S2.SS3.p3.1.m1.1.1.6">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.1.m1.1c">\text{covloss}_{t}\leq\sum_{i}a_{i}^{t}=1</annotation></semantics></math>.
Equation (<a href="#S2.E12" title="In 2.3 Coverage mechanism ‣ 2 Our Models ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a>) differs from the coverage loss used in Machine Translation.
In MT, we assume that there should be a roughly one-to-one translation ratio; accordingly the final coverage vector is penalized if it is more or less than 1.
Our loss function is more flexible: because summarization should not require uniform coverage, we only penalize the overlap between each attention distribution and the coverage so far – preventing repeated attention.
Finally, the coverage loss, reweighted by some hyperparameter <math id="S2.SS3.p3.2.m2.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S2.SS3.p3.2.m2.1a"><mi id="S2.SS3.p3.2.m2.1.1" xref="S2.SS3.p3.2.m2.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.2.m2.1b"><ci id="S2.SS3.p3.2.m2.1.1.cmml" xref="S2.SS3.p3.2.m2.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.2.m2.1c">\lambda</annotation></semantics></math>, is added to the primary loss function to yield a new composite loss function:</p>
<table id="S2.E13" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E13.m1.4" class="ltx_Math" alttext="\text{loss}_{t}=-\log P(w^{*}_{t})+\lambda\sum\nolimits_{i}\min(a_{i}^{t},c_{i}^{t})" display="block"><semantics id="S2.E13.m1.4a"><mrow id="S2.E13.m1.4.4" xref="S2.E13.m1.4.4.cmml"><msub id="S2.E13.m1.4.4.5" xref="S2.E13.m1.4.4.5.cmml"><mtext id="S2.E13.m1.4.4.5.2" xref="S2.E13.m1.4.4.5.2a.cmml">loss</mtext><mi id="S2.E13.m1.4.4.5.3" xref="S2.E13.m1.4.4.5.3.cmml">t</mi></msub><mo id="S2.E13.m1.4.4.4" xref="S2.E13.m1.4.4.4.cmml">=</mo><mrow id="S2.E13.m1.4.4.3" xref="S2.E13.m1.4.4.3.cmml"><mrow id="S2.E13.m1.2.2.1.1" xref="S2.E13.m1.2.2.1.1.cmml"><mo rspace="0.167em" id="S2.E13.m1.2.2.1.1a" xref="S2.E13.m1.2.2.1.1.cmml">−</mo><mrow id="S2.E13.m1.2.2.1.1.1" xref="S2.E13.m1.2.2.1.1.1.cmml"><mrow id="S2.E13.m1.2.2.1.1.1.3" xref="S2.E13.m1.2.2.1.1.1.3.cmml"><mi id="S2.E13.m1.2.2.1.1.1.3.1" xref="S2.E13.m1.2.2.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S2.E13.m1.2.2.1.1.1.3a" xref="S2.E13.m1.2.2.1.1.1.3.cmml">⁡</mo><mi id="S2.E13.m1.2.2.1.1.1.3.2" xref="S2.E13.m1.2.2.1.1.1.3.2.cmml">P</mi></mrow><mo lspace="0em" rspace="0em" id="S2.E13.m1.2.2.1.1.1.2" xref="S2.E13.m1.2.2.1.1.1.2.cmml">​</mo><mrow id="S2.E13.m1.2.2.1.1.1.1.1" xref="S2.E13.m1.2.2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E13.m1.2.2.1.1.1.1.1.2" xref="S2.E13.m1.2.2.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.E13.m1.2.2.1.1.1.1.1.1" xref="S2.E13.m1.2.2.1.1.1.1.1.1.cmml"><mi id="S2.E13.m1.2.2.1.1.1.1.1.1.2.2" xref="S2.E13.m1.2.2.1.1.1.1.1.1.2.2.cmml">w</mi><mi id="S2.E13.m1.2.2.1.1.1.1.1.1.3" xref="S2.E13.m1.2.2.1.1.1.1.1.1.3.cmml">t</mi><mo id="S2.E13.m1.2.2.1.1.1.1.1.1.2.3" xref="S2.E13.m1.2.2.1.1.1.1.1.1.2.3.cmml">∗</mo></msubsup><mo stretchy="false" id="S2.E13.m1.2.2.1.1.1.1.1.3" xref="S2.E13.m1.2.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S2.E13.m1.4.4.3.4" xref="S2.E13.m1.4.4.3.4.cmml">+</mo><mrow id="S2.E13.m1.4.4.3.3" xref="S2.E13.m1.4.4.3.3.cmml"><mi id="S2.E13.m1.4.4.3.3.4" xref="S2.E13.m1.4.4.3.3.4.cmml">λ</mi><mo lspace="0em" rspace="0em" id="S2.E13.m1.4.4.3.3.3" xref="S2.E13.m1.4.4.3.3.3.cmml">​</mo><mrow id="S2.E13.m1.4.4.3.3.2" xref="S2.E13.m1.4.4.3.3.2.cmml"><msub id="S2.E13.m1.4.4.3.3.2.3" xref="S2.E13.m1.4.4.3.3.2.3.cmml"><mo id="S2.E13.m1.4.4.3.3.2.3.2" xref="S2.E13.m1.4.4.3.3.2.3.2.cmml">∑</mo><mi id="S2.E13.m1.4.4.3.3.2.3.3" xref="S2.E13.m1.4.4.3.3.2.3.3.cmml">i</mi></msub><mrow id="S2.E13.m1.4.4.3.3.2.2.2" xref="S2.E13.m1.4.4.3.3.2.2.3.cmml"><mi id="S2.E13.m1.1.1" xref="S2.E13.m1.1.1.cmml">min</mi><mo id="S2.E13.m1.4.4.3.3.2.2.2a" xref="S2.E13.m1.4.4.3.3.2.2.3.cmml">⁡</mo><mrow id="S2.E13.m1.4.4.3.3.2.2.2.2" xref="S2.E13.m1.4.4.3.3.2.2.3.cmml"><mo stretchy="false" id="S2.E13.m1.4.4.3.3.2.2.2.2.3" xref="S2.E13.m1.4.4.3.3.2.2.3.cmml">(</mo><msubsup id="S2.E13.m1.3.3.2.2.1.1.1.1.1" xref="S2.E13.m1.3.3.2.2.1.1.1.1.1.cmml"><mi id="S2.E13.m1.3.3.2.2.1.1.1.1.1.2.2" xref="S2.E13.m1.3.3.2.2.1.1.1.1.1.2.2.cmml">a</mi><mi id="S2.E13.m1.3.3.2.2.1.1.1.1.1.2.3" xref="S2.E13.m1.3.3.2.2.1.1.1.1.1.2.3.cmml">i</mi><mi id="S2.E13.m1.3.3.2.2.1.1.1.1.1.3" xref="S2.E13.m1.3.3.2.2.1.1.1.1.1.3.cmml">t</mi></msubsup><mo id="S2.E13.m1.4.4.3.3.2.2.2.2.4" xref="S2.E13.m1.4.4.3.3.2.2.3.cmml">,</mo><msubsup id="S2.E13.m1.4.4.3.3.2.2.2.2.2" xref="S2.E13.m1.4.4.3.3.2.2.2.2.2.cmml"><mi id="S2.E13.m1.4.4.3.3.2.2.2.2.2.2.2" xref="S2.E13.m1.4.4.3.3.2.2.2.2.2.2.2.cmml">c</mi><mi id="S2.E13.m1.4.4.3.3.2.2.2.2.2.2.3" xref="S2.E13.m1.4.4.3.3.2.2.2.2.2.2.3.cmml">i</mi><mi id="S2.E13.m1.4.4.3.3.2.2.2.2.2.3" xref="S2.E13.m1.4.4.3.3.2.2.2.2.2.3.cmml">t</mi></msubsup><mo stretchy="false" id="S2.E13.m1.4.4.3.3.2.2.2.2.5" xref="S2.E13.m1.4.4.3.3.2.2.3.cmml">)</mo></mrow></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E13.m1.4b"><apply id="S2.E13.m1.4.4.cmml" xref="S2.E13.m1.4.4"><eq id="S2.E13.m1.4.4.4.cmml" xref="S2.E13.m1.4.4.4"></eq><apply id="S2.E13.m1.4.4.5.cmml" xref="S2.E13.m1.4.4.5"><csymbol cd="ambiguous" id="S2.E13.m1.4.4.5.1.cmml" xref="S2.E13.m1.4.4.5">subscript</csymbol><ci id="S2.E13.m1.4.4.5.2a.cmml" xref="S2.E13.m1.4.4.5.2"><mtext id="S2.E13.m1.4.4.5.2.cmml" xref="S2.E13.m1.4.4.5.2">loss</mtext></ci><ci id="S2.E13.m1.4.4.5.3.cmml" xref="S2.E13.m1.4.4.5.3">𝑡</ci></apply><apply id="S2.E13.m1.4.4.3.cmml" xref="S2.E13.m1.4.4.3"><plus id="S2.E13.m1.4.4.3.4.cmml" xref="S2.E13.m1.4.4.3.4"></plus><apply id="S2.E13.m1.2.2.1.1.cmml" xref="S2.E13.m1.2.2.1.1"><minus id="S2.E13.m1.2.2.1.1.2.cmml" xref="S2.E13.m1.2.2.1.1"></minus><apply id="S2.E13.m1.2.2.1.1.1.cmml" xref="S2.E13.m1.2.2.1.1.1"><times id="S2.E13.m1.2.2.1.1.1.2.cmml" xref="S2.E13.m1.2.2.1.1.1.2"></times><apply id="S2.E13.m1.2.2.1.1.1.3.cmml" xref="S2.E13.m1.2.2.1.1.1.3"><log id="S2.E13.m1.2.2.1.1.1.3.1.cmml" xref="S2.E13.m1.2.2.1.1.1.3.1"></log><ci id="S2.E13.m1.2.2.1.1.1.3.2.cmml" xref="S2.E13.m1.2.2.1.1.1.3.2">𝑃</ci></apply><apply id="S2.E13.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.E13.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E13.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S2.E13.m1.2.2.1.1.1.1.1">subscript</csymbol><apply id="S2.E13.m1.2.2.1.1.1.1.1.1.2.cmml" xref="S2.E13.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E13.m1.2.2.1.1.1.1.1.1.2.1.cmml" xref="S2.E13.m1.2.2.1.1.1.1.1">superscript</csymbol><ci id="S2.E13.m1.2.2.1.1.1.1.1.1.2.2.cmml" xref="S2.E13.m1.2.2.1.1.1.1.1.1.2.2">𝑤</ci><times id="S2.E13.m1.2.2.1.1.1.1.1.1.2.3.cmml" xref="S2.E13.m1.2.2.1.1.1.1.1.1.2.3"></times></apply><ci id="S2.E13.m1.2.2.1.1.1.1.1.1.3.cmml" xref="S2.E13.m1.2.2.1.1.1.1.1.1.3">𝑡</ci></apply></apply></apply><apply id="S2.E13.m1.4.4.3.3.cmml" xref="S2.E13.m1.4.4.3.3"><times id="S2.E13.m1.4.4.3.3.3.cmml" xref="S2.E13.m1.4.4.3.3.3"></times><ci id="S2.E13.m1.4.4.3.3.4.cmml" xref="S2.E13.m1.4.4.3.3.4">𝜆</ci><apply id="S2.E13.m1.4.4.3.3.2.cmml" xref="S2.E13.m1.4.4.3.3.2"><apply id="S2.E13.m1.4.4.3.3.2.3.cmml" xref="S2.E13.m1.4.4.3.3.2.3"><csymbol cd="ambiguous" id="S2.E13.m1.4.4.3.3.2.3.1.cmml" xref="S2.E13.m1.4.4.3.3.2.3">subscript</csymbol><sum id="S2.E13.m1.4.4.3.3.2.3.2.cmml" xref="S2.E13.m1.4.4.3.3.2.3.2"></sum><ci id="S2.E13.m1.4.4.3.3.2.3.3.cmml" xref="S2.E13.m1.4.4.3.3.2.3.3">𝑖</ci></apply><apply id="S2.E13.m1.4.4.3.3.2.2.3.cmml" xref="S2.E13.m1.4.4.3.3.2.2.2"><min id="S2.E13.m1.1.1.cmml" xref="S2.E13.m1.1.1"></min><apply id="S2.E13.m1.3.3.2.2.1.1.1.1.1.cmml" xref="S2.E13.m1.3.3.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E13.m1.3.3.2.2.1.1.1.1.1.1.cmml" xref="S2.E13.m1.3.3.2.2.1.1.1.1.1">superscript</csymbol><apply id="S2.E13.m1.3.3.2.2.1.1.1.1.1.2.cmml" xref="S2.E13.m1.3.3.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E13.m1.3.3.2.2.1.1.1.1.1.2.1.cmml" xref="S2.E13.m1.3.3.2.2.1.1.1.1.1">subscript</csymbol><ci id="S2.E13.m1.3.3.2.2.1.1.1.1.1.2.2.cmml" xref="S2.E13.m1.3.3.2.2.1.1.1.1.1.2.2">𝑎</ci><ci id="S2.E13.m1.3.3.2.2.1.1.1.1.1.2.3.cmml" xref="S2.E13.m1.3.3.2.2.1.1.1.1.1.2.3">𝑖</ci></apply><ci id="S2.E13.m1.3.3.2.2.1.1.1.1.1.3.cmml" xref="S2.E13.m1.3.3.2.2.1.1.1.1.1.3">𝑡</ci></apply><apply id="S2.E13.m1.4.4.3.3.2.2.2.2.2.cmml" xref="S2.E13.m1.4.4.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E13.m1.4.4.3.3.2.2.2.2.2.1.cmml" xref="S2.E13.m1.4.4.3.3.2.2.2.2.2">superscript</csymbol><apply id="S2.E13.m1.4.4.3.3.2.2.2.2.2.2.cmml" xref="S2.E13.m1.4.4.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E13.m1.4.4.3.3.2.2.2.2.2.2.1.cmml" xref="S2.E13.m1.4.4.3.3.2.2.2.2.2">subscript</csymbol><ci id="S2.E13.m1.4.4.3.3.2.2.2.2.2.2.2.cmml" xref="S2.E13.m1.4.4.3.3.2.2.2.2.2.2.2">𝑐</ci><ci id="S2.E13.m1.4.4.3.3.2.2.2.2.2.2.3.cmml" xref="S2.E13.m1.4.4.3.3.2.2.2.2.2.2.3">𝑖</ci></apply><ci id="S2.E13.m1.4.4.3.3.2.2.2.2.2.3.cmml" xref="S2.E13.m1.4.4.3.3.2.2.2.2.2.3">𝑡</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E13.m1.4c">\text{loss}_{t}=-\log P(w^{*}_{t})+\lambda\sum\nolimits_{i}\min(a_{i}^{t},c_{i}^{t})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(13)</span></td>
</tr></tbody>
</table>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Related Work</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p"><span id="S3.p1.1.1" class="ltx_text ltx_font_bold">Neural abstractive summarization.</span>
<cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib22" title="" class="ltx_ref">Rush et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Rush et al. (<a href="#bib.bib22" title="" class="ltx_ref">2015</a>)</cite> were the first to apply modern neural networks to abstractive text summarization, achieving state-of-the-art performance on DUC-2004 and Gigaword, two sentence-level summarization datasets.
Their approach, which is centered on the attention mechanism, has been augmented with recurrent decoders <cite class="ltx_cite ltx_citemacro_cite">Chopra et al. (<a href="#bib.bib4" title="" class="ltx_ref">2016</a>)</cite>, Abstract Meaning Representations <cite class="ltx_cite ltx_citemacro_cite">Takase et al. (<a href="#bib.bib27" title="" class="ltx_ref">2016</a>)</cite>, hierarchical networks <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite>, variational autoencoders <cite class="ltx_cite ltx_citemacro_cite">Miao and Blunsom (<a href="#bib.bib17" title="" class="ltx_ref">2016</a>)</cite>, and direct optimization of the performance metric <cite class="ltx_cite ltx_citemacro_cite">Ranzato et al. (<a href="#bib.bib21" title="" class="ltx_ref">2016</a>)</cite>, further improving performance on those datasets.</p>
</div>
<div id="S3.p2" class="ltx_para">
<p id="S3.p2.1" class="ltx_p">However, large-scale datasets for summarization of <span id="S3.p2.1.1" class="ltx_text ltx_font_italic">longer</span> text are rare.
<cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib19" title="" class="ltx_ref">Nallapati et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite> adapted the DeepMind question-answering dataset <cite class="ltx_cite ltx_citemacro_cite">Hermann et al. (<a href="#bib.bib9" title="" class="ltx_ref">2015</a>)</cite> for summarization, resulting in the <span id="S3.p2.1.2" class="ltx_text ltx_font_italic">CNN/Daily Mail</span> dataset, and provided the first abstractive baselines.
The same authors then published a neural <em id="S3.p2.1.3" class="ltx_emph ltx_font_italic">extractive</em> approach <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib18" title="" class="ltx_ref">2017</a>)</cite>, which uses hierarchical RNNs to select sentences, and found that it significantly outperformed their abstractive result with respect to the ROUGE metric.
To our knowledge, these are the only two published results on the full dataset.</p>
</div>
<div id="S3.p3" class="ltx_para">
<p id="S3.p3.1" class="ltx_p">Prior to modern neural methods, abstractive summarization received less attention than extractive summarization, but <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib10" title="" class="ltx_ref">Jing</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Jing (<a href="#bib.bib10" title="" class="ltx_ref">2000</a>)</cite> explored cutting unimportant parts of sentences to create summaries, and <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib3" title="" class="ltx_ref">Cheung and Penn</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Cheung and Penn (<a href="#bib.bib3" title="" class="ltx_ref">2014</a>)</cite> explore sentence fusion using dependency trees.</p>
</div>
<div id="S3.p4" class="ltx_para">
<p id="S3.p4.1" class="ltx_p"><span id="S3.p4.1.1" class="ltx_text ltx_font_bold">Pointer-generator networks.</span>
The pointer network <cite class="ltx_cite ltx_citemacro_cite">Vinyals et al. (<a href="#bib.bib29" title="" class="ltx_ref">2015</a>)</cite> is a sequence-to-sequence model that uses the soft attention distribution of <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib1" title="" class="ltx_ref">Bahdanau et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Bahdanau et al. (<a href="#bib.bib1" title="" class="ltx_ref">2015</a>)</cite> to produce an output sequence consisting of elements from the input sequence.
The pointer network has been used to create hybrid approaches for NMT <cite class="ltx_cite ltx_citemacro_cite">Gulcehre et al. (<a href="#bib.bib8" title="" class="ltx_ref">2016</a>)</cite>, language modeling <cite class="ltx_cite ltx_citemacro_cite">Merity et al. (<a href="#bib.bib15" title="" class="ltx_ref">2016</a>)</cite>, and summarization <cite class="ltx_cite ltx_citemacro_cite">Gu et al. (<a href="#bib.bib7" title="" class="ltx_ref">2016</a>); Gulcehre et al. (<a href="#bib.bib8" title="" class="ltx_ref">2016</a>); Miao and Blunsom (<a href="#bib.bib17" title="" class="ltx_ref">2016</a>); Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>); Zeng et al. (<a href="#bib.bib31" title="" class="ltx_ref">2016</a>)</cite>.</p>
</div>
<div id="S3.p5" class="ltx_para">
<p id="S3.p5.2" class="ltx_p">Our approach is close to the Forced-Attention Sentence Compression model of <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib17" title="" class="ltx_ref">Miao and Blunsom</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Miao and Blunsom (<a href="#bib.bib17" title="" class="ltx_ref">2016</a>)</cite> and the CopyNet model of <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib7" title="" class="ltx_ref">Gu et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Gu et al. (<a href="#bib.bib7" title="" class="ltx_ref">2016</a>)</cite>, with some small differences:
(i) We calculate an explicit switch probability <math id="S3.p5.1.m1.1" class="ltx_Math" alttext="p_{\text{gen}}" display="inline"><semantics id="S3.p5.1.m1.1a"><msub id="S3.p5.1.m1.1.1" xref="S3.p5.1.m1.1.1.cmml"><mi id="S3.p5.1.m1.1.1.2" xref="S3.p5.1.m1.1.1.2.cmml">p</mi><mtext id="S3.p5.1.m1.1.1.3" xref="S3.p5.1.m1.1.1.3a.cmml">gen</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.p5.1.m1.1b"><apply id="S3.p5.1.m1.1.1.cmml" xref="S3.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S3.p5.1.m1.1.1.1.cmml" xref="S3.p5.1.m1.1.1">subscript</csymbol><ci id="S3.p5.1.m1.1.1.2.cmml" xref="S3.p5.1.m1.1.1.2">𝑝</ci><ci id="S3.p5.1.m1.1.1.3a.cmml" xref="S3.p5.1.m1.1.1.3"><mtext mathsize="70%" id="S3.p5.1.m1.1.1.3.cmml" xref="S3.p5.1.m1.1.1.3">gen</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p5.1.m1.1c">p_{\text{gen}}</annotation></semantics></math>, whereas Gu et al. induce competition through a shared softmax function.
(ii) We recycle the attention distribution to serve as the copy distribution, but Gu et al. use two separate distributions.
(iii) When a word appears multiple times in the source text, we sum probability mass from all corresponding parts of the attention distribution, whereas Miao and Blunsom do not.
Our reasoning is that (i) calculating an explicit <math id="S3.p5.2.m2.1" class="ltx_Math" alttext="p_{\text{gen}}" display="inline"><semantics id="S3.p5.2.m2.1a"><msub id="S3.p5.2.m2.1.1" xref="S3.p5.2.m2.1.1.cmml"><mi id="S3.p5.2.m2.1.1.2" xref="S3.p5.2.m2.1.1.2.cmml">p</mi><mtext id="S3.p5.2.m2.1.1.3" xref="S3.p5.2.m2.1.1.3a.cmml">gen</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.p5.2.m2.1b"><apply id="S3.p5.2.m2.1.1.cmml" xref="S3.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S3.p5.2.m2.1.1.1.cmml" xref="S3.p5.2.m2.1.1">subscript</csymbol><ci id="S3.p5.2.m2.1.1.2.cmml" xref="S3.p5.2.m2.1.1.2">𝑝</ci><ci id="S3.p5.2.m2.1.1.3a.cmml" xref="S3.p5.2.m2.1.1.3"><mtext mathsize="70%" id="S3.p5.2.m2.1.1.3.cmml" xref="S3.p5.2.m2.1.1.3">gen</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p5.2.m2.1c">p_{\text{gen}}</annotation></semantics></math> usefully enables us to raise or lower the probability of all generated words or all copy words at once, rather than individually, (ii) the two distributions serve such similar purposes that we find our simpler approach suffices, and (iii) we observe that the pointer mechanism often copies a word while attending to multiple occurrences of it in the source text.</p>
</div>
<div id="S3.p6" class="ltx_para">
<p id="S3.p6.1" class="ltx_p">Our approach is considerably different from that of
<cite class="ltx_cite ltx_citemacro_citet">Gulcehre et al. (<a href="#bib.bib8" title="" class="ltx_ref">2016</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite>.
Those works train their pointer components to activate only for out-of-vocabulary words or named entities (whereas we allow our model to freely learn when to use the pointer), and they do not mix the probabilities from the copy distribution and the vocabulary distribution.
We believe the mixture approach described here is better for abstractive summarization – in section <a href="#S6" title="6 Results ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> we show that the copy mechanism is vital for accurately reproducing rare but in-vocabulary words, and in section <a href="#S7.SS2" title="7.2 How abstractive is our model? ‣ 7 Discussion ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.2</span></a> we observe that the mixture model enables the language model and copy mechanism to work together to perform abstractive copying.</p>
</div>
<div id="S3.p7" class="ltx_para">
<p id="S3.p7.1" class="ltx_p"><span id="S3.p7.1.1" class="ltx_text ltx_font_bold">Coverage.</span>
Originating from Statistical Machine Translation <cite class="ltx_cite ltx_citemacro_cite">Koehn (<a href="#bib.bib11" title="" class="ltx_ref">2009</a>)</cite>, coverage was adapted for NMT by <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib28" title="" class="ltx_ref">Tu et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Tu et al. (<a href="#bib.bib28" title="" class="ltx_ref">2016</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib16" title="" class="ltx_ref">Mi et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Mi et al. (<a href="#bib.bib16" title="" class="ltx_ref">2016</a>)</cite>, who both use a GRU to update the coverage vector each step.
We find that a simpler approach – summing the attention distributions to obtain the coverage vector – suffices.
In this respect our approach is similar to <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib30" title="" class="ltx_ref">Xu et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Xu et al. (<a href="#bib.bib30" title="" class="ltx_ref">2015</a>)</cite>, who apply a coverage-like method to image captioning,
and <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib2" title="" class="ltx_ref">Chen et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Chen et al. (<a href="#bib.bib2" title="" class="ltx_ref">2016</a>)</cite>, who also incorporate a coverage mechanism (which they call ‘distraction’) as described in equation (<a href="#S2.E11" title="In 2.3 Coverage mechanism ‣ 2 Our Models ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>) into neural summarization of longer text.</p>
</div>
<div id="S3.p8" class="ltx_para">
<p id="S3.p8.1" class="ltx_p"><span id="S3.p8.1.1" class="ltx_text ltx_font_italic">Temporal attention</span> is a related technique that has been applied to NMT <cite class="ltx_cite ltx_citemacro_cite">Sankaran et al. (<a href="#bib.bib24" title="" class="ltx_ref">2016</a>)</cite> and summarization <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite>.
In this approach, each attention distribution is divided by the sum of the previous, which effectively dampens repeated attention.
We tried this method but found it too destructive, distorting the signal from the attention mechanism and reducing performance.
We hypothesize that an early intervention method such as coverage is preferable to a post hoc method such as temporal attention – it is better to <span id="S3.p8.1.2" class="ltx_text ltx_font_italic">inform</span> the attention mechanism to help it make better decisions, than to <span id="S3.p8.1.3" class="ltx_text ltx_font_italic">override</span> its decisions altogether.
This theory is supported by the large boost that coverage gives our ROUGE scores (see Table <a href="#S4.T1" title="Table 1 ‣ 4 Dataset ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>), compared to the smaller boost given by temporal attention for the same task <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite>.</p>
</div>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Dataset</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">We use the <span id="S4.p1.1.1" class="ltx_text ltx_font_italic">CNN/Daily Mail</span> dataset <cite class="ltx_cite ltx_citemacro_cite">Hermann et al. (<a href="#bib.bib9" title="" class="ltx_ref">2015</a>); Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite>, which contains online news articles (781 tokens on average) paired with multi-sentence summaries (3.75 sentences or 56 tokens on average).
We used scripts supplied by <cite class="ltx_cite ltx_citemacro_citet">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite> to obtain the same version of the the data, which has 287,226 training pairs, 13,368 validation pairs and 11,490 test pairs.
Both the dataset’s published results <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>, <a href="#bib.bib18" title="" class="ltx_ref">2017</a>)</cite> use the <span id="S4.p1.1.2" class="ltx_text ltx_font_italic">anonymized</span> version of the data, which has been pre-processed to replace each named entity, e.g., <span id="S4.p1.1.3" class="ltx_text ltx_font_italic">The United Nations</span>, with its own unique identifier for the example pair, e.g., <span id="S4.p1.1.4" class="ltx_text ltx_font_typewriter">@entity5</span>.
By contrast, we operate directly on the original text (or <span id="S4.p1.1.5" class="ltx_text ltx_font_italic">non-anonymized</span> version of the data),<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>at <a href="www.github.com/abisee/pointer-generator" title="" class="ltx_ref ltx_url ltx_font_typewriter">www.github.com/abisee/pointer-generator</a></span></span></span> which we believe is the favorable problem to solve because it requires no pre-processing.</p>
</div>
<figure id="S4.T1" class="ltx_table">
<table id="S4.T1.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T1.6.7.1" class="ltx_tr">
<td id="S4.T1.6.7.1.1" class="ltx_td ltx_border_l ltx_border_r ltx_border_t"></td>
<td id="S4.T1.6.7.1.2" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t" colspan="3">ROUGE</td>
<td id="S4.T1.6.7.1.3" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t" colspan="2">METEOR</td>
</tr>
<tr id="S4.T1.6.8.2" class="ltx_tr">
<td id="S4.T1.6.8.2.1" class="ltx_td ltx_border_l ltx_border_r"></td>
<td id="S4.T1.6.8.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1</td>
<td id="S4.T1.6.8.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">2</td>
<td id="S4.T1.6.8.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">L</td>
<td id="S4.T1.6.8.2.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">exact match</td>
<td id="S4.T1.6.8.2.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">+ stem/syn/para</td>
</tr>
<tr id="S4.T1.6.9.3" class="ltx_tr">
<td id="S4.T1.6.9.3.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">abstractive model <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite>*</td>
<td id="S4.T1.6.9.3.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">35.46</td>
<td id="S4.T1.6.9.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">13.30</td>
<td id="S4.T1.6.9.3.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">32.65</td>
<td id="S4.T1.6.9.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T1.6.9.3.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
</tr>
<tr id="S4.T1.6.10.4" class="ltx_tr">
<td id="S4.T1.6.10.4.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">seq-to-seq + attn baseline (150k vocab)</td>
<td id="S4.T1.6.10.4.2" class="ltx_td ltx_align_center ltx_border_r">30.49</td>
<td id="S4.T1.6.10.4.3" class="ltx_td ltx_align_center ltx_border_r">11.17</td>
<td id="S4.T1.6.10.4.4" class="ltx_td ltx_align_center ltx_border_r">28.08</td>
<td id="S4.T1.6.10.4.5" class="ltx_td ltx_align_center ltx_border_r">11.65</td>
<td id="S4.T1.6.10.4.6" class="ltx_td ltx_align_center ltx_border_r">12.86</td>
</tr>
<tr id="S4.T1.6.11.5" class="ltx_tr">
<td id="S4.T1.6.11.5.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">seq-to-seq + attn baseline (50k vocab)</td>
<td id="S4.T1.6.11.5.2" class="ltx_td ltx_align_center ltx_border_r">31.33</td>
<td id="S4.T1.6.11.5.3" class="ltx_td ltx_align_center ltx_border_r">11.81</td>
<td id="S4.T1.6.11.5.4" class="ltx_td ltx_align_center ltx_border_r">28.83</td>
<td id="S4.T1.6.11.5.5" class="ltx_td ltx_align_center ltx_border_r">12.03</td>
<td id="S4.T1.6.11.5.6" class="ltx_td ltx_align_center ltx_border_r">13.20</td>
</tr>
<tr id="S4.T1.6.12.6" class="ltx_tr">
<td id="S4.T1.6.12.6.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">pointer-generator</td>
<td id="S4.T1.6.12.6.2" class="ltx_td ltx_align_center ltx_border_r">36.44</td>
<td id="S4.T1.6.12.6.3" class="ltx_td ltx_align_center ltx_border_r">15.66</td>
<td id="S4.T1.6.12.6.4" class="ltx_td ltx_align_center ltx_border_r">33.42</td>
<td id="S4.T1.6.12.6.5" class="ltx_td ltx_align_center ltx_border_r">15.35</td>
<td id="S4.T1.6.12.6.6" class="ltx_td ltx_align_center ltx_border_r">16.65</td>
</tr>
<tr id="S4.T1.6.13.7" class="ltx_tr">
<td id="S4.T1.6.13.7.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">pointer-generator + coverage</td>
<td id="S4.T1.6.13.7.2" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T1.6.13.7.2.1" class="ltx_text ltx_font_bold">39.53</span></td>
<td id="S4.T1.6.13.7.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T1.6.13.7.3.1" class="ltx_text ltx_font_bold">17.28</span></td>
<td id="S4.T1.6.13.7.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T1.6.13.7.4.1" class="ltx_text ltx_font_bold">36.38</span></td>
<td id="S4.T1.6.13.7.5" class="ltx_td ltx_align_center ltx_border_r">17.32</td>
<td id="S4.T1.6.13.7.6" class="ltx_td ltx_align_center ltx_border_r">18.72</td>
</tr>
<tr id="S4.T1.6.14.8" class="ltx_tr">
<td id="S4.T1.6.14.8.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">lead-3 baseline (ours)</td>
<td id="S4.T1.6.14.8.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">40.34</td>
<td id="S4.T1.6.14.8.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">17.70</td>
<td id="S4.T1.6.14.8.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">36.57</td>
<td id="S4.T1.6.14.8.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">20.48</td>
<td id="S4.T1.6.14.8.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">22.21</td>
</tr>
<tr id="S4.T1.3.3" class="ltx_tr">
<td id="S4.T1.3.3.4" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">lead-3 baseline <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib18" title="" class="ltx_ref">2017</a>)</cite>*</td>
<td id="S4.T1.1.1.1" class="ltx_td ltx_align_center ltx_border_r">39.2</td>
<td id="S4.T1.2.2.2" class="ltx_td ltx_align_center ltx_border_r">15.7</td>
<td id="S4.T1.3.3.3" class="ltx_td ltx_align_center ltx_border_r">35.5</td>
<td id="S4.T1.3.3.5" class="ltx_td ltx_align_center ltx_border_r">-</td>
<td id="S4.T1.3.3.6" class="ltx_td ltx_align_center ltx_border_r">-</td>
</tr>
<tr id="S4.T1.6.6" class="ltx_tr">
<td id="S4.T1.6.6.4" class="ltx_td ltx_align_left ltx_border_b ltx_border_l ltx_border_r">extractive model <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib18" title="" class="ltx_ref">2017</a>)</cite>*</td>
<td id="S4.T1.4.4.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">39.6</td>
<td id="S4.T1.5.5.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">16.2</td>
<td id="S4.T1.6.6.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">35.3</td>
<td id="S4.T1.6.6.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">-</td>
<td id="S4.T1.6.6.6" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">-</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>ROUGE F<sub id="S4.T1.14.1" class="ltx_sub">1</sub> and METEOR scores on the test set.
Models and baselines in the top half are abstractive, while those in the bottom half are extractive.
Those marked with * were trained and evaluated on the anonymized dataset, and so are not strictly comparable to our results on the original text.
All our ROUGE scores have a 95% confidence interval of at most <math id="S4.T1.11.m2.1" class="ltx_Math" alttext="\pm 0.25" display="inline"><semantics id="S4.T1.11.m2.1b"><mrow id="S4.T1.11.m2.1.1" xref="S4.T1.11.m2.1.1.cmml"><mo id="S4.T1.11.m2.1.1b" xref="S4.T1.11.m2.1.1.cmml">±</mo><mn id="S4.T1.11.m2.1.1.2" xref="S4.T1.11.m2.1.1.2.cmml">0.25</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.11.m2.1c"><apply id="S4.T1.11.m2.1.1.cmml" xref="S4.T1.11.m2.1.1"><csymbol cd="latexml" id="S4.T1.11.m2.1.1.1.cmml" xref="S4.T1.11.m2.1.1">plus-or-minus</csymbol><cn type="float" id="S4.T1.11.m2.1.1.2.cmml" xref="S4.T1.11.m2.1.1.2">0.25</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.11.m2.1d">\pm 0.25</annotation></semantics></math> as reported by the official ROUGE script.
The METEOR improvement from the 50k baseline to the pointer-generator model, and from the pointer-generator to the pointer-generator+coverage model, were both found to be statistically significant using an approximate randomization test with <math id="S4.T1.12.m3.1" class="ltx_Math" alttext="p&lt;0.01" display="inline"><semantics id="S4.T1.12.m3.1b"><mrow id="S4.T1.12.m3.1.1" xref="S4.T1.12.m3.1.1.cmml"><mi id="S4.T1.12.m3.1.1.2" xref="S4.T1.12.m3.1.1.2.cmml">p</mi><mo id="S4.T1.12.m3.1.1.1" xref="S4.T1.12.m3.1.1.1.cmml">&lt;</mo><mn id="S4.T1.12.m3.1.1.3" xref="S4.T1.12.m3.1.1.3.cmml">0.01</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.12.m3.1c"><apply id="S4.T1.12.m3.1.1.cmml" xref="S4.T1.12.m3.1.1"><lt id="S4.T1.12.m3.1.1.1.cmml" xref="S4.T1.12.m3.1.1.1"></lt><ci id="S4.T1.12.m3.1.1.2.cmml" xref="S4.T1.12.m3.1.1.2">𝑝</ci><cn type="float" id="S4.T1.12.m3.1.1.3.cmml" xref="S4.T1.12.m3.1.1.3">0.01</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.12.m3.1d">p&lt;0.01</annotation></semantics></math>.
</figcaption>
</figure>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Experiments</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">For all experiments, our model has 256-dimensional hidden states and 128-dimensional word embeddings.
For the pointer-generator models, we use a vocabulary of 50k words for both source and target – note that due to the pointer network’s ability to handle OOV words, we can use a smaller vocabulary size than <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib19" title="" class="ltx_ref">Nallapati et al.</a></cite>’s <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite> 150k source and 60k target vocabularies.
For the baseline model, we also try a larger vocabulary size of 150k.</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.5" class="ltx_p">Note that the pointer and the coverage mechanism introduce very few additional parameters to the network: for the models with vocabulary size 50k, the baseline model has 21,499,600 parameters, the pointer-generator adds 1153 extra parameters (<math id="S5.p2.1.m1.1" class="ltx_Math" alttext="w_{h^{*}}" display="inline"><semantics id="S5.p2.1.m1.1a"><msub id="S5.p2.1.m1.1.1" xref="S5.p2.1.m1.1.1.cmml"><mi id="S5.p2.1.m1.1.1.2" xref="S5.p2.1.m1.1.1.2.cmml">w</mi><msup id="S5.p2.1.m1.1.1.3" xref="S5.p2.1.m1.1.1.3.cmml"><mi id="S5.p2.1.m1.1.1.3.2" xref="S5.p2.1.m1.1.1.3.2.cmml">h</mi><mo id="S5.p2.1.m1.1.1.3.3" xref="S5.p2.1.m1.1.1.3.3.cmml">∗</mo></msup></msub><annotation-xml encoding="MathML-Content" id="S5.p2.1.m1.1b"><apply id="S5.p2.1.m1.1.1.cmml" xref="S5.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S5.p2.1.m1.1.1.1.cmml" xref="S5.p2.1.m1.1.1">subscript</csymbol><ci id="S5.p2.1.m1.1.1.2.cmml" xref="S5.p2.1.m1.1.1.2">𝑤</ci><apply id="S5.p2.1.m1.1.1.3.cmml" xref="S5.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S5.p2.1.m1.1.1.3.1.cmml" xref="S5.p2.1.m1.1.1.3">superscript</csymbol><ci id="S5.p2.1.m1.1.1.3.2.cmml" xref="S5.p2.1.m1.1.1.3.2">ℎ</ci><times id="S5.p2.1.m1.1.1.3.3.cmml" xref="S5.p2.1.m1.1.1.3.3"></times></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.1.m1.1c">w_{h^{*}}</annotation></semantics></math>, <math id="S5.p2.2.m2.1" class="ltx_Math" alttext="w_{s}" display="inline"><semantics id="S5.p2.2.m2.1a"><msub id="S5.p2.2.m2.1.1" xref="S5.p2.2.m2.1.1.cmml"><mi id="S5.p2.2.m2.1.1.2" xref="S5.p2.2.m2.1.1.2.cmml">w</mi><mi id="S5.p2.2.m2.1.1.3" xref="S5.p2.2.m2.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S5.p2.2.m2.1b"><apply id="S5.p2.2.m2.1.1.cmml" xref="S5.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S5.p2.2.m2.1.1.1.cmml" xref="S5.p2.2.m2.1.1">subscript</csymbol><ci id="S5.p2.2.m2.1.1.2.cmml" xref="S5.p2.2.m2.1.1.2">𝑤</ci><ci id="S5.p2.2.m2.1.1.3.cmml" xref="S5.p2.2.m2.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.2.m2.1c">w_{s}</annotation></semantics></math>, <math id="S5.p2.3.m3.1" class="ltx_Math" alttext="w_{x}" display="inline"><semantics id="S5.p2.3.m3.1a"><msub id="S5.p2.3.m3.1.1" xref="S5.p2.3.m3.1.1.cmml"><mi id="S5.p2.3.m3.1.1.2" xref="S5.p2.3.m3.1.1.2.cmml">w</mi><mi id="S5.p2.3.m3.1.1.3" xref="S5.p2.3.m3.1.1.3.cmml">x</mi></msub><annotation-xml encoding="MathML-Content" id="S5.p2.3.m3.1b"><apply id="S5.p2.3.m3.1.1.cmml" xref="S5.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S5.p2.3.m3.1.1.1.cmml" xref="S5.p2.3.m3.1.1">subscript</csymbol><ci id="S5.p2.3.m3.1.1.2.cmml" xref="S5.p2.3.m3.1.1.2">𝑤</ci><ci id="S5.p2.3.m3.1.1.3.cmml" xref="S5.p2.3.m3.1.1.3">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.3.m3.1c">w_{x}</annotation></semantics></math> and <math id="S5.p2.4.m4.1" class="ltx_Math" alttext="b_{\text{ptr}}" display="inline"><semantics id="S5.p2.4.m4.1a"><msub id="S5.p2.4.m4.1.1" xref="S5.p2.4.m4.1.1.cmml"><mi id="S5.p2.4.m4.1.1.2" xref="S5.p2.4.m4.1.1.2.cmml">b</mi><mtext id="S5.p2.4.m4.1.1.3" xref="S5.p2.4.m4.1.1.3a.cmml">ptr</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.p2.4.m4.1b"><apply id="S5.p2.4.m4.1.1.cmml" xref="S5.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S5.p2.4.m4.1.1.1.cmml" xref="S5.p2.4.m4.1.1">subscript</csymbol><ci id="S5.p2.4.m4.1.1.2.cmml" xref="S5.p2.4.m4.1.1.2">𝑏</ci><ci id="S5.p2.4.m4.1.1.3a.cmml" xref="S5.p2.4.m4.1.1.3"><mtext mathsize="70%" id="S5.p2.4.m4.1.1.3.cmml" xref="S5.p2.4.m4.1.1.3">ptr</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.4.m4.1c">b_{\text{ptr}}</annotation></semantics></math> in equation <a href="#S2.E8" title="In 2.2 Pointer-generator network ‣ 2 Our Models ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>), and coverage adds 512 extra parameters (<math id="S5.p2.5.m5.1" class="ltx_Math" alttext="w_{c}" display="inline"><semantics id="S5.p2.5.m5.1a"><msub id="S5.p2.5.m5.1.1" xref="S5.p2.5.m5.1.1.cmml"><mi id="S5.p2.5.m5.1.1.2" xref="S5.p2.5.m5.1.1.2.cmml">w</mi><mi id="S5.p2.5.m5.1.1.3" xref="S5.p2.5.m5.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S5.p2.5.m5.1b"><apply id="S5.p2.5.m5.1.1.cmml" xref="S5.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S5.p2.5.m5.1.1.1.cmml" xref="S5.p2.5.m5.1.1">subscript</csymbol><ci id="S5.p2.5.m5.1.1.2.cmml" xref="S5.p2.5.m5.1.1.2">𝑤</ci><ci id="S5.p2.5.m5.1.1.3.cmml" xref="S5.p2.5.m5.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.5.m5.1c">w_{c}</annotation></semantics></math> in equation <a href="#S2.E11" title="In 2.3 Coverage mechanism ‣ 2 Our Models ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>).</p>
</div>
<div id="S5.p3" class="ltx_para">
<p id="S5.p3.1" class="ltx_p">Unlike <cite class="ltx_cite ltx_citemacro_citet">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite>, we do not pre-train the word embeddings – they are learned from scratch during training.
We train using Adagrad <cite class="ltx_cite ltx_citemacro_cite">Duchi et al. (<a href="#bib.bib6" title="" class="ltx_ref">2011</a>)</cite> with learning rate 0.15 and an initial accumulator value of 0.1.
(This was found to work best of Stochastic Gradient Descent, Adadelta, Momentum, Adam and RMSProp).
We use gradient clipping with a maximum gradient norm of 2, but do not use any form of regularization.
We use loss on the validation set to implement early stopping.</p>
</div>
<div id="S5.p4" class="ltx_para">
<p id="S5.p4.1" class="ltx_p">During training and at test time we truncate the article to 400 tokens and limit the length of the summary to 100 tokens for training and 120 tokens at test time.<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>The upper limit of 120 is mostly invisible: the beam search algorithm is self-stopping and almost never reaches the 120th step.</span></span></span>
This is done to expedite training and testing, but we also found that truncating the article can <span id="S5.p4.1.1" class="ltx_text ltx_font_italic">raise</span> the performance of the model (see section <a href="#S7.SS1" title="7.1 Comparison with extractive systems ‣ 7 Discussion ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.1</span></a> for more details).
For training, we found it efficient to start with highly-truncated sequences, then raise the maximum length once converged.
We train on a single Tesla K40m GPU with a batch size of 16.
At test time our summaries are produced using beam search with beam size 4.</p>
</div>
<div id="S5.p5" class="ltx_para">
<p id="S5.p5.2" class="ltx_p">We trained both our baseline models for about 600,000 iterations (33 epochs) – this is similar to the 35 epochs required by <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib19" title="" class="ltx_ref">Nallapati et al.</a></cite>’s <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite> best model.
Training took 4 days and 14 hours for the 50k vocabulary model, and 8 days 21 hours for the 150k vocabulary model.
We found the pointer-generator model quicker to train, requiring less than 230,000 training iterations (12.8 epochs); a total of 3 days and 4 hours.
In particular, the pointer-generator model makes much quicker progress in the early phases of training.
To obtain our final coverage model, we added the coverage mechanism with coverage loss weighted to <math id="S5.p5.1.m1.1" class="ltx_Math" alttext="\lambda=1" display="inline"><semantics id="S5.p5.1.m1.1a"><mrow id="S5.p5.1.m1.1.1" xref="S5.p5.1.m1.1.1.cmml"><mi id="S5.p5.1.m1.1.1.2" xref="S5.p5.1.m1.1.1.2.cmml">λ</mi><mo id="S5.p5.1.m1.1.1.1" xref="S5.p5.1.m1.1.1.1.cmml">=</mo><mn id="S5.p5.1.m1.1.1.3" xref="S5.p5.1.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.p5.1.m1.1b"><apply id="S5.p5.1.m1.1.1.cmml" xref="S5.p5.1.m1.1.1"><eq id="S5.p5.1.m1.1.1.1.cmml" xref="S5.p5.1.m1.1.1.1"></eq><ci id="S5.p5.1.m1.1.1.2.cmml" xref="S5.p5.1.m1.1.1.2">𝜆</ci><cn type="integer" id="S5.p5.1.m1.1.1.3.cmml" xref="S5.p5.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p5.1.m1.1c">\lambda=1</annotation></semantics></math> (as described in equation <a href="#S2.E13" title="In 2.3 Coverage mechanism ‣ 2 Our Models ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>), and trained for a further 3000 iterations (about 2 hours).
In this time the coverage loss converged to about 0.2, down from an initial value of about 0.5.
We also tried a more aggressive value of <math id="S5.p5.2.m2.1" class="ltx_Math" alttext="\lambda=2" display="inline"><semantics id="S5.p5.2.m2.1a"><mrow id="S5.p5.2.m2.1.1" xref="S5.p5.2.m2.1.1.cmml"><mi id="S5.p5.2.m2.1.1.2" xref="S5.p5.2.m2.1.1.2.cmml">λ</mi><mo id="S5.p5.2.m2.1.1.1" xref="S5.p5.2.m2.1.1.1.cmml">=</mo><mn id="S5.p5.2.m2.1.1.3" xref="S5.p5.2.m2.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.p5.2.m2.1b"><apply id="S5.p5.2.m2.1.1.cmml" xref="S5.p5.2.m2.1.1"><eq id="S5.p5.2.m2.1.1.1.cmml" xref="S5.p5.2.m2.1.1.1"></eq><ci id="S5.p5.2.m2.1.1.2.cmml" xref="S5.p5.2.m2.1.1.2">𝜆</ci><cn type="integer" id="S5.p5.2.m2.1.1.3.cmml" xref="S5.p5.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p5.2.m2.1c">\lambda=2</annotation></semantics></math>; this reduced coverage loss but increased the primary loss function, thus we did not use it.</p>
</div>
<div id="S5.p6" class="ltx_para">
<p id="S5.p6.1" class="ltx_p">We tried training the coverage model without the loss function, hoping that the attention mechanism may learn by itself not to attend repeatedly to the same locations, but we found this to be ineffective, with no discernible reduction in repetition.
We also tried training with coverage from the first iteration rather than as a separate training phase, but found that in the early phase of training, the coverage objective interfered with the main objective, reducing overall performance.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Results</h2>

<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.1 </span>Preliminaries</h3>

<div id="S6.SS1.p1" class="ltx_para">
<p id="S6.SS1.p1.1" class="ltx_p">Our results are given in Table <a href="#S4.T1" title="Table 1 ‣ 4 Dataset ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
We evaluate our models with the standard ROUGE metric <cite class="ltx_cite ltx_citemacro_cite">Lin (<a href="#bib.bib14" title="" class="ltx_ref">2004b</a>)</cite>, reporting the F<sub id="S6.SS1.p1.1.1" class="ltx_sub">1</sub> scores for ROUGE-1, ROUGE-2 and ROUGE-L (which respectively measure the word-overlap, bigram-overlap, and longest common sequence between the reference summary and the summary to be evaluated).
We obtain our ROUGE scores using the <span id="S6.SS1.p1.1.2" class="ltx_text ltx_font_typewriter">pyrouge</span> package.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span><a href="pypi.python.org/pypi/pyrouge/0.1.3" title="" class="ltx_ref ltx_url ltx_font_typewriter">pypi.python.org/pypi/pyrouge/0.1.3</a></span></span></span>
We also evaluate with the METEOR metric <cite class="ltx_cite ltx_citemacro_cite">Denkowski and Lavie (<a href="#bib.bib5" title="" class="ltx_ref">2014</a>)</cite>, both in exact match mode (rewarding only exact matches between words) and full mode (which additionally rewards matching stems, synonyms and paraphrases).<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span><a href="www.cs.cmu.edu/~alavie/METEOR" title="" class="ltx_ref ltx_url ltx_font_typewriter">www.cs.cmu.edu/~alavie/METEOR</a></span></span></span></p>
</div>
<div id="S6.SS1.p2" class="ltx_para">
<p id="S6.SS1.p2.1" class="ltx_p">In addition to our own models, we also report the lead-3 baseline (which uses the first three
sentences of the article as a summary), and compare to the only existing
abstractive <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite> and extractive <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib18" title="" class="ltx_ref">2017</a>)</cite> models on the full dataset.
The output of our models is available online.<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span><a href="www.github.com/abisee/pointer-generator" title="" class="ltx_ref ltx_url ltx_font_typewriter">www.github.com/abisee/pointer-generator</a></span></span></span></p>
</div>
<div id="S6.SS1.p3" class="ltx_para">
<p id="S6.SS1.p3.1" class="ltx_p">Given that we generate plain-text summaries but Nallapati et
al. <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>, <a href="#bib.bib18" title="" class="ltx_ref">2017</a>)</cite>
generate anonymized summaries (see Section <a href="#S4" title="4 Dataset ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>), our ROUGE scores are not strictly comparable.
There is evidence to suggest that the original-text dataset may result in higher ROUGE scores in general than the anonymized dataset – the lead-3 baseline is higher on the former than the latter.
One possible explanation is that multi-word named entities lead to a higher rate of <math id="S6.SS1.p3.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S6.SS1.p3.1.m1.1a"><mi id="S6.SS1.p3.1.m1.1.1" xref="S6.SS1.p3.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S6.SS1.p3.1.m1.1b"><ci id="S6.SS1.p3.1.m1.1.1.cmml" xref="S6.SS1.p3.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p3.1.m1.1c">n</annotation></semantics></math>-gram overlap.
Unfortunately, ROUGE is the only available means of comparison with Nallapati et al.’s work.
Nevertheless, given that the disparity in the lead-3 scores is (+1.1 ROUGE-1, +2.0 ROUGE-2, +1.1 ROUGE-L) points respectively, and our best model scores exceed <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib19" title="" class="ltx_ref">Nallapati et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite> by (+4.07 ROUGE-1, +3.98 ROUGE-2, +3.73 ROUGE-L) points, we may estimate that we outperform the only previous abstractive system by at least 2 ROUGE points all-round.</p>
</div>
<figure id="S6.F4" class="ltx_figure"><svg id="S6.F4.pic1" class="ltx_picture ltx_centering" height="222.16" overflow="visible" version="1.1" width="285.29"><g transform="translate(0,222.16) matrix(1 0 0 -1 0 0) translate(45.91,0) translate(0,104.92) matrix(1.0 0.0 0.0 1.0 -45.91 -104.92)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g class="ltx_nestedsvg" transform="matrix(1 0 0 1 0 0) translate(65.83,0) translate(0,104.92)"><g stroke-width="0.4pt" fill="#BFBFBF" stroke="#BFBFBF" color="#BFBFBF"><path d="M -19.93 0 L 219.18 0 M -19.93 31.74 L 219.18 31.74 M -19.93 63.48 L 219.18 63.48 M -19.93 95.21 L 219.18 95.21" style="fill:none"></path></g><g stroke-width="0.2pt" fill="#808080" fill-opacity="0" stroke="#808080" stroke-opacity="0" color="#808080"><path d="M 0 -5.91 L 0 0 M 49.81 -5.91 L 49.81 0 M 99.63 -5.91 L 99.63 0 M 149.44 -5.91 L 149.44 0 M 199.25 -5.91 L 199.25 0 M 0 101.12 L 0 95.21 M 49.81 101.12 L 49.81 95.21 M 99.63 101.12 L 99.63 95.21 M 149.44 101.12 L 149.44 95.21 M 199.25 101.12 L 199.25 95.21" style="fill:none"></path></g><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M -19.93 0 L -14.02 0 M -19.93 31.74 L -14.02 31.74 M -19.93 63.48 L -14.02 63.48 M -19.93 95.21 L -14.02 95.21 M 219.18 0 L 213.27 0 M 219.18 31.74 L 213.27 31.74 M 219.18 63.48 L 213.27 63.48 M 219.18 95.21 L 213.27 95.21" style="fill:none"></path></g><g stroke="#000000" fill="#000000" stroke-width="0.4pt"><path d="M -19.93 0 L -19.93 95.21 L 219.18 95.21 L 219.18 0 L -19.93 0 Z" style="fill:none"></path><g transform="matrix(0.86603 0.5 -0.5 0.86603 -37.02 -37.89)" fill="#000000" stroke="#000000"><foreignObject width="47.78" height="11.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S6.F4.pic1.5.5.5.5.5.1.1" class="ltx_text">1-grams</span></foreignObject></g><g transform="matrix(0.86603 0.5 -0.5 0.86603 12.8 -37.89)" fill="#000000" stroke="#000000"><foreignObject width="47.78" height="11.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S6.F4.pic1.6.6.6.6.6.1.1" class="ltx_text">2-grams</span></foreignObject></g><g transform="matrix(0.86603 0.5 -0.5 0.86603 62.61 -37.89)" fill="#000000" stroke="#000000"><foreignObject width="47.78" height="11.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S6.F4.pic1.7.7.7.7.7.1.1" class="ltx_text">3-grams</span></foreignObject></g><g transform="matrix(0.86603 0.5 -0.5 0.86603 112.42 -37.89)" fill="#000000" stroke="#000000"><foreignObject width="47.78" height="11.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S6.F4.pic1.8.8.8.8.8.1.1" class="ltx_text">4-grams</span></foreignObject></g><g transform="matrix(0.86603 0.5 -0.5 0.86603 155.01 -41.6)" fill="#000000" stroke="#000000"><foreignObject width="56.27" height="8.51" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S6.F4.pic1.9.9.9.9.9.1.1" class="ltx_text">sentences</span></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -31.73 -4.46)" fill="#000000" stroke="#000000"><foreignObject width="6.92" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S6.F4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S6.F4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S6.F4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S6.F4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S6.F4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S6.F4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S6.F4.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">0</cn></annotation-xml></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -38.65 27.28)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S6.F4.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S6.F4.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S6.F4.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S6.F4.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S6.F4.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S6.F4.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S6.F4.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.F4.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1c">10</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -38.65 59.02)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S6.F4.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S6.F4.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S6.F4.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S6.F4.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S6.F4.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S6.F4.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S6.F4.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.F4.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1c">20</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -38.65 90.76)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S6.F4.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="30" display="inline"><semantics id="S6.F4.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S6.F4.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S6.F4.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">30</mn><annotation-xml encoding="MathML-Content" id="S6.F4.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S6.F4.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S6.F4.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1">30</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.F4.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1c">30</annotation></semantics></math></foreignObject></g><clipPath id="pgfcp1"><path d="M -19.93 0 L 219.18 0 L 219.18 95.21 L -19.93 95.21 Z"></path></clipPath><g clip-path="url(#pgfcp1)"><g fill="#DB4437" stroke="#DB4437" color="#DB4437"><g fill="#F4C7C3"><path d="M -17.3 0 h 9.69 v 91.25 h -9.69 Z M 32.52 0 h 9.69 v 48.1 h -9.69 Z M 82.33 0 h 9.69 v 42.2 h -9.69 Z M 132.14 0 h 9.69 v 38.95 h -9.69 Z M 181.96 0 h 9.69 v 21.11 h -9.69 Z"></path><g stroke-width="0.4pt"><path d="M 0 0 L 4.29 4.29" style="fill:none"></path></g><defs><pattern id="pgfupat2" href="#pgfpat3"><g fill="#DB4437" stroke="#DB4437"><use href="#pgfsym3"></use></g></pattern></defs></g><g fill="url(#pgfupat2)"><path d="M -17.3 0 h 9.69 v 91.25 h -9.69 Z M 32.52 0 h 9.69 v 48.1 h -9.69 Z M 82.33 0 h 9.69 v 42.2 h -9.69 Z M 132.14 0 h 9.69 v 38.95 h -9.69 Z M 181.96 0 h 9.69 v 21.11 h -9.69 Z"></path></g></g><g></g><g stroke="#4285F4"><g fill="#E6E6FF"><path d="M 0 0 h 138.37 v 3.46 h -138.37 Z" style="stroke:none"></path></g><g fill="#D9D9FF"><path d="M 0 2.77 h 138.37 v 3.46 h -138.37 Z" style="stroke:none"></path></g><g fill="url(#pgfpat17)"><path d="M -4.84 0 h 9.69 v 67.5 h -9.69 Z M 44.97 0 h 9.69 v 10.69 h -9.69 Z M 94.78 0 h 9.69 v 4.99 h -9.69 Z M 144.6 0 h 9.69 v 3.43 h -9.69 Z M 194.41 0 h 9.69 v 0.39 h -9.69 Z"></path></g></g><g></g><g fill="#0F9D58" stroke="#0F9D58" color="#0F9D58"><g fill="#B7E2CD"><path d="M 7.61 0 h 9.69 v 62.03 h -9.69 Z M 57.42 0 h 9.69 v 4.9 h -9.69 Z M 107.24 0 h 9.69 v 0.71 h -9.69 Z M 157.05 0 h 9.69 v 0.16 h -9.69 Z M 206.86 0 h 9.69 v 0 h -9.69 Z"></path><g stroke-width="0.4pt"><path d="M 0 4.15 L 4.29 -0.14" style="fill:none"></path></g><defs><pattern id="pgfupat3" href="#pgfpat4"><g fill="#0F9D58" stroke="#0F9D58"><use href="#pgfsym4"></use></g></pattern></defs></g><g fill="url(#pgfupat3)"><path d="M 7.61 0 h 9.69 v 62.03 h -9.69 Z M 57.42 0 h 9.69 v 4.9 h -9.69 Z M 107.24 0 h 9.69 v 0.71 h -9.69 Z M 157.05 0 h 9.69 v 0.16 h -9.69 Z M 206.86 0 h 9.69 v 0 h -9.69 Z"></path></g></g><g></g></g><g transform="matrix(0.0 1.0 -1.0 0.0 -50.84 -17.41)" fill="#000000" stroke="#000000"><foreignObject width="130.03" height="13.07" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S6.F4.pic1.10.10.10.10.10.1.1" class="ltx_text">% that are duplicates</span></foreignObject></g><g fill="#FFFFFF" stroke="#000000"><path d="M -5.68 -104.65 h 210.61 v 56.76 h -210.61 Z"></path></g><g fill="#FFFFFF" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 -1.53 -101.88)"><g class="ltx_tikzmatrix" transform="matrix(1 0 0 -1 0 42.62)"><g class="ltx_tikzmatrix_row" transform="matrix(1 0 0 1 0 8.58)"><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 0 0) translate(6.5,0)" fill="#DB4437" stroke="#DB4437" color="#DB4437"><g fill="#F4C7C3"><path d="M 6.23 8.3 L -6.23 8.3 L -6.23 -2.77 L 6.23 -2.77 Z M -6.23 -2.77"></path><defs><pattern id="pgfupat4" href="#pgfpat3"><g fill="#DB4437" stroke="#DB4437"><use href="#pgfsym3"></use></g></pattern></defs></g><g fill="url(#pgfupat4)"><path d="M 6.23 8.3 L -6.23 8.3 L -6.23 -2.77 L 6.23 -2.77 Z M -6.23 -2.77"></path></g></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 13.28 0) translate(-0.28,0) matrix(1.0 0.0 0.0 1.0 3.04 -3.58)" fill="#000000" stroke="#000000"><foreignObject width="184.26" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S6.F4.pic1.11.11.11.11.11.1.1.1.1.1" class="ltx_text">pointer-generator, no coverage</span></foreignObject></g></g><g class="ltx_tikzmatrix_row" transform="matrix(1 0 0 1 0 25.58)"><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 0 0) translate(6.5,0)" fill="url(#pgfpat17)" stroke="#4285F4"><path d="M 6.23 8.3 L -6.23 8.3 L -6.23 -2.77 L 6.23 -2.77 Z M -6.23 -2.77"></path></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 13.28 0) translate(-0.28,0) matrix(1.0 0.0 0.0 1.0 3.04 -3.58)" fill="#000000" stroke="#000000"><foreignObject width="176.58" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S6.F4.pic1.12.12.12.12.12.2.2.1.1.1" class="ltx_text">pointer-generator + coverage</span></foreignObject></g></g><g class="ltx_tikzmatrix_row" transform="matrix(1 0 0 1 0 42.62)"><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 0 0) translate(6.5,0)" fill="#0F9D58" stroke="#0F9D58" color="#0F9D58"><g fill="#B7E2CD"><path d="M 6.23 8.3 L -6.23 8.3 L -6.23 -2.77 L 6.23 -2.77 Z M -6.23 -2.77"></path><defs><pattern id="pgfupat5" href="#pgfpat4"><g fill="#0F9D58" stroke="#0F9D58"><use href="#pgfsym4"></use></g></pattern></defs></g><g fill="url(#pgfupat5)"><path d="M 6.23 8.3 L -6.23 8.3 L -6.23 -2.77 L 6.23 -2.77 Z M -6.23 -2.77"></path></g></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 13.28 0) translate(-0.28,0) matrix(1.0 0.0 0.0 1.0 3.04 -3.77)" fill="#000000" stroke="#000000"><foreignObject width="122.11" height="9.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S6.F4.pic1.13.13.13.13.13.3.3.1.1.1" class="ltx_text">reference summaries</span></foreignObject></g></g></g></g></g></g></g></svg>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Coverage eliminates undesirable repetition.
Summaries from our <span id="S6.F4.6.1" class="ltx_text ltx_font_bold" style="color:#DB4437;">non-coverage model</span> contain many duplicated <math id="S6.F4.2.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S6.F4.2.m1.1b"><mi id="S6.F4.2.m1.1.1" xref="S6.F4.2.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S6.F4.2.m1.1c"><ci id="S6.F4.2.m1.1.1.cmml" xref="S6.F4.2.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.F4.2.m1.1d">n</annotation></semantics></math>-grams while our <span id="S6.F4.7.2" class="ltx_text ltx_font_bold" style="color:#4285F4;">coverage model</span> produces a similar number as the <span id="S6.F4.8.3" class="ltx_text ltx_font_bold" style="color:#0F9D58;">reference summaries</span>.</figcaption>
</figure>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.2 </span>Observations</h3>

<div id="S6.SS2.p1" class="ltx_para">
<p id="S6.SS2.p1.1" class="ltx_p">We find that both our baseline models perform poorly with respect to ROUGE and METEOR, and in fact the larger vocabulary size (150k) does not seem to help.
Even the better-performing baseline (with 50k vocabulary) produces summaries with several common problems.
Factual details are frequently reproduced incorrectly, often replacing an uncommon (but in-vocabulary) word with a more-common alternative.
For example in Figure <a href="#S0.F1" title="Figure 1 ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, the baseline model appears to struggle with the rare word <span id="S6.SS2.p1.1.1" class="ltx_text ltx_font_italic">thwart</span>, producing <span id="S6.SS2.p1.1.2" class="ltx_text ltx_font_italic">destabilize</span> instead, which leads to the fabricated phrase <span id="S6.SS2.p1.1.3" class="ltx_text ltx_font_italic">destabilize nigeria’s economy</span>.
Even more catastrophically, the summaries sometimes devolve into repetitive nonsense, such as the third sentence produced by the baseline model in Figure <a href="#S0.F1" title="Figure 1 ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
In addition, the baseline model can’t reproduce out-of-vocabulary words (such as <span id="S6.SS2.p1.1.4" class="ltx_text ltx_font_italic">muhammadu buhari</span> in Figure <a href="#S0.F1" title="Figure 1 ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).
Further examples of all these problems are provided in the supplementary material.</p>
</div>
<div id="S6.SS2.p2" class="ltx_para">
<p id="S6.SS2.p2.1" class="ltx_p">Our pointer-generator model achieves much better ROUGE and METEOR scores than the baseline, despite many fewer training epochs.
The difference in the summaries is also marked: out-of-vocabulary words are handled easily, factual details are almost always copied correctly, and there are no fabrications (see Figure <a href="#S0.F1" title="Figure 1 ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).
However, repetition is still very common.</p>
</div>
<div id="S6.SS2.p3" class="ltx_para">
<p id="S6.SS2.p3.1" class="ltx_p">Our pointer-generator model with coverage improves the ROUGE and METEOR scores further, convincingly surpassing the best abstractive model of <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib19" title="" class="ltx_ref">Nallapati et al.</a></cite> <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib19" title="" class="ltx_ref">2016</a>)</cite> by several ROUGE points.
Despite the brevity of the coverage training phase (about 1% of the total training time), the repetition problem is almost completely eliminated, which can be seen both qualitatively (Figure <a href="#S0.F1" title="Figure 1 ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>) and quantitatively (Figure <a href="#S6.F4" title="Figure 4 ‣ 6.1 Preliminaries ‣ 6 Results ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>).
However, our best model does not quite surpass the ROUGE scores of the
lead-3 baseline, nor the current best extractive model <cite class="ltx_cite ltx_citemacro_cite">Nallapati et al. (<a href="#bib.bib18" title="" class="ltx_ref">2017</a>)</cite>.
We discuss this issue in section <a href="#S7.SS1" title="7.1 Comparison with extractive systems ‣ 7 Discussion ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.1</span></a>.</p>
</div>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Discussion</h2>

<figure id="S7.F5" class="ltx_figure">
<p id="S7.F5.1" class="ltx_p">
<span id="S7.F5.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:426.8pt;">
<span id="S7.F5.1.1.1" class="ltx_p"><span id="S7.F5.1.1.1.1" class="ltx_text ltx_font_bold">Article:</span> smugglers lure arab and african migrants by offering discounts to get onto overcrowded ships if people bring more potential passengers, a cnn investigation has revealed. (…) 
<br class="ltx_break"><span id="S7.F5.1.1.1.2" class="ltx_text ltx_font_bold">Summary:</span> cnn investigation <span id="S7.F5.1.1.1.3" class="ltx_text ltx_font_bold" style="color:#4285F4;">uncovers</span> the <span id="S7.F5.1.1.1.4" class="ltx_text ltx_font_bold" style="color:#4285F4;">business inside</span> a <span id="S7.F5.1.1.1.5" class="ltx_text ltx_font_bold" style="color:#4285F4;">human smuggling ring</span>.

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="S7.F5.1.1.1.6" class="ltx_text ltx_font_bold">Article:</span> eyewitness video showing white north charleston police officer michael slager shooting to death an unarmed black man has exposed discrepancies in the reports of the first officers on the scene. (…) 
<br class="ltx_break"><span id="S7.F5.1.1.1.7" class="ltx_text ltx_font_bold">Summary:</span> more <span id="S7.F5.1.1.1.8" class="ltx_text ltx_font_bold" style="color:#4285F4;">questions than answers emerge</span> in <span id="S7.F5.1.1.1.9" class="ltx_text ltx_font_bold" style="color:#4285F4;">controversial s.c.</span> police shooting.</span>
</span></p>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Examples of highly abstractive reference summaries (<span id="S7.F5.3.1" class="ltx_text ltx_font_bold" style="color:#4285F4;">bold</span> denotes novel words).</figcaption>
</figure>
<section id="S7.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.1 </span>Comparison with extractive systems</h3>

<div id="S7.SS1.p1" class="ltx_para">
<p id="S7.SS1.p1.1" class="ltx_p">It is clear from Table <a href="#S4.T1" title="Table 1 ‣ 4 Dataset ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> that extractive systems tend to achieve higher ROUGE scores than abstractive, and that the extractive lead-3 baseline is extremely strong (even the best extractive system beats it by only a small margin).
We offer two possible explanations for these observations.</p>
</div>
<div id="S7.SS1.p2" class="ltx_para">
<p id="S7.SS1.p2.1" class="ltx_p">Firstly, news articles tend to be structured with the most important information at the start; this partially explains the strength of the lead-3 baseline.
Indeed, we found that using only the first 400 tokens (about 20 sentences) of the article yielded significantly higher ROUGE scores than using the first 800 tokens.</p>
</div>
<div id="S7.SS1.p3" class="ltx_para">
<p id="S7.SS1.p3.1" class="ltx_p">Secondly, the nature of the task and the ROUGE metric make extractive approaches and the lead-3 baseline difficult to beat.
The choice of content for the reference summaries is quite subjective – sometimes the sentences form a self-contained summary; other times they simply showcase a few interesting details from the article.
Given that the articles contain 39 sentences on average, there are many equally valid ways to choose 3 or 4 highlights in this style.
Abstraction introduces even more options (choice of phrasing), further decreasing the likelihood of matching the reference summary.
For example, <span id="S7.SS1.p3.1.1" class="ltx_text ltx_font_italic">smugglers profit from desperate migrants</span> is a valid alternative abstractive summary for the first example in Figure <a href="#S7.F5" title="Figure 5 ‣ 7 Discussion ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, but it scores 0 ROUGE with respect to the reference summary.
This inflexibility of ROUGE is exacerbated by only having one reference summary, which has been shown to lower ROUGE’s reliability compared to multiple reference summaries <cite class="ltx_cite ltx_citemacro_cite">Lin (<a href="#bib.bib13" title="" class="ltx_ref">2004a</a>)</cite>.</p>
</div>
<div id="S7.SS1.p4" class="ltx_para">
<p id="S7.SS1.p4.1" class="ltx_p">Due to the subjectivity of the task and thus the diversity of valid summaries, it seems that ROUGE rewards safe strategies such as selecting the first-appearing content, or preserving original phrasing.
While the reference summaries <span id="S7.SS1.p4.1.1" class="ltx_text ltx_font_italic">do</span> sometimes deviate from these techniques, those deviations are unpredictable enough that the safer strategy obtains higher ROUGE scores on average.
This may explain why extractive systems tend to obtain higher ROUGE scores than abstractive, and even extractive systems do not significantly exceed the lead-3 baseline.</p>
</div>
<div id="S7.SS1.p5" class="ltx_para">
<p id="S7.SS1.p5.1" class="ltx_p">To explore this issue further, we evaluated our systems with the METEOR metric, which rewards not only exact word matches, but also matching stems, synonyms and paraphrases (from a pre-defined list).
We observe that all our models receive over 1 METEOR point boost by the inclusion of stem, synonym and paraphrase matching, indicating that they may be performing some abstraction.
However, we again observe that the lead-3 baseline is not surpassed by our models.
It may be that news article style makes the lead-3 baseline very strong with respect to any metric.
We believe that investigating this issue further is an important direction for future work.</p>
</div>
<figure id="S7.F6" class="ltx_figure"><svg id="S7.F6.pic1" class="ltx_picture ltx_centering" height="209.42" overflow="visible" version="1.1" width="311.44"><g transform="translate(0,209.42) matrix(1 0 0 -1 0 0) translate(50.9,0) translate(0,105.14) matrix(1.0 0.0 0.0 1.0 -50.9 -105.14)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g class="ltx_nestedsvg" transform="matrix(1 0 0 1 0 0) translate(70.83,0) translate(0,105.14)"><g stroke-width="0.4pt" fill="#BFBFBF" stroke="#BFBFBF" color="#BFBFBF"><path d="M -19.93 0 L 219.18 0 M -19.93 19.04 L 219.18 19.04 M -19.93 38.09 L 219.18 38.09 M -19.93 57.13 L 219.18 57.13 M -19.93 76.17 L 219.18 76.17 M -19.93 95.21 L 219.18 95.21" style="fill:none"></path></g><g stroke-width="0.2pt" fill="#808080" fill-opacity="0" stroke="#808080" stroke-opacity="0" color="#808080"><path d="M 0 -5.91 L 0 0 M 49.81 -5.91 L 49.81 0 M 99.63 -5.91 L 99.63 0 M 149.44 -5.91 L 149.44 0 M 199.25 -5.91 L 199.25 0 M 0 101.12 L 0 95.21 M 49.81 101.12 L 49.81 95.21 M 99.63 101.12 L 99.63 95.21 M 149.44 101.12 L 149.44 95.21 M 199.25 101.12 L 199.25 95.21" style="fill:none"></path></g><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M -19.93 0 L -14.02 0 M -19.93 19.04 L -14.02 19.04 M -19.93 38.09 L -14.02 38.09 M -19.93 57.13 L -14.02 57.13 M -19.93 76.17 L -14.02 76.17 M -19.93 95.21 L -14.02 95.21 M 219.18 0 L 213.27 0 M 219.18 19.04 L 213.27 19.04 M 219.18 38.09 L 213.27 38.09 M 219.18 57.13 L 213.27 57.13 M 219.18 76.17 L 213.27 76.17 M 219.18 95.21 L 213.27 95.21" style="fill:none"></path></g><g stroke="#000000" fill="#000000" stroke-width="0.4pt"><path d="M -19.93 0 L -19.93 95.21 L 219.18 95.21 L 219.18 0 L -19.93 0 Z" style="fill:none"></path><g transform="matrix(0.86603 0.5 -0.5 0.86603 -37.02 -37.89)" fill="#000000" stroke="#000000"><foreignObject width="47.78" height="11.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S7.F6.pic1.7.7.7.7.7.1.1" class="ltx_text">1-grams</span></foreignObject></g><g transform="matrix(0.86603 0.5 -0.5 0.86603 12.8 -37.89)" fill="#000000" stroke="#000000"><foreignObject width="47.78" height="11.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S7.F6.pic1.8.8.8.8.8.1.1" class="ltx_text">2-grams</span></foreignObject></g><g transform="matrix(0.86603 0.5 -0.5 0.86603 62.61 -37.89)" fill="#000000" stroke="#000000"><foreignObject width="47.78" height="11.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S7.F6.pic1.9.9.9.9.9.1.1" class="ltx_text">3-grams</span></foreignObject></g><g transform="matrix(0.86603 0.5 -0.5 0.86603 112.42 -37.89)" fill="#000000" stroke="#000000"><foreignObject width="47.78" height="11.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S7.F6.pic1.10.10.10.10.10.1.1" class="ltx_text">4-grams</span></foreignObject></g><g transform="matrix(0.86603 0.5 -0.5 0.86603 155.01 -41.6)" fill="#000000" stroke="#000000"><foreignObject width="56.27" height="8.51" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S7.F6.pic1.11.11.11.11.11.1.1" class="ltx_text">sentences</span></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -31.73 -4.46)" fill="#000000" stroke="#000000"><foreignObject width="6.92" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S7.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S7.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S7.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S7.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S7.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S7.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S7.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">0</cn></annotation-xml></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -38.65 14.58)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S7.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S7.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S7.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S7.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S7.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S7.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S7.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1c">20</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -38.65 33.63)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S7.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="40" display="inline"><semantics id="S7.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S7.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S7.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">40</mn><annotation-xml encoding="MathML-Content" id="S7.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S7.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S7.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1">40</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1c">40</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -38.65 52.67)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S7.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="60" display="inline"><semantics id="S7.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S7.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S7.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">60</mn><annotation-xml encoding="MathML-Content" id="S7.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S7.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S7.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1">60</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1c">60</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -38.65 71.71)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S7.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="80" display="inline"><semantics id="S7.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S7.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S7.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">80</mn><annotation-xml encoding="MathML-Content" id="S7.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S7.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S7.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.m1.1.1">80</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.m1.1c">80</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -45.57 90.76)" fill="#000000" stroke="#000000"><foreignObject width="20.76" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S7.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="S7.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S7.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S7.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S7.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S7.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S7.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.1.1.1.1.1.1.1.1.1.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.1.1.1.1.1.1.1.1.1.m1.1c">100</annotation></semantics></math></foreignObject></g><clipPath id="pgfcp6"><path d="M -19.93 0 L 219.18 0 L 219.18 95.21 L -19.93 95.21 Z"></path></clipPath><g clip-path="url(#pgfcp6)"><g stroke="#4285F4" fill="url(#pgfpat17)"><path d="M -17.3 0 h 9.69 v 0.06 h -9.69 Z M 32.52 0 h 9.69 v 2.42 h -9.69 Z M 82.33 0 h 9.69 v 6.33 h -9.69 Z M 132.14 0 h 9.69 v 10.12 h -9.69 Z M 181.96 0 h 9.69 v 62.05 h -9.69 Z"></path></g><g></g><g fill="#F4B400" stroke="#F4B400" color="#F4B400"><g fill="#FCE9B3"><path d="M -4.84 0 h 9.69 v 5.86 h -9.69 Z M 44.97 0 h 9.69 v 20.18 h -9.69 Z M 94.78 0 h 9.69 v 33.84 h -9.69 Z M 144.6 0 h 9.69 v 43.85 h -9.69 Z M 194.41 0 h 9.69 v 92.47 h -9.69 Z"></path><defs><pattern id="pgfupat7" href="#pgfpat3"><g fill="#F4B400" stroke="#F4B400"><use href="#pgfsym3"></use></g></pattern></defs></g><g fill="url(#pgfupat7)"><path d="M -4.84 0 h 9.69 v 5.86 h -9.69 Z M 44.97 0 h 9.69 v 20.18 h -9.69 Z M 94.78 0 h 9.69 v 33.84 h -9.69 Z M 144.6 0 h 9.69 v 43.85 h -9.69 Z M 194.41 0 h 9.69 v 92.47 h -9.69 Z"></path></g></g><g></g><g fill="#0F9D58" stroke="#0F9D58" color="#0F9D58"><g fill="#B7E2CD"><path d="M 7.61 0 h 9.69 v 10.75 h -9.69 Z M 57.42 0 h 9.69 v 46.68 h -9.69 Z M 107.24 0 h 9.69 v 66.5 h -9.69 Z M 157.05 0 h 9.69 v 75.97 h -9.69 Z M 206.86 0 h 9.69 v 93.99 h -9.69 Z"></path><defs><pattern id="pgfupat8" href="#pgfpat4"><g fill="#0F9D58" stroke="#0F9D58"><use href="#pgfsym4"></use></g></pattern></defs></g><g fill="url(#pgfupat8)"><path d="M 7.61 0 h 9.69 v 10.75 h -9.69 Z M 57.42 0 h 9.69 v 46.68 h -9.69 Z M 107.24 0 h 9.69 v 66.5 h -9.69 Z M 157.05 0 h 9.69 v 75.97 h -9.69 Z M 206.86 0 h 9.69 v 93.99 h -9.69 Z"></path></g></g><g></g></g><g transform="matrix(0.0 1.0 -1.0 0.0 -55.84 -2.57)" fill="#000000" stroke="#000000"><foreignObject width="100.74" height="11.15" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S7.F6.pic1.12.12.12.12.12.1.1" class="ltx_text">% that are novel</span></foreignObject></g><g fill="#FFFFFF" stroke="#000000"><path d="M -41.08 -104.86 h 281.41 v 56.98 h -281.41 Z"></path></g><g fill="#FFFFFF" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 -36.93 -102.09)"><g class="ltx_tikzmatrix" transform="matrix(1 0 0 -1 0 42.83)"><g class="ltx_tikzmatrix_row" transform="matrix(1 0 0 1 0 8.58)"><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 0 0) translate(6.5,0)" fill="url(#pgfpat17)" stroke="#4285F4"><path d="M 6.23 8.3 L -6.23 8.3 L -6.23 -2.77 L 6.23 -2.77 Z M -6.23 -2.77"></path></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 13.28 0) translate(-0.28,0) matrix(1.0 0.0 0.0 1.0 3.04 -3.58)" fill="#000000" stroke="#000000"><foreignObject width="176.58" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S7.F6.pic1.13.13.13.13.13.1.1.1.1.1" class="ltx_text">pointer-generator + coverage</span></foreignObject></g></g><g class="ltx_tikzmatrix_row" transform="matrix(1 0 0 1 0 25.61)"><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 0 0) translate(6.5,0)" fill="#F4B400" stroke="#F4B400" color="#F4B400"><g fill="#FCE9B3"><path d="M 6.23 8.3 L -6.23 8.3 L -6.23 -2.77 L 6.23 -2.77 Z M -6.23 -2.77"></path><defs><pattern id="pgfupat9" href="#pgfpat3"><g fill="#F4B400" stroke="#F4B400"><use href="#pgfsym3"></use></g></pattern></defs></g><g fill="url(#pgfupat9)"><path d="M 6.23 8.3 L -6.23 8.3 L -6.23 -2.77 L 6.23 -2.77 Z M -6.23 -2.77"></path></g></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 13.28 0) translate(-0.28,0) matrix(1.0 0.0 0.0 1.0 3.04 -3.77)" fill="#000000" stroke="#000000"><foreignObject width="254.29" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S7.F6.pic1.14.14.14.14.14.2.2.1.1.1" class="ltx_text">sequence-to-sequence + attention baseline</span></foreignObject></g></g><g class="ltx_tikzmatrix_row" transform="matrix(1 0 0 1 0 42.83)"><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 0 0) translate(6.5,0)" fill="#0F9D58" stroke="#0F9D58" color="#0F9D58"><g fill="#B7E2CD"><path d="M 6.23 8.3 L -6.23 8.3 L -6.23 -2.77 L 6.23 -2.77 Z M -6.23 -2.77"></path><defs><pattern id="pgfupat10" href="#pgfpat4"><g fill="#0F9D58" stroke="#0F9D58"><use href="#pgfsym4"></use></g></pattern></defs></g><g fill="url(#pgfupat10)"><path d="M 6.23 8.3 L -6.23 8.3 L -6.23 -2.77 L 6.23 -2.77 Z M -6.23 -2.77"></path></g></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 13.28 0) translate(-0.28,0) matrix(1.0 0.0 0.0 1.0 3.04 -3.77)" fill="#000000" stroke="#000000"><foreignObject width="122.11" height="9.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S7.F6.pic1.15.15.15.15.15.3.3.1.1.1" class="ltx_text">reference summaries</span></foreignObject></g></g></g></g></g></g></g></svg>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Although our <span id="S7.F6.10.1" class="ltx_text ltx_font_bold" style="color:#4285F4;">best model</span> is abstractive, it does not produce novel <math id="S7.F6.4.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S7.F6.4.m1.1b"><mi id="S7.F6.4.m1.1.1" xref="S7.F6.4.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S7.F6.4.m1.1c"><ci id="S7.F6.4.m1.1.1.cmml" xref="S7.F6.4.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.F6.4.m1.1d">n</annotation></semantics></math>-grams (i.e., <math id="S7.F6.5.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S7.F6.5.m2.1b"><mi id="S7.F6.5.m2.1.1" xref="S7.F6.5.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S7.F6.5.m2.1c"><ci id="S7.F6.5.m2.1.1.cmml" xref="S7.F6.5.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.F6.5.m2.1d">n</annotation></semantics></math>-grams that don’t appear in the source text) as often as the <span id="S7.F6.11.2" class="ltx_text ltx_font_bold" style="color:#0F9D58;">reference summaries</span>. The <span id="S7.F6.12.3" class="ltx_text ltx_font_bold" style="color:#DCA200;">baseline model</span> produces more novel <math id="S7.F6.6.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S7.F6.6.m3.1b"><mi id="S7.F6.6.m3.1.1" xref="S7.F6.6.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S7.F6.6.m3.1c"><ci id="S7.F6.6.m3.1.1.cmml" xref="S7.F6.6.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.F6.6.m3.1d">n</annotation></semantics></math>-grams, but many of these are erroneous (see section <a href="#S7.SS2" title="7.2 How abstractive is our model? ‣ 7 Discussion ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.2</span></a>).</figcaption>
</figure>
<figure id="S7.F7" class="ltx_figure">
<p id="S7.F7.1" class="ltx_p">
<span id="S7.F7.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:426.8pt;">
<span id="S7.F7.1.1.1" class="ltx_p"><span id="S7.F7.1.1.1.1" class="ltx_text ltx_font_bold">Article:</span> andy murray (…) is into the semi-finals of the miami open , but not before getting a scare from 21 year-old austrian dominic thiem, who pushed him to 4-4 in the second set before going down 3-6 6-4, 6-1 in an hour and three quarters. (…) 
<br class="ltx_break"><span id="S7.F7.1.1.1.2" class="ltx_text ltx_font_bold">Summary:</span> andy murray <span id="S7.F7.1.1.1.3" class="ltx_text ltx_font_bold" style="color:#4285F4;">defeated</span> dominic thiem 3-6 6-4, 6-1 in an hour and three quarters.

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="S7.F7.1.1.1.4" class="ltx_text ltx_font_bold">Article:</span> (…) wayne rooney smashes home during manchester united ’s 3-1 win over aston villa on saturday. (…) 
<br class="ltx_break"><span id="S7.F7.1.1.1.5" class="ltx_text ltx_font_bold">Summary:</span> manchester united <span id="S7.F7.1.1.1.6" class="ltx_text ltx_font_bold" style="color:#4285F4;">beat</span> aston villa 3-1 at old trafford on saturday.</span>
</span></p>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>Examples of abstractive summaries produced by our model (<span id="S7.F7.3.1" class="ltx_text ltx_font_bold" style="color:#4285F4;">bold</span> denotes novel words).</figcaption>
</figure>
</section>
<section id="S7.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.2 </span>How abstractive is our model?</h3>

<div id="S7.SS2.p1" class="ltx_para">
<p id="S7.SS2.p1.1" class="ltx_p">We have shown that our pointer mechanism makes our abstractive system more reliable, copying factual details correctly more often.
But does the ease of copying make our system any less <span id="S7.SS2.p1.1.1" class="ltx_text ltx_font_italic">abstractive</span>?</p>
</div>
<div id="S7.SS2.p2" class="ltx_para">
<p id="S7.SS2.p2.2" class="ltx_p">Figure <a href="#S7.F6" title="Figure 6 ‣ 7.1 Comparison with extractive systems ‣ 7 Discussion ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> shows that our final model’s summaries contain a much lower rate of novel <math id="S7.SS2.p2.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S7.SS2.p2.1.m1.1a"><mi id="S7.SS2.p2.1.m1.1.1" xref="S7.SS2.p2.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S7.SS2.p2.1.m1.1b"><ci id="S7.SS2.p2.1.m1.1.1.cmml" xref="S7.SS2.p2.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p2.1.m1.1c">n</annotation></semantics></math>-grams (i.e., those that don’t appear in the article) than the reference summaries, indicating a lower degree of abstraction.
Note that the baseline model produces novel <math id="S7.SS2.p2.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S7.SS2.p2.2.m2.1a"><mi id="S7.SS2.p2.2.m2.1.1" xref="S7.SS2.p2.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S7.SS2.p2.2.m2.1b"><ci id="S7.SS2.p2.2.m2.1.1.cmml" xref="S7.SS2.p2.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p2.2.m2.1c">n</annotation></semantics></math>-grams more frequently – however, this statistic includes all the incorrectly copied words, <span id="S7.SS2.p2.2.1" class="ltx_text ltx_font_italic">UNK</span> tokens and fabrications alongside the good instances of abstraction.</p>
</div>
<div id="S7.SS2.p3" class="ltx_para">
<p id="S7.SS2.p3.1" class="ltx_p">In particular, Figure <a href="#S7.F6" title="Figure 6 ‣ 7.1 Comparison with extractive systems ‣ 7 Discussion ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> shows that our final model copies whole article sentences 35% of the time; by comparison the reference summaries do so only 1.3% of the time.
This is a main area for improvement, as we would like our model to move beyond simple sentence extraction.
However, we observe that the other 65% encompasses a range of abstractive techniques.
Article sentences are truncated to form grammatically-correct shorter versions, and new sentences are composed by stitching together fragments.
Unnecessary interjections, clauses and parenthesized phrases are sometimes omitted from copied passages.
Some of these abilities are demonstrated in Figure <a href="#S0.F1" title="Figure 1 ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, and the supplementary material contains more examples.</p>
</div>
<div id="S7.SS2.p4" class="ltx_para">
<p id="S7.SS2.p4.4" class="ltx_p">Figure <a href="#S7.F7" title="Figure 7 ‣ 7.1 Comparison with extractive systems ‣ 7 Discussion ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> shows two examples of more impressive abstraction – both with similar structure.
The dataset contains many sports stories whose summaries follow the <span id="S7.SS2.p4.4.4" class="ltx_text ltx_font_italic">X beat Y <math id="S7.SS2.p4.1.1.m1.1" class="ltx_Math" alttext="\langle" display="inline"><semantics id="S7.SS2.p4.1.1.m1.1a"><mo stretchy="false" id="S7.SS2.p4.1.1.m1.1.1" xref="S7.SS2.p4.1.1.m1.1.1.cmml">⟨</mo><annotation-xml encoding="MathML-Content" id="S7.SS2.p4.1.1.m1.1b"><ci id="S7.SS2.p4.1.1.m1.1.1.cmml" xref="S7.SS2.p4.1.1.m1.1.1">⟨</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p4.1.1.m1.1c">\langle</annotation></semantics></math>score<math id="S7.SS2.p4.2.2.m2.1" class="ltx_Math" alttext="\rangle" display="inline"><semantics id="S7.SS2.p4.2.2.m2.1a"><mo stretchy="false" id="S7.SS2.p4.2.2.m2.1.1" xref="S7.SS2.p4.2.2.m2.1.1.cmml">⟩</mo><annotation-xml encoding="MathML-Content" id="S7.SS2.p4.2.2.m2.1b"><ci id="S7.SS2.p4.2.2.m2.1.1.cmml" xref="S7.SS2.p4.2.2.m2.1.1">⟩</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p4.2.2.m2.1c">\rangle</annotation></semantics></math> on <math id="S7.SS2.p4.3.3.m3.1" class="ltx_Math" alttext="\langle" display="inline"><semantics id="S7.SS2.p4.3.3.m3.1a"><mo stretchy="false" id="S7.SS2.p4.3.3.m3.1.1" xref="S7.SS2.p4.3.3.m3.1.1.cmml">⟨</mo><annotation-xml encoding="MathML-Content" id="S7.SS2.p4.3.3.m3.1b"><ci id="S7.SS2.p4.3.3.m3.1.1.cmml" xref="S7.SS2.p4.3.3.m3.1.1">⟨</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p4.3.3.m3.1c">\langle</annotation></semantics></math>day<math id="S7.SS2.p4.4.4.m4.1" class="ltx_Math" alttext="\rangle" display="inline"><semantics id="S7.SS2.p4.4.4.m4.1a"><mo stretchy="false" id="S7.SS2.p4.4.4.m4.1.1" xref="S7.SS2.p4.4.4.m4.1.1.cmml">⟩</mo><annotation-xml encoding="MathML-Content" id="S7.SS2.p4.4.4.m4.1b"><ci id="S7.SS2.p4.4.4.m4.1.1.cmml" xref="S7.SS2.p4.4.4.m4.1.1">⟩</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p4.4.4.m4.1c">\rangle</annotation></semantics></math></span> template, which may explain why our model is most confidently abstractive on these examples.
In general however, our model does not routinely produce summaries like those in Figure <a href="#S7.F7" title="Figure 7 ‣ 7.1 Comparison with extractive systems ‣ 7 Discussion ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, and is not close to producing summaries like in Figure <a href="#S7.F5" title="Figure 5 ‣ 7 Discussion ‣ Get To The Point: Summarization with Pointer-Generator Networks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<div id="S7.SS2.p5" class="ltx_para">
<p id="S7.SS2.p5.4" class="ltx_p">The value of the generation probability <math id="S7.SS2.p5.1.m1.1" class="ltx_Math" alttext="p_{\text{gen}}" display="inline"><semantics id="S7.SS2.p5.1.m1.1a"><msub id="S7.SS2.p5.1.m1.1.1" xref="S7.SS2.p5.1.m1.1.1.cmml"><mi id="S7.SS2.p5.1.m1.1.1.2" xref="S7.SS2.p5.1.m1.1.1.2.cmml">p</mi><mtext id="S7.SS2.p5.1.m1.1.1.3" xref="S7.SS2.p5.1.m1.1.1.3a.cmml">gen</mtext></msub><annotation-xml encoding="MathML-Content" id="S7.SS2.p5.1.m1.1b"><apply id="S7.SS2.p5.1.m1.1.1.cmml" xref="S7.SS2.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S7.SS2.p5.1.m1.1.1.1.cmml" xref="S7.SS2.p5.1.m1.1.1">subscript</csymbol><ci id="S7.SS2.p5.1.m1.1.1.2.cmml" xref="S7.SS2.p5.1.m1.1.1.2">𝑝</ci><ci id="S7.SS2.p5.1.m1.1.1.3a.cmml" xref="S7.SS2.p5.1.m1.1.1.3"><mtext mathsize="70%" id="S7.SS2.p5.1.m1.1.1.3.cmml" xref="S7.SS2.p5.1.m1.1.1.3">gen</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p5.1.m1.1c">p_{\text{gen}}</annotation></semantics></math> also gives a measure of the abstractiveness of our model.
During training, <math id="S7.SS2.p5.2.m2.1" class="ltx_Math" alttext="p_{\text{gen}}" display="inline"><semantics id="S7.SS2.p5.2.m2.1a"><msub id="S7.SS2.p5.2.m2.1.1" xref="S7.SS2.p5.2.m2.1.1.cmml"><mi id="S7.SS2.p5.2.m2.1.1.2" xref="S7.SS2.p5.2.m2.1.1.2.cmml">p</mi><mtext id="S7.SS2.p5.2.m2.1.1.3" xref="S7.SS2.p5.2.m2.1.1.3a.cmml">gen</mtext></msub><annotation-xml encoding="MathML-Content" id="S7.SS2.p5.2.m2.1b"><apply id="S7.SS2.p5.2.m2.1.1.cmml" xref="S7.SS2.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S7.SS2.p5.2.m2.1.1.1.cmml" xref="S7.SS2.p5.2.m2.1.1">subscript</csymbol><ci id="S7.SS2.p5.2.m2.1.1.2.cmml" xref="S7.SS2.p5.2.m2.1.1.2">𝑝</ci><ci id="S7.SS2.p5.2.m2.1.1.3a.cmml" xref="S7.SS2.p5.2.m2.1.1.3"><mtext mathsize="70%" id="S7.SS2.p5.2.m2.1.1.3.cmml" xref="S7.SS2.p5.2.m2.1.1.3">gen</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p5.2.m2.1c">p_{\text{gen}}</annotation></semantics></math> starts with a value of about 0.30 then increases, converging to about 0.53 by the end of training.
This indicates that the model first learns to mostly copy, then learns to generate about half the time.
However at test time, <math id="S7.SS2.p5.3.m3.1" class="ltx_Math" alttext="p_{\text{gen}}" display="inline"><semantics id="S7.SS2.p5.3.m3.1a"><msub id="S7.SS2.p5.3.m3.1.1" xref="S7.SS2.p5.3.m3.1.1.cmml"><mi id="S7.SS2.p5.3.m3.1.1.2" xref="S7.SS2.p5.3.m3.1.1.2.cmml">p</mi><mtext id="S7.SS2.p5.3.m3.1.1.3" xref="S7.SS2.p5.3.m3.1.1.3a.cmml">gen</mtext></msub><annotation-xml encoding="MathML-Content" id="S7.SS2.p5.3.m3.1b"><apply id="S7.SS2.p5.3.m3.1.1.cmml" xref="S7.SS2.p5.3.m3.1.1"><csymbol cd="ambiguous" id="S7.SS2.p5.3.m3.1.1.1.cmml" xref="S7.SS2.p5.3.m3.1.1">subscript</csymbol><ci id="S7.SS2.p5.3.m3.1.1.2.cmml" xref="S7.SS2.p5.3.m3.1.1.2">𝑝</ci><ci id="S7.SS2.p5.3.m3.1.1.3a.cmml" xref="S7.SS2.p5.3.m3.1.1.3"><mtext mathsize="70%" id="S7.SS2.p5.3.m3.1.1.3.cmml" xref="S7.SS2.p5.3.m3.1.1.3">gen</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p5.3.m3.1c">p_{\text{gen}}</annotation></semantics></math> is heavily skewed towards copying, with a mean value of 0.17.
The disparity is likely due to the fact that during training, the model receives word-by-word supervision in the form of the reference summary, but at test time it does not.
Nonetheless, the generator module is useful even when the model is copying.
We find that <math id="S7.SS2.p5.4.m4.1" class="ltx_Math" alttext="p_{\text{gen}}" display="inline"><semantics id="S7.SS2.p5.4.m4.1a"><msub id="S7.SS2.p5.4.m4.1.1" xref="S7.SS2.p5.4.m4.1.1.cmml"><mi id="S7.SS2.p5.4.m4.1.1.2" xref="S7.SS2.p5.4.m4.1.1.2.cmml">p</mi><mtext id="S7.SS2.p5.4.m4.1.1.3" xref="S7.SS2.p5.4.m4.1.1.3a.cmml">gen</mtext></msub><annotation-xml encoding="MathML-Content" id="S7.SS2.p5.4.m4.1b"><apply id="S7.SS2.p5.4.m4.1.1.cmml" xref="S7.SS2.p5.4.m4.1.1"><csymbol cd="ambiguous" id="S7.SS2.p5.4.m4.1.1.1.cmml" xref="S7.SS2.p5.4.m4.1.1">subscript</csymbol><ci id="S7.SS2.p5.4.m4.1.1.2.cmml" xref="S7.SS2.p5.4.m4.1.1.2">𝑝</ci><ci id="S7.SS2.p5.4.m4.1.1.3a.cmml" xref="S7.SS2.p5.4.m4.1.1.3"><mtext mathsize="70%" id="S7.SS2.p5.4.m4.1.1.3.cmml" xref="S7.SS2.p5.4.m4.1.1.3">gen</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p5.4.m4.1c">p_{\text{gen}}</annotation></semantics></math> is highest at times of uncertainty such as the beginning of sentences, the join between stitched-together fragments, and when producing periods that truncate a copied sentence.
Our mixture model allows the network to copy while simultaneously consulting the language model – enabling operations like stitching and truncation to be performed with grammaticality.
In any case, encouraging the pointer-generator model to write more abstractively, while retaining the accuracy advantages of the pointer module, is an exciting direction for future work.</p>
</div>
</section>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8 </span>Conclusion</h2>

<div id="S8.p1" class="ltx_para">
<p id="S8.p1.1" class="ltx_p">In this work we presented a hybrid pointer-generator architecture with coverage, and showed that it reduces inaccuracies and repetition.
We applied our model to a new and challenging long-text dataset, and significantly outperformed the abstractive state-of-the-art result.
Our model exhibits many abstractive abilities, but attaining higher levels of abstraction remains an open research question.</p>
</div>
</section>
<section id="S9" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">9 </span>Acknowledgment</h2>

<div id="S9.p1" class="ltx_para">
<p id="S9.p1.1" class="ltx_p">We thank the ACL reviewers for their helpful comments.
This work was begun while the first author was an intern at Google Brain and continued at Stanford. Stanford University gratefully acknowledges the support of the DARPA DEFT Program AFRL contract no. FA8750-13-2-0040. Any opinions in this material are those of the authors alone.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bahdanau et al. (2015)</span>
<span class="ltx_bibblock">
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.

</span>
<span class="ltx_bibblock">Neural machine translation by jointly learning to align and
translate.

</span>
<span class="ltx_bibblock">In <span id="bib.bib1.1.1" class="ltx_text ltx_font_italic">International Conference on Learning Representations</span>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2016)</span>
<span class="ltx_bibblock">
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, and Hui Jiang. 2016.

</span>
<span class="ltx_bibblock">Distraction-based neural networks for modeling documents.

</span>
<span class="ltx_bibblock">In <span id="bib.bib2.1.1" class="ltx_text ltx_font_italic">International Joint Conference on Artificial Intelligence</span>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cheung and Penn (2014)</span>
<span class="ltx_bibblock">
Jackie Chi Kit Cheung and Gerald Penn. 2014.

</span>
<span class="ltx_bibblock">Unsupervised sentence enhancement for automatic summarization.

</span>
<span class="ltx_bibblock">In <span id="bib.bib3.1.1" class="ltx_text ltx_font_italic">Empirical Methods in Natural Language Processing</span>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chopra et al. (2016)</span>
<span class="ltx_bibblock">
Sumit Chopra, Michael Auli, and Alexander M Rush. 2016.

</span>
<span class="ltx_bibblock">Abstractive sentence summarization with attentive recurrent neural
networks.

</span>
<span class="ltx_bibblock">In <span id="bib.bib4.1.1" class="ltx_text ltx_font_italic">North American Chapter of the Association for Computational
Linguistics</span>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Denkowski and Lavie (2014)</span>
<span class="ltx_bibblock">
Michael Denkowski and Alon Lavie. 2014.

</span>
<span class="ltx_bibblock">Meteor universal: Language specific translation evaluation for any
target language.

</span>
<span class="ltx_bibblock">In <span id="bib.bib5.1.1" class="ltx_text ltx_font_italic">EACL 2014 Workshop on Statistical Machine Translation</span>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Duchi et al. (2011)</span>
<span class="ltx_bibblock">
John Duchi, Elad Hazan, and Yoram Singer. 2011.

</span>
<span class="ltx_bibblock">Adaptive subgradient methods for online learning and stochastic
optimization.

</span>
<span class="ltx_bibblock"><span id="bib.bib6.1.1" class="ltx_text ltx_font_italic">Journal of Machine Learning Research</span> 12:2121–2159.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gu et al. (2016)</span>
<span class="ltx_bibblock">
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016.

</span>
<span class="ltx_bibblock">Incorporating copying mechanism in sequence-to-sequence learning.

</span>
<span class="ltx_bibblock">In <span id="bib.bib7.1.1" class="ltx_text ltx_font_italic">Association for Computational Linguistics</span>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gulcehre et al. (2016)</span>
<span class="ltx_bibblock">
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio.
2016.

</span>
<span class="ltx_bibblock">Pointing the unknown words.

</span>
<span class="ltx_bibblock">In <span id="bib.bib8.1.1" class="ltx_text ltx_font_italic">Association for Computational Linguistics</span>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hermann et al. (2015)</span>
<span class="ltx_bibblock">
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will
Kay, Mustafa Suleyman, and Phil Blunsom. 2015.

</span>
<span class="ltx_bibblock">Teaching machines to read and comprehend.

</span>
<span class="ltx_bibblock">In <span id="bib.bib9.1.1" class="ltx_text ltx_font_italic">Neural Information Processing Systems</span>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jing (2000)</span>
<span class="ltx_bibblock">
Hongyan Jing. 2000.

</span>
<span class="ltx_bibblock">Sentence reduction for automatic text summarization.

</span>
<span class="ltx_bibblock">In <span id="bib.bib10.1.1" class="ltx_text ltx_font_italic">Applied natural language processing</span>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Koehn (2009)</span>
<span class="ltx_bibblock">
Philipp Koehn. 2009.

</span>
<span class="ltx_bibblock"><span id="bib.bib11.1.1" class="ltx_text ltx_font_italic">Statistical machine translation</span>.

</span>
<span class="ltx_bibblock">Cambridge University Press.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kupiec et al. (1995)</span>
<span class="ltx_bibblock">
Julian Kupiec, Jan Pedersen, and Francine Chen. 1995.

</span>
<span class="ltx_bibblock">A trainable document summarizer.

</span>
<span class="ltx_bibblock">In <span id="bib.bib12.1.1" class="ltx_text ltx_font_italic">International ACM SIGIR conference on Research and
development in information retrieval</span>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin (2004a)</span>
<span class="ltx_bibblock">
Chin-Yew Lin. 2004a.

</span>
<span class="ltx_bibblock">Looking for a few good metrics: Automatic summarization
evaluation-how many samples are enough?

</span>
<span class="ltx_bibblock">In <span id="bib.bib13.1.1" class="ltx_text ltx_font_italic">NACSIS/NII Test Collection for Information Retrieval (NTCIR)
Workshop</span>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin (2004b)</span>
<span class="ltx_bibblock">
Chin-Yew Lin. 2004b.

</span>
<span class="ltx_bibblock">Rouge: A package for automatic evaluation of summaries.

</span>
<span class="ltx_bibblock">In <span id="bib.bib14.1.1" class="ltx_text ltx_font_italic">Text summarization branches out: ACL workshop</span>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Merity et al. (2016)</span>
<span class="ltx_bibblock">
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.

</span>
<span class="ltx_bibblock">Pointer sentinel mixture models.

</span>
<span class="ltx_bibblock">In <span id="bib.bib15.1.1" class="ltx_text ltx_font_italic">NIPS 2016 Workshop on Multi-class and Multi-label Learning in
Extremely Large Label Spaces</span>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mi et al. (2016)</span>
<span class="ltx_bibblock">
Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016.

</span>
<span class="ltx_bibblock">Coverage embedding models for neural machine translation.

</span>
<span class="ltx_bibblock">In <span id="bib.bib16.1.1" class="ltx_text ltx_font_italic">Empirical Methods in Natural Language Processing</span>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Miao and Blunsom (2016)</span>
<span class="ltx_bibblock">
Yishu Miao and Phil Blunsom. 2016.

</span>
<span class="ltx_bibblock">Language as a latent variable: Discrete generative models for
sentence compression.

</span>
<span class="ltx_bibblock">In <span id="bib.bib17.1.1" class="ltx_text ltx_font_italic">Empirical Methods in Natural Language Processing</span>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nallapati et al. (2017)</span>
<span class="ltx_bibblock">
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017.

</span>
<span class="ltx_bibblock">SummaRuNNer: A recurrent neural network based sequence model for
extractive summarization of documents.

</span>
<span class="ltx_bibblock">In <span id="bib.bib18.1.1" class="ltx_text ltx_font_italic">Association for the Advancement of Artificial
Intelligence</span>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nallapati et al. (2016)</span>
<span class="ltx_bibblock">
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre,
and Bing Xiang. 2016.

</span>
<span class="ltx_bibblock">Abstractive text summarization using sequence-to-sequence RNNs and
beyond.

</span>
<span class="ltx_bibblock">In <span id="bib.bib19.1.1" class="ltx_text ltx_font_italic">Computational Natural Language Learning</span>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Paice (1990)</span>
<span class="ltx_bibblock">
Chris D Paice. 1990.

</span>
<span class="ltx_bibblock">Constructing literature abstracts by computer: techniques and
prospects.

</span>
<span class="ltx_bibblock"><span id="bib.bib20.1.1" class="ltx_text ltx_font_italic">Information Processing &amp; Management</span> 26(1):171–186.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ranzato et al. (2016)</span>
<span class="ltx_bibblock">
Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016.

</span>
<span class="ltx_bibblock">Sequence level training with recurrent neural networks.

</span>
<span class="ltx_bibblock">In <span id="bib.bib21.1.1" class="ltx_text ltx_font_italic">International Conference on Learning Representations</span>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rush et al. (2015)</span>
<span class="ltx_bibblock">
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015.

</span>
<span class="ltx_bibblock">A neural attention model for abstractive sentence summarization.

</span>
<span class="ltx_bibblock">In <span id="bib.bib22.1.1" class="ltx_text ltx_font_italic">Empirical Methods in Natural Language Processing</span>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Saggion and Poibeau (2013)</span>
<span class="ltx_bibblock">
Horacio Saggion and Thierry Poibeau. 2013.

</span>
<span class="ltx_bibblock">Automatic text summarization: Past, present and future.

</span>
<span class="ltx_bibblock">In <span id="bib.bib23.1.1" class="ltx_text ltx_font_italic">Multi-source, Multilingual Information Extraction and
Summarization</span>, Springer, pages 3–21.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sankaran et al. (2016)</span>
<span class="ltx_bibblock">
Baskaran Sankaran, Haitao Mi, Yaser Al-Onaizan, and Abe Ittycheriah. 2016.

</span>
<span class="ltx_bibblock">Temporal attention model for neural machine translation.

</span>
<span class="ltx_bibblock"><span id="bib.bib24.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1608.02927</span> .

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sutskever et al. (2014)</span>
<span class="ltx_bibblock">
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.

</span>
<span class="ltx_bibblock">Sequence to sequence learning with neural networks.

</span>
<span class="ltx_bibblock">In <span id="bib.bib25.1.1" class="ltx_text ltx_font_italic">Neural Information Processing Systems</span>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Suzuki and Nagata (2016)</span>
<span class="ltx_bibblock">
Jun Suzuki and Masaaki Nagata. 2016.

</span>
<span class="ltx_bibblock">RNN-based encoder-decoder approach with word frequency estimation.

</span>
<span class="ltx_bibblock"><span id="bib.bib26.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1701.00138</span> .

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Takase et al. (2016)</span>
<span class="ltx_bibblock">
Sho Takase, Jun Suzuki, Naoaki Okazaki, Tsutomu Hirao, and Masaaki Nagata.
2016.

</span>
<span class="ltx_bibblock">Neural headline generation on abstract meaning representation.

</span>
<span class="ltx_bibblock">In <span id="bib.bib27.1.1" class="ltx_text ltx_font_italic">Empirical Methods in Natural Language Processing</span>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tu et al. (2016)</span>
<span class="ltx_bibblock">
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016.

</span>
<span class="ltx_bibblock">Modeling coverage for neural machine translation.

</span>
<span class="ltx_bibblock">In <span id="bib.bib28.1.1" class="ltx_text ltx_font_italic">Association for Computational Linguistics</span>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vinyals et al. (2015)</span>
<span class="ltx_bibblock">
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015.

</span>
<span class="ltx_bibblock">Pointer networks.

</span>
<span class="ltx_bibblock">In <span id="bib.bib29.1.1" class="ltx_text ltx_font_italic">Neural Information Processing Systems</span>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2015)</span>
<span class="ltx_bibblock">
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan
Salakhutdinov, Richard S Zemel, and Yoshua Bengio. 2015.

</span>
<span class="ltx_bibblock">Show, attend and tell: Neural image caption generation with visual
attention.

</span>
<span class="ltx_bibblock">In <span id="bib.bib30.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zeng et al. (2016)</span>
<span class="ltx_bibblock">
Wenyuan Zeng, Wenjie Luo, Sanja Fidler, and Raquel Urtasun. 2016.

</span>
<span class="ltx_bibblock">Efficient summarization with read-again and copy mechanism.

</span>
<span class="ltx_bibblock"><span id="bib.bib31.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1611.03382</span> .

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="Ax1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">Supplementary Material</h2>

<div id="Ax1.p1" class="ltx_para">
<p id="Ax1.p1.1" class="ltx_p">This appendix provides examples from the test set, with side-by-side comparisons of the reference summaries and the summaries produced by our models.
In each example:</p>
<ul id="Ax1.I1" class="ltx_itemize">
<li id="Ax1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="Ax1.I1.i1.p1" class="ltx_para">
<p id="Ax1.I1.i1.p1.1" class="ltx_p"><span id="Ax1.I1.i1.p1.1.1" class="ltx_text ltx_font_italic">italics</span> denote out-of-vocabulary words</p>
</div>
</li>
<li id="Ax1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="Ax1.I1.i2.p1" class="ltx_para">
<p id="Ax1.I1.i2.p1.1" class="ltx_p"><span id="Ax1.I1.i2.p1.1.1" class="ltx_text ltx_font_bold" style="color:#DB4437;">red</span> denotes factual errors in the summaries</p>
</div>
</li>
<li id="Ax1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="Ax1.I1.i3.p1" class="ltx_para">
<p id="Ax1.I1.i3.p1.1" class="ltx_p"><span id="Ax1.I1.i3.p1.1.1" class="ltx_text" style="background-color:#B7E2CD;">green shading</span> intensity represents the value of the generation probability <math id="Ax1.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="p_{\text{gen}}" display="inline"><semantics id="Ax1.I1.i3.p1.1.m1.1a"><msub id="Ax1.I1.i3.p1.1.m1.1.1" xref="Ax1.I1.i3.p1.1.m1.1.1.cmml"><mi id="Ax1.I1.i3.p1.1.m1.1.1.2" xref="Ax1.I1.i3.p1.1.m1.1.1.2.cmml">p</mi><mtext id="Ax1.I1.i3.p1.1.m1.1.1.3" xref="Ax1.I1.i3.p1.1.m1.1.1.3a.cmml">gen</mtext></msub><annotation-xml encoding="MathML-Content" id="Ax1.I1.i3.p1.1.m1.1b"><apply id="Ax1.I1.i3.p1.1.m1.1.1.cmml" xref="Ax1.I1.i3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="Ax1.I1.i3.p1.1.m1.1.1.1.cmml" xref="Ax1.I1.i3.p1.1.m1.1.1">subscript</csymbol><ci id="Ax1.I1.i3.p1.1.m1.1.1.2.cmml" xref="Ax1.I1.i3.p1.1.m1.1.1.2">𝑝</ci><ci id="Ax1.I1.i3.p1.1.m1.1.1.3a.cmml" xref="Ax1.I1.i3.p1.1.m1.1.1.3"><mtext mathsize="70%" id="Ax1.I1.i3.p1.1.m1.1.1.3.cmml" xref="Ax1.I1.i3.p1.1.m1.1.1.3">gen</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Ax1.I1.i3.p1.1.m1.1c">p_{\text{gen}}</annotation></semantics></math></p>
</div>
</li>
<li id="Ax1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="Ax1.I1.i4.p1" class="ltx_para">
<p id="Ax1.I1.i4.p1.1" class="ltx_p"><span id="Ax1.I1.i4.p1.1.1" class="ltx_text" style="background-color:#FCE9B3;">yellow shading</span> intensity represents final value of the coverage vector at the end of final model’s summarization process.</p>
</div>
</li>
</ul>
</div>
<figure id="Ax1.F8" class="ltx_figure">
<p id="Ax1.F8.1" class="ltx_p">
<span id="Ax1.F8.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:448.4pt;">
<span id="Ax1.F8.1.1.1" class="ltx_p"><span id="Ax1.F8.1.1.1.1" class="ltx_text ltx_font_bold">Article (truncated):</span> <span id="Ax1.F8.1.1.1.2" class="ltx_text" style="background-color:#FCE7AE;">andy</span> <span id="Ax1.F8.1.1.1.3" class="ltx_text" style="background-color:#FDF5DC;">murray</span> <span id="Ax1.F8.1.1.1.4" class="ltx_text" style="background-color:#FEFBF1;">came</span> <span id="Ax1.F8.1.1.1.5" class="ltx_text" style="background-color:#FFFFFF;">close</span> <span id="Ax1.F8.1.1.1.6" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F8.1.1.1.7" class="ltx_text" style="background-color:#FFFFFF;">giving</span> <span id="Ax1.F8.1.1.1.8" class="ltx_text" style="background-color:#FFFFFF;">himself</span> <span id="Ax1.F8.1.1.1.9" class="ltx_text" style="background-color:#FFFFFF;">some</span> <span id="Ax1.F8.1.1.1.10" class="ltx_text" style="background-color:#FFFFFF;">extra</span> <span id="Ax1.F8.1.1.1.11" class="ltx_text" style="background-color:#FFFFFF;">preparation</span> <span id="Ax1.F8.1.1.1.12" class="ltx_text" style="background-color:#FFFFFF;">time</span> <span id="Ax1.F8.1.1.1.13" class="ltx_text" style="background-color:#FFFFFF;">for</span> <span id="Ax1.F8.1.1.1.14" class="ltx_text" style="background-color:#FFFFFF;">his</span> <span id="Ax1.F8.1.1.1.15" class="ltx_text" style="background-color:#FFFFFF;">wedding</span> <span id="Ax1.F8.1.1.1.16" class="ltx_text" style="background-color:#FFFFFF;">next</span> <span id="Ax1.F8.1.1.1.17" class="ltx_text" style="background-color:#FFFFFF;">week</span> <span id="Ax1.F8.1.1.1.18" class="ltx_text" style="background-color:#FFFFFF;">before</span> <span id="Ax1.F8.1.1.1.19" class="ltx_text" style="background-color:#FFFFFF;">ensuring</span> <span id="Ax1.F8.1.1.1.20" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F8.1.1.1.21" class="ltx_text" style="background-color:#FFFFFF;">he</span> <span id="Ax1.F8.1.1.1.22" class="ltx_text" style="background-color:#FFFFFF;">still</span> <span id="Ax1.F8.1.1.1.23" class="ltx_text" style="background-color:#FFFFFF;">has</span> <span id="Ax1.F8.1.1.1.24" class="ltx_text" style="background-color:#FFFFFF;">unfinished</span> <span id="Ax1.F8.1.1.1.25" class="ltx_text" style="background-color:#FFFFFF;">tennis</span> <span id="Ax1.F8.1.1.1.26" class="ltx_text" style="background-color:#FFFFFF;">business</span> <span id="Ax1.F8.1.1.1.27" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F8.1.1.1.28" class="ltx_text" style="background-color:#FFFFFF;">attend</span> <span id="Ax1.F8.1.1.1.29" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F8.1.1.1.30" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F8.1.1.1.31" class="ltx_text" style="background-color:#FFFEFD;">the</span> <span id="Ax1.F8.1.1.1.32" class="ltx_text" style="background-color:#FFFDF8;">world</span> <span id="Ax1.F8.1.1.1.33" class="ltx_text" style="background-color:#FFFFFF;">no</span> <span id="Ax1.F8.1.1.1.34" class="ltx_text" style="background-color:#FFFFFF;">4</span> <span id="Ax1.F8.1.1.1.35" class="ltx_text" style="background-color:#FFFFFF;">is</span> <span id="Ax1.F8.1.1.1.36" class="ltx_text" style="background-color:#FFFFFF;">into</span> <span id="Ax1.F8.1.1.1.37" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F8.1.1.1.38" class="ltx_text" style="background-color:#FFFEFC;">semi-finals</span> <span id="Ax1.F8.1.1.1.39" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F8.1.1.1.40" class="ltx_text" style="background-color:#FEFBF1;">the</span> <span id="Ax1.F8.1.1.1.41" class="ltx_text" style="background-color:#FFFEFA;">miami</span> <span id="Ax1.F8.1.1.1.42" class="ltx_text" style="background-color:#FFFFFF;">open</span> <span id="Ax1.F8.1.1.1.43" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F8.1.1.1.44" class="ltx_text" style="background-color:#FFFEFC;">but</span> <span id="Ax1.F8.1.1.1.45" class="ltx_text" style="background-color:#FFFFFF;">not</span> <span id="Ax1.F8.1.1.1.46" class="ltx_text" style="background-color:#FFFFFF;">before</span> <span id="Ax1.F8.1.1.1.47" class="ltx_text" style="background-color:#FFFFFF;">getting</span> <span id="Ax1.F8.1.1.1.48" class="ltx_text" style="background-color:#FFFEFD;">a</span> <span id="Ax1.F8.1.1.1.49" class="ltx_text" style="background-color:#FFFFFE;">scare</span> <span id="Ax1.F8.1.1.1.50" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F8.1.1.1.51" class="ltx_text" style="background-color:#FFFDF7;">21</span> <span id="Ax1.F8.1.1.1.52" class="ltx_text" style="background-color:#FFFFFF;">year-old</span> <span id="Ax1.F8.1.1.1.53" class="ltx_text" style="background-color:#FFFCF4;">austrian</span> <span id="Ax1.F8.1.1.1.54" class="ltx_text" style="background-color:#FFFFFF;">dominic</span> <span id="Ax1.F8.1.1.1.55" class="ltx_text ltx_font_italic" style="background-color:#FFFEFB;">thiem</span> <span id="Ax1.F8.1.1.1.56" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F8.1.1.1.57" class="ltx_text" style="background-color:#FFFFFF;">who</span> <span id="Ax1.F8.1.1.1.58" class="ltx_text" style="background-color:#FFFEFD;">pushed</span> <span id="Ax1.F8.1.1.1.59" class="ltx_text" style="background-color:#FFFFFF;">him</span> <span id="Ax1.F8.1.1.1.60" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F8.1.1.1.61" class="ltx_text" style="background-color:#FFFFFF;">4-4</span> <span id="Ax1.F8.1.1.1.62" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F8.1.1.1.63" class="ltx_text" style="background-color:#FFFEFC;">the</span> <span id="Ax1.F8.1.1.1.64" class="ltx_text" style="background-color:#FFFDF7;">second</span> <span id="Ax1.F8.1.1.1.65" class="ltx_text" style="background-color:#FFFEFC;">set</span> <span id="Ax1.F8.1.1.1.66" class="ltx_text" style="background-color:#FFFFFF;">before</span> <span id="Ax1.F8.1.1.1.67" class="ltx_text" style="background-color:#FFFFFF;">going</span> <span id="Ax1.F8.1.1.1.68" class="ltx_text" style="background-color:#FFFFFF;">down</span> <span id="Ax1.F8.1.1.1.69" class="ltx_text" style="background-color:#FDEEC6;">3-6</span> <span id="Ax1.F8.1.1.1.70" class="ltx_text" style="background-color:#FBE6A8;">6-4</span> <span id="Ax1.F8.1.1.1.71" class="ltx_text" style="background-color:#FCE8B1;">,</span> <span id="Ax1.F8.1.1.1.72" class="ltx_text" style="background-color:#FEF8E8;">6-1</span> <span id="Ax1.F8.1.1.1.73" class="ltx_text" style="background-color:#FBE4A3;">in</span> <span id="Ax1.F8.1.1.1.74" class="ltx_text" style="background-color:#FBE6AA;">an</span> <span id="Ax1.F8.1.1.1.75" class="ltx_text" style="background-color:#FBE6A8;">hour</span> <span id="Ax1.F8.1.1.1.76" class="ltx_text" style="background-color:#FBE5A5;">and</span> <span id="Ax1.F8.1.1.1.77" class="ltx_text" style="background-color:#FBE4A3;">three</span> <span id="Ax1.F8.1.1.1.78" class="ltx_text" style="background-color:#FBE4A3;">quarters</span> <span id="Ax1.F8.1.1.1.79" class="ltx_text" style="background-color:#FBE4A3;">.</span> <span id="Ax1.F8.1.1.1.80" class="ltx_text" style="background-color:#FCEEC4;">murray</span> <span id="Ax1.F8.1.1.1.81" class="ltx_text" style="background-color:#FCE8B0;">was</span> <span id="Ax1.F8.1.1.1.82" class="ltx_text" style="background-color:#FBE39F;">awaiting</span> <span id="Ax1.F8.1.1.1.83" class="ltx_text" style="background-color:#FBE5A7;">the</span> <span id="Ax1.F8.1.1.1.84" class="ltx_text" style="background-color:#FBE5A6;">winner</span> <span id="Ax1.F8.1.1.1.85" class="ltx_text" style="background-color:#FBE4A3;">from</span> <span id="Ax1.F8.1.1.1.86" class="ltx_text" style="background-color:#FCE8AF;">the</span> <span id="Ax1.F8.1.1.1.87" class="ltx_text" style="background-color:#FBE4A3;">last</span> <span id="Ax1.F8.1.1.1.88" class="ltx_text" style="background-color:#FBE4A2;">eight</span> <span id="Ax1.F8.1.1.1.89" class="ltx_text" style="background-color:#FBE4A4;">match</span> <span id="Ax1.F8.1.1.1.90" class="ltx_text" style="background-color:#FBE4A2;">between</span> <span id="Ax1.F8.1.1.1.91" class="ltx_text" style="background-color:#FBE199;">tomas</span> <span id="Ax1.F8.1.1.1.92" class="ltx_text" style="background-color:#FBE39E;">berdych</span> <span id="Ax1.F8.1.1.1.93" class="ltx_text" style="background-color:#FBE4A3;">and</span> <span id="Ax1.F8.1.1.1.94" class="ltx_text" style="background-color:#FBE4A3;">argentina</span> <span id="Ax1.F8.1.1.1.95" class="ltx_text" style="background-color:#FBE4A3;">’s</span> <span id="Ax1.F8.1.1.1.96" class="ltx_text" style="background-color:#FBE3A1;">juan</span> <span id="Ax1.F8.1.1.1.97" class="ltx_text" style="background-color:#FBE4A3;">monaco</span> <span id="Ax1.F8.1.1.1.98" class="ltx_text" style="background-color:#FBE4A3;">.</span> <span id="Ax1.F8.1.1.1.99" class="ltx_text" style="background-color:#FDF4D9;">prior</span> <span id="Ax1.F8.1.1.1.100" class="ltx_text" style="background-color:#FBE4A3;">to</span> <span id="Ax1.F8.1.1.1.101" class="ltx_text" style="background-color:#FBE5A5;">this</span> <span id="Ax1.F8.1.1.1.102" class="ltx_text" style="background-color:#FBE4A3;">tournament</span> <span id="Ax1.F8.1.1.1.103" class="ltx_text ltx_font_italic" style="background-color:#FBE4A1;">thiem</span> <span id="Ax1.F8.1.1.1.104" class="ltx_text" style="background-color:#FBE5A5;">lost</span> <span id="Ax1.F8.1.1.1.105" class="ltx_text" style="background-color:#FBE4A4;">in</span> <span id="Ax1.F8.1.1.1.106" class="ltx_text" style="background-color:#FCE9B4;">the</span> <span id="Ax1.F8.1.1.1.107" class="ltx_text" style="background-color:#FCE7AE;">second</span> <span id="Ax1.F8.1.1.1.108" class="ltx_text" style="background-color:#FBE5A5;">round</span> <span id="Ax1.F8.1.1.1.109" class="ltx_text" style="background-color:#FBE5A6;">of</span> <span id="Ax1.F8.1.1.1.110" class="ltx_text" style="background-color:#FBE5A6;">a</span> <span id="Ax1.F8.1.1.1.111" class="ltx_text" style="background-color:#FBE3A1;">challenger</span> <span id="Ax1.F8.1.1.1.112" class="ltx_text" style="background-color:#FBE4A3;">event</span> <span id="Ax1.F8.1.1.1.113" class="ltx_text" style="background-color:#FBE3A1;">to</span> <span id="Ax1.F8.1.1.1.114" class="ltx_text" style="background-color:#FBE4A3;">soon-to-be</span> <span id="Ax1.F8.1.1.1.115" class="ltx_text" style="background-color:#FBE4A1;">new</span> <span id="Ax1.F8.1.1.1.116" class="ltx_text" style="background-color:#FBE4A2;">brit</span> <span id="Ax1.F8.1.1.1.117" class="ltx_text ltx_font_italic" style="background-color:#FBE3A1;">aljaz</span> <span id="Ax1.F8.1.1.1.118" class="ltx_text" style="background-color:#FBE39E;">bedene</span> <span id="Ax1.F8.1.1.1.119" class="ltx_text" style="background-color:#FBE4A3;">.</span> <span id="Ax1.F8.1.1.1.120" class="ltx_text" style="background-color:#FFFEFB;">andy</span> <span id="Ax1.F8.1.1.1.121" class="ltx_text" style="background-color:#FCEDC1;">murray</span> <span id="Ax1.F8.1.1.1.122" class="ltx_text" style="background-color:#FCE9B4;">pumps</span> <span id="Ax1.F8.1.1.1.123" class="ltx_text" style="background-color:#FFFFFF;">his</span> <span id="Ax1.F8.1.1.1.124" class="ltx_text" style="background-color:#FFFFFF;">first</span> <span id="Ax1.F8.1.1.1.125" class="ltx_text" style="background-color:#FFFFFF;">after</span> <span id="Ax1.F8.1.1.1.126" class="ltx_text" style="background-color:#FFFFFF;">defeating</span> <span id="Ax1.F8.1.1.1.127" class="ltx_text" style="background-color:#FCE8B2;">dominic</span> <span id="Ax1.F8.1.1.1.128" class="ltx_text ltx_font_italic" style="background-color:#FBE5A6;">thiem</span> <span id="Ax1.F8.1.1.1.129" class="ltx_text" style="background-color:#FEF7E4;">to</span> <span id="Ax1.F8.1.1.1.130" class="ltx_text" style="background-color:#FFFFFF;">reach</span> <span id="Ax1.F8.1.1.1.131" class="ltx_text" style="background-color:#FFFEFD;">the</span> <span id="Ax1.F8.1.1.1.132" class="ltx_text" style="background-color:#FFFEFB;">miami</span> <span id="Ax1.F8.1.1.1.133" class="ltx_text" style="background-color:#FFFFFF;">open</span> <span id="Ax1.F8.1.1.1.134" class="ltx_text" style="background-color:#FFFEFB;">semi</span> <span id="Ax1.F8.1.1.1.135" class="ltx_text" style="background-color:#FFFFFF;">finals</span> <span id="Ax1.F8.1.1.1.136" class="ltx_text" style="background-color:#FFFFFE;">.</span> <span id="Ax1.F8.1.1.1.137" class="ltx_text ltx_font_italic" style="background-color:#FEF7E5;">muray</span> <span id="Ax1.F8.1.1.1.138" class="ltx_text" style="background-color:#FFFFFF;">throws</span> <span id="Ax1.F8.1.1.1.139" class="ltx_text" style="background-color:#FFFFFF;">his</span> <span id="Ax1.F8.1.1.1.140" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">sweatband</span> <span id="Ax1.F8.1.1.1.141" class="ltx_text" style="background-color:#FFFFFF;">into</span> <span id="Ax1.F8.1.1.1.142" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F8.1.1.1.143" class="ltx_text" style="background-color:#FFFFFF;">crowd</span> <span id="Ax1.F8.1.1.1.144" class="ltx_text" style="background-color:#FFFFFF;">after</span> <span id="Ax1.F8.1.1.1.145" class="ltx_text" style="background-color:#FFFFFF;">completing</span> <span id="Ax1.F8.1.1.1.146" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F8.1.1.1.147" class="ltx_text" style="background-color:#FFFEFB;">3-6</span> <span id="Ax1.F8.1.1.1.148" class="ltx_text" style="background-color:#FFFDF9;">,</span> <span id="Ax1.F8.1.1.1.149" class="ltx_text" style="background-color:#FFFCF6;">6-4</span> <span id="Ax1.F8.1.1.1.150" class="ltx_text" style="background-color:#FEFBF1;">,</span> <span id="Ax1.F8.1.1.1.151" class="ltx_text" style="background-color:#FCECBF;">6-1</span> <span id="Ax1.F8.1.1.1.152" class="ltx_text" style="background-color:#FFFFFF;">victory</span> <span id="Ax1.F8.1.1.1.153" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F8.1.1.1.154" class="ltx_text" style="background-color:#FFFFFE;">florida</span> <span id="Ax1.F8.1.1.1.155" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F8.1.1.1.156" class="ltx_text" style="background-color:#FEF8E8;">murray</span> <span id="Ax1.F8.1.1.1.157" class="ltx_text" style="background-color:#FFFFFF;">shakes</span> <span id="Ax1.F8.1.1.1.158" class="ltx_text" style="background-color:#FFFFFF;">hands</span> <span id="Ax1.F8.1.1.1.159" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F8.1.1.1.160" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">thiem</span> <span id="Ax1.F8.1.1.1.161" class="ltx_text" style="background-color:#FFFFFF;">who</span> <span id="Ax1.F8.1.1.1.162" class="ltx_text" style="background-color:#FFFFFF;">he</span> <span id="Ax1.F8.1.1.1.163" class="ltx_text" style="background-color:#FFFFFF;">described</span> <span id="Ax1.F8.1.1.1.164" class="ltx_text" style="background-color:#FFFFFF;">as</span> <span id="Ax1.F8.1.1.1.165" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F8.1.1.1.166" class="ltx_text" style="background-color:#FFFFFF;">’</span> <span id="Ax1.F8.1.1.1.167" class="ltx_text" style="background-color:#FFFFFF;">strong</span> <span id="Ax1.F8.1.1.1.168" class="ltx_text" style="background-color:#FFFFFF;">guy</span> <span id="Ax1.F8.1.1.1.169" class="ltx_text" style="background-color:#FFFFFF;">’</span> <span id="Ax1.F8.1.1.1.170" class="ltx_text" style="background-color:#FFFFFF;">after</span> <span id="Ax1.F8.1.1.1.171" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F8.1.1.1.172" class="ltx_text" style="background-color:#FFFFFF;">game</span> <span id="Ax1.F8.1.1.1.173" class="ltx_text" style="background-color:#FFFFFE;">.</span> <span id="Ax1.F8.1.1.1.174" class="ltx_text" style="background-color:#FFFFFD;">and</span> <span id="Ax1.F8.1.1.1.175" class="ltx_text" style="background-color:#FEF6E1;">murray</span> <span id="Ax1.F8.1.1.1.176" class="ltx_text" style="background-color:#FFFDFA;">has</span> <span id="Ax1.F8.1.1.1.177" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F8.1.1.1.178" class="ltx_text" style="background-color:#FFFFFF;">fairly</span> <span id="Ax1.F8.1.1.1.179" class="ltx_text" style="background-color:#FFFFFF;">simple</span> <span id="Ax1.F8.1.1.1.180" class="ltx_text" style="background-color:#FFFFFF;">message</span> <span id="Ax1.F8.1.1.1.181" class="ltx_text" style="background-color:#FFFFFF;">for</span> <span id="Ax1.F8.1.1.1.182" class="ltx_text" style="background-color:#FFFFFF;">any</span> <span id="Ax1.F8.1.1.1.183" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F8.1.1.1.184" class="ltx_text" style="background-color:#FFFFFF;">his</span> <span id="Ax1.F8.1.1.1.185" class="ltx_text" style="background-color:#FFFFFF;">fellow</span> <span id="Ax1.F8.1.1.1.186" class="ltx_text" style="background-color:#FFFFFF;">british</span> <span id="Ax1.F8.1.1.1.187" class="ltx_text" style="background-color:#FFFFFF;">tennis</span> <span id="Ax1.F8.1.1.1.188" class="ltx_text" style="background-color:#FFFFFF;">players</span> <span id="Ax1.F8.1.1.1.189" class="ltx_text" style="background-color:#FFFFFF;">who</span> <span id="Ax1.F8.1.1.1.190" class="ltx_text" style="background-color:#FFFFFF;">might</span> <span id="Ax1.F8.1.1.1.191" class="ltx_text" style="background-color:#FFFFFF;">be</span> <span id="Ax1.F8.1.1.1.192" class="ltx_text" style="background-color:#FFFFFF;">agitated</span> <span id="Ax1.F8.1.1.1.193" class="ltx_text" style="background-color:#FFFFFF;">about</span> <span id="Ax1.F8.1.1.1.194" class="ltx_text" style="background-color:#FFFFFF;">his</span> <span id="Ax1.F8.1.1.1.195" class="ltx_text" style="background-color:#FFFFFF;">imminent</span> <span id="Ax1.F8.1.1.1.196" class="ltx_text" style="background-color:#FFFFFF;">arrival</span> <span id="Ax1.F8.1.1.1.197" class="ltx_text" style="background-color:#FFFFFF;">into</span> <span id="Ax1.F8.1.1.1.198" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F8.1.1.1.199" class="ltx_text" style="background-color:#FFFFFF;">home</span> <span id="Ax1.F8.1.1.1.200" class="ltx_text" style="background-color:#FFFFFF;">ranks</span> <span id="Ax1.F8.1.1.1.201" class="ltx_text" style="background-color:#FFFFFF;">:</span> <span id="Ax1.F8.1.1.1.202" class="ltx_text" style="background-color:#FFFFFF;">do</span> <span id="Ax1.F8.1.1.1.203" class="ltx_text" style="background-color:#FFFFFF;">n’t</span> <span id="Ax1.F8.1.1.1.204" class="ltx_text" style="background-color:#FFFFFF;">complain</span> <span id="Ax1.F8.1.1.1.205" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F8.1.1.1.206" class="ltx_text" style="background-color:#FEF6E1;">instead</span> <span id="Ax1.F8.1.1.1.207" class="ltx_text" style="background-color:#FFFFFE;">the</span> <span id="Ax1.F8.1.1.1.208" class="ltx_text" style="background-color:#FFFEFD;">british</span> <span id="Ax1.F8.1.1.1.209" class="ltx_text" style="background-color:#FFFFFF;">no</span> <span id="Ax1.F8.1.1.1.210" class="ltx_text" style="background-color:#FFFFFF;">1</span> <span id="Ax1.F8.1.1.1.211" class="ltx_text" style="background-color:#FFFFFF;">believes</span> <span id="Ax1.F8.1.1.1.212" class="ltx_text" style="background-color:#FFFFFF;">his</span> <span id="Ax1.F8.1.1.1.213" class="ltx_text" style="background-color:#FFFFFF;">colleagues</span> <span id="Ax1.F8.1.1.1.214" class="ltx_text" style="background-color:#FFFFFF;">should</span> <span id="Ax1.F8.1.1.1.215" class="ltx_text" style="background-color:#FFFFFF;">use</span> <span id="Ax1.F8.1.1.1.216" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F8.1.1.1.217" class="ltx_text" style="background-color:#FFFFFF;">assimilation</span> <span id="Ax1.F8.1.1.1.218" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F8.1.1.1.219" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F8.1.1.1.220" class="ltx_text" style="background-color:#FFFFFF;">world</span> <span id="Ax1.F8.1.1.1.221" class="ltx_text" style="background-color:#FFFFFF;">number</span> <span id="Ax1.F8.1.1.1.222" class="ltx_text" style="background-color:#FFFFFF;">83</span> <span id="Ax1.F8.1.1.1.223" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F8.1.1.1.224" class="ltx_text" style="background-color:#FFFFFF;">originally</span> <span id="Ax1.F8.1.1.1.225" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F8.1.1.1.226" class="ltx_text" style="background-color:#FFFFFD;">slovenia</span> <span id="Ax1.F8.1.1.1.227" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F8.1.1.1.228" class="ltx_text" style="background-color:#FFFFFF;">as</span> <span id="Ax1.F8.1.1.1.229" class="ltx_text" style="background-color:#FFFFFF;">motivation</span> <span id="Ax1.F8.1.1.1.230" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F8.1.1.1.231" class="ltx_text" style="background-color:#FFFFFF;">better</span> <span id="Ax1.F8.1.1.1.232" class="ltx_text" style="background-color:#FFFFFF;">themselves</span> <span id="Ax1.F8.1.1.1.233" class="ltx_text" style="background-color:#FFFFFF;">.</span>

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F8.1.1.1.234" class="ltx_text ltx_font_bold">Reference Summary:</span> 
<br class="ltx_break">british no 1 defeated dominic <span id="Ax1.F8.1.1.1.235" class="ltx_text ltx_font_italic">thiem</span> in miami open quarter finals .
<br class="ltx_break">andy murray celebrated his 500th career win in the previous round .
<br class="ltx_break">third seed will play the winner of tomas berdych and juan monaco in the semi finals of the atp masters 1000 event in key biscayne .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F8.1.1.1.236" class="ltx_text ltx_font_bold">Baseline:</span> 
<br class="ltx_break">andy murray beat UNK <span id="Ax1.F8.1.1.1.237" class="ltx_text ltx_font_bold" style="color:#DB4437;">bedene 6-3</span> , 6-4 , 6-1 in an hour and three quarters . 
<br class="ltx_break">british no 1 believes his colleagues should use the <span id="Ax1.F8.1.1.1.238" class="ltx_text ltx_font_bold" style="color:#DB4437;">maze</span> of the world number 83 , originally from slovenia , as motivation to better themselves .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F8.1.1.1.239" class="ltx_text ltx_font_bold">Pointer-Generator, No Coverage:</span> 
<br class="ltx_break">andy murray came close to giving himself some extra preparation for his wedding next week before getting a scare from 21 year-old austrian dominic <span id="Ax1.F8.1.1.1.240" class="ltx_text ltx_font_italic">thiem</span> . 
<br class="ltx_break">the world no 4 is into the semi-finals of the miami open , but not before getting a scare from 21 year-old austrian dominic <span id="Ax1.F8.1.1.1.241" class="ltx_text ltx_font_italic">thiem</span> .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F8.1.1.1.242" class="ltx_text ltx_font_bold">Pointer-Generator, With Coverage:</span> 
<br class="ltx_break"><span id="Ax1.F8.1.1.1.243" class="ltx_text" style="background-color:#F8FCFA;">andy</span> <span id="Ax1.F8.1.1.1.244" class="ltx_text" style="background-color:#FCFEFD;">murray</span> <span id="Ax1.F8.1.1.1.245" class="ltx_text" style="background-color:#C2E6D5;">defeated</span> <span id="Ax1.F8.1.1.1.246" class="ltx_text" style="background-color:#F8FCFA;">dominic</span> <span id="Ax1.F8.1.1.1.247" class="ltx_text ltx_font_italic" style="background-color:#FEFFFF;">thiem</span> <span id="Ax1.F8.1.1.1.248" class="ltx_text" style="background-color:#EEF8F3;">3-6</span> <span id="Ax1.F8.1.1.1.249" class="ltx_text" style="background-color:#F9FCFB;">6-4</span> <span id="Ax1.F8.1.1.1.250" class="ltx_text" style="background-color:#EBF7F1;">,</span> <span id="Ax1.F8.1.1.1.251" class="ltx_text" style="background-color:#F9FDFB;">6-1</span> <span id="Ax1.F8.1.1.1.252" class="ltx_text" style="background-color:#C3E6D5;">in</span> <span id="Ax1.F8.1.1.1.253" class="ltx_text" style="background-color:#F6FBF9;">an</span> <span id="Ax1.F8.1.1.1.254" class="ltx_text" style="background-color:#FEFEFE;">hour</span> <span id="Ax1.F8.1.1.1.255" class="ltx_text" style="background-color:#DEF2E8;">and</span> <span id="Ax1.F8.1.1.1.256" class="ltx_text" style="background-color:#F8FCFA;">three</span> <span id="Ax1.F8.1.1.1.257" class="ltx_text" style="background-color:#F9FCFB;">quarters</span> <span id="Ax1.F8.1.1.1.258" class="ltx_text" style="background-color:#AFDFC7;">.</span>
<br class="ltx_break"><span id="Ax1.F8.1.1.1.259" class="ltx_text" style="background-color:#F2FAF6;">murray</span> <span id="Ax1.F8.1.1.1.260" class="ltx_text" style="background-color:#C7E8D8;">was</span> <span id="Ax1.F8.1.1.1.261" class="ltx_text" style="background-color:#DEF2E8;">awaiting</span> <span id="Ax1.F8.1.1.1.262" class="ltx_text" style="background-color:#F7FCF9;">the</span> <span id="Ax1.F8.1.1.1.263" class="ltx_text" style="background-color:#F5FBF8;">winner</span> <span id="Ax1.F8.1.1.1.264" class="ltx_text" style="background-color:#DDF1E7;">from</span> <span id="Ax1.F8.1.1.1.265" class="ltx_text" style="background-color:#FAFDFC;">the</span> <span id="Ax1.F8.1.1.1.266" class="ltx_text" style="background-color:#FAFDFC;">last</span> <span id="Ax1.F8.1.1.1.267" class="ltx_text" style="background-color:#FBFDFC;">eight</span> <span id="Ax1.F8.1.1.1.268" class="ltx_text" style="background-color:#F8FCFA;">match</span> <span id="Ax1.F8.1.1.1.269" class="ltx_text" style="background-color:#EEF8F3;">between</span> <span id="Ax1.F8.1.1.1.270" class="ltx_text" style="background-color:#FCFEFD;">tomas</span> <span id="Ax1.F8.1.1.1.271" class="ltx_text" style="background-color:#FDFEFE;">berdych</span> <span id="Ax1.F8.1.1.1.272" class="ltx_text" style="background-color:#D8EFE4;">and</span> <span id="Ax1.F8.1.1.1.273" class="ltx_text" style="background-color:#FEFFFF;">argentina</span> <span id="Ax1.F8.1.1.1.274" class="ltx_text" style="background-color:#F5FBF8;">’s</span> <span id="Ax1.F8.1.1.1.275" class="ltx_text" style="background-color:#FFFFFF;">juan</span> <span id="Ax1.F8.1.1.1.276" class="ltx_text" style="background-color:#FFFFFF;">monaco</span> <span id="Ax1.F8.1.1.1.277" class="ltx_text" style="background-color:#E8F6EF;">.</span>
<br class="ltx_break"><span id="Ax1.F8.1.1.1.278" class="ltx_text" style="background-color:#D9F0E5;">prior</span> <span id="Ax1.F8.1.1.1.279" class="ltx_text" style="background-color:#E2F3EA;">to</span> <span id="Ax1.F8.1.1.1.280" class="ltx_text" style="background-color:#F5FBF8;">this</span> <span id="Ax1.F8.1.1.1.281" class="ltx_text" style="background-color:#F9FDFB;">tournament</span> <span id="Ax1.F8.1.1.1.282" class="ltx_text" style="background-color:#FCFEFD;">thiem</span> <span id="Ax1.F8.1.1.1.283" class="ltx_text" style="background-color:#FDFEFE;">lost</span> <span id="Ax1.F8.1.1.1.284" class="ltx_text" style="background-color:#FBFEFD;">in</span> <span id="Ax1.F8.1.1.1.285" class="ltx_text" style="background-color:#FEFFFE;">the</span> <span id="Ax1.F8.1.1.1.286" class="ltx_text" style="background-color:#FEFFFE;">second</span> <span id="Ax1.F8.1.1.1.287" class="ltx_text" style="background-color:#FEFFFE;">round</span> <span id="Ax1.F8.1.1.1.288" class="ltx_text" style="background-color:#F8FCFA;">of</span> <span id="Ax1.F8.1.1.1.289" class="ltx_text" style="background-color:#FCFEFD;">a</span> <span id="Ax1.F8.1.1.1.290" class="ltx_text" style="background-color:#FDFEFE;">challenger</span> <span id="Ax1.F8.1.1.1.291" class="ltx_text" style="background-color:#FDFEFE;">event</span> <span id="Ax1.F8.1.1.1.292" class="ltx_text" style="background-color:#F8FCFA;">to</span> <span id="Ax1.F8.1.1.1.293" class="ltx_text" style="background-color:#FBFEFC;">soon-to-be</span> <span id="Ax1.F8.1.1.1.294" class="ltx_text" style="background-color:#FDFEFE;">new</span> <span id="Ax1.F8.1.1.1.295" class="ltx_text" style="background-color:#FDFEFE;">brit</span> <span id="Ax1.F8.1.1.1.296" class="ltx_text ltx_font_italic" style="background-color:#FEFFFE;">aljaz</span> <span id="Ax1.F8.1.1.1.297" class="ltx_text" style="background-color:#FFFFFF;">bedene</span> <span id="Ax1.F8.1.1.1.298" class="ltx_text" style="background-color:#F4FBF8;">.</span></span>
</span></p>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>
The baseline model reports the wrong score <span id="Ax1.F8.9.1" class="ltx_text ltx_font_italic">6-3</span>, substitutes <span id="Ax1.F8.10.2" class="ltx_text ltx_font_italic">bedene</span> for <span id="Ax1.F8.11.3" class="ltx_text ltx_font_italic">thiem</span> and struggles with the uncommon word <span id="Ax1.F8.12.4" class="ltx_text ltx_font_italic">assimilation</span>.
The pointer-network models accurately reproduce the out-of-vocabulary words <span id="Ax1.F8.13.5" class="ltx_text ltx_font_italic">thiem</span> and <span id="Ax1.F8.14.6" class="ltx_text ltx_font_italic">aljaz</span>.
Note that the final model produces the novel word <span id="Ax1.F8.15.7" class="ltx_text ltx_font_italic">defeated</span> to incorporate several fragments into a single sentence.
</figcaption>
</figure>
<figure id="Ax1.F9" class="ltx_figure">
<p id="Ax1.F9.1" class="ltx_p">
<span id="Ax1.F9.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:448.4pt;">
<span id="Ax1.F9.1.1.1" class="ltx_p"><span id="Ax1.F9.1.1.1.1" class="ltx_text ltx_font_bold">Article (truncated):</span> <span id="Ax1.F9.1.1.1.2" class="ltx_text" style="background-color:#FDF3D6;">munster</span> <span id="Ax1.F9.1.1.1.3" class="ltx_text" style="background-color:#FFFFFF;">have</span> <span id="Ax1.F9.1.1.1.4" class="ltx_text" style="background-color:#FFFEFA;">signed</span> <span id="Ax1.F9.1.1.1.5" class="ltx_text" style="background-color:#FEF9EC;">new</span> <span id="Ax1.F9.1.1.1.6" class="ltx_text" style="background-color:#FFFFFE;">zealand</span> <span id="Ax1.F9.1.1.1.7" class="ltx_text" style="background-color:#FFFEFC;">international</span> <span id="Ax1.F9.1.1.1.8" class="ltx_text" style="background-color:#FDF3D7;">francis</span> <span id="Ax1.F9.1.1.1.9" class="ltx_text ltx_font_italic" style="background-color:#FEF6DF;">saili</span> <span id="Ax1.F9.1.1.1.10" class="ltx_text" style="background-color:#FFFCF6;">on</span> <span id="Ax1.F9.1.1.1.11" class="ltx_text" style="background-color:#FFFDF7;">a</span> <span id="Ax1.F9.1.1.1.12" class="ltx_text" style="background-color:#FFFEFA;">two-year</span> <span id="Ax1.F9.1.1.1.13" class="ltx_text" style="background-color:#FFFCF4;">deal</span> <span id="Ax1.F9.1.1.1.14" class="ltx_text" style="background-color:#FFFFFE;">.</span> <span id="Ax1.F9.1.1.1.15" class="ltx_text" style="background-color:#FEF7E5;">utility</span> <span id="Ax1.F9.1.1.1.16" class="ltx_text" style="background-color:#FFFFFF;">back</span> <span id="Ax1.F9.1.1.1.17" class="ltx_text ltx_font_italic" style="background-color:#FFFEFD;">saili</span> <span id="Ax1.F9.1.1.1.18" class="ltx_text" style="background-color:#FFFEFB;">,</span> <span id="Ax1.F9.1.1.1.19" class="ltx_text" style="background-color:#FFFFFF;">who</span> <span id="Ax1.F9.1.1.1.20" class="ltx_text" style="background-color:#FEF9EA;">made</span> <span id="Ax1.F9.1.1.1.21" class="ltx_text" style="background-color:#FFFFFF;">his</span> <span id="Ax1.F9.1.1.1.22" class="ltx_text" style="background-color:#FFFFFF;">all</span> <span id="Ax1.F9.1.1.1.23" class="ltx_text" style="background-color:#FFFFFF;">blacks</span> <span id="Ax1.F9.1.1.1.24" class="ltx_text" style="background-color:#FFFFFF;">debut</span> <span id="Ax1.F9.1.1.1.25" class="ltx_text" style="background-color:#FFFFFF;">against</span> <span id="Ax1.F9.1.1.1.26" class="ltx_text" style="background-color:#FFFFFE;">argentina</span> <span id="Ax1.F9.1.1.1.27" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F9.1.1.1.28" class="ltx_text" style="background-color:#FEF7E4;">2013</span> <span id="Ax1.F9.1.1.1.29" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F9.1.1.1.30" class="ltx_text" style="background-color:#FEF6E1;">will</span> <span id="Ax1.F9.1.1.1.31" class="ltx_text" style="background-color:#FFFEFD;">move</span> <span id="Ax1.F9.1.1.1.32" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F9.1.1.1.33" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F9.1.1.1.34" class="ltx_text" style="background-color:#FFFFFF;">province</span> <span id="Ax1.F9.1.1.1.35" class="ltx_text" style="background-color:#FFFCF4;">later</span> <span id="Ax1.F9.1.1.1.36" class="ltx_text" style="background-color:#FFFDF7;">this</span> <span id="Ax1.F9.1.1.1.37" class="ltx_text" style="background-color:#FFFDF9;">year</span> <span id="Ax1.F9.1.1.1.38" class="ltx_text" style="background-color:#FFFFFF;">after</span> <span id="Ax1.F9.1.1.1.39" class="ltx_text" style="background-color:#FFFFFE;">the</span> <span id="Ax1.F9.1.1.1.40" class="ltx_text" style="background-color:#FFFFFF;">completion</span> <span id="Ax1.F9.1.1.1.41" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F9.1.1.1.42" class="ltx_text" style="background-color:#FFFEFD;">his</span> <span id="Ax1.F9.1.1.1.43" class="ltx_text" style="background-color:#FFFEFD;">2015</span> <span id="Ax1.F9.1.1.1.44" class="ltx_text" style="background-color:#FFFFFF;">contractual</span> <span id="Ax1.F9.1.1.1.45" class="ltx_text" style="background-color:#FFFFFF;">commitments</span> <span id="Ax1.F9.1.1.1.46" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F9.1.1.1.47" class="ltx_text" style="background-color:#FDEFC8;">the</span> <span id="Ax1.F9.1.1.1.48" class="ltx_text" style="background-color:#FBE5A7;">24-year-old</span> <span id="Ax1.F9.1.1.1.49" class="ltx_text" style="background-color:#FEFAED;">currently</span> <span id="Ax1.F9.1.1.1.50" class="ltx_text" style="background-color:#FFFDF9;">plays</span> <span id="Ax1.F9.1.1.1.51" class="ltx_text" style="background-color:#FFFFFF;">for</span> <span id="Ax1.F9.1.1.1.52" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">auckland-based</span> <span id="Ax1.F9.1.1.1.53" class="ltx_text" style="background-color:#FFFFFE;">super</span> <span id="Ax1.F9.1.1.1.54" class="ltx_text" style="background-color:#FFFFFF;">rugby</span> <span id="Ax1.F9.1.1.1.55" class="ltx_text" style="background-color:#FFFFFF;">side</span> <span id="Ax1.F9.1.1.1.56" class="ltx_text" style="background-color:#FFFFFE;">the</span> <span id="Ax1.F9.1.1.1.57" class="ltx_text" style="background-color:#FFFEFD;">blues</span> <span id="Ax1.F9.1.1.1.58" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F9.1.1.1.59" class="ltx_text" style="background-color:#FEFAEF;">was</span> <span id="Ax1.F9.1.1.1.60" class="ltx_text" style="background-color:#FCEAB8;">part</span> <span id="Ax1.F9.1.1.1.61" class="ltx_text" style="background-color:#FCE8B1;">of</span> <span id="Ax1.F9.1.1.1.62" class="ltx_text" style="background-color:#FCE9B5;">the</span> <span id="Ax1.F9.1.1.1.63" class="ltx_text" style="background-color:#FCEAB7;">new</span> <span id="Ax1.F9.1.1.1.64" class="ltx_text" style="background-color:#FBE4A2;">zealand</span> <span id="Ax1.F9.1.1.1.65" class="ltx_text" style="background-color:#FBE4A2;">under-20</span> <span id="Ax1.F9.1.1.1.66" class="ltx_text" style="background-color:#FBE3A0;">side</span> <span id="Ax1.F9.1.1.1.67" class="ltx_text" style="background-color:#FBE4A2;">that</span> <span id="Ax1.F9.1.1.1.68" class="ltx_text" style="background-color:#FBE5A6;">won</span> <span id="Ax1.F9.1.1.1.69" class="ltx_text" style="background-color:#FCECBF;">the</span> <span id="Ax1.F9.1.1.1.70" class="ltx_text" style="background-color:#FBE4A4;">junior</span> <span id="Ax1.F9.1.1.1.71" class="ltx_text" style="background-color:#FBE4A3;">world</span> <span id="Ax1.F9.1.1.1.72" class="ltx_text" style="background-color:#FBE39F;">championship</span> <span id="Ax1.F9.1.1.1.73" class="ltx_text" style="background-color:#FCECBF;">in</span> <span id="Ax1.F9.1.1.1.74" class="ltx_text" style="background-color:#FBE7AD;">italy</span> <span id="Ax1.F9.1.1.1.75" class="ltx_text" style="background-color:#FBE3A0;">in</span> <span id="Ax1.F9.1.1.1.76" class="ltx_text" style="background-color:#FDEFC7;">2011</span> <span id="Ax1.F9.1.1.1.77" class="ltx_text" style="background-color:#FBE5A6;">.</span> <span id="Ax1.F9.1.1.1.78" class="ltx_text ltx_font_italic" style="background-color:#FEF6E0;">saili</span> <span id="Ax1.F9.1.1.1.79" class="ltx_text" style="background-color:#FCECBE;">’s</span> <span id="Ax1.F9.1.1.1.80" class="ltx_text" style="background-color:#FBE4A2;">signature</span> <span id="Ax1.F9.1.1.1.81" class="ltx_text" style="background-color:#FBE3A0;">is</span> <span id="Ax1.F9.1.1.1.82" class="ltx_text" style="background-color:#FBE3A0;">something</span> <span id="Ax1.F9.1.1.1.83" class="ltx_text" style="background-color:#FBE3A0;">of</span> <span id="Ax1.F9.1.1.1.84" class="ltx_text" style="background-color:#FBE3A0;">a</span> <span id="Ax1.F9.1.1.1.85" class="ltx_text" style="background-color:#FBE39F;">coup</span> <span id="Ax1.F9.1.1.1.86" class="ltx_text" style="background-color:#FBE3A0;">for</span> <span id="Ax1.F9.1.1.1.87" class="ltx_text" style="background-color:#FBE3A0;">munster</span> <span id="Ax1.F9.1.1.1.88" class="ltx_text" style="background-color:#FBE39F;">and</span> <span id="Ax1.F9.1.1.1.89" class="ltx_text" style="background-color:#FBE4A3;">head</span> <span id="Ax1.F9.1.1.1.90" class="ltx_text" style="background-color:#FBE39F;">coach</span> <span id="Ax1.F9.1.1.1.91" class="ltx_text" style="background-color:#FBE199;">anthony</span> <span id="Ax1.F9.1.1.1.92" class="ltx_text" style="background-color:#FBE39F;">foley</span> <span id="Ax1.F9.1.1.1.93" class="ltx_text" style="background-color:#FBE5A5;">believes</span> <span id="Ax1.F9.1.1.1.94" class="ltx_text" style="background-color:#FFFFFE;">he</span> <span id="Ax1.F9.1.1.1.95" class="ltx_text" style="background-color:#FFFFFF;">will</span> <span id="Ax1.F9.1.1.1.96" class="ltx_text" style="background-color:#FFFFFF;">be</span> <span id="Ax1.F9.1.1.1.97" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F9.1.1.1.98" class="ltx_text" style="background-color:#FFFFFF;">great</span> <span id="Ax1.F9.1.1.1.99" class="ltx_text" style="background-color:#FFFFFF;">addition</span> <span id="Ax1.F9.1.1.1.100" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F9.1.1.1.101" class="ltx_text" style="background-color:#FFFFFF;">their</span> <span id="Ax1.F9.1.1.1.102" class="ltx_text" style="background-color:#FFFFFF;">backline</span> <span id="Ax1.F9.1.1.1.103" class="ltx_text" style="background-color:#FFFFFE;">.</span> <span id="Ax1.F9.1.1.1.104" class="ltx_text" style="background-color:#FEFAEF;">francis</span> <span id="Ax1.F9.1.1.1.105" class="ltx_text ltx_font_italic" style="background-color:#FDEFC8;">saili</span> <span id="Ax1.F9.1.1.1.106" class="ltx_text" style="background-color:#FCEDC3;">has</span> <span id="Ax1.F9.1.1.1.107" class="ltx_text" style="background-color:#FBE5A5;">signed</span> <span id="Ax1.F9.1.1.1.108" class="ltx_text" style="background-color:#FCE8B2;">a</span> <span id="Ax1.F9.1.1.1.109" class="ltx_text" style="background-color:#FBE39F;">two-year</span> <span id="Ax1.F9.1.1.1.110" class="ltx_text" style="background-color:#FBE7AD;">deal</span> <span id="Ax1.F9.1.1.1.111" class="ltx_text" style="background-color:#FBE3A0;">to</span> <span id="Ax1.F9.1.1.1.112" class="ltx_text" style="background-color:#FBE4A3;">join</span> <span id="Ax1.F9.1.1.1.113" class="ltx_text" style="background-color:#FBE4A2;">munster</span> <span id="Ax1.F9.1.1.1.114" class="ltx_text" style="background-color:#FEF9EB;">and</span> <span id="Ax1.F9.1.1.1.115" class="ltx_text" style="background-color:#FFFEFD;">will</span> <span id="Ax1.F9.1.1.1.116" class="ltx_text" style="background-color:#FFFFFE;">link</span> <span id="Ax1.F9.1.1.1.117" class="ltx_text" style="background-color:#FFFFFF;">up</span> <span id="Ax1.F9.1.1.1.118" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F9.1.1.1.119" class="ltx_text" style="background-color:#FFFFFE;">them</span> <span id="Ax1.F9.1.1.1.120" class="ltx_text" style="background-color:#FCEEC5;">later</span> <span id="Ax1.F9.1.1.1.121" class="ltx_text" style="background-color:#FBE5A6;">this</span> <span id="Ax1.F9.1.1.1.122" class="ltx_text" style="background-color:#FBE5A5;">year</span> <span id="Ax1.F9.1.1.1.123" class="ltx_text" style="background-color:#FBE3A0;">.</span> <span id="Ax1.F9.1.1.1.124" class="ltx_text" style="background-color:#FFFFFE;">’</span> <span id="Ax1.F9.1.1.1.125" class="ltx_text" style="background-color:#FFFFFF;">we</span> <span id="Ax1.F9.1.1.1.126" class="ltx_text" style="background-color:#FFFFFF;">are</span> <span id="Ax1.F9.1.1.1.127" class="ltx_text" style="background-color:#FFFFFF;">really</span> <span id="Ax1.F9.1.1.1.128" class="ltx_text" style="background-color:#FFFFFF;">pleased</span> <span id="Ax1.F9.1.1.1.129" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F9.1.1.1.130" class="ltx_text" style="background-color:#FFFEFB;">francis</span> <span id="Ax1.F9.1.1.1.131" class="ltx_text" style="background-color:#FFFFFE;">has</span> <span id="Ax1.F9.1.1.1.132" class="ltx_text" style="background-color:#FFFFFF;">committed</span> <span id="Ax1.F9.1.1.1.133" class="ltx_text" style="background-color:#FFFFFF;">his</span> <span id="Ax1.F9.1.1.1.134" class="ltx_text" style="background-color:#FFFFFF;">future</span> <span id="Ax1.F9.1.1.1.135" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F9.1.1.1.136" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F9.1.1.1.137" class="ltx_text" style="background-color:#FFFFFF;">province</span> <span id="Ax1.F9.1.1.1.138" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F9.1.1.1.139" class="ltx_text" style="background-color:#FFFFFF;">’</span> <span id="Ax1.F9.1.1.1.140" class="ltx_text" style="background-color:#FEF6E1;">foley</span> <span id="Ax1.F9.1.1.1.141" class="ltx_text" style="background-color:#FFFFFF;">told</span> <span id="Ax1.F9.1.1.1.142" class="ltx_text" style="background-color:#FFFFFF;">munster</span> <span id="Ax1.F9.1.1.1.143" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F9.1.1.1.144" class="ltx_text" style="background-color:#FFFFFF;">official</span> <span id="Ax1.F9.1.1.1.145" class="ltx_text" style="background-color:#FFFFFF;">website</span> <span id="Ax1.F9.1.1.1.146" class="ltx_text" style="background-color:#FFFDF9;">.</span> <span id="Ax1.F9.1.1.1.147" class="ltx_text" style="background-color:#FFFFFE;">’</span> <span id="Ax1.F9.1.1.1.148" class="ltx_text" style="background-color:#FFFDF7;">he</span> <span id="Ax1.F9.1.1.1.149" class="ltx_text" style="background-color:#FFFFFF;">is</span> <span id="Ax1.F9.1.1.1.150" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F9.1.1.1.151" class="ltx_text" style="background-color:#FFFFFF;">talented</span> <span id="Ax1.F9.1.1.1.152" class="ltx_text" style="background-color:#FFFFFF;">centre</span> <span id="Ax1.F9.1.1.1.153" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F9.1.1.1.154" class="ltx_text" style="background-color:#FFFFFF;">an</span> <span id="Ax1.F9.1.1.1.155" class="ltx_text" style="background-color:#FFFFFF;">impressive</span> <span id="Ax1.F9.1.1.1.156" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">skill-set</span> <span id="Ax1.F9.1.1.1.157" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F9.1.1.1.158" class="ltx_text" style="background-color:#FFFFFE;">he</span> <span id="Ax1.F9.1.1.1.159" class="ltx_text" style="background-color:#FFFFFF;">possesses</span> <span id="Ax1.F9.1.1.1.160" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F9.1.1.1.161" class="ltx_text" style="background-color:#FFFFFF;">physical</span> <span id="Ax1.F9.1.1.1.162" class="ltx_text" style="background-color:#FFFFFF;">attributes</span> <span id="Ax1.F9.1.1.1.163" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F9.1.1.1.164" class="ltx_text" style="background-color:#FFFFFF;">excel</span> <span id="Ax1.F9.1.1.1.165" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F9.1.1.1.166" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F9.1.1.1.167" class="ltx_text" style="background-color:#FFFFFF;">northern</span> <span id="Ax1.F9.1.1.1.168" class="ltx_text" style="background-color:#FFFFFF;">hemisphere</span> <span id="Ax1.F9.1.1.1.169" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F9.1.1.1.170" class="ltx_text" style="background-color:#FFFFFE;">’</span> <span id="Ax1.F9.1.1.1.171" class="ltx_text" style="background-color:#FFFFFE;">i</span> <span id="Ax1.F9.1.1.1.172" class="ltx_text" style="background-color:#FFFFFF;">believe</span> <span id="Ax1.F9.1.1.1.173" class="ltx_text" style="background-color:#FFFFFF;">he</span> <span id="Ax1.F9.1.1.1.174" class="ltx_text" style="background-color:#FFFFFF;">will</span> <span id="Ax1.F9.1.1.1.175" class="ltx_text" style="background-color:#FFFFFF;">be</span> <span id="Ax1.F9.1.1.1.176" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F9.1.1.1.177" class="ltx_text" style="background-color:#FFFFFF;">great</span> <span id="Ax1.F9.1.1.1.178" class="ltx_text" style="background-color:#FFFFFF;">addition</span> <span id="Ax1.F9.1.1.1.179" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F9.1.1.1.180" class="ltx_text" style="background-color:#FFFFFF;">our</span> <span id="Ax1.F9.1.1.1.181" class="ltx_text" style="background-color:#FFFFFF;">backline</span> <span id="Ax1.F9.1.1.1.182" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F9.1.1.1.183" class="ltx_text" style="background-color:#FFFFFF;">we</span> <span id="Ax1.F9.1.1.1.184" class="ltx_text" style="background-color:#FFFFFF;">look</span> <span id="Ax1.F9.1.1.1.185" class="ltx_text" style="background-color:#FFFFFE;">forward</span> <span id="Ax1.F9.1.1.1.186" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F9.1.1.1.187" class="ltx_text" style="background-color:#FFFFFF;">welcoming</span> <span id="Ax1.F9.1.1.1.188" class="ltx_text" style="background-color:#FFFFFF;">him</span> <span id="Ax1.F9.1.1.1.189" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F9.1.1.1.190" class="ltx_text" style="background-color:#FFFFFF;">munster</span> <span id="Ax1.F9.1.1.1.191" class="ltx_text" style="background-color:#FFFEFA;">.</span> <span id="Ax1.F9.1.1.1.192" class="ltx_text" style="background-color:#FFFFFE;">’</span> <span id="Ax1.F9.1.1.1.193" class="ltx_text ltx_font_italic" style="background-color:#FCECBF;">saili</span> <span id="Ax1.F9.1.1.1.194" class="ltx_text" style="background-color:#FDF4DB;">has</span> <span id="Ax1.F9.1.1.1.195" class="ltx_text" style="background-color:#FFFEFC;">been</span> <span id="Ax1.F9.1.1.1.196" class="ltx_text" style="background-color:#FFFFFD;">capped</span> <span id="Ax1.F9.1.1.1.197" class="ltx_text" style="background-color:#FFFFFF;">twice</span> <span id="Ax1.F9.1.1.1.198" class="ltx_text" style="background-color:#FFFFFF;">by</span> <span id="Ax1.F9.1.1.1.199" class="ltx_text" style="background-color:#FFFFFF;">new</span> <span id="Ax1.F9.1.1.1.200" class="ltx_text" style="background-color:#FFFFFE;">zealand</span> <span id="Ax1.F9.1.1.1.201" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F9.1.1.1.202" class="ltx_text" style="background-color:#FFFDF9;">was</span> <span id="Ax1.F9.1.1.1.203" class="ltx_text" style="background-color:#FEF9E9;">part</span> <span id="Ax1.F9.1.1.1.204" class="ltx_text" style="background-color:#FEFAED;">of</span> <span id="Ax1.F9.1.1.1.205" class="ltx_text" style="background-color:#FEFAEF;">the</span> <span id="Ax1.F9.1.1.1.206" class="ltx_text" style="background-color:#FEFBF2;">under</span> <span id="Ax1.F9.1.1.1.207" class="ltx_text" style="background-color:#FFFFFF;">20</span> <span id="Ax1.F9.1.1.1.208" class="ltx_text" style="background-color:#FFFFFF;">side</span> <span id="Ax1.F9.1.1.1.209" class="ltx_text" style="background-color:#FFFFFE;">that</span> <span id="Ax1.F9.1.1.1.210" class="ltx_text" style="background-color:#FFFDF8;">won</span> <span id="Ax1.F9.1.1.1.211" class="ltx_text" style="background-color:#FEF8E8;">the</span> <span id="Ax1.F9.1.1.1.212" class="ltx_text" style="background-color:#FEFBF3;">junior</span> <span id="Ax1.F9.1.1.1.213" class="ltx_text" style="background-color:#FEFAEF;">championship</span> <span id="Ax1.F9.1.1.1.214" class="ltx_text" style="background-color:#FEF6DF;">in</span> <span id="Ax1.F9.1.1.1.215" class="ltx_text" style="background-color:#FEF7E5;">2011</span> <span id="Ax1.F9.1.1.1.216" class="ltx_text" style="background-color:#FFFDF9;">.</span>

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F9.1.1.1.217" class="ltx_text ltx_font_bold">Reference Summary:</span> 
<br class="ltx_break">utility back francis <span id="Ax1.F9.1.1.1.218" class="ltx_text ltx_font_italic">saili</span> will join up with munster later this year .
<br class="ltx_break">the new zealand international has signed a two-year contract .
<br class="ltx_break"><span id="Ax1.F9.1.1.1.219" class="ltx_text ltx_font_italic">saili</span> made his debut for the all blacks against argentina in 2013 .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F9.1.1.1.220" class="ltx_text ltx_font_bold">Baseline:</span> 
<br class="ltx_break"><span id="Ax1.F9.1.1.1.221" class="ltx_text ltx_font_bold" style="color:#DB4437;">dutch</span> international francis UNK has signed a two-year deal to join <span id="Ax1.F9.1.1.1.222" class="ltx_text ltx_font_bold" style="color:#DB4437;">irish</span> UNK super rugby side the blues .
<br class="ltx_break">UNK ’s signature is something of a coup for munster and his head coach anthony foley believes he will be a great addition to their <span id="Ax1.F9.1.1.1.223" class="ltx_text ltx_font_bold" style="color:#DB4437;">respective prospects</span> . 
<br class="ltx_break">UNK has been capped twice by new zealand .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F9.1.1.1.224" class="ltx_text ltx_font_bold">Pointer-Generator, No Coverage:</span> 
<br class="ltx_break">new zealand international francis <span id="Ax1.F9.1.1.1.225" class="ltx_text ltx_font_italic">saili</span> will move to the province later this year .
<br class="ltx_break">utility back <span id="Ax1.F9.1.1.1.226" class="ltx_text ltx_font_italic">saili</span> made his all blacks debut against argentina in 2013 .
<br class="ltx_break">utility back <span id="Ax1.F9.1.1.1.227" class="ltx_text ltx_font_italic">saili</span> will move to the province later this year .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F9.1.1.1.228" class="ltx_text ltx_font_bold">Pointer-Generator, With Coverage:</span> 
<br class="ltx_break"><span id="Ax1.F9.1.1.1.229" class="ltx_text" style="background-color:#FAFDFB;">francis</span> <span id="Ax1.F9.1.1.1.230" class="ltx_text ltx_font_italic" style="background-color:#F6FBF9;">saili</span> <span id="Ax1.F9.1.1.1.231" class="ltx_text" style="background-color:#C3E6D5;">has</span> <span id="Ax1.F9.1.1.1.232" class="ltx_text" style="background-color:#DFF2E9;">signed</span> <span id="Ax1.F9.1.1.1.233" class="ltx_text" style="background-color:#F2FAF6;">a</span> <span id="Ax1.F9.1.1.1.234" class="ltx_text" style="background-color:#FCFEFD;">two-year</span> <span id="Ax1.F9.1.1.1.235" class="ltx_text" style="background-color:#F8FCFA;">deal</span> <span id="Ax1.F9.1.1.1.236" class="ltx_text" style="background-color:#DFF2E8;">to</span> <span id="Ax1.F9.1.1.1.237" class="ltx_text" style="background-color:#F2FAF6;">join</span> <span id="Ax1.F9.1.1.1.238" class="ltx_text" style="background-color:#ECF7F2;">munster</span> <span id="Ax1.F9.1.1.1.239" class="ltx_text" style="background-color:#BAE3CF;">later</span> <span id="Ax1.F9.1.1.1.240" class="ltx_text" style="background-color:#C7E8D8;">this</span> <span id="Ax1.F9.1.1.1.241" class="ltx_text" style="background-color:#DEF1E8;">year</span> <span id="Ax1.F9.1.1.1.242" class="ltx_text" style="background-color:#A0D8BD;">.</span>
<br class="ltx_break"><span id="Ax1.F9.1.1.1.243" class="ltx_text" style="background-color:#BAE3CF;">the</span> <span id="Ax1.F9.1.1.1.244" class="ltx_text" style="background-color:#FAFDFC;">24-year-old</span> <span id="Ax1.F9.1.1.1.245" class="ltx_text" style="background-color:#E2F3EA;">was</span> <span id="Ax1.F9.1.1.1.246" class="ltx_text" style="background-color:#E5F4ED;">part</span> <span id="Ax1.F9.1.1.1.247" class="ltx_text" style="background-color:#C2E6D4;">of</span> <span id="Ax1.F9.1.1.1.248" class="ltx_text" style="background-color:#E8F6EF;">the</span> <span id="Ax1.F9.1.1.1.249" class="ltx_text" style="background-color:#F5FBF8;">new</span> <span id="Ax1.F9.1.1.1.250" class="ltx_text" style="background-color:#FDFEFE;">zealand</span> <span id="Ax1.F9.1.1.1.251" class="ltx_text" style="background-color:#DCF1E7;">under-20</span> <span id="Ax1.F9.1.1.1.252" class="ltx_text" style="background-color:#F1F9F5;">side</span> <span id="Ax1.F9.1.1.1.253" class="ltx_text" style="background-color:#DBF0E6;">that</span> <span id="Ax1.F9.1.1.1.254" class="ltx_text" style="background-color:#DEF1E8;">won</span> <span id="Ax1.F9.1.1.1.255" class="ltx_text" style="background-color:#F5FBF8;">the</span> <span id="Ax1.F9.1.1.1.256" class="ltx_text" style="background-color:#F8FCFA;">junior</span> <span id="Ax1.F9.1.1.1.257" class="ltx_text" style="background-color:#FCFEFD;">world</span> <span id="Ax1.F9.1.1.1.258" class="ltx_text" style="background-color:#FCFEFD;">championship</span> <span id="Ax1.F9.1.1.1.259" class="ltx_text" style="background-color:#DEF2E8;">in</span> <span id="Ax1.F9.1.1.1.260" class="ltx_text" style="background-color:#FDFEFE;">italy</span> <span id="Ax1.F9.1.1.1.261" class="ltx_text" style="background-color:#D1ECDF;">in</span> <span id="Ax1.F9.1.1.1.262" class="ltx_text" style="background-color:#FCFEFD;">2011</span> <span id="Ax1.F9.1.1.1.263" class="ltx_text" style="background-color:#B8E2CE;">.</span>
<br class="ltx_break"><span id="Ax1.F9.1.1.1.264" class="ltx_text ltx_font_italic" style="background-color:#E1F3EA;">saili</span> <span id="Ax1.F9.1.1.1.265" class="ltx_text" style="background-color:#DEF2E8;">’s</span> <span id="Ax1.F9.1.1.1.266" class="ltx_text" style="background-color:#F8FCFA;">signature</span> <span id="Ax1.F9.1.1.1.267" class="ltx_text" style="background-color:#F4FBF7;">is</span> <span id="Ax1.F9.1.1.1.268" class="ltx_text" style="background-color:#FBFEFD;">something</span> <span id="Ax1.F9.1.1.1.269" class="ltx_text" style="background-color:#FDFEFD;">of</span> <span id="Ax1.F9.1.1.1.270" class="ltx_text" style="background-color:#FDFEFE;">a</span> <span id="Ax1.F9.1.1.1.271" class="ltx_text" style="background-color:#FEFFFE;">coup</span> <span id="Ax1.F9.1.1.1.272" class="ltx_text" style="background-color:#F7FCF9;">for</span> <span id="Ax1.F9.1.1.1.273" class="ltx_text" style="background-color:#FCFEFD;">munster</span> <span id="Ax1.F9.1.1.1.274" class="ltx_text" style="background-color:#EFF9F4;">and</span> <span id="Ax1.F9.1.1.1.275" class="ltx_text" style="background-color:#FEFEFE;">head</span> <span id="Ax1.F9.1.1.1.276" class="ltx_text" style="background-color:#FEFEFE;">coach</span> <span id="Ax1.F9.1.1.1.277" class="ltx_text" style="background-color:#FAFDFC;">anthony</span> <span id="Ax1.F9.1.1.1.278" class="ltx_text" style="background-color:#FFFFFF;">foley</span> <span id="Ax1.F9.1.1.1.279" class="ltx_text" style="background-color:#DAF0E5;">.</span>
<br class="ltx_break"></span>
</span></p>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>The baseline model incorrectly substitutes <span id="Ax1.F9.9.1" class="ltx_text ltx_font_italic">dutch</span> for <span id="Ax1.F9.10.2" class="ltx_text ltx_font_italic">new zealand</span> (perhaps reflecting the European bias of the dataset), fabricates <span id="Ax1.F9.11.3" class="ltx_text ltx_font_italic">irish</span>, and struggles with out-of-vocabulary words <span id="Ax1.F9.12.4" class="ltx_text ltx_font_italic">saili</span> and <span id="Ax1.F9.13.5" class="ltx_text ltx_font_italic">auckland-based</span>.
Though it is not clear why, the phrase <span id="Ax1.F9.14.6" class="ltx_text ltx_font_italic">addition to our backline</span> is changed to the nonsensical <span id="Ax1.F9.15.7" class="ltx_text ltx_font_italic">addition to their respective prospects</span>.
The pointer-generator model fixes these accuracy problems, and the addition of coverage fixes the repetition problem.
Note that the final model skips over large passages of text to produce shorter sentences.</figcaption>
</figure>
<figure id="Ax1.F10" class="ltx_figure">
<p id="Ax1.F10.1" class="ltx_p">
<span id="Ax1.F10.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:448.4pt;">
<span id="Ax1.F10.1.1.1" class="ltx_p"><span id="Ax1.F10.1.1.1.1" class="ltx_text ltx_font_bold">Article (truncated):</span> <span id="Ax1.F10.1.1.1.2" class="ltx_text" style="background-color:#FFFFFF;">right</span> <span id="Ax1.F10.1.1.1.3" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F10.1.1.1.4" class="ltx_text" style="background-color:#FFFFFE;">the</span> <span id="Ax1.F10.1.1.1.5" class="ltx_text" style="background-color:#FFFFFF;">moment</span> <span id="Ax1.F10.1.1.1.6" class="ltx_text" style="background-color:#FFFFFE;">he</span> <span id="Ax1.F10.1.1.1.7" class="ltx_text" style="background-color:#FFFFFF;">breezed</span> <span id="Ax1.F10.1.1.1.8" class="ltx_text" style="background-color:#FFFFFF;">through</span> <span id="Ax1.F10.1.1.1.9" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F10.1.1.1.10" class="ltx_text" style="background-color:#FFFFFF;">doors</span> <span id="Ax1.F10.1.1.1.11" class="ltx_text" style="background-color:#FFFFFE;">at</span> <span id="Ax1.F10.1.1.1.12" class="ltx_text" style="background-color:#FCEDC3;">old</span> <span id="Ax1.F10.1.1.1.13" class="ltx_text" style="background-color:#FBE5A5;">trafford</span> <span id="Ax1.F10.1.1.1.14" class="ltx_text" style="background-color:#FEF7E3;">,</span> <span id="Ax1.F10.1.1.1.15" class="ltx_text" style="background-color:#FCECBC;">louis</span> <span id="Ax1.F10.1.1.1.16" class="ltx_text" style="background-color:#FEF6E1;">van</span> <span id="Ax1.F10.1.1.1.17" class="ltx_text" style="background-color:#FEFAED;">gaal</span> <span id="Ax1.F10.1.1.1.18" class="ltx_text" style="background-color:#FEF8E7;">was</span> <span id="Ax1.F10.1.1.1.19" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F10.1.1.1.20" class="ltx_text" style="background-color:#FFFFFF;">man</span> <span id="Ax1.F10.1.1.1.21" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F10.1.1.1.22" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F10.1.1.1.23" class="ltx_text" style="background-color:#FFFFFF;">plan</span> <span id="Ax1.F10.1.1.1.24" class="ltx_text" style="background-color:#FFFEFD;">.</span> <span id="Ax1.F10.1.1.1.25" class="ltx_text" style="background-color:#FFFEFC;">the</span> <span id="Ax1.F10.1.1.1.26" class="ltx_text" style="background-color:#FFFFFE;">first</span> <span id="Ax1.F10.1.1.1.27" class="ltx_text" style="background-color:#FFFFFF;">season</span> <span id="Ax1.F10.1.1.1.28" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F10.1.1.1.29" class="ltx_text" style="background-color:#FFFFFE;">he</span> <span id="Ax1.F10.1.1.1.30" class="ltx_text" style="background-color:#FFFFFF;">stated</span> <span id="Ax1.F10.1.1.1.31" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F10.1.1.1.32" class="ltx_text" style="background-color:#FFFFFF;">would</span> <span id="Ax1.F10.1.1.1.33" class="ltx_text" style="background-color:#FFFFFF;">see</span> <span id="Ax1.F10.1.1.1.34" class="ltx_text" style="background-color:#FFFFFF;">him</span> <span id="Ax1.F10.1.1.1.35" class="ltx_text" style="background-color:#FFFFFF;">deliver</span> <span id="Ax1.F10.1.1.1.36" class="ltx_text" style="background-color:#FDF2D1;">manchester</span> <span id="Ax1.F10.1.1.1.37" class="ltx_text" style="background-color:#FCEDC2;">united</span> <span id="Ax1.F10.1.1.1.38" class="ltx_text" style="background-color:#FFFEFC;">back</span> <span id="Ax1.F10.1.1.1.39" class="ltx_text" style="background-color:#FFFFFF;">into</span> <span id="Ax1.F10.1.1.1.40" class="ltx_text" style="background-color:#FFFEFC;">their</span> <span id="Ax1.F10.1.1.1.41" class="ltx_text" style="background-color:#FFFFFF;">rightful</span> <span id="Ax1.F10.1.1.1.42" class="ltx_text" style="background-color:#FFFFFF;">place</span> <span id="Ax1.F10.1.1.1.43" class="ltx_text" style="background-color:#FFFFFE;">in</span> <span id="Ax1.F10.1.1.1.44" class="ltx_text" style="background-color:#FFFEFD;">the</span> <span id="Ax1.F10.1.1.1.45" class="ltx_text" style="background-color:#FFFFFE;">champions</span> <span id="Ax1.F10.1.1.1.46" class="ltx_text" style="background-color:#FFFEFA;">league</span> <span id="Ax1.F10.1.1.1.47" class="ltx_text" style="background-color:#FDF1CE;">.</span> <span id="Ax1.F10.1.1.1.48" class="ltx_text" style="background-color:#FEFBF2;">he</span> <span id="Ax1.F10.1.1.1.49" class="ltx_text" style="background-color:#FFFFFF;">would</span> <span id="Ax1.F10.1.1.1.50" class="ltx_text" style="background-color:#FFFFFF;">restore</span> <span id="Ax1.F10.1.1.1.51" class="ltx_text" style="background-color:#FFFFFF;">them</span> <span id="Ax1.F10.1.1.1.52" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F10.1.1.1.53" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F10.1.1.1.54" class="ltx_text" style="background-color:#FFFFFF;">premier</span> <span id="Ax1.F10.1.1.1.55" class="ltx_text" style="background-color:#FFFFFF;">league</span> <span id="Ax1.F10.1.1.1.56" class="ltx_text" style="background-color:#FFFFFF;">top</span> <span id="Ax1.F10.1.1.1.57" class="ltx_text" style="background-color:#FFFFFF;">four</span> <span id="Ax1.F10.1.1.1.58" class="ltx_text" style="background-color:#FFFFFF;">but</span> <span id="Ax1.F10.1.1.1.59" class="ltx_text ltx_font_italic" style="background-color:#FFFDF8;">loftier</span> <span id="Ax1.F10.1.1.1.60" class="ltx_text" style="background-color:#FFFFFF;">aims</span> <span id="Ax1.F10.1.1.1.61" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F10.1.1.1.62" class="ltx_text" style="background-color:#FFFFFF;">silverware</span> <span id="Ax1.F10.1.1.1.63" class="ltx_text" style="background-color:#FFFFFF;">would</span> <span id="Ax1.F10.1.1.1.64" class="ltx_text" style="background-color:#FFFFFF;">have</span> <span id="Ax1.F10.1.1.1.65" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F10.1.1.1.66" class="ltx_text" style="background-color:#FFFFFE;">wait</span> <span id="Ax1.F10.1.1.1.67" class="ltx_text" style="background-color:#FFFFFE;">.</span> <span id="Ax1.F10.1.1.1.68" class="ltx_text" style="background-color:#FFFDF9;">his</span> <span id="Ax1.F10.1.1.1.69" class="ltx_text" style="background-color:#FFFFFE;">three-year</span> <span id="Ax1.F10.1.1.1.70" class="ltx_text" style="background-color:#FFFFFF;">vision</span> <span id="Ax1.F10.1.1.1.71" class="ltx_text" style="background-color:#FFFFFF;">would</span> <span id="Ax1.F10.1.1.1.72" class="ltx_text" style="background-color:#FFFFFF;">allow</span> <span id="Ax1.F10.1.1.1.73" class="ltx_text" style="background-color:#FFFFFF;">for</span> <span id="Ax1.F10.1.1.1.74" class="ltx_text" style="background-color:#FFFFFF;">such</span> <span id="Ax1.F10.1.1.1.75" class="ltx_text" style="background-color:#FFFFFF;">thoughts</span> <span id="Ax1.F10.1.1.1.76" class="ltx_text" style="background-color:#FFFFFF;">but</span> <span id="Ax1.F10.1.1.1.77" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F10.1.1.1.78" class="ltx_text" style="background-color:#FFFFFF;">first</span> <span id="Ax1.F10.1.1.1.79" class="ltx_text" style="background-color:#FFFFFF;">things</span> <span id="Ax1.F10.1.1.1.80" class="ltx_text" style="background-color:#FFFFFF;">first</span> <span id="Ax1.F10.1.1.1.81" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F10.1.1.1.82" class="ltx_text" style="background-color:#FEF9EB;">united</span> <span id="Ax1.F10.1.1.1.83" class="ltx_text" style="background-color:#FCEBBB;">needed</span> <span id="Ax1.F10.1.1.1.84" class="ltx_text" style="background-color:#FBE5A7;">to</span> <span id="Ax1.F10.1.1.1.85" class="ltx_text" style="background-color:#FBE5A6;">be</span> <span id="Ax1.F10.1.1.1.86" class="ltx_text" style="background-color:#FBE5A5;">dining</span> <span id="Ax1.F10.1.1.1.87" class="ltx_text" style="background-color:#FBE5A5;">from</span> <span id="Ax1.F10.1.1.1.88" class="ltx_text" style="background-color:#FBE5A6;">european</span> <span id="Ax1.F10.1.1.1.89" class="ltx_text" style="background-color:#FBE4A4;">football</span> <span id="Ax1.F10.1.1.1.90" class="ltx_text" style="background-color:#FBE4A5;">’s</span> <span id="Ax1.F10.1.1.1.91" class="ltx_text" style="background-color:#FBE6AA;">top</span> <span id="Ax1.F10.1.1.1.92" class="ltx_text" style="background-color:#FBE4A2;">table</span> <span id="Ax1.F10.1.1.1.93" class="ltx_text" style="background-color:#FBE5A5;">again</span> <span id="Ax1.F10.1.1.1.94" class="ltx_text" style="background-color:#FBE4A4;">.</span> <span id="Ax1.F10.1.1.1.95" class="ltx_text" style="background-color:#FCEAB8;">louis</span> <span id="Ax1.F10.1.1.1.96" class="ltx_text" style="background-color:#FCE9B4;">van</span> <span id="Ax1.F10.1.1.1.97" class="ltx_text" style="background-color:#FCEAB7;">gaal</span> <span id="Ax1.F10.1.1.1.98" class="ltx_text" style="background-color:#FCECBF;">is</span> <span id="Ax1.F10.1.1.1.99" class="ltx_text" style="background-color:#FBE5A5;">close</span> <span id="Ax1.F10.1.1.1.100" class="ltx_text" style="background-color:#FBE5A5;">to</span> <span id="Ax1.F10.1.1.1.101" class="ltx_text" style="background-color:#FBE4A4;">delivering</span> <span id="Ax1.F10.1.1.1.102" class="ltx_text" style="background-color:#FBE6AB;">his</span> <span id="Ax1.F10.1.1.1.103" class="ltx_text ltx_font_italic" style="background-color:#FDF0CB;">first-season</span> <span id="Ax1.F10.1.1.1.104" class="ltx_text" style="background-color:#FBE29D;">aim</span> <span id="Ax1.F10.1.1.1.105" class="ltx_text" style="background-color:#FBE5A5;">of</span> <span id="Ax1.F10.1.1.1.106" class="ltx_text" style="background-color:#FBE4A4;">returning</span> <span id="Ax1.F10.1.1.1.107" class="ltx_text" style="background-color:#FBE5A6;">man</span> <span id="Ax1.F10.1.1.1.108" class="ltx_text" style="background-color:#FBE3A0;">united</span> <span id="Ax1.F10.1.1.1.109" class="ltx_text" style="background-color:#FBE5A7;">into</span> <span id="Ax1.F10.1.1.1.110" class="ltx_text" style="background-color:#FBE6AA;">champions</span> <span id="Ax1.F10.1.1.1.111" class="ltx_text" style="background-color:#FBE6AA;">league</span> <span id="Ax1.F10.1.1.1.112" class="ltx_text" style="background-color:#FDF3D5;">.</span> <span id="Ax1.F10.1.1.1.113" class="ltx_text" style="background-color:#FDF3D5;">wayne</span> <span id="Ax1.F10.1.1.1.114" class="ltx_text" style="background-color:#FFFFFE;">rooney</span> <span id="Ax1.F10.1.1.1.115" class="ltx_text" style="background-color:#FFFFFF;">smashes</span> <span id="Ax1.F10.1.1.1.116" class="ltx_text" style="background-color:#FFFFFF;">home</span> <span id="Ax1.F10.1.1.1.117" class="ltx_text" style="background-color:#FEFBF2;">during</span> <span id="Ax1.F10.1.1.1.118" class="ltx_text" style="background-color:#FEF9EB;">manchester</span> <span id="Ax1.F10.1.1.1.119" class="ltx_text" style="background-color:#FEF8E9;">united</span> <span id="Ax1.F10.1.1.1.120" class="ltx_text" style="background-color:#FEFBF2;">’s</span> <span id="Ax1.F10.1.1.1.121" class="ltx_text" style="background-color:#FBE5A5;">3-1</span> <span id="Ax1.F10.1.1.1.122" class="ltx_text" style="background-color:#FFFFFF;">win</span> <span id="Ax1.F10.1.1.1.123" class="ltx_text" style="background-color:#FEF6E0;">over</span> <span id="Ax1.F10.1.1.1.124" class="ltx_text" style="background-color:#FBE199;">aston</span> <span id="Ax1.F10.1.1.1.125" class="ltx_text" style="background-color:#FCEBBC;">villa</span> <span id="Ax1.F10.1.1.1.126" class="ltx_text" style="background-color:#FDF1CF;">on</span> <span id="Ax1.F10.1.1.1.127" class="ltx_text" style="background-color:#FBE5A5;">saturday</span> <span id="Ax1.F10.1.1.1.128" class="ltx_text" style="background-color:#FBE29E;">.</span> <span id="Ax1.F10.1.1.1.129" class="ltx_text" style="background-color:#FDF4D8;">united</span> <span id="Ax1.F10.1.1.1.130" class="ltx_text" style="background-color:#FCE9B5;">’s</span> <span id="Ax1.F10.1.1.1.131" class="ltx_text" style="background-color:#FFFFFE;">win</span> <span id="Ax1.F10.1.1.1.132" class="ltx_text" style="background-color:#FFFFFF;">over</span> <span id="Ax1.F10.1.1.1.133" class="ltx_text" style="background-color:#FFFEFA;">aston</span> <span id="Ax1.F10.1.1.1.134" class="ltx_text" style="background-color:#FEF8E7;">villa</span> <span id="Ax1.F10.1.1.1.135" class="ltx_text" style="background-color:#FFFFFF;">took</span> <span id="Ax1.F10.1.1.1.136" class="ltx_text" style="background-color:#FFFFFF;">them</span> <span id="Ax1.F10.1.1.1.137" class="ltx_text" style="background-color:#FFFFFF;">third</span> <span id="Ax1.F10.1.1.1.138" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F10.1.1.1.139" class="ltx_text" style="background-color:#FFFFFE;">eight</span> <span id="Ax1.F10.1.1.1.140" class="ltx_text" style="background-color:#FFFFFF;">points</span> <span id="Ax1.F10.1.1.1.141" class="ltx_text" style="background-color:#FFFFFF;">ahead</span> <span id="Ax1.F10.1.1.1.142" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F10.1.1.1.143" class="ltx_text" style="background-color:#FFFFFF;">fifth-placed</span> <span id="Ax1.F10.1.1.1.144" class="ltx_text" style="background-color:#FFFFFE;">liverpool</span> <span id="Ax1.F10.1.1.1.145" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F10.1.1.1.146" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F10.1.1.1.147" class="ltx_text" style="background-color:#FFFFFF;">table</span> <span id="Ax1.F10.1.1.1.148" class="ltx_text" style="background-color:#FFFFFE;">.</span> <span id="Ax1.F10.1.1.1.149" class="ltx_text" style="background-color:#FFFEFA;">april</span> <span id="Ax1.F10.1.1.1.150" class="ltx_text" style="background-color:#FFFFFF;">12</span> <span id="Ax1.F10.1.1.1.151" class="ltx_text" style="background-color:#FFFFFF;">manchester</span> <span id="Ax1.F10.1.1.1.152" class="ltx_text" style="background-color:#FFFDF8;">city</span> <span id="Ax1.F10.1.1.1.153" class="ltx_text" style="background-color:#FFFEFA;">(</span> <span id="Ax1.F10.1.1.1.154" class="ltx_text" style="background-color:#FFFFFF;">h</span> <span id="Ax1.F10.1.1.1.155" class="ltx_text" style="background-color:#FFFFFF;">)</span> <span id="Ax1.F10.1.1.1.156" class="ltx_text" style="background-color:#FFFFFE;">.</span> <span id="Ax1.F10.1.1.1.157" class="ltx_text" style="background-color:#FFFEFD;">april</span> <span id="Ax1.F10.1.1.1.158" class="ltx_text" style="background-color:#FFFFFF;">18</span> <span id="Ax1.F10.1.1.1.159" class="ltx_text" style="background-color:#FFFFFF;">chelsea</span> <span id="Ax1.F10.1.1.1.160" class="ltx_text" style="background-color:#FFFFFE;">(</span> <span id="Ax1.F10.1.1.1.161" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F10.1.1.1.162" class="ltx_text" style="background-color:#FFFFFF;">)</span> <span id="Ax1.F10.1.1.1.163" class="ltx_text" style="background-color:#FFFFFE;">.</span> <span id="Ax1.F10.1.1.1.164" class="ltx_text" style="background-color:#FFFEFD;">april</span> <span id="Ax1.F10.1.1.1.165" class="ltx_text" style="background-color:#FFFFFF;">26</span> <span id="Ax1.F10.1.1.1.166" class="ltx_text" style="background-color:#FFFFFE;">everton</span> <span id="Ax1.F10.1.1.1.167" class="ltx_text" style="background-color:#FFFFFF;">(</span> <span id="Ax1.F10.1.1.1.168" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F10.1.1.1.169" class="ltx_text" style="background-color:#FFFFFF;">)</span> <span id="Ax1.F10.1.1.1.170" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F10.1.1.1.171" class="ltx_text" style="background-color:#FFFFFE;">may</span> <span id="Ax1.F10.1.1.1.172" class="ltx_text" style="background-color:#FFFFFF;">2</span> <span id="Ax1.F10.1.1.1.173" class="ltx_text" style="background-color:#FFFFFE;">west</span> <span id="Ax1.F10.1.1.1.174" class="ltx_text" style="background-color:#FFFFFF;">bromwich</span> <span id="Ax1.F10.1.1.1.175" class="ltx_text" style="background-color:#FFFFFF;">albion</span> <span id="Ax1.F10.1.1.1.176" class="ltx_text" style="background-color:#FFFDF7;">(</span> <span id="Ax1.F10.1.1.1.177" class="ltx_text" style="background-color:#FFFFFF;">h</span> <span id="Ax1.F10.1.1.1.178" class="ltx_text" style="background-color:#FFFFFF;">)</span> <span id="Ax1.F10.1.1.1.179" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F10.1.1.1.180" class="ltx_text" style="background-color:#FFFFFE;">may</span> <span id="Ax1.F10.1.1.1.181" class="ltx_text" style="background-color:#FFFFFF;">9</span> <span id="Ax1.F10.1.1.1.182" class="ltx_text" style="background-color:#FFFFFE;">crystal</span> <span id="Ax1.F10.1.1.1.183" class="ltx_text" style="background-color:#FFFFFF;">palace</span> <span id="Ax1.F10.1.1.1.184" class="ltx_text" style="background-color:#FFFFFF;">(</span> <span id="Ax1.F10.1.1.1.185" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F10.1.1.1.186" class="ltx_text" style="background-color:#FFFFFF;">)</span> <span id="Ax1.F10.1.1.1.187" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F10.1.1.1.188" class="ltx_text" style="background-color:#FFFFFE;">may</span> <span id="Ax1.F10.1.1.1.189" class="ltx_text" style="background-color:#FFFFFF;">17</span> <span id="Ax1.F10.1.1.1.190" class="ltx_text" style="background-color:#FFFFFE;">arsenal</span> <span id="Ax1.F10.1.1.1.191" class="ltx_text" style="background-color:#FEF7E3;">(</span> <span id="Ax1.F10.1.1.1.192" class="ltx_text" style="background-color:#FFFFFF;">h</span> <span id="Ax1.F10.1.1.1.193" class="ltx_text" style="background-color:#FFFFFF;">)</span> <span id="Ax1.F10.1.1.1.194" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F10.1.1.1.195" class="ltx_text" style="background-color:#FFFFFE;">may</span> <span id="Ax1.F10.1.1.1.196" class="ltx_text" style="background-color:#FFFFFF;">24</span> <span id="Ax1.F10.1.1.1.197" class="ltx_text" style="background-color:#FFFFFE;">hull</span> <span id="Ax1.F10.1.1.1.198" class="ltx_text" style="background-color:#FFFFFF;">city</span> <span id="Ax1.F10.1.1.1.199" class="ltx_text" style="background-color:#FFFFFF;">(</span> <span id="Ax1.F10.1.1.1.200" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F10.1.1.1.201" class="ltx_text" style="background-color:#FFFFFF;">)</span> <span id="Ax1.F10.1.1.1.202" class="ltx_text" style="background-color:#FFFFFE;">.</span> <span id="Ax1.F10.1.1.1.203" class="ltx_text" style="background-color:#FFFFFF;">one</span> <span id="Ax1.F10.1.1.1.204" class="ltx_text" style="background-color:#FFFFFF;">season</span> <span id="Ax1.F10.1.1.1.205" class="ltx_text" style="background-color:#FFFFFF;">out</span> <span id="Ax1.F10.1.1.1.206" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F10.1.1.1.207" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F10.1.1.1.208" class="ltx_text" style="background-color:#FFFFFF;">champions</span> <span id="Ax1.F10.1.1.1.209" class="ltx_text" style="background-color:#FFFFFF;">league</span> <span id="Ax1.F10.1.1.1.210" class="ltx_text" style="background-color:#FFFFFF;">was</span> <span id="Ax1.F10.1.1.1.211" class="ltx_text" style="background-color:#FFFFFF;">far</span> <span id="Ax1.F10.1.1.1.212" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F10.1.1.1.213" class="ltx_text" style="background-color:#FFFFFF;">ideal</span> <span id="Ax1.F10.1.1.1.214" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F10.1.1.1.215" class="ltx_text" style="background-color:#FFFFFF;">but</span> <span id="Ax1.F10.1.1.1.216" class="ltx_text" style="background-color:#FFFFFF;">two</span> <span id="Ax1.F10.1.1.1.217" class="ltx_text" style="background-color:#FFFFFF;">seasons</span> <span id="Ax1.F10.1.1.1.218" class="ltx_text" style="background-color:#FFFFFF;">would</span> <span id="Ax1.F10.1.1.1.219" class="ltx_text" style="background-color:#FFFFFF;">be</span> <span id="Ax1.F10.1.1.1.220" class="ltx_text" style="background-color:#FFFFFF;">an</span> <span id="Ax1.F10.1.1.1.221" class="ltx_text" style="background-color:#FFFFFF;">absolute</span> <span id="Ax1.F10.1.1.1.222" class="ltx_text" style="background-color:#FFFFFF;">disaster</span> <span id="Ax1.F10.1.1.1.223" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F10.1.1.1.224" class="ltx_text" style="background-color:#FFFFFF;">something</span> <span id="Ax1.F10.1.1.1.225" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F10.1.1.1.226" class="ltx_text" style="background-color:#FFFFFF;">he</span> <span id="Ax1.F10.1.1.1.227" class="ltx_text" style="background-color:#FFFFFF;">understood</span> <span id="Ax1.F10.1.1.1.228" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F10.1.1.1.229" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F10.1.1.1.230" class="ltx_text" style="background-color:#FFFFFF;">would</span> <span id="Ax1.F10.1.1.1.231" class="ltx_text" style="background-color:#FFFFFF;">not</span> <span id="Ax1.F10.1.1.1.232" class="ltx_text" style="background-color:#FFFFFF;">be</span> <span id="Ax1.F10.1.1.1.233" class="ltx_text" style="background-color:#FFFFFF;">tolerated</span> <span id="Ax1.F10.1.1.1.234" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F10.1.1.1.235" class="ltx_text" style="background-color:#FFFFFE;">in</span> <span id="Ax1.F10.1.1.1.236" class="ltx_text" style="background-color:#FFFFFF;">november</span> <span id="Ax1.F10.1.1.1.237" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F10.1.1.1.238" class="ltx_text" style="background-color:#FFFFFF;">even</span> <span id="Ax1.F10.1.1.1.239" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F10.1.1.1.240" class="ltx_text" style="background-color:#FFFFFF;">was</span> <span id="Ax1.F10.1.1.1.241" class="ltx_text" style="background-color:#FFFFFF;">looking</span> <span id="Ax1.F10.1.1.1.242" class="ltx_text" style="background-color:#FFFFFF;">optimistic</span> <span id="Ax1.F10.1.1.1.243" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F10.1.1.1.244" class="ltx_text" style="background-color:#FFFFFE;">a</span> <span id="Ax1.F10.1.1.1.245" class="ltx_text" style="background-color:#FFFFFF;">1-0</span> <span id="Ax1.F10.1.1.1.246" class="ltx_text" style="background-color:#FFFFFF;">defeat</span> <span id="Ax1.F10.1.1.1.247" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F10.1.1.1.248" class="ltx_text" style="background-color:#FFFFFE;">manchester</span> <span id="Ax1.F10.1.1.1.249" class="ltx_text" style="background-color:#FFFEFD;">city</span> <span id="Ax1.F10.1.1.1.250" class="ltx_text" style="background-color:#FFFFFF;">meant</span> <span id="Ax1.F10.1.1.1.251" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F10.1.1.1.252" class="ltx_text" style="background-color:#FFFFFE;">united</span> <span id="Ax1.F10.1.1.1.253" class="ltx_text" style="background-color:#FEF8E7;">had</span> <span id="Ax1.F10.1.1.1.254" class="ltx_text" style="background-color:#FFFFFF;">taken</span> <span id="Ax1.F10.1.1.1.255" class="ltx_text" style="background-color:#FFFFFF;">just</span> <span id="Ax1.F10.1.1.1.256" class="ltx_text" style="background-color:#FFFFFF;">13</span> <span id="Ax1.F10.1.1.1.257" class="ltx_text" style="background-color:#FFFFFF;">points</span> <span id="Ax1.F10.1.1.1.258" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F10.1.1.1.259" class="ltx_text" style="background-color:#FFFFFE;">their</span> <span id="Ax1.F10.1.1.1.260" class="ltx_text" style="background-color:#FFFFFF;">opening</span> <span id="Ax1.F10.1.1.1.261" class="ltx_text" style="background-color:#FFFFFF;">10</span> <span id="Ax1.F10.1.1.1.262" class="ltx_text" style="background-color:#FFFFFF;">matches</span> <span id="Ax1.F10.1.1.1.263" class="ltx_text" style="background-color:#FFFFFF;">-</span> <span id="Ax1.F10.1.1.1.264" class="ltx_text" style="background-color:#FFFFFF;">it</span> <span id="Ax1.F10.1.1.1.265" class="ltx_text" style="background-color:#FFFFFF;">was</span> <span id="Ax1.F10.1.1.1.266" class="ltx_text" style="background-color:#FFFFFF;">their</span> <span id="Ax1.F10.1.1.1.267" class="ltx_text" style="background-color:#FFFFFF;">worst</span> <span id="Ax1.F10.1.1.1.268" class="ltx_text" style="background-color:#FFFFFF;">start</span> <span id="Ax1.F10.1.1.1.269" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F10.1.1.1.270" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F10.1.1.1.271" class="ltx_text" style="background-color:#FFFFFF;">league</span> <span id="Ax1.F10.1.1.1.272" class="ltx_text" style="background-color:#FFFFFF;">campaign</span> <span id="Ax1.F10.1.1.1.273" class="ltx_text" style="background-color:#FFFFFF;">since</span> <span id="Ax1.F10.1.1.1.274" class="ltx_text" style="background-color:#FFFFFF;">1986</span> <span id="Ax1.F10.1.1.1.275" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F10.1.1.1.276" class="ltx_text" style="background-color:#FFFFFF;">when</span> <span id="Ax1.F10.1.1.1.277" class="ltx_text" style="background-color:#FFFFFF;">ron</span> <span id="Ax1.F10.1.1.1.278" class="ltx_text" style="background-color:#FFFFFF;">atkinson</span> <span id="Ax1.F10.1.1.1.279" class="ltx_text" style="background-color:#FFFFFF;">only</span> <span id="Ax1.F10.1.1.1.280" class="ltx_text" style="background-color:#FFFFFF;">lasted</span> <span id="Ax1.F10.1.1.1.281" class="ltx_text" style="background-color:#FFFFFF;">until</span> <span id="Ax1.F10.1.1.1.282" class="ltx_text" style="background-color:#FFFFFF;">november</span> <span id="Ax1.F10.1.1.1.283" class="ltx_text" style="background-color:#FFFFFF;">.</span>

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F10.1.1.1.284" class="ltx_text ltx_font_bold">Reference Summary:</span> 
<br class="ltx_break">man united have an eight-point cushion from <span id="Ax1.F10.1.1.1.285" class="ltx_text ltx_font_italic">fifth-place</span> liverpool .
<br class="ltx_break">van gaal looks likely to deliver on his promise of top four finish .
<br class="ltx_break">but the dutchman has a three-year vision mapped out .
<br class="ltx_break">next season will have to see united mount sustained challenge for title .
<br class="ltx_break">they must also reach the later stages of the champions league .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F10.1.1.1.286" class="ltx_text ltx_font_bold">Baseline:</span> 
<br class="ltx_break">manchester united beat aston villa 3-1 at old trafford on saturday . 
<br class="ltx_break">louis van gaal is close to delivering his UNK aim of returning man united into the premier league top four . 
<br class="ltx_break">louis van gaal is close to delivering his UNK aim of returning man united into champions league .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F10.1.1.1.287" class="ltx_text ltx_font_bold">Pointer-Generator, No Coverage:</span> 
<br class="ltx_break">louis van gaal is close to delivering his <span id="Ax1.F10.1.1.1.288" class="ltx_text ltx_font_italic">first-season</span> aim of returning man united into champions league. 
<br class="ltx_break">united ’s win over aston villa took them third , eight points ahead of fifth-placed liverpool in the table . 
<br class="ltx_break">louis van gaal is close to delivering his <span id="Ax1.F10.1.1.1.289" class="ltx_text ltx_font_italic">first-season</span> aim of returning man united into champions league.

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F10.1.1.1.290" class="ltx_text ltx_font_bold">Pointer-Generator, With Coverage:</span> 
<br class="ltx_break"><span id="Ax1.F10.1.1.1.291" class="ltx_text" style="background-color:#F7FCF9;">manchester</span> <span id="Ax1.F10.1.1.1.292" class="ltx_text" style="background-color:#E4F4EC;">united</span> <span id="Ax1.F10.1.1.1.293" class="ltx_text" style="background-color:#A8DCC2;">beat</span> <span id="Ax1.F10.1.1.1.294" class="ltx_text" style="background-color:#ECF7F2;">aston</span> <span id="Ax1.F10.1.1.1.295" class="ltx_text" style="background-color:#FBFDFC;">villa</span> <span id="Ax1.F10.1.1.1.296" class="ltx_text" style="background-color:#ECF7F2;">3-1</span> <span id="Ax1.F10.1.1.1.297" class="ltx_text" style="background-color:#A2D9BE;">at</span> <span id="Ax1.F10.1.1.1.298" class="ltx_text" style="background-color:#EBF7F1;">old</span> <span id="Ax1.F10.1.1.1.299" class="ltx_text" style="background-color:#FAFDFC;">trafford</span> <span id="Ax1.F10.1.1.1.300" class="ltx_text" style="background-color:#A8DCC2;">on</span> <span id="Ax1.F10.1.1.1.301" class="ltx_text" style="background-color:#E6F5ED;">saturday</span> <span id="Ax1.F10.1.1.1.302" class="ltx_text" style="background-color:#A1D9BD;">.</span>
<br class="ltx_break"><span id="Ax1.F10.1.1.1.303" class="ltx_text" style="background-color:#E8F6EF;">louis</span> <span id="Ax1.F10.1.1.1.304" class="ltx_text" style="background-color:#F3FAF7;">van</span> <span id="Ax1.F10.1.1.1.305" class="ltx_text" style="background-color:#F7FCF9;">gaal</span> <span id="Ax1.F10.1.1.1.306" class="ltx_text" style="background-color:#B0DFC8;">is</span> <span id="Ax1.F10.1.1.1.307" class="ltx_text" style="background-color:#DEF2E8;">close</span> <span id="Ax1.F10.1.1.1.308" class="ltx_text" style="background-color:#B1DFC9;">to</span> <span id="Ax1.F10.1.1.1.309" class="ltx_text" style="background-color:#C7E8D8;">delivering</span> <span id="Ax1.F10.1.1.1.310" class="ltx_text" style="background-color:#E3F4EC;">his</span> <span id="Ax1.F10.1.1.1.311" class="ltx_text ltx_font_italic" style="background-color:#F7FCF9;">first-season</span> <span id="Ax1.F10.1.1.1.312" class="ltx_text" style="background-color:#F6FBF9;">aim</span> <span id="Ax1.F10.1.1.1.313" class="ltx_text" style="background-color:#CDEBDC;">of</span> <span id="Ax1.F10.1.1.1.314" class="ltx_text" style="background-color:#D5EEE2;">returning</span> <span id="Ax1.F10.1.1.1.315" class="ltx_text" style="background-color:#DBF0E6;">man</span> <span id="Ax1.F10.1.1.1.316" class="ltx_text" style="background-color:#EAF7F0;">united</span> <span id="Ax1.F10.1.1.1.317" class="ltx_text" style="background-color:#BEE4D1;">into</span> <span id="Ax1.F10.1.1.1.318" class="ltx_text" style="background-color:#F8FCFA;">champions</span> <span id="Ax1.F10.1.1.1.319" class="ltx_text" style="background-color:#F6FBF8;">league</span> <span id="Ax1.F10.1.1.1.320" class="ltx_text" style="background-color:#AEDEC7;">.</span>
<br class="ltx_break"><span id="Ax1.F10.1.1.1.321" class="ltx_text" style="background-color:#E0F3EA;">united</span> <span id="Ax1.F10.1.1.1.322" class="ltx_text" style="background-color:#ADDEC6;">needed</span> <span id="Ax1.F10.1.1.1.323" class="ltx_text" style="background-color:#D4EDE1;">to</span> <span id="Ax1.F10.1.1.1.324" class="ltx_text" style="background-color:#DFF2E9;">be</span> <span id="Ax1.F10.1.1.1.325" class="ltx_text" style="background-color:#E6F5EE;">dining</span> <span id="Ax1.F10.1.1.1.326" class="ltx_text" style="background-color:#DCF1E7;">from</span> <span id="Ax1.F10.1.1.1.327" class="ltx_text" style="background-color:#E2F3EB;">european</span> <span id="Ax1.F10.1.1.1.328" class="ltx_text" style="background-color:#F4FBF8;">football</span> <span id="Ax1.F10.1.1.1.329" class="ltx_text" style="background-color:#F4FBF8;">’s</span> <span id="Ax1.F10.1.1.1.330" class="ltx_text" style="background-color:#FDFEFD;">top</span> <span id="Ax1.F10.1.1.1.331" class="ltx_text" style="background-color:#FAFDFC;">table</span> <span id="Ax1.F10.1.1.1.332" class="ltx_text" style="background-color:#ECF7F2;">again</span> <span id="Ax1.F10.1.1.1.333" class="ltx_text" style="background-color:#AEDEC6;">.</span>
<br class="ltx_break"></span>
</span></p>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>In this example, both our baseline model and final model produce a completely abstractive first sentence, using a novel word <span id="Ax1.F10.3.1" class="ltx_text ltx_font_italic">beat</span>.</figcaption>
</figure>
<figure id="Ax1.F11" class="ltx_figure">
<p id="Ax1.F11.1" class="ltx_p">
<span id="Ax1.F11.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:448.4pt;">
<span id="Ax1.F11.1.1.1" class="ltx_p"><span id="Ax1.F11.1.1.1.1" class="ltx_text ltx_font_bold">Article (truncated):</span> <span id="Ax1.F11.1.1.1.2" class="ltx_text" style="background-color:#FFFFFF;">having</span> <span id="Ax1.F11.1.1.1.3" class="ltx_text" style="background-color:#FFFFFF;">been</span> <span id="Ax1.F11.1.1.1.4" class="ltx_text" style="background-color:#FFFDF7;">on</span> <span id="Ax1.F11.1.1.1.5" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.6" class="ltx_text" style="background-color:#FFFFFF;">receiving</span> <span id="Ax1.F11.1.1.1.7" class="ltx_text" style="background-color:#FFFFFF;">end</span> <span id="Ax1.F11.1.1.1.8" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F11.1.1.1.9" class="ltx_text" style="background-color:#FFFFFE;">a</span> <span id="Ax1.F11.1.1.1.10" class="ltx_text" style="background-color:#FFFEFB;">6-1</span> <span id="Ax1.F11.1.1.1.11" class="ltx_text" style="background-color:#FFFFFF;">thumping</span> <span id="Ax1.F11.1.1.1.12" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F11.1.1.1.13" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F11.1.1.1.14" class="ltx_text" style="background-color:#FFFFFE;">defeat</span> <span id="Ax1.F11.1.1.1.15" class="ltx_text" style="background-color:#FFFFFF;">like</span> <span id="Ax1.F11.1.1.1.16" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F11.1.1.1.17" class="ltx_text" style="background-color:#FFFFFF;">could</span> <span id="Ax1.F11.1.1.1.18" class="ltx_text" style="background-color:#FFFFFF;">be</span> <span id="Ax1.F11.1.1.1.19" class="ltx_text" style="background-color:#FFFFFF;">justifiably</span> <span id="Ax1.F11.1.1.1.20" class="ltx_text" style="background-color:#FFFFFE;">met</span> <span id="Ax1.F11.1.1.1.21" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F11.1.1.1.22" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F11.1.1.1.23" class="ltx_text" style="background-color:#FFFFFF;">backlash</span> <span id="Ax1.F11.1.1.1.24" class="ltx_text" style="background-color:#FFFFFF;">by</span> <span id="Ax1.F11.1.1.1.25" class="ltx_text" style="background-color:#FFFEFD;">angry</span> <span id="Ax1.F11.1.1.1.26" class="ltx_text" style="background-color:#FFFFFE;">supporters</span> <span id="Ax1.F11.1.1.1.27" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F11.1.1.1.28" class="ltx_text" style="background-color:#FFFCF4;">watching</span> <span id="Ax1.F11.1.1.1.29" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F11.1.1.1.30" class="ltx_text" style="background-color:#FFFFFF;">3-1</span> <span id="Ax1.F11.1.1.1.31" class="ltx_text" style="background-color:#FFFFFF;">first</span> <span id="Ax1.F11.1.1.1.32" class="ltx_text" style="background-color:#FFFFFF;">leg</span> <span id="Ax1.F11.1.1.1.33" class="ltx_text" style="background-color:#FFFFFF;">aggregate</span> <span id="Ax1.F11.1.1.1.34" class="ltx_text" style="background-color:#FFFFFF;">advantage</span> <span id="Ax1.F11.1.1.1.35" class="ltx_text" style="background-color:#FFFFFF;">turn</span> <span id="Ax1.F11.1.1.1.36" class="ltx_text" style="background-color:#FFFFFF;">into</span> <span id="Ax1.F11.1.1.1.37" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F11.1.1.1.38" class="ltx_text" style="background-color:#FFFFFF;">7-4</span> <span id="Ax1.F11.1.1.1.39" class="ltx_text" style="background-color:#FFFFFF;">deficit</span> <span id="Ax1.F11.1.1.1.40" class="ltx_text" style="background-color:#FFFFFF;">come</span> <span id="Ax1.F11.1.1.1.41" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.42" class="ltx_text" style="background-color:#FFFFFF;">end</span> <span id="Ax1.F11.1.1.1.43" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F11.1.1.1.44" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.45" class="ltx_text" style="background-color:#FFFFFF;">reverse</span> <span id="Ax1.F11.1.1.1.46" class="ltx_text" style="background-color:#FFFFFF;">encounter</span> <span id="Ax1.F11.1.1.1.47" class="ltx_text" style="background-color:#FFFFFF;">too</span> <span id="Ax1.F11.1.1.1.48" class="ltx_text" style="background-color:#FFFFFF;">could</span> <span id="Ax1.F11.1.1.1.49" class="ltx_text" style="background-color:#FFFFFF;">send</span> <span id="Ax1.F11.1.1.1.50" class="ltx_text" style="background-color:#FFFFFF;">many</span> <span id="Ax1.F11.1.1.1.51" class="ltx_text" style="background-color:#FFFEFC;">fans</span> <span id="Ax1.F11.1.1.1.52" class="ltx_text ltx_font_italic" style="background-color:#FFFEFA;">apoplectic</span> <span id="Ax1.F11.1.1.1.53" class="ltx_text" style="background-color:#FFFFFF;">at</span> <span id="Ax1.F11.1.1.1.54" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.55" class="ltx_text" style="background-color:#FFFFFF;">capitulation</span> <span id="Ax1.F11.1.1.1.56" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F11.1.1.1.57" class="ltx_text" style="background-color:#FFFEFD;">their</span> <span id="Ax1.F11.1.1.1.58" class="ltx_text" style="background-color:#FFFCF4;">side</span> <span id="Ax1.F11.1.1.1.59" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F11.1.1.1.60" class="ltx_text" style="background-color:#FEF8E8;">however</span> <span id="Ax1.F11.1.1.1.61" class="ltx_text" style="background-color:#FCE8B0;">that</span> <span id="Ax1.F11.1.1.1.62" class="ltx_text" style="background-color:#FCE7AF;">does</span> <span id="Ax1.F11.1.1.1.63" class="ltx_text" style="background-color:#FCE8AF;">n’t</span> <span id="Ax1.F11.1.1.1.64" class="ltx_text" style="background-color:#FCE8AF;">appear</span> <span id="Ax1.F11.1.1.1.65" class="ltx_text" style="background-color:#FCE8B0;">the</span> <span id="Ax1.F11.1.1.1.66" class="ltx_text" style="background-color:#FCE7AF;">case</span> <span id="Ax1.F11.1.1.1.67" class="ltx_text" style="background-color:#FCE8AF;">for</span> <span id="Ax1.F11.1.1.1.68" class="ltx_text" style="background-color:#FCE8AF;">those</span> <span id="Ax1.F11.1.1.1.69" class="ltx_text" style="background-color:#FCE7AF;">devoted</span> <span id="Ax1.F11.1.1.1.70" class="ltx_text" style="background-color:#FCE8AF;">to</span> <span id="Ax1.F11.1.1.1.71" class="ltx_text" style="background-color:#FBE6AB;">porto</span> <span id="Ax1.F11.1.1.1.72" class="ltx_text" style="background-color:#FCE8B0;">.</span> <span id="Ax1.F11.1.1.1.73" class="ltx_text" style="background-color:#FDEFC8;">porto</span> <span id="Ax1.F11.1.1.1.74" class="ltx_text" style="background-color:#FDF4DA;">supporters</span> <span id="Ax1.F11.1.1.1.75" class="ltx_text" style="background-color:#FFFFFF;">gave</span> <span id="Ax1.F11.1.1.1.76" class="ltx_text" style="background-color:#FFFFFF;">their</span> <span id="Ax1.F11.1.1.1.77" class="ltx_text" style="background-color:#FFFFFE;">team</span> <span id="Ax1.F11.1.1.1.78" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F11.1.1.1.79" class="ltx_text" style="background-color:#FFFFFF;">hero</span> <span id="Ax1.F11.1.1.1.80" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F11.1.1.1.81" class="ltx_text" style="background-color:#FFFFFF;">welcome</span> <span id="Ax1.F11.1.1.1.82" class="ltx_text" style="background-color:#FFFEFB;">following</span> <span id="Ax1.F11.1.1.1.83" class="ltx_text" style="background-color:#FFFFFF;">their</span> <span id="Ax1.F11.1.1.1.84" class="ltx_text" style="background-color:#FFFFFF;">6-1</span> <span id="Ax1.F11.1.1.1.85" class="ltx_text" style="background-color:#FFFFFF;">defeat</span> <span id="Ax1.F11.1.1.1.86" class="ltx_text" style="background-color:#FFFFFF;">at</span> <span id="Ax1.F11.1.1.1.87" class="ltx_text" style="background-color:#FFFFFE;">bayern</span> <span id="Ax1.F11.1.1.1.88" class="ltx_text" style="background-color:#FFFFFF;">munich</span> <span id="Ax1.F11.1.1.1.89" class="ltx_text" style="background-color:#FFFFFF;">on</span> <span id="Ax1.F11.1.1.1.90" class="ltx_text" style="background-color:#FFFFFF;">tuesday</span> <span id="Ax1.F11.1.1.1.91" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F11.1.1.1.92" class="ltx_text" style="background-color:#FEF7E3;">porto</span> <span id="Ax1.F11.1.1.1.93" class="ltx_text" style="background-color:#FEF5DD;">star</span> <span id="Ax1.F11.1.1.1.94" class="ltx_text" style="background-color:#FDF0CD;">striker</span> <span id="Ax1.F11.1.1.1.95" class="ltx_text" style="background-color:#FBE199;">jackson</span> <span id="Ax1.F11.1.1.1.96" class="ltx_text" style="background-color:#FCE8AF;">martinez</span> <span id="Ax1.F11.1.1.1.97" class="ltx_text" style="background-color:#FCECBF;">was</span> <span id="Ax1.F11.1.1.1.98" class="ltx_text" style="background-color:#FCECBF;">one</span> <span id="Ax1.F11.1.1.1.99" class="ltx_text" style="background-color:#FCEAB9;">of</span> <span id="Ax1.F11.1.1.1.100" class="ltx_text" style="background-color:#FCECBD;">many</span> <span id="Ax1.F11.1.1.1.101" class="ltx_text" style="background-color:#FCEBBB;">players</span> <span id="Ax1.F11.1.1.1.102" class="ltx_text" style="background-color:#FCE7AE;">to</span> <span id="Ax1.F11.1.1.1.103" class="ltx_text" style="background-color:#FCE8B0;">look</span> <span id="Ax1.F11.1.1.1.104" class="ltx_text" style="background-color:#FCE8B0;">perplexed</span> <span id="Ax1.F11.1.1.1.105" class="ltx_text" style="background-color:#FCE8AF;">by</span> <span id="Ax1.F11.1.1.1.106" class="ltx_text" style="background-color:#FEF7E2;">their</span> <span id="Ax1.F11.1.1.1.107" class="ltx_text" style="background-color:#FCEBBA;">warm</span> <span id="Ax1.F11.1.1.1.108" class="ltx_text" style="background-color:#FBE6A9;">reception</span> <span id="Ax1.F11.1.1.1.109" class="ltx_text" style="background-color:#FCE8B0;">.</span> <span id="Ax1.F11.1.1.1.110" class="ltx_text" style="background-color:#FEF5DD;">porto</span> <span id="Ax1.F11.1.1.1.111" class="ltx_text" style="background-color:#FDF1CF;">boss</span> <span id="Ax1.F11.1.1.1.112" class="ltx_text ltx_font_italic" style="background-color:#FCEAB6;">julen</span> <span id="Ax1.F11.1.1.1.113" class="ltx_text ltx_font_italic" style="background-color:#FCE9B3;">lopetegui</span> <span id="Ax1.F11.1.1.1.114" class="ltx_text" style="background-color:#FFFFFE;">(</span> <span id="Ax1.F11.1.1.1.115" class="ltx_text" style="background-color:#FFFFFF;">left</span> <span id="Ax1.F11.1.1.1.116" class="ltx_text" style="background-color:#FFFFFF;">)</span> <span id="Ax1.F11.1.1.1.117" class="ltx_text" style="background-color:#FDF2D2;">was</span> <span id="Ax1.F11.1.1.1.118" class="ltx_text" style="background-color:#FBE3A1;">hugged</span> <span id="Ax1.F11.1.1.1.119" class="ltx_text" style="background-color:#FCE8B0;">by</span> <span id="Ax1.F11.1.1.1.120" class="ltx_text" style="background-color:#FCECBD;">fans</span> <span id="Ax1.F11.1.1.1.121" class="ltx_text" style="background-color:#FDEFC7;">congratulating</span> <span id="Ax1.F11.1.1.1.122" class="ltx_text" style="background-color:#FCE8B0;">him</span> <span id="Ax1.F11.1.1.1.123" class="ltx_text" style="background-color:#FCE8AF;">on</span> <span id="Ax1.F11.1.1.1.124" class="ltx_text" style="background-color:#FCECBE;">their</span> <span id="Ax1.F11.1.1.1.125" class="ltx_text" style="background-color:#FDF1CE;">champions</span> <span id="Ax1.F11.1.1.1.126" class="ltx_text" style="background-color:#FCE8AF;">league</span> <span id="Ax1.F11.1.1.1.127" class="ltx_text" style="background-color:#FCE8B0;">run</span> <span id="Ax1.F11.1.1.1.128" class="ltx_text" style="background-color:#FCE9B5;">.</span> <span id="Ax1.F11.1.1.1.129" class="ltx_text" style="background-color:#FEF9EA;">police</span> <span id="Ax1.F11.1.1.1.130" class="ltx_text" style="background-color:#FCEBBA;">escorts</span> <span id="Ax1.F11.1.1.1.131" class="ltx_text" style="background-color:#FBE6AA;">were</span> <span id="Ax1.F11.1.1.1.132" class="ltx_text" style="background-color:#FBE6AA;">needed</span> <span id="Ax1.F11.1.1.1.133" class="ltx_text" style="background-color:#FCE8AF;">to</span> <span id="Ax1.F11.1.1.1.134" class="ltx_text" style="background-color:#FCE8AF;">keep</span> <span id="Ax1.F11.1.1.1.135" class="ltx_text" style="background-color:#FCE8B1;">the</span> <span id="Ax1.F11.1.1.1.136" class="ltx_text" style="background-color:#FCEBBB;">delirious</span> <span id="Ax1.F11.1.1.1.137" class="ltx_text" style="background-color:#FBE7AC;">supporters</span> <span id="Ax1.F11.1.1.1.138" class="ltx_text" style="background-color:#FCE8B2;">at</span> <span id="Ax1.F11.1.1.1.139" class="ltx_text" style="background-color:#FCE7AE;">bay</span> <span id="Ax1.F11.1.1.1.140" class="ltx_text" style="background-color:#FCE8AF;">as</span> <span id="Ax1.F11.1.1.1.141" class="ltx_text" style="background-color:#FFFFFE;">the</span> <span id="Ax1.F11.1.1.1.142" class="ltx_text" style="background-color:#FFFCF4;">porto</span> <span id="Ax1.F11.1.1.1.143" class="ltx_text" style="background-color:#FFFFFE;">team</span> <span id="Ax1.F11.1.1.1.144" class="ltx_text" style="background-color:#FFFFFF;">bus</span> <span id="Ax1.F11.1.1.1.145" class="ltx_text" style="background-color:#FFFFFF;">drove</span> <span id="Ax1.F11.1.1.1.146" class="ltx_text" style="background-color:#FFFFFF;">past</span> <span id="Ax1.F11.1.1.1.147" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F11.1.1.1.148" class="ltx_text" style="background-color:#FFFDF7;">the</span> <span id="Ax1.F11.1.1.1.149" class="ltx_text" style="background-color:#FFFCF6;">team</span> <span id="Ax1.F11.1.1.1.150" class="ltx_text" style="background-color:#FFFFFF;">bus</span> <span id="Ax1.F11.1.1.1.151" class="ltx_text" style="background-color:#FFFFFF;">was</span> <span id="Ax1.F11.1.1.1.152" class="ltx_text" style="background-color:#FFFCF5;">met</span> <span id="Ax1.F11.1.1.1.153" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F11.1.1.1.154" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F11.1.1.1.155" class="ltx_text" style="background-color:#FFFFFF;">cacophony</span> <span id="Ax1.F11.1.1.1.156" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F11.1.1.1.157" class="ltx_text" style="background-color:#FFFFFE;">noise</span> <span id="Ax1.F11.1.1.1.158" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F11.1.1.1.159" class="ltx_text" style="background-color:#FEFBF2;">porto</span> <span id="Ax1.F11.1.1.1.160" class="ltx_text" style="background-color:#FFFDF9;">supporters</span> <span id="Ax1.F11.1.1.1.161" class="ltx_text" style="background-color:#FFFEFC;">proudly</span> <span id="Ax1.F11.1.1.1.162" class="ltx_text" style="background-color:#FFFEFD;">chanting</span> <span id="Ax1.F11.1.1.1.163" class="ltx_text" style="background-color:#FFFFFF;">about</span> <span id="Ax1.F11.1.1.1.164" class="ltx_text" style="background-color:#FFFEFD;">their</span> <span id="Ax1.F11.1.1.1.165" class="ltx_text" style="background-color:#FFFCF5;">club</span> <span id="Ax1.F11.1.1.1.166" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F11.1.1.1.167" class="ltx_text" style="background-color:#FEFAED;">on</span> <span id="Ax1.F11.1.1.1.168" class="ltx_text" style="background-color:#FFFFFE;">their</span> <span id="Ax1.F11.1.1.1.169" class="ltx_text" style="background-color:#FFFEFD;">return</span> <span id="Ax1.F11.1.1.1.170" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F11.1.1.1.171" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F11.1.1.1.172" class="ltx_text" style="background-color:#FFFFFF;">humiliating</span> <span id="Ax1.F11.1.1.1.173" class="ltx_text" style="background-color:#FFFFFF;">champions</span> <span id="Ax1.F11.1.1.1.174" class="ltx_text" style="background-color:#FFFFFF;">league</span> <span id="Ax1.F11.1.1.1.175" class="ltx_text" style="background-color:#FFFFFE;">quarter-final</span> <span id="Ax1.F11.1.1.1.176" class="ltx_text" style="background-color:#FFFFFF;">loss</span> <span id="Ax1.F11.1.1.1.177" class="ltx_text" style="background-color:#FFFFFF;">at</span> <span id="Ax1.F11.1.1.1.178" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.179" class="ltx_text" style="background-color:#FFFFFF;">hands</span> <span id="Ax1.F11.1.1.1.180" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F11.1.1.1.181" class="ltx_text" style="background-color:#FFFDF7;">bayern</span> <span id="Ax1.F11.1.1.1.182" class="ltx_text" style="background-color:#FFFFFF;">munich</span> <span id="Ax1.F11.1.1.1.183" class="ltx_text" style="background-color:#FFFFFF;">on</span> <span id="Ax1.F11.1.1.1.184" class="ltx_text" style="background-color:#FFFFFF;">tuesday</span> <span id="Ax1.F11.1.1.1.185" class="ltx_text" style="background-color:#FFFFFF;">night</span> <span id="Ax1.F11.1.1.1.186" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F11.1.1.1.187" class="ltx_text" style="background-color:#FEF8E8;">the</span> <span id="Ax1.F11.1.1.1.188" class="ltx_text" style="background-color:#FFFDF9;">squad</span> <span id="Ax1.F11.1.1.1.189" class="ltx_text" style="background-color:#FFFFFF;">were</span> <span id="Ax1.F11.1.1.1.190" class="ltx_text" style="background-color:#FFFFFE;">given</span> <span id="Ax1.F11.1.1.1.191" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F11.1.1.1.192" class="ltx_text" style="background-color:#FFFFFF;">heroes</span> <span id="Ax1.F11.1.1.1.193" class="ltx_text" style="background-color:#FFFFFF;">reception</span> <span id="Ax1.F11.1.1.1.194" class="ltx_text" style="background-color:#FFFFFF;">as</span> <span id="Ax1.F11.1.1.1.195" class="ltx_text" style="background-color:#FFFFFF;">they</span> <span id="Ax1.F11.1.1.1.196" class="ltx_text" style="background-color:#FFFFFF;">arrived</span> <span id="Ax1.F11.1.1.1.197" class="ltx_text" style="background-color:#FFFFFF;">back</span> <span id="Ax1.F11.1.1.1.198" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F11.1.1.1.199" class="ltx_text" style="background-color:#FFFFFF;">portugal</span> <span id="Ax1.F11.1.1.1.200" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F11.1.1.1.201" class="ltx_text" style="background-color:#FFFEFA;">in</span> <span id="Ax1.F11.1.1.1.202" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.203" class="ltx_text" style="background-color:#FFFFFF;">early</span> <span id="Ax1.F11.1.1.1.204" class="ltx_text" style="background-color:#FFFFFF;">hours</span> <span id="Ax1.F11.1.1.1.205" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F11.1.1.1.206" class="ltx_text" style="background-color:#FFFFFF;">wednesday</span> <span id="Ax1.F11.1.1.1.207" class="ltx_text" style="background-color:#FFFFFF;">morning</span> <span id="Ax1.F11.1.1.1.208" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F11.1.1.1.209" class="ltx_text" style="background-color:#FEFAEF;">fans</span> <span id="Ax1.F11.1.1.1.210" class="ltx_text" style="background-color:#FFFFFD;">mobbed</span> <span id="Ax1.F11.1.1.1.211" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.212" class="ltx_text" style="background-color:#FFFFFF;">squad</span> <span id="Ax1.F11.1.1.1.213" class="ltx_text" style="background-color:#FFFFFF;">congratulating</span> <span id="Ax1.F11.1.1.1.214" class="ltx_text" style="background-color:#FFFFFF;">them</span> <span id="Ax1.F11.1.1.1.215" class="ltx_text" style="background-color:#FFFFFF;">on</span> <span id="Ax1.F11.1.1.1.216" class="ltx_text" style="background-color:#FFFFFE;">their</span> <span id="Ax1.F11.1.1.1.217" class="ltx_text" style="background-color:#FEF7E4;">run</span> <span id="Ax1.F11.1.1.1.218" class="ltx_text" style="background-color:#FFFFFE;">in</span> <span id="Ax1.F11.1.1.1.219" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.220" class="ltx_text" style="background-color:#FFFFFF;">tournament</span> <span id="Ax1.F11.1.1.1.221" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F11.1.1.1.222" class="ltx_text" style="background-color:#FEFBF3;">star</span> <span id="Ax1.F11.1.1.1.223" class="ltx_text" style="background-color:#FFFEFC;">striker</span> <span id="Ax1.F11.1.1.1.224" class="ltx_text" style="background-color:#FFFDF9;">jackson</span> <span id="Ax1.F11.1.1.1.225" class="ltx_text" style="background-color:#FFFEFB;">martinez</span> <span id="Ax1.F11.1.1.1.226" class="ltx_text" style="background-color:#FFFFFD;">and</span> <span id="Ax1.F11.1.1.1.227" class="ltx_text" style="background-color:#FFFFFF;">ricardo</span> <span id="Ax1.F11.1.1.1.228" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">quaresma</span> <span id="Ax1.F11.1.1.1.229" class="ltx_text" style="background-color:#FFFFFF;">were</span> <span id="Ax1.F11.1.1.1.230" class="ltx_text" style="background-color:#FFFDFA;">one</span> <span id="Ax1.F11.1.1.1.231" class="ltx_text" style="background-color:#FFFCF6;">of</span> <span id="Ax1.F11.1.1.1.232" class="ltx_text" style="background-color:#FFFDF8;">many</span> <span id="Ax1.F11.1.1.1.233" class="ltx_text" style="background-color:#FEFAED;">porto</span> <span id="Ax1.F11.1.1.1.234" class="ltx_text" style="background-color:#FFFEFC;">players</span> <span id="Ax1.F11.1.1.1.235" class="ltx_text" style="background-color:#FFFFFE;">who</span> <span id="Ax1.F11.1.1.1.236" class="ltx_text" style="background-color:#FFFFFF;">looked</span> <span id="Ax1.F11.1.1.1.237" class="ltx_text" style="background-color:#FFFFFE;">perplexed</span> <span id="Ax1.F11.1.1.1.238" class="ltx_text" style="background-color:#FFFFFF;">as</span> <span id="Ax1.F11.1.1.1.239" class="ltx_text" style="background-color:#FFFFFF;">they</span> <span id="Ax1.F11.1.1.1.240" class="ltx_text" style="background-color:#FFFFFF;">were</span> <span id="Ax1.F11.1.1.1.241" class="ltx_text" style="background-color:#FFFFFF;">hugged</span> <span id="Ax1.F11.1.1.1.242" class="ltx_text" style="background-color:#FFFFFE;">by</span> <span id="Ax1.F11.1.1.1.243" class="ltx_text" style="background-color:#FEF6E1;">fans</span> <span id="Ax1.F11.1.1.1.244" class="ltx_text" style="background-color:#FEF9EC;">before</span> <span id="Ax1.F11.1.1.1.245" class="ltx_text" style="background-color:#FFFFFF;">they</span> <span id="Ax1.F11.1.1.1.246" class="ltx_text" style="background-color:#FFFFFF;">making</span> <span id="Ax1.F11.1.1.1.247" class="ltx_text" style="background-color:#FFFFFF;">their</span> <span id="Ax1.F11.1.1.1.248" class="ltx_text" style="background-color:#FFFFFF;">way</span> <span id="Ax1.F11.1.1.1.249" class="ltx_text" style="background-color:#FFFFFF;">on</span> <span id="Ax1.F11.1.1.1.250" class="ltx_text" style="background-color:#FFFEFA;">to</span> <span id="Ax1.F11.1.1.1.251" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.252" class="ltx_text" style="background-color:#FFFFFE;">team</span> <span id="Ax1.F11.1.1.1.253" class="ltx_text" style="background-color:#FFFFFF;">bus</span> <span id="Ax1.F11.1.1.1.254" class="ltx_text" style="background-color:#FFFFFF;">-</span> <span id="Ax1.F11.1.1.1.255" class="ltx_text" style="background-color:#FFFFFF;">set</span> <span id="Ax1.F11.1.1.1.256" class="ltx_text" style="background-color:#FFFFFF;">upon</span> <span id="Ax1.F11.1.1.1.257" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F11.1.1.1.258" class="ltx_text" style="background-color:#FFFFFF;">cacophony</span> <span id="Ax1.F11.1.1.1.259" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F11.1.1.1.260" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">fiercely-proud</span> <span id="Ax1.F11.1.1.1.261" class="ltx_text" style="background-color:#FFFFFF;">chanting</span> <span id="Ax1.F11.1.1.1.262" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F11.1.1.1.263" class="ltx_text" style="background-color:#FEF6DF;">it</span> <span id="Ax1.F11.1.1.1.264" class="ltx_text" style="background-color:#FFFFFF;">was</span> <span id="Ax1.F11.1.1.1.265" class="ltx_text" style="background-color:#FFFFFE;">the</span> <span id="Ax1.F11.1.1.1.266" class="ltx_text" style="background-color:#FFFFFF;">first</span> <span id="Ax1.F11.1.1.1.267" class="ltx_text" style="background-color:#FFFFFF;">time</span> <span id="Ax1.F11.1.1.1.268" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F11.1.1.1.269" class="ltx_text" style="background-color:#FFFCF4;">porto</span> <span id="Ax1.F11.1.1.1.270" class="ltx_text" style="background-color:#FFFFFE;">,</span> <span id="Ax1.F11.1.1.1.271" class="ltx_text" style="background-color:#FFFFFF;">who</span> <span id="Ax1.F11.1.1.1.272" class="ltx_text" style="background-color:#FFFFFF;">had</span> <span id="Ax1.F11.1.1.1.273" class="ltx_text" style="background-color:#FFFFFF;">been</span> <span id="Ax1.F11.1.1.1.274" class="ltx_text" style="background-color:#FFFFFF;">unbeaten</span> <span id="Ax1.F11.1.1.1.275" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F11.1.1.1.276" class="ltx_text" style="background-color:#FFFFFF;">this</span> <span id="Ax1.F11.1.1.1.277" class="ltx_text" style="background-color:#FFFFFF;">season</span> <span id="Ax1.F11.1.1.1.278" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F11.1.1.1.279" class="ltx_text" style="background-color:#FFFFFF;">tournament</span> <span id="Ax1.F11.1.1.1.280" class="ltx_text" style="background-color:#FFFFFF;">up</span> <span id="Ax1.F11.1.1.1.281" class="ltx_text" style="background-color:#FFFFFF;">until</span> <span id="Ax1.F11.1.1.1.282" class="ltx_text" style="background-color:#FFFFFF;">tuesday</span> <span id="Ax1.F11.1.1.1.283" class="ltx_text" style="background-color:#FFFFFF;">night</span> <span id="Ax1.F11.1.1.1.284" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F11.1.1.1.285" class="ltx_text" style="background-color:#FFFFFF;">had</span> <span id="Ax1.F11.1.1.1.286" class="ltx_text" style="background-color:#FFFFFF;">reached</span> <span id="Ax1.F11.1.1.1.287" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.288" class="ltx_text" style="background-color:#FFFFFF;">quarter-finals</span> <span id="Ax1.F11.1.1.1.289" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F11.1.1.1.290" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.291" class="ltx_text" style="background-color:#FFFFFF;">champions</span> <span id="Ax1.F11.1.1.1.292" class="ltx_text" style="background-color:#FFFFFF;">league</span> <span id="Ax1.F11.1.1.1.293" class="ltx_text" style="background-color:#FFFFFF;">since</span> <span id="Ax1.F11.1.1.1.294" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F11.1.1.1.295" class="ltx_text" style="background-color:#FFFFFF;">2008-09</span> <span id="Ax1.F11.1.1.1.296" class="ltx_text" style="background-color:#FFFFFF;">season</span> <span id="Ax1.F11.1.1.1.297" class="ltx_text" style="background-color:#FFFFFF;">.</span>

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F11.1.1.1.298" class="ltx_text ltx_font_bold">Reference Summary:</span> 
<br class="ltx_break">bayern munich beat porto 6-1 in their champions league tie on tuesday .
<br class="ltx_break">result saw bayern win quarter-final encounter 7-4 on aggregate .
<br class="ltx_break">it was the first-time porto had reached that stage since the 2008-09 season .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F11.1.1.1.299" class="ltx_text ltx_font_bold">Baseline:</span> 
<br class="ltx_break">porto <span id="Ax1.F11.1.1.1.300" class="ltx_text ltx_font_bold" style="color:#DB4437;">beat</span> bayern munich <span id="Ax1.F11.1.1.1.301" class="ltx_text ltx_font_bold" style="color:#DB4437;">2-0</span> in the champions league on tuesday night . 
<br class="ltx_break">porto star <span id="Ax1.F11.1.1.1.302" class="ltx_text ltx_font_bold" style="color:#DB4437;">james</span> UNK was one of many players involved in the match . 
<br class="ltx_break">the squad were given a <span id="Ax1.F11.1.1.1.303" class="ltx_text ltx_font_bold" style="color:#DB4437;">trophy</span> as they arrived back in portugal .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F11.1.1.1.304" class="ltx_text ltx_font_bold">Pointer-Generator, No Coverage:</span> 
<br class="ltx_break">porto star striker jackson martinez was one of many players to look perplexed by their warm reception. 
<br class="ltx_break">porto star striker jackson martinez was one of many players to look perplexed by their warm reception. 
<br class="ltx_break">porto star striker jackson martinez was one of many players to look perplexed by their warm reception.

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F11.1.1.1.305" class="ltx_text ltx_font_bold">Pointer-Generator, With Coverage:</span> 
<br class="ltx_break"><span id="Ax1.F11.1.1.1.306" class="ltx_text" style="background-color:#F8FCFA;">porto</span> <span id="Ax1.F11.1.1.1.307" class="ltx_text" style="background-color:#D7EFE3;">star</span> <span id="Ax1.F11.1.1.1.308" class="ltx_text" style="background-color:#EFF8F4;">striker</span> <span id="Ax1.F11.1.1.1.309" class="ltx_text" style="background-color:#F9FCFB;">jackson</span> <span id="Ax1.F11.1.1.1.310" class="ltx_text" style="background-color:#FBFDFC;">martinez</span> <span id="Ax1.F11.1.1.1.311" class="ltx_text" style="background-color:#D4EEE1;">was</span> <span id="Ax1.F11.1.1.1.312" class="ltx_text" style="background-color:#DAF0E5;">one</span> <span id="Ax1.F11.1.1.1.313" class="ltx_text" style="background-color:#DDF1E7;">of</span> <span id="Ax1.F11.1.1.1.314" class="ltx_text" style="background-color:#E9F6F0;">many</span> <span id="Ax1.F11.1.1.1.315" class="ltx_text" style="background-color:#E6F5EE;">players</span> <span id="Ax1.F11.1.1.1.316" class="ltx_text" style="background-color:#BDE4D1;">to</span> <span id="Ax1.F11.1.1.1.317" class="ltx_text" style="background-color:#D6EEE3;">look</span> <span id="Ax1.F11.1.1.1.318" class="ltx_text" style="background-color:#E5F4ED;">perplexed</span> <span id="Ax1.F11.1.1.1.319" class="ltx_text" style="background-color:#DBF0E6;">by</span> <span id="Ax1.F11.1.1.1.320" class="ltx_text" style="background-color:#F8FCFA;">their</span> <span id="Ax1.F11.1.1.1.321" class="ltx_text" style="background-color:#F4FAF7;">warm</span> <span id="Ax1.F11.1.1.1.322" class="ltx_text" style="background-color:#F6FBF8;">reception</span> <span id="Ax1.F11.1.1.1.323" class="ltx_text" style="background-color:#B4E1CB;">.</span>
<br class="ltx_break"><span id="Ax1.F11.1.1.1.324" class="ltx_text" style="background-color:#EEF8F3;">porto</span> <span id="Ax1.F11.1.1.1.325" class="ltx_text" style="background-color:#E3F3EB;">boss</span> <span id="Ax1.F11.1.1.1.326" class="ltx_text" style="background-color:#F6FBF9;">julen</span> <span id="Ax1.F11.1.1.1.327" class="ltx_text" style="background-color:#FEFFFE;">lopetegui</span> <span id="Ax1.F11.1.1.1.328" class="ltx_text" style="background-color:#E5F4ED;">was</span> <span id="Ax1.F11.1.1.1.329" class="ltx_text" style="background-color:#E5F4ED;">hugged</span> <span id="Ax1.F11.1.1.1.330" class="ltx_text" style="background-color:#D5EEE2;">by</span> <span id="Ax1.F11.1.1.1.331" class="ltx_text" style="background-color:#F7FCF9;">fans</span> <span id="Ax1.F11.1.1.1.332" class="ltx_text" style="background-color:#E2F3EB;">congratulating</span> <span id="Ax1.F11.1.1.1.333" class="ltx_text" style="background-color:#FCFEFD;">him</span> <span id="Ax1.F11.1.1.1.334" class="ltx_text" style="background-color:#EBF7F1;">on</span> <span id="Ax1.F11.1.1.1.335" class="ltx_text" style="background-color:#F4FBF7;">their</span> <span id="Ax1.F11.1.1.1.336" class="ltx_text" style="background-color:#F9FDFB;">champions</span> <span id="Ax1.F11.1.1.1.337" class="ltx_text" style="background-color:#FAFDFB;">league</span> <span id="Ax1.F11.1.1.1.338" class="ltx_text" style="background-color:#EFF9F4;">run</span> <span id="Ax1.F11.1.1.1.339" class="ltx_text" style="background-color:#BEE5D2;">.</span>
<br class="ltx_break"><span id="Ax1.F11.1.1.1.340" class="ltx_text" style="background-color:#E1F3EA;">however</span> <span id="Ax1.F11.1.1.1.341" class="ltx_text" style="background-color:#DFF2E9;">that</span> <span id="Ax1.F11.1.1.1.342" class="ltx_text" style="background-color:#D3EDE0;">does</span> <span id="Ax1.F11.1.1.1.343" class="ltx_text" style="background-color:#DFF2E9;">n’t</span> <span id="Ax1.F11.1.1.1.344" class="ltx_text" style="background-color:#E8F6EF;">appear</span> <span id="Ax1.F11.1.1.1.345" class="ltx_text" style="background-color:#F7FCF9;">the</span> <span id="Ax1.F11.1.1.1.346" class="ltx_text" style="background-color:#FBFDFC;">case</span> <span id="Ax1.F11.1.1.1.347" class="ltx_text" style="background-color:#F9FDFB;">for</span> <span id="Ax1.F11.1.1.1.348" class="ltx_text" style="background-color:#FBFEFC;">those</span> <span id="Ax1.F11.1.1.1.349" class="ltx_text" style="background-color:#F8FCFA;">devoted</span> <span id="Ax1.F11.1.1.1.350" class="ltx_text" style="background-color:#F8FCFA;">to</span> <span id="Ax1.F11.1.1.1.351" class="ltx_text" style="background-color:#FDFEFE;">porto</span> <span id="Ax1.F11.1.1.1.352" class="ltx_text" style="background-color:#D3EDE1;">.</span>
<br class="ltx_break"><span id="Ax1.F11.1.1.1.353" class="ltx_text" style="background-color:#CDEADC;">police</span> <span id="Ax1.F11.1.1.1.354" class="ltx_text" style="background-color:#EAF7F1;">escorts</span> <span id="Ax1.F11.1.1.1.355" class="ltx_text" style="background-color:#DFF2E9;">were</span> <span id="Ax1.F11.1.1.1.356" class="ltx_text" style="background-color:#F7FCF9;">needed</span> <span id="Ax1.F11.1.1.1.357" class="ltx_text" style="background-color:#DDF1E8;">to</span> <span id="Ax1.F11.1.1.1.358" class="ltx_text" style="background-color:#F5FBF8;">keep</span> <span id="Ax1.F11.1.1.1.359" class="ltx_text" style="background-color:#F5FBF8;">the</span> <span id="Ax1.F11.1.1.1.360" class="ltx_text" style="background-color:#F7FCF9;">delirious</span> <span id="Ax1.F11.1.1.1.361" class="ltx_text" style="background-color:#FAFDFC;">supporters</span> <span id="Ax1.F11.1.1.1.362" class="ltx_text" style="background-color:#F6FBF9;">at</span> <span id="Ax1.F11.1.1.1.363" class="ltx_text" style="background-color:#FEFEFE;">bay</span> <span id="Ax1.F11.1.1.1.364" class="ltx_text" style="background-color:#BDE4D1;">.</span>
<br class="ltx_break"></span>
</span></p>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>The baseline model makes several factual inaccuracies: it claims <span id="Ax1.F11.11.1" class="ltx_text ltx_font_italic">porto</span> beat <span id="Ax1.F11.12.2" class="ltx_text ltx_font_italic">bayern munich</span> not vice versa, the score is changed from <span id="Ax1.F11.13.3" class="ltx_text ltx_font_italic">7-4</span> to <span id="Ax1.F11.14.4" class="ltx_text ltx_font_italic">2-0</span>, <span id="Ax1.F11.15.5" class="ltx_text ltx_font_italic">jackson</span> is changed to <span id="Ax1.F11.16.6" class="ltx_text ltx_font_italic">james</span> and <span id="Ax1.F11.17.7" class="ltx_text ltx_font_italic">a heroes reception</span> is replaced with <span id="Ax1.F11.18.8" class="ltx_text ltx_font_italic">a trophy</span>.
Our final model produces sentences that are individually accurate, but they do not make sense as a whole.
Note that the final model omits the parenthesized phrase <span id="Ax1.F11.19.9" class="ltx_text ltx_font_italic">( left )</span> from its second sentence.</figcaption>
</figure>
<figure id="Ax1.F12" class="ltx_figure">
<p id="Ax1.F12.1" class="ltx_p">
<span id="Ax1.F12.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:448.4pt;">
<span id="Ax1.F12.1.1.1" class="ltx_p"><span id="Ax1.F12.1.1.1.1" class="ltx_text ltx_font_bold">Article:</span> <span id="Ax1.F12.1.1.1.2" class="ltx_text" style="background-color:#FFFFFF;">(</span> <span id="Ax1.F12.1.1.1.3" class="ltx_text" style="background-color:#FFFFFF;">cnn</span> <span id="Ax1.F12.1.1.1.4" class="ltx_text" style="background-color:#FFFFFF;">)</span> <span id="Ax1.F12.1.1.1.5" class="ltx_text" style="background-color:#FFFFFF;">”</span> <span id="Ax1.F12.1.1.1.6" class="ltx_text" style="background-color:#FFFFFE;">it</span> <span id="Ax1.F12.1.1.1.7" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F12.1.1.1.8" class="ltx_text" style="background-color:#FFFFFE;">showtime</span> <span id="Ax1.F12.1.1.1.9" class="ltx_text" style="background-color:#FFFFFF;">!</span> <span id="Ax1.F12.1.1.1.10" class="ltx_text" style="background-color:#FFFFFF;">”</span> <span id="Ax1.F12.1.1.1.11" class="ltx_text" style="background-color:#FEF7E2;">michael</span> <span id="Ax1.F12.1.1.1.12" class="ltx_text" style="background-color:#FCEAB6;">keaton</span> <span id="Ax1.F12.1.1.1.13" class="ltx_text" style="background-color:#FDF3D5;">paid</span> <span id="Ax1.F12.1.1.1.14" class="ltx_text" style="background-color:#FCEAB8;">homage</span> <span id="Ax1.F12.1.1.1.15" class="ltx_text" style="background-color:#FCEBBC;">–</span> <span id="Ax1.F12.1.1.1.16" class="ltx_text" style="background-color:#FFFFFE;">ever</span> <span id="Ax1.F12.1.1.1.17" class="ltx_text" style="background-color:#FFFFFF;">so</span> <span id="Ax1.F12.1.1.1.18" class="ltx_text" style="background-color:#FFFFFF;">slightly</span> <span id="Ax1.F12.1.1.1.19" class="ltx_text" style="background-color:#FFFFFF;">–</span> <span id="Ax1.F12.1.1.1.20" class="ltx_text" style="background-color:#FFFEFC;">to</span> <span id="Ax1.F12.1.1.1.21" class="ltx_text" style="background-color:#FDEFC9;">his</span> <span id="Ax1.F12.1.1.1.22" class="ltx_text" style="background-color:#FDEEC5;">roles</span> <span id="Ax1.F12.1.1.1.23" class="ltx_text" style="background-color:#FCE9B5;">in</span> <span id="Ax1.F12.1.1.1.24" class="ltx_text" style="background-color:#FCECC0;">“</span> <span id="Ax1.F12.1.1.1.25" class="ltx_text ltx_font_italic" style="background-color:#FBE199;">beetlejuice</span> <span id="Ax1.F12.1.1.1.26" class="ltx_text" style="background-color:#FCEAB6;">”</span> <span id="Ax1.F12.1.1.1.27" class="ltx_text" style="background-color:#FCEAB7;">and</span> <span id="Ax1.F12.1.1.1.28" class="ltx_text" style="background-color:#FCE8B1;">“</span> <span id="Ax1.F12.1.1.1.29" class="ltx_text" style="background-color:#FDF4DA;">batman</span> <span id="Ax1.F12.1.1.1.30" class="ltx_text" style="background-color:#FCEAB6;">”</span> <span id="Ax1.F12.1.1.1.31" class="ltx_text" style="background-color:#FCEAB8;">in</span> <span id="Ax1.F12.1.1.1.32" class="ltx_text" style="background-color:#FCEAB7;">his</span> <span id="Ax1.F12.1.1.1.33" class="ltx_text" style="background-color:#FCE9B3;">third</span> <span id="Ax1.F12.1.1.1.34" class="ltx_text" style="background-color:#FCE8B1;">turn</span> <span id="Ax1.F12.1.1.1.35" class="ltx_text" style="background-color:#FCEAB6;">hosting</span> <span id="Ax1.F12.1.1.1.36" class="ltx_text" style="background-color:#FCEDC0;">“</span> <span id="Ax1.F12.1.1.1.37" class="ltx_text" style="background-color:#FBE4A4;">saturday</span> <span id="Ax1.F12.1.1.1.38" class="ltx_text" style="background-color:#FCEAB6;">night</span> <span id="Ax1.F12.1.1.1.39" class="ltx_text" style="background-color:#FCEAB6;">live</span> <span id="Ax1.F12.1.1.1.40" class="ltx_text" style="background-color:#FCEAB6;">”</span> <span id="Ax1.F12.1.1.1.41" class="ltx_text" style="background-color:#FCEDC0;">this</span> <span id="Ax1.F12.1.1.1.42" class="ltx_text" style="background-color:#FFFFFE;">weekend</span> <span id="Ax1.F12.1.1.1.43" class="ltx_text" style="background-color:#FFFCF6;">.</span> <span id="Ax1.F12.1.1.1.44" class="ltx_text" style="background-color:#FDEEC7;">keaton</span> <span id="Ax1.F12.1.1.1.45" class="ltx_text" style="background-color:#FCEAB9;">acknowledged</span> <span id="Ax1.F12.1.1.1.46" class="ltx_text" style="background-color:#FCEAB6;">in</span> <span id="Ax1.F12.1.1.1.47" class="ltx_text" style="background-color:#FCE9B5;">his</span> <span id="Ax1.F12.1.1.1.48" class="ltx_text" style="background-color:#FCE9B5;">opening</span> <span id="Ax1.F12.1.1.1.49" class="ltx_text" style="background-color:#FCE9B5;">monologue</span> <span id="Ax1.F12.1.1.1.50" class="ltx_text" style="background-color:#FCEAB8;">that</span> <span id="Ax1.F12.1.1.1.51" class="ltx_text" style="background-color:#FCEAB8;">a</span> <span id="Ax1.F12.1.1.1.52" class="ltx_text" style="background-color:#FCEAB8;">lot</span> <span id="Ax1.F12.1.1.1.53" class="ltx_text" style="background-color:#FCEAB6;">has</span> <span id="Ax1.F12.1.1.1.54" class="ltx_text" style="background-color:#FCEAB7;">changed</span> <span id="Ax1.F12.1.1.1.55" class="ltx_text" style="background-color:#FCE9B5;">since</span> <span id="Ax1.F12.1.1.1.56" class="ltx_text" style="background-color:#FCE9B2;">he</span> <span id="Ax1.F12.1.1.1.57" class="ltx_text" style="background-color:#FCEEC3;">first</span> <span id="Ax1.F12.1.1.1.58" class="ltx_text" style="background-color:#FBE6A9;">hosted</span> <span id="Ax1.F12.1.1.1.59" class="ltx_text" style="background-color:#FCE9B4;">the</span> <span id="Ax1.F12.1.1.1.60" class="ltx_text" style="background-color:#FCEAB8;">comedy</span> <span id="Ax1.F12.1.1.1.61" class="ltx_text" style="background-color:#FCEBBA;">sketch</span> <span id="Ax1.F12.1.1.1.62" class="ltx_text" style="background-color:#FCEBBB;">show</span> <span id="Ax1.F12.1.1.1.63" class="ltx_text" style="background-color:#FBE5A7;">in</span> <span id="Ax1.F12.1.1.1.64" class="ltx_text" style="background-color:#FCE9B3;">1982</span> <span id="Ax1.F12.1.1.1.65" class="ltx_text" style="background-color:#FCE9B5;">.</span> <span id="Ax1.F12.1.1.1.66" class="ltx_text" style="background-color:#FFFDF9;">“</span> <span id="Ax1.F12.1.1.1.67" class="ltx_text" style="background-color:#FFFFFF;">i</span> <span id="Ax1.F12.1.1.1.68" class="ltx_text" style="background-color:#FFFFFF;">had</span> <span id="Ax1.F12.1.1.1.69" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.70" class="ltx_text" style="background-color:#FFFEFD;">baby</span> <span id="Ax1.F12.1.1.1.71" class="ltx_text" style="background-color:#FFFFFF;">–</span> <span id="Ax1.F12.1.1.1.72" class="ltx_text" style="background-color:#FFFFFF;">he</span> <span id="Ax1.F12.1.1.1.73" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F12.1.1.1.74" class="ltx_text" style="background-color:#FFFFFF;">31</span> <span id="Ax1.F12.1.1.1.75" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F12.1.1.1.76" class="ltx_text" style="background-color:#FFFFFF;">i</span> <span id="Ax1.F12.1.1.1.77" class="ltx_text" style="background-color:#FFFFFF;">also</span> <span id="Ax1.F12.1.1.1.78" class="ltx_text" style="background-color:#FFFFFF;">have</span> <span id="Ax1.F12.1.1.1.79" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.80" class="ltx_text" style="background-color:#FFFFFF;">new</span> <span id="Ax1.F12.1.1.1.81" class="ltx_text" style="background-color:#FFFFFF;">girlfriend</span> <span id="Ax1.F12.1.1.1.82" class="ltx_text" style="background-color:#FFFFFF;">–</span> <span id="Ax1.F12.1.1.1.83" class="ltx_text" style="background-color:#FFFFFF;">she</span> <span id="Ax1.F12.1.1.1.84" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F12.1.1.1.85" class="ltx_text" style="background-color:#FFFFFF;">28</span> <span id="Ax1.F12.1.1.1.86" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F12.1.1.1.87" class="ltx_text" style="background-color:#FFFFFF;">”</span> <span id="Ax1.F12.1.1.1.88" class="ltx_text" style="background-color:#FFFFFF;">he</span> <span id="Ax1.F12.1.1.1.89" class="ltx_text" style="background-color:#FFFFFF;">said</span> <span id="Ax1.F12.1.1.1.90" class="ltx_text" style="background-color:#FFFEFD;">.</span> <span id="Ax1.F12.1.1.1.91" class="ltx_text" style="background-color:#FEFBF3;">fans</span> <span id="Ax1.F12.1.1.1.92" class="ltx_text" style="background-color:#FFFFFF;">who</span> <span id="Ax1.F12.1.1.1.93" class="ltx_text" style="background-color:#FFFFFF;">were</span> <span id="Ax1.F12.1.1.1.94" class="ltx_text" style="background-color:#FFFFFF;">hoping</span> <span id="Ax1.F12.1.1.1.95" class="ltx_text" style="background-color:#FFFFFF;">for</span> <span id="Ax1.F12.1.1.1.96" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.97" class="ltx_text" style="background-color:#FFFFFF;">full-blown</span> <span id="Ax1.F12.1.1.1.98" class="ltx_text" style="background-color:#FFFFFF;">revival</span> <span id="Ax1.F12.1.1.1.99" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F12.1.1.1.100" class="ltx_text" style="background-color:#FFFFFE;">keaton</span> <span id="Ax1.F12.1.1.1.101" class="ltx_text" style="background-color:#FFFEFA;">’s</span> <span id="Ax1.F12.1.1.1.102" class="ltx_text" style="background-color:#FFFFFF;">most</span> <span id="Ax1.F12.1.1.1.103" class="ltx_text" style="background-color:#FFFFFF;">memorable</span> <span id="Ax1.F12.1.1.1.104" class="ltx_text" style="background-color:#FFFFFF;">characters</span> <span id="Ax1.F12.1.1.1.105" class="ltx_text" style="background-color:#FFFFFF;">might</span> <span id="Ax1.F12.1.1.1.106" class="ltx_text" style="background-color:#FFFFFF;">have</span> <span id="Ax1.F12.1.1.1.107" class="ltx_text" style="background-color:#FFFFFE;">been</span> <span id="Ax1.F12.1.1.1.108" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.109" class="ltx_text" style="background-color:#FFFFFF;">little</span> <span id="Ax1.F12.1.1.1.110" class="ltx_text" style="background-color:#FFFFFF;">disappointed</span> <span id="Ax1.F12.1.1.1.111" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F12.1.1.1.112" class="ltx_text" style="background-color:#FEFBF3;">snl</span> <span id="Ax1.F12.1.1.1.113" class="ltx_text" style="background-color:#FFFFFF;">cast</span> <span id="Ax1.F12.1.1.1.114" class="ltx_text" style="background-color:#FFFFFF;">members</span> <span id="Ax1.F12.1.1.1.115" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">taran</span> <span id="Ax1.F12.1.1.1.116" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">killam</span> <span id="Ax1.F12.1.1.1.117" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F12.1.1.1.118" class="ltx_text" style="background-color:#FFFFFF;">bobby</span> <span id="Ax1.F12.1.1.1.119" class="ltx_text" style="background-color:#FFFFFF;">moynihan</span> <span id="Ax1.F12.1.1.1.120" class="ltx_text" style="background-color:#FFFFFF;">begged</span> <span id="Ax1.F12.1.1.1.121" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F12.1.1.1.122" class="ltx_text" style="background-color:#FFFFFF;">actor</span> <span id="Ax1.F12.1.1.1.123" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F12.1.1.1.124" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.125" class="ltx_text" style="background-color:#FFFFFF;">song</span> <span id="Ax1.F12.1.1.1.126" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F12.1.1.1.127" class="ltx_text" style="background-color:#FFFDF9;">“</span> <span id="Ax1.F12.1.1.1.128" class="ltx_text" style="background-color:#FFFEFA;">play</span> <span id="Ax1.F12.1.1.1.129" class="ltx_text" style="background-color:#FFFFFF;">”</span> <span id="Ax1.F12.1.1.1.130" class="ltx_text" style="background-color:#FFFFFE;">batman</span> <span id="Ax1.F12.1.1.1.131" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F12.1.1.1.132" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">beetlejuice</span> <span id="Ax1.F12.1.1.1.133" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F12.1.1.1.134" class="ltx_text" style="background-color:#FFFFFF;">them</span> <span id="Ax1.F12.1.1.1.135" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F12.1.1.1.136" class="ltx_text" style="background-color:#FFFEFC;">all</span> <span id="Ax1.F12.1.1.1.137" class="ltx_text" style="background-color:#FFFFFF;">they</span> <span id="Ax1.F12.1.1.1.138" class="ltx_text" style="background-color:#FFFFFF;">got</span> <span id="Ax1.F12.1.1.1.139" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F12.1.1.1.140" class="ltx_text" style="background-color:#FFFFFF;">response</span> <span id="Ax1.F12.1.1.1.141" class="ltx_text" style="background-color:#FFFFFF;">were</span> <span id="Ax1.F12.1.1.1.142" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.143" class="ltx_text" style="background-color:#FFFFFF;">couple</span> <span id="Ax1.F12.1.1.1.144" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F12.1.1.1.145" class="ltx_text" style="background-color:#FFFFFF;">one-liners</span> <span id="Ax1.F12.1.1.1.146" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F12.1.1.1.147" class="ltx_text" style="background-color:#FFFEFC;">overall</span> <span id="Ax1.F12.1.1.1.148" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F12.1.1.1.149" class="ltx_text" style="background-color:#FEFAED;">keaton</span> <span id="Ax1.F12.1.1.1.150" class="ltx_text" style="background-color:#FEF7E4;">’s</span> <span id="Ax1.F12.1.1.1.151" class="ltx_text" style="background-color:#FFFFFF;">performance</span> <span id="Ax1.F12.1.1.1.152" class="ltx_text" style="background-color:#FFFFFF;">drew</span> <span id="Ax1.F12.1.1.1.153" class="ltx_text" style="background-color:#FFFFFF;">high</span> <span id="Ax1.F12.1.1.1.154" class="ltx_text" style="background-color:#FFFFFF;">marks</span> <span id="Ax1.F12.1.1.1.155" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F12.1.1.1.156" class="ltx_text" style="background-color:#FFFFFF;">viewers</span> <span id="Ax1.F12.1.1.1.157" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F12.1.1.1.158" class="ltx_text" style="background-color:#FFFEFD;">critics</span> <span id="Ax1.F12.1.1.1.159" class="ltx_text" style="background-color:#FFFFFF;">for</span> <span id="Ax1.F12.1.1.1.160" class="ltx_text" style="background-color:#FFFFFF;">its</span> <span id="Ax1.F12.1.1.1.161" class="ltx_text" style="background-color:#FFFFFF;">“</span> <span id="Ax1.F12.1.1.1.162" class="ltx_text" style="background-color:#FFFFFF;">deadpan</span> <span id="Ax1.F12.1.1.1.163" class="ltx_text" style="background-color:#FFFFFF;">”</span> <span id="Ax1.F12.1.1.1.164" class="ltx_text" style="background-color:#FFFFFF;">manner</span> <span id="Ax1.F12.1.1.1.165" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F12.1.1.1.166" class="ltx_text" style="background-color:#FFFFFF;">“</span> <span id="Ax1.F12.1.1.1.167" class="ltx_text" style="background-color:#FFFFFF;">unpredictable</span> <span id="Ax1.F12.1.1.1.168" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">weirdness</span> <span id="Ax1.F12.1.1.1.169" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F12.1.1.1.170" class="ltx_text" style="background-color:#FFFFFF;">”</span> <span id="Ax1.F12.1.1.1.171" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F12.1.1.1.172" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F12.1.1.1.173" class="ltx_text" style="background-color:#FFFFFF;">words</span> <span id="Ax1.F12.1.1.1.174" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F12.1.1.1.175" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">a.v</span> <span id="Ax1.F12.1.1.1.176" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F12.1.1.1.177" class="ltx_text" style="background-color:#FFFDF7;">club</span> <span id="Ax1.F12.1.1.1.178" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F12.1.1.1.179" class="ltx_text" style="background-color:#FFFFFF;">dennis</span> <span id="Ax1.F12.1.1.1.180" class="ltx_text" style="background-color:#FFFFFF;">perkins</span> <span id="Ax1.F12.1.1.1.181" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F12.1.1.1.182" class="ltx_text" style="background-color:#FFFDF9;">fans</span> <span id="Ax1.F12.1.1.1.183" class="ltx_text" style="background-color:#FFFFFF;">also</span> <span id="Ax1.F12.1.1.1.184" class="ltx_text" style="background-color:#FFFFFF;">delighted</span> <span id="Ax1.F12.1.1.1.185" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F12.1.1.1.186" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.187" class="ltx_text" style="background-color:#FFFFFF;">cameo</span> <span id="Ax1.F12.1.1.1.188" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F12.1.1.1.189" class="ltx_text" style="background-color:#FFFFFD;">“</span> <span id="Ax1.F12.1.1.1.190" class="ltx_text" style="background-color:#FFFFFE;">walking</span> <span id="Ax1.F12.1.1.1.191" class="ltx_text" style="background-color:#FFFFFF;">dead</span> <span id="Ax1.F12.1.1.1.192" class="ltx_text" style="background-color:#FFFFFF;">”</span> <span id="Ax1.F12.1.1.1.193" class="ltx_text" style="background-color:#FFFFFF;">star</span> <span id="Ax1.F12.1.1.1.194" class="ltx_text" style="background-color:#FFFFFF;">norman</span> <span id="Ax1.F12.1.1.1.195" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">reedus</span> <span id="Ax1.F12.1.1.1.196" class="ltx_text" style="background-color:#FFFFFF;">during</span> <span id="Ax1.F12.1.1.1.197" class="ltx_text" style="background-color:#FFFFFF;">weekend</span> <span id="Ax1.F12.1.1.1.198" class="ltx_text" style="background-color:#FFFFFF;">update</span> <span id="Ax1.F12.1.1.1.199" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F12.1.1.1.200" class="ltx_text" style="background-color:#FEF9EA;">keaton</span> <span id="Ax1.F12.1.1.1.201" class="ltx_text" style="background-color:#FFFFFE;">scored</span> <span id="Ax1.F12.1.1.1.202" class="ltx_text" style="background-color:#FFFFFF;">some</span> <span id="Ax1.F12.1.1.1.203" class="ltx_text" style="background-color:#FFFFFF;">laughs</span> <span id="Ax1.F12.1.1.1.204" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F12.1.1.1.205" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F12.1.1.1.206" class="ltx_text" style="background-color:#FFFFFF;">audience</span> <span id="Ax1.F12.1.1.1.207" class="ltx_text" style="background-color:#FFFFFF;">as</span> <span id="Ax1.F12.1.1.1.208" class="ltx_text" style="background-color:#FFFFFF;">an</span> <span id="Ax1.F12.1.1.1.209" class="ltx_text" style="background-color:#FFFFFF;">ad</span> <span id="Ax1.F12.1.1.1.210" class="ltx_text" style="background-color:#FFFFFF;">executive</span> <span id="Ax1.F12.1.1.1.211" class="ltx_text" style="background-color:#FFFFFF;">who</span> <span id="Ax1.F12.1.1.1.212" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F12.1.1.1.213" class="ltx_text" style="background-color:#FFFFFF;">not</span> <span id="Ax1.F12.1.1.1.214" class="ltx_text" style="background-color:#FFFFFF;">very</span> <span id="Ax1.F12.1.1.1.215" class="ltx_text" style="background-color:#FFFFFF;">good</span> <span id="Ax1.F12.1.1.1.216" class="ltx_text" style="background-color:#FFFFFF;">at</span> <span id="Ax1.F12.1.1.1.217" class="ltx_text" style="background-color:#FFFFFF;">his</span> <span id="Ax1.F12.1.1.1.218" class="ltx_text" style="background-color:#FFFFFE;">job</span> <span id="Ax1.F12.1.1.1.219" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F12.1.1.1.220" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.221" class="ltx_text" style="background-color:#FFFFFF;">confused</span> <span id="Ax1.F12.1.1.1.222" class="ltx_text" style="background-color:#FFFFFF;">grandfather</span> <span id="Ax1.F12.1.1.1.223" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F12.1.1.1.224" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.225" class="ltx_text" style="background-color:#FFFFFF;">high</span> <span id="Ax1.F12.1.1.1.226" class="ltx_text" style="background-color:#FFFFFF;">school</span> <span id="Ax1.F12.1.1.1.227" class="ltx_text" style="background-color:#FFFFFF;">teacher</span> <span id="Ax1.F12.1.1.1.228" class="ltx_text" style="background-color:#FFFFFF;">who</span> <span id="Ax1.F12.1.1.1.229" class="ltx_text" style="background-color:#FFFFFF;">gets</span> <span id="Ax1.F12.1.1.1.230" class="ltx_text" style="background-color:#FFFFFF;">asked</span> <span id="Ax1.F12.1.1.1.231" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F12.1.1.1.232" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F12.1.1.1.233" class="ltx_text" style="background-color:#FFFEFD;">prom</span> <span id="Ax1.F12.1.1.1.234" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F12.1.1.1.235" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.236" class="ltx_text" style="background-color:#FFFFFF;">riff</span> <span id="Ax1.F12.1.1.1.237" class="ltx_text" style="background-color:#FFFFFF;">on</span> <span id="Ax1.F12.1.1.1.238" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F12.1.1.1.239" class="ltx_text" style="background-color:#FFFFFF;">romantic</span> <span id="Ax1.F12.1.1.1.240" class="ltx_text" style="background-color:#FFFFFF;">comedy</span> <span id="Ax1.F12.1.1.1.241" class="ltx_text" style="background-color:#FFFFFF;">“</span> <span id="Ax1.F12.1.1.1.242" class="ltx_text" style="background-color:#FFFFFF;">she</span> <span id="Ax1.F12.1.1.1.243" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F12.1.1.1.244" class="ltx_text" style="background-color:#FFFFFF;">all</span> <span id="Ax1.F12.1.1.1.245" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F12.1.1.1.246" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F12.1.1.1.247" class="ltx_text" style="background-color:#FFFFFF;">”</span> <span id="Ax1.F12.1.1.1.248" class="ltx_text" style="background-color:#FFFEFB;">other</span> <span id="Ax1.F12.1.1.1.249" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">crowd-pleasing</span> <span id="Ax1.F12.1.1.1.250" class="ltx_text" style="background-color:#FFFFFF;">spots</span> <span id="Ax1.F12.1.1.1.251" class="ltx_text" style="background-color:#FFFFFF;">included</span> <span id="Ax1.F12.1.1.1.252" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.253" class="ltx_text" style="background-color:#FFFFFF;">scientology</span> <span id="Ax1.F12.1.1.1.254" class="ltx_text" style="background-color:#FFFFFF;">parody</span> <span id="Ax1.F12.1.1.1.255" class="ltx_text" style="background-color:#FFFFFF;">music</span> <span id="Ax1.F12.1.1.1.256" class="ltx_text" style="background-color:#FFFFFF;">video</span> <span id="Ax1.F12.1.1.1.257" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F12.1.1.1.258" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.259" class="ltx_text" style="background-color:#FFFFFF;">news</span> <span id="Ax1.F12.1.1.1.260" class="ltx_text" style="background-color:#FFFFFF;">conference</span> <span id="Ax1.F12.1.1.1.261" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">spoofing</span> <span id="Ax1.F12.1.1.1.262" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F12.1.1.1.263" class="ltx_text" style="background-color:#FFFFFF;">ncaa</span> <span id="Ax1.F12.1.1.1.264" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">student-athlete</span> <span id="Ax1.F12.1.1.1.265" class="ltx_text" style="background-color:#FFFFFF;">debate</span> <span id="Ax1.F12.1.1.1.266" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F12.1.1.1.267" class="ltx_text" style="background-color:#FFFDF7;">the</span> <span id="Ax1.F12.1.1.1.268" class="ltx_text" style="background-color:#FFFFFF;">show</span> <span id="Ax1.F12.1.1.1.269" class="ltx_text" style="background-color:#FFFFFF;">also</span> <span id="Ax1.F12.1.1.1.270" class="ltx_text" style="background-color:#FFFFFF;">poked</span> <span id="Ax1.F12.1.1.1.271" class="ltx_text" style="background-color:#FFFFFF;">fun</span> <span id="Ax1.F12.1.1.1.272" class="ltx_text" style="background-color:#FFFFFF;">at</span> <span id="Ax1.F12.1.1.1.273" class="ltx_text" style="background-color:#FFFFFE;">cnn</span> <span id="Ax1.F12.1.1.1.274" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F12.1.1.1.275" class="ltx_text" style="background-color:#FFFFFF;">cast</span> <span id="Ax1.F12.1.1.1.276" class="ltx_text" style="background-color:#FFFFFF;">member</span> <span id="Ax1.F12.1.1.1.277" class="ltx_text" style="background-color:#FFFFFF;">cecily</span> <span id="Ax1.F12.1.1.1.278" class="ltx_text" style="background-color:#FFFFFF;">strong</span> <span id="Ax1.F12.1.1.1.279" class="ltx_text" style="background-color:#FFFFFF;">playing</span> <span id="Ax1.F12.1.1.1.280" class="ltx_text" style="background-color:#FFFFFF;">anchor</span> <span id="Ax1.F12.1.1.1.281" class="ltx_text" style="background-color:#FFFFFF;">brooke</span> <span id="Ax1.F12.1.1.1.282" class="ltx_text" style="background-color:#FFFFFF;">baldwin</span> <span id="Ax1.F12.1.1.1.283" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F12.1.1.1.284" class="ltx_text" style="background-color:#FFFEFB;">baldwin</span> <span id="Ax1.F12.1.1.1.285" class="ltx_text" style="background-color:#FFFFFF;">said</span> <span id="Ax1.F12.1.1.1.286" class="ltx_text" style="background-color:#FFFFFF;">on</span> <span id="Ax1.F12.1.1.1.287" class="ltx_text" style="background-color:#FFFFFF;">twitter</span> <span id="Ax1.F12.1.1.1.288" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F12.1.1.1.289" class="ltx_text" style="background-color:#FFFFFF;">she</span> <span id="Ax1.F12.1.1.1.290" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F12.1.1.1.291" class="ltx_text" style="background-color:#FFFFFF;">taking</span> <span id="Ax1.F12.1.1.1.292" class="ltx_text" style="background-color:#FFFFFF;">it</span> <span id="Ax1.F12.1.1.1.293" class="ltx_text" style="background-color:#FFFFFF;">as</span> <span id="Ax1.F12.1.1.1.294" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.295" class="ltx_text" style="background-color:#FFFFFF;">“</span> <span id="Ax1.F12.1.1.1.296" class="ltx_text" style="background-color:#FFFFFF;">crazy</span> <span id="Ax1.F12.1.1.1.297" class="ltx_text" style="background-color:#FFFFFF;">compliment</span> <span id="Ax1.F12.1.1.1.298" class="ltx_text" style="background-color:#FFFFFF;">”</span> <span id="Ax1.F12.1.1.1.299" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F12.1.1.1.300" class="ltx_text" style="background-color:#FFFFFF;">shared</span> <span id="Ax1.F12.1.1.1.301" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F12.1.1.1.302" class="ltx_text" style="background-color:#FFFFFF;">clip</span> <span id="Ax1.F12.1.1.1.303" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F12.1.1.1.304" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F12.1.1.1.305" class="ltx_text" style="background-color:#FFFFFF;">skit</span> <span id="Ax1.F12.1.1.1.306" class="ltx_text" style="background-color:#FFFFFF;">.</span>

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F12.1.1.1.307" class="ltx_text ltx_font_bold">Reference Summary:</span> 
<br class="ltx_break">michael keaton hosted “ saturday night live ” for the first time in 1982 .
<br class="ltx_break">in 2015 , his nods to starring roles in “ <span id="Ax1.F12.1.1.1.308" class="ltx_text ltx_font_italic">beetlejuice</span> ” and “ batman ” are brief .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F12.1.1.1.309" class="ltx_text ltx_font_bold">Baseline:</span> 
<br class="ltx_break">“ snl ” wins “ saturday night live ” . 
<br class="ltx_break">keaton acknowledged in his opening “ saturday night live ” . 
<br class="ltx_break">keaton acknowledged in his opening night ’s “ UNK ” . 
<br class="ltx_break">the show also pokes fun at cnn with cast member <span id="Ax1.F12.1.1.1.310" class="ltx_text ltx_font_bold" style="color:#DB4437;">mariah carey</span> .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F12.1.1.1.311" class="ltx_text ltx_font_bold">Pointer-Generator, No Coverage:</span> 
<br class="ltx_break">“ <span id="Ax1.F12.1.1.1.312" class="ltx_text ltx_font_italic">beetlejuice</span> ” michael keaton paid homage to his roles in “ <span id="Ax1.F12.1.1.1.313" class="ltx_text ltx_font_italic">beetlejuice</span> ” and “ batman ” in his third turn hosting “ saturday night live ” this weekend . 
<br class="ltx_break">keaton acknowledged in his opening monologue that a lot has changed since he first hosted the comedy sketch show in 1982 .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F12.1.1.1.314" class="ltx_text ltx_font_bold">Pointer-Generator, With Coverage:</span> 
<br class="ltx_break"><span id="Ax1.F12.1.1.1.315" class="ltx_text" style="background-color:#EBF7F1;">michael</span> <span id="Ax1.F12.1.1.1.316" class="ltx_text" style="background-color:#FBFDFC;">keaton</span> <span id="Ax1.F12.1.1.1.317" class="ltx_text" style="background-color:#BAE3CF;">paid</span> <span id="Ax1.F12.1.1.1.318" class="ltx_text" style="background-color:#E6F5ED;">homage</span> <span id="Ax1.F12.1.1.1.319" class="ltx_text" style="background-color:#B8E2CD;">to</span> <span id="Ax1.F12.1.1.1.320" class="ltx_text" style="background-color:#E6F5ED;">his</span> <span id="Ax1.F12.1.1.1.321" class="ltx_text" style="background-color:#F2FAF6;">roles</span> <span id="Ax1.F12.1.1.1.322" class="ltx_text" style="background-color:#D3EDE0;">in</span> <span id="Ax1.F12.1.1.1.323" class="ltx_text" style="background-color:#F6FBF9;">“</span> <span id="Ax1.F12.1.1.1.324" class="ltx_text ltx_font_italic" style="background-color:#FEFFFE;">beetlejuice</span> <span id="Ax1.F12.1.1.1.325" class="ltx_text" style="background-color:#EEF8F3;">”</span> <span id="Ax1.F12.1.1.1.326" class="ltx_text" style="background-color:#EFF8F4;">and</span> <span id="Ax1.F12.1.1.1.327" class="ltx_text" style="background-color:#F4FAF7;">“</span> <span id="Ax1.F12.1.1.1.328" class="ltx_text" style="background-color:#FEFFFE;">batman</span> <span id="Ax1.F12.1.1.1.329" class="ltx_text" style="background-color:#B7E2CD;">”</span> <span id="Ax1.F12.1.1.1.330" class="ltx_text" style="background-color:#CEEBDD;">in</span> <span id="Ax1.F12.1.1.1.331" class="ltx_text" style="background-color:#F7FCF9;">his</span> <span id="Ax1.F12.1.1.1.332" class="ltx_text" style="background-color:#FCFEFD;">third</span> <span id="Ax1.F12.1.1.1.333" class="ltx_text" style="background-color:#EAF6F0;">turn</span> <span id="Ax1.F12.1.1.1.334" class="ltx_text" style="background-color:#E0F3EA;">hosting</span> <span id="Ax1.F12.1.1.1.335" class="ltx_text" style="background-color:#D5EEE1;">“</span> <span id="Ax1.F12.1.1.1.336" class="ltx_text" style="background-color:#FDFEFD;">saturday</span> <span id="Ax1.F12.1.1.1.337" class="ltx_text" style="background-color:#FFFFFF;">night</span> <span id="Ax1.F12.1.1.1.338" class="ltx_text" style="background-color:#FDFEFE;">live</span> <span id="Ax1.F12.1.1.1.339" class="ltx_text" style="background-color:#C2E6D4;">”</span> <span id="Ax1.F12.1.1.1.340" class="ltx_text" style="background-color:#BFE5D2;">.</span>
<br class="ltx_break"><span id="Ax1.F12.1.1.1.341" class="ltx_text" style="background-color:#E7F5EE;">keaton</span> <span id="Ax1.F12.1.1.1.342" class="ltx_text" style="background-color:#C7E8D8;">acknowledged</span> <span id="Ax1.F12.1.1.1.343" class="ltx_text" style="background-color:#E8F6EF;">in</span> <span id="Ax1.F12.1.1.1.344" class="ltx_text" style="background-color:#FAFDFC;">his</span> <span id="Ax1.F12.1.1.1.345" class="ltx_text" style="background-color:#FDFEFD;">opening</span> <span id="Ax1.F12.1.1.1.346" class="ltx_text" style="background-color:#FCFEFD;">monologue</span> <span id="Ax1.F12.1.1.1.347" class="ltx_text" style="background-color:#EBF7F1;">that</span> <span id="Ax1.F12.1.1.1.348" class="ltx_text" style="background-color:#FAFDFB;">a</span> <span id="Ax1.F12.1.1.1.349" class="ltx_text" style="background-color:#FEFFFF;">lot</span> <span id="Ax1.F12.1.1.1.350" class="ltx_text" style="background-color:#FBFDFC;">has</span> <span id="Ax1.F12.1.1.1.351" class="ltx_text" style="background-color:#FDFEFE;">changed</span> <span id="Ax1.F12.1.1.1.352" class="ltx_text" style="background-color:#EEF8F3;">since</span> <span id="Ax1.F12.1.1.1.353" class="ltx_text" style="background-color:#F7FCF9;">he</span> <span id="Ax1.F12.1.1.1.354" class="ltx_text" style="background-color:#FCFEFD;">first</span> <span id="Ax1.F12.1.1.1.355" class="ltx_text" style="background-color:#F1F9F5;">hosted</span> <span id="Ax1.F12.1.1.1.356" class="ltx_text" style="background-color:#FDFEFD;">the</span> <span id="Ax1.F12.1.1.1.357" class="ltx_text" style="background-color:#FEFFFE;">comedy</span> <span id="Ax1.F12.1.1.1.358" class="ltx_text" style="background-color:#FFFFFF;">sketch</span> <span id="Ax1.F12.1.1.1.359" class="ltx_text" style="background-color:#FEFFFE;">show</span> <span id="Ax1.F12.1.1.1.360" class="ltx_text" style="background-color:#F8FCFA;">in</span> <span id="Ax1.F12.1.1.1.361" class="ltx_text" style="background-color:#FEFFFF;">1982</span> <span id="Ax1.F12.1.1.1.362" class="ltx_text" style="background-color:#DBF0E6;">.</span>
<br class="ltx_break"></span>
</span></p>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 12: </span>Baseline model replaces <span id="Ax1.F12.6.1" class="ltx_text ltx_font_italic">cecily strong</span> with <span id="Ax1.F12.7.2" class="ltx_text ltx_font_italic">mariah carey</span>, and produces generally nonsensical output.
The baseline model may be struggling with the out-of-vocabulary word <span id="Ax1.F12.8.3" class="ltx_text ltx_font_italic">beetlejuice</span>, or perhaps the unusual non-news format of the article.
Note that the final model omits <span id="Ax1.F12.9.4" class="ltx_text ltx_font_italic">– ever so slightly –</span> from its first sentence.</figcaption>
</figure>
<figure id="Ax1.F13" class="ltx_figure">
<p id="Ax1.F13.1" class="ltx_p">
<span id="Ax1.F13.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:448.4pt;">
<span id="Ax1.F13.1.1.1" class="ltx_p"><span id="Ax1.F13.1.1.1.1" class="ltx_text ltx_font_bold">Article (truncated):</span> <span id="Ax1.F13.1.1.1.2" class="ltx_text" style="background-color:#FFFEFB;">they</span> <span id="Ax1.F13.1.1.1.3" class="ltx_text" style="background-color:#FFFEFD;">are</span> <span id="Ax1.F13.1.1.1.4" class="ltx_text" style="background-color:#FFFFFF;">supposed</span> <span id="Ax1.F13.1.1.1.5" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F13.1.1.1.6" class="ltx_text" style="background-color:#FFFFFF;">be</span> <span id="Ax1.F13.1.1.1.7" class="ltx_text" style="background-color:#FFFFFD;">the</span> <span id="Ax1.F13.1.1.1.8" class="ltx_text" style="background-color:#FFFFFE;">dream</span> <span id="Ax1.F13.1.1.1.9" class="ltx_text" style="background-color:#FFFFFE;">team</span> <span id="Ax1.F13.1.1.1.10" class="ltx_text" style="background-color:#FFFFFF;">who</span> <span id="Ax1.F13.1.1.1.11" class="ltx_text" style="background-color:#FFFFFF;">can</span> <span id="Ax1.F13.1.1.1.12" class="ltx_text" style="background-color:#FFFFFF;">solve</span> <span id="Ax1.F13.1.1.1.13" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F13.1.1.1.14" class="ltx_text" style="background-color:#FFFFFF;">conundrum</span> <span id="Ax1.F13.1.1.1.15" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F13.1.1.1.16" class="ltx_text" style="background-color:#FFFFFF;">how</span> <span id="Ax1.F13.1.1.1.17" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F13.1.1.1.18" class="ltx_text" style="background-color:#FFFFFE;">win</span> <span id="Ax1.F13.1.1.1.19" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F13.1.1.1.20" class="ltx_text" style="background-color:#FFFFFF;">election</span> <span id="Ax1.F13.1.1.1.21" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F13.1.1.1.22" class="ltx_text" style="background-color:#FEF6E1;">but</span> <span id="Ax1.F13.1.1.1.23" class="ltx_text" style="background-color:#FDF4DA;">david</span> <span id="Ax1.F13.1.1.1.24" class="ltx_text" style="background-color:#FFFEFB;">cameron</span> <span id="Ax1.F13.1.1.1.25" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F13.1.1.1.26" class="ltx_text" style="background-color:#FFFCF5;">boris</span> <span id="Ax1.F13.1.1.1.27" class="ltx_text" style="background-color:#FFFFFF;">johnson</span> <span id="Ax1.F13.1.1.1.28" class="ltx_text" style="background-color:#FFFFFF;">were</span> <span id="Ax1.F13.1.1.1.29" class="ltx_text" style="background-color:#FFFFFF;">left</span> <span id="Ax1.F13.1.1.1.30" class="ltx_text" style="background-color:#FFFFFF;">scratching</span> <span id="Ax1.F13.1.1.1.31" class="ltx_text" style="background-color:#FFFFFF;">their</span> <span id="Ax1.F13.1.1.1.32" class="ltx_text" style="background-color:#FFFFFF;">heads</span> <span id="Ax1.F13.1.1.1.33" class="ltx_text" style="background-color:#FFFFFE;">today</span> <span id="Ax1.F13.1.1.1.34" class="ltx_text" style="background-color:#FFFFFF;">as</span> <span id="Ax1.F13.1.1.1.35" class="ltx_text" style="background-color:#FFFEFD;">they</span> <span id="Ax1.F13.1.1.1.36" class="ltx_text" style="background-color:#FFFEFD;">struggled</span> <span id="Ax1.F13.1.1.1.37" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F13.1.1.1.38" class="ltx_text" style="background-color:#FFFFFE;">a</span> <span id="Ax1.F13.1.1.1.39" class="ltx_text" style="background-color:#FFFEFB;">children</span> <span id="Ax1.F13.1.1.1.40" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F13.1.1.1.41" class="ltx_text" style="background-color:#FFFFFE;">jigsaw</span> <span id="Ax1.F13.1.1.1.42" class="ltx_text" style="background-color:#FFFFFE;">teaching</span> <span id="Ax1.F13.1.1.1.43" class="ltx_text" style="background-color:#FFFFFF;">toddlers</span> <span id="Ax1.F13.1.1.1.44" class="ltx_text" style="background-color:#FFFFFF;">about</span> <span id="Ax1.F13.1.1.1.45" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F13.1.1.1.46" class="ltx_text" style="background-color:#FFFFFF;">seasons</span> <span id="Ax1.F13.1.1.1.47" class="ltx_text" style="background-color:#FFFFFE;">.</span> <span id="Ax1.F13.1.1.1.48" class="ltx_text" style="background-color:#FFFEFA;">as</span> <span id="Ax1.F13.1.1.1.49" class="ltx_text" style="background-color:#FFFEFB;">the</span> <span id="Ax1.F13.1.1.1.50" class="ltx_text" style="background-color:#FDF4D8;">london</span> <span id="Ax1.F13.1.1.1.51" class="ltx_text" style="background-color:#FBE5A5;">mayor</span> <span id="Ax1.F13.1.1.1.52" class="ltx_text" style="background-color:#FBE6AB;">tried</span> <span id="Ax1.F13.1.1.1.53" class="ltx_text" style="background-color:#FBE7AC;">to</span> <span id="Ax1.F13.1.1.1.54" class="ltx_text" style="background-color:#FBE6A9;">hammer</span> <span id="Ax1.F13.1.1.1.55" class="ltx_text" style="background-color:#FCE8B1;">ill-fitting</span> <span id="Ax1.F13.1.1.1.56" class="ltx_text" style="background-color:#FBE3A0;">pieces</span> <span id="Ax1.F13.1.1.1.57" class="ltx_text" style="background-color:#FBE5A8;">together</span> <span id="Ax1.F13.1.1.1.58" class="ltx_text" style="background-color:#FBE6A8;">with</span> <span id="Ax1.F13.1.1.1.59" class="ltx_text" style="background-color:#FCEBBB;">his</span> <span id="Ax1.F13.1.1.1.60" class="ltx_text" style="background-color:#FBE4A3;">hands</span> <span id="Ax1.F13.1.1.1.61" class="ltx_text" style="background-color:#FBE6AA;">,</span> <span id="Ax1.F13.1.1.1.62" class="ltx_text" style="background-color:#FFFCF4;">the</span> <span id="Ax1.F13.1.1.1.63" class="ltx_text" style="background-color:#FDF2D4;">prime</span> <span id="Ax1.F13.1.1.1.64" class="ltx_text" style="background-color:#FBE5A7;">minister</span> <span id="Ax1.F13.1.1.1.65" class="ltx_text" style="background-color:#FBE6AA;">tried</span> <span id="Ax1.F13.1.1.1.66" class="ltx_text" style="background-color:#FBE6A9;">out</span> <span id="Ax1.F13.1.1.1.67" class="ltx_text" style="background-color:#FBE5A8;">what</span> <span id="Ax1.F13.1.1.1.68" class="ltx_text" style="background-color:#FBE5A7;">could</span> <span id="Ax1.F13.1.1.1.69" class="ltx_text" style="background-color:#FBE6A8;">be</span> <span id="Ax1.F13.1.1.1.70" class="ltx_text" style="background-color:#FCE8B0;">a</span> <span id="Ax1.F13.1.1.1.71" class="ltx_text" style="background-color:#FBE6A9;">new</span> <span id="Ax1.F13.1.1.1.72" class="ltx_text" style="background-color:#FBE6AA;">election</span> <span id="Ax1.F13.1.1.1.73" class="ltx_text" style="background-color:#FBE199;">slogan</span> <span id="Ax1.F13.1.1.1.74" class="ltx_text" style="background-color:#FBE5A7;">,</span> <span id="Ax1.F13.1.1.1.75" class="ltx_text" style="background-color:#FFFFFF;">telling</span> <span id="Ax1.F13.1.1.1.76" class="ltx_text" style="background-color:#FFFFFF;">him</span> <span id="Ax1.F13.1.1.1.77" class="ltx_text" style="background-color:#FFFFFF;">:</span> <span id="Ax1.F13.1.1.1.78" class="ltx_text" style="background-color:#FFFFFF;">’</span> <span id="Ax1.F13.1.1.1.79" class="ltx_text" style="background-color:#FFFFFF;">if</span> <span id="Ax1.F13.1.1.1.80" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F13.1.1.1.81" class="ltx_text" style="background-color:#FFFFFF;">doubt</span> <span id="Ax1.F13.1.1.1.82" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F13.1.1.1.83" class="ltx_text" style="background-color:#FFFFFF;">wedge</span> <span id="Ax1.F13.1.1.1.84" class="ltx_text" style="background-color:#FFFFFF;">it</span> <span id="Ax1.F13.1.1.1.85" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F13.1.1.1.86" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F13.1.1.1.87" class="ltx_text" style="background-color:#FFFFFF;">’</span> <span id="Ax1.F13.1.1.1.88" class="ltx_text" style="background-color:#FFFEFD;">after</span> <span id="Ax1.F13.1.1.1.89" class="ltx_text" style="background-color:#FFFFFE;">being</span> <span id="Ax1.F13.1.1.1.90" class="ltx_text" style="background-color:#FFFFFE;">put</span> <span id="Ax1.F13.1.1.1.91" class="ltx_text" style="background-color:#FFFFFF;">right</span> <span id="Ax1.F13.1.1.1.92" class="ltx_text" style="background-color:#FFFFFF;">by</span> <span id="Ax1.F13.1.1.1.93" class="ltx_text" style="background-color:#FFFFFE;">a</span> <span id="Ax1.F13.1.1.1.94" class="ltx_text" style="background-color:#FFFDFA;">four-year-old</span> <span id="Ax1.F13.1.1.1.95" class="ltx_text" style="background-color:#FFFFFF;">who</span> <span id="Ax1.F13.1.1.1.96" class="ltx_text" style="background-color:#FFFFFF;">spotted</span> <span id="Ax1.F13.1.1.1.97" class="ltx_text" style="background-color:#FFFFFF;">their</span> <span id="Ax1.F13.1.1.1.98" class="ltx_text" style="background-color:#FFFFFF;">errors</span> <span id="Ax1.F13.1.1.1.99" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F13.1.1.1.100" class="ltx_text" style="background-color:#FEFAED;">the</span> <span id="Ax1.F13.1.1.1.101" class="ltx_text" style="background-color:#FEF8E6;">pair</span> <span id="Ax1.F13.1.1.1.102" class="ltx_text" style="background-color:#FFFFFF;">had</span> <span id="Ax1.F13.1.1.1.103" class="ltx_text" style="background-color:#FFFFFF;">more</span> <span id="Ax1.F13.1.1.1.104" class="ltx_text" style="background-color:#FFFFFF;">fun</span> <span id="Ax1.F13.1.1.1.105" class="ltx_text" style="background-color:#FFFFFF;">finger</span> <span id="Ax1.F13.1.1.1.106" class="ltx_text" style="background-color:#FFFFFF;">painting</span> <span id="Ax1.F13.1.1.1.107" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F13.1.1.1.108" class="ltx_text" style="background-color:#FFFEFA;">tory</span> <span id="Ax1.F13.1.1.1.109" class="ltx_text" style="background-color:#FFFFFF;">blue</span> <span id="Ax1.F13.1.1.1.110" class="ltx_text" style="background-color:#FFFFFE;">paint</span> <span id="Ax1.F13.1.1.1.111" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F13.1.1.1.112" class="ltx_text" style="background-color:#FEFAEE;">david</span> <span id="Ax1.F13.1.1.1.113" class="ltx_text" style="background-color:#FFFFFE;">cameron</span> <span id="Ax1.F13.1.1.1.114" class="ltx_text" style="background-color:#FFFEFC;">and</span> <span id="Ax1.F13.1.1.1.115" class="ltx_text" style="background-color:#FFFCF6;">boris</span> <span id="Ax1.F13.1.1.1.116" class="ltx_text" style="background-color:#FFFFFF;">johnson</span> <span id="Ax1.F13.1.1.1.117" class="ltx_text" style="background-color:#FFFFFF;">were</span> <span id="Ax1.F13.1.1.1.118" class="ltx_text" style="background-color:#FFFFFE;">left</span> <span id="Ax1.F13.1.1.1.119" class="ltx_text" style="background-color:#FFFFFF;">stumped</span> <span id="Ax1.F13.1.1.1.120" class="ltx_text" style="background-color:#FFFFFF;">by</span> <span id="Ax1.F13.1.1.1.121" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F13.1.1.1.122" class="ltx_text" style="background-color:#FFFEFD;">puzzle</span> <span id="Ax1.F13.1.1.1.123" class="ltx_text" style="background-color:#FFFFFF;">at</span> <span id="Ax1.F13.1.1.1.124" class="ltx_text" style="background-color:#FFFFFF;">advantage</span> <span id="Ax1.F13.1.1.1.125" class="ltx_text" style="background-color:#FFFFFE;">children</span> <span id="Ax1.F13.1.1.1.126" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F13.1.1.1.127" class="ltx_text" style="background-color:#FFFFFF;">day</span> <span id="Ax1.F13.1.1.1.128" class="ltx_text" style="background-color:#FFFFFF;">nursery</span> <span id="Ax1.F13.1.1.1.129" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F13.1.1.1.130" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">surbiton</span> <span id="Ax1.F13.1.1.1.131" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F13.1.1.1.132" class="ltx_text" style="background-color:#FFFFFF;">as</span> <span id="Ax1.F13.1.1.1.133" class="ltx_text" style="background-color:#FFFDF7;">three-year-old</span> <span id="Ax1.F13.1.1.1.134" class="ltx_text" style="background-color:#FFFEFC;">stephanie</span> <span id="Ax1.F13.1.1.1.135" class="ltx_text" style="background-color:#FFFFFF;">looked</span> <span id="Ax1.F13.1.1.1.136" class="ltx_text" style="background-color:#FFFFFF;">on</span> <span id="Ax1.F13.1.1.1.137" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F13.1.1.1.138" class="ltx_text" style="background-color:#FFFEFC;">when</span> <span id="Ax1.F13.1.1.1.139" class="ltx_text" style="background-color:#FFFEFD;">they</span> <span id="Ax1.F13.1.1.1.140" class="ltx_text" style="background-color:#FFFEFC;">tried</span> <span id="Ax1.F13.1.1.1.141" class="ltx_text" style="background-color:#FFFEFA;">to</span> <span id="Ax1.F13.1.1.1.142" class="ltx_text" style="background-color:#FFFFFF;">put</span> <span id="Ax1.F13.1.1.1.143" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F13.1.1.1.144" class="ltx_text" style="background-color:#FFFFFF;">puzzle</span> <span id="Ax1.F13.1.1.1.145" class="ltx_text" style="background-color:#FFFFFF;">back</span> <span id="Ax1.F13.1.1.1.146" class="ltx_text" style="background-color:#FFFFFF;">together</span> <span id="Ax1.F13.1.1.1.147" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F13.1.1.1.148" class="ltx_text" style="background-color:#FEF9EA;">they</span> <span id="Ax1.F13.1.1.1.149" class="ltx_text" style="background-color:#FCEDC0;">hit</span> <span id="Ax1.F13.1.1.1.150" class="ltx_text" style="background-color:#FCEAB7;">trouble</span> <span id="Ax1.F13.1.1.1.151" class="ltx_text" style="background-color:#FCE8B0;">after</span> <span id="Ax1.F13.1.1.1.152" class="ltx_text" style="background-color:#FCEAB8;">it</span> <span id="Ax1.F13.1.1.1.153" class="ltx_text" style="background-color:#FCECBD;">proved</span> <span id="Ax1.F13.1.1.1.154" class="ltx_text" style="background-color:#FCECBD;">to</span> <span id="Ax1.F13.1.1.1.155" class="ltx_text" style="background-color:#FCEAB7;">be</span> <span id="Ax1.F13.1.1.1.156" class="ltx_text" style="background-color:#FCECBD;">more</span> <span id="Ax1.F13.1.1.1.157" class="ltx_text" style="background-color:#FCEBBA;">difficult</span> <span id="Ax1.F13.1.1.1.158" class="ltx_text" style="background-color:#FCEDC3;">than</span> <span id="Ax1.F13.1.1.1.159" class="ltx_text" style="background-color:#FDEEC6;">expected</span> <span id="Ax1.F13.1.1.1.160" class="ltx_text" style="background-color:#FCEAB7;">.</span> <span id="Ax1.F13.1.1.1.161" class="ltx_text" style="background-color:#FFFEFB;">the</span> <span id="Ax1.F13.1.1.1.162" class="ltx_text" style="background-color:#FEFAED;">conservative</span> <span id="Ax1.F13.1.1.1.163" class="ltx_text" style="background-color:#FFFEFD;">duo</span> <span id="Ax1.F13.1.1.1.164" class="ltx_text" style="background-color:#FFFFFE;">made</span> <span id="Ax1.F13.1.1.1.165" class="ltx_text" style="background-color:#FFFFFF;">their</span> <span id="Ax1.F13.1.1.1.166" class="ltx_text" style="background-color:#FFFFFF;">first</span> <span id="Ax1.F13.1.1.1.167" class="ltx_text" style="background-color:#FFFFFF;">appearance</span> <span id="Ax1.F13.1.1.1.168" class="ltx_text" style="background-color:#FFFFFF;">together</span> <span id="Ax1.F13.1.1.1.169" class="ltx_text" style="background-color:#FFFFFE;">on</span> <span id="Ax1.F13.1.1.1.170" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F13.1.1.1.171" class="ltx_text" style="background-color:#FFFFFE;">campaign</span> <span id="Ax1.F13.1.1.1.172" class="ltx_text" style="background-color:#FFFFFF;">trail</span> <span id="Ax1.F13.1.1.1.173" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F13.1.1.1.174" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F13.1.1.1.175" class="ltx_text" style="background-color:#FFFFFE;">visit</span> <span id="Ax1.F13.1.1.1.176" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F13.1.1.1.177" class="ltx_text" style="background-color:#FFFFFF;">advantage</span> <span id="Ax1.F13.1.1.1.178" class="ltx_text" style="background-color:#FFFFFF;">day</span> <span id="Ax1.F13.1.1.1.179" class="ltx_text" style="background-color:#FFFFFE;">nursery</span> <span id="Ax1.F13.1.1.1.180" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F13.1.1.1.181" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">surbiton</span> <span id="Ax1.F13.1.1.1.182" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F13.1.1.1.183" class="ltx_text" style="background-color:#FFFFFF;">south</span> <span id="Ax1.F13.1.1.1.184" class="ltx_text" style="background-color:#FFFFFF;">west</span> <span id="Ax1.F13.1.1.1.185" class="ltx_text" style="background-color:#FFFFFF;">london</span> <span id="Ax1.F13.1.1.1.186" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F13.1.1.1.187" class="ltx_text" style="background-color:#FEFBF0;">they</span> <span id="Ax1.F13.1.1.1.188" class="ltx_text" style="background-color:#FFFDF7;">were</span> <span id="Ax1.F13.1.1.1.189" class="ltx_text" style="background-color:#FFFFFF;">supposed</span> <span id="Ax1.F13.1.1.1.190" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F13.1.1.1.191" class="ltx_text" style="background-color:#FFFFFF;">be</span> <span id="Ax1.F13.1.1.1.192" class="ltx_text" style="background-color:#FFFFFF;">highlighting</span> <span id="Ax1.F13.1.1.1.193" class="ltx_text" style="background-color:#FFFEFD;">tory</span> <span id="Ax1.F13.1.1.1.194" class="ltx_text" style="background-color:#FFFFFF;">plans</span> <span id="Ax1.F13.1.1.1.195" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F13.1.1.1.196" class="ltx_text" style="background-color:#FFFFFF;">double</span> <span id="Ax1.F13.1.1.1.197" class="ltx_text" style="background-color:#FFFFFF;">free</span> <span id="Ax1.F13.1.1.1.198" class="ltx_text" style="background-color:#FFFFFF;">childcare</span> <span id="Ax1.F13.1.1.1.199" class="ltx_text" style="background-color:#FFFFFF;">for</span> <span id="Ax1.F13.1.1.1.200" class="ltx_text" style="background-color:#FFFFFF;">600,000</span> <span id="Ax1.F13.1.1.1.201" class="ltx_text" style="background-color:#FFFFFF;">children</span> <span id="Ax1.F13.1.1.1.202" class="ltx_text" style="background-color:#FFFFFF;">aged</span> <span id="Ax1.F13.1.1.1.203" class="ltx_text" style="background-color:#FFFFFF;">three</span> <span id="Ax1.F13.1.1.1.204" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F13.1.1.1.205" class="ltx_text" style="background-color:#FFFFFF;">four</span> <span id="Ax1.F13.1.1.1.206" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F13.1.1.1.207" class="ltx_text" style="background-color:#FFFCF4;">they</span> <span id="Ax1.F13.1.1.1.208" class="ltx_text" style="background-color:#FFFEFD;">sat</span> <span id="Ax1.F13.1.1.1.209" class="ltx_text" style="background-color:#FFFFFF;">at</span> <span id="Ax1.F13.1.1.1.210" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F13.1.1.1.211" class="ltx_text" style="background-color:#FFFFFF;">table</span> <span id="Ax1.F13.1.1.1.212" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F13.1.1.1.213" class="ltx_text" style="background-color:#FFFFFF;">joshua</span> <span id="Ax1.F13.1.1.1.214" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F13.1.1.1.215" class="ltx_text" style="background-color:#FFFFFF;">stephanie</span> <span id="Ax1.F13.1.1.1.216" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F13.1.1.1.217" class="ltx_text" style="background-color:#FFFFFF;">both</span> <span id="Ax1.F13.1.1.1.218" class="ltx_text" style="background-color:#FFFFFF;">aged</span> <span id="Ax1.F13.1.1.1.219" class="ltx_text" style="background-color:#FFFFFF;">three</span> <span id="Ax1.F13.1.1.1.220" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F13.1.1.1.221" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F13.1.1.1.222" class="ltx_text" style="background-color:#FFFFFF;">four-year-olds</span> <span id="Ax1.F13.1.1.1.223" class="ltx_text" style="background-color:#FFFFFF;">hamish</span> <span id="Ax1.F13.1.1.1.224" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F13.1.1.1.225" class="ltx_text" style="background-color:#FFFFFF;">leo</span> <span id="Ax1.F13.1.1.1.226" class="ltx_text" style="background-color:#FFFFFF;">where</span> <span id="Ax1.F13.1.1.1.227" class="ltx_text" style="background-color:#FFFFFF;">they</span> <span id="Ax1.F13.1.1.1.228" class="ltx_text" style="background-color:#FFFFFF;">broke</span> <span id="Ax1.F13.1.1.1.229" class="ltx_text" style="background-color:#FFFFFF;">up</span> <span id="Ax1.F13.1.1.1.230" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F13.1.1.1.231" class="ltx_text" style="background-color:#FFFFFF;">puzzle</span> <span id="Ax1.F13.1.1.1.232" class="ltx_text" style="background-color:#FFFFFF;">which</span> <span id="Ax1.F13.1.1.1.233" class="ltx_text" style="background-color:#FFFFFF;">depicted</span> <span id="Ax1.F13.1.1.1.234" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F13.1.1.1.235" class="ltx_text" style="background-color:#FFFFFF;">four</span> <span id="Ax1.F13.1.1.1.236" class="ltx_text" style="background-color:#FFFFFF;">seasons</span> <span id="Ax1.F13.1.1.1.237" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F13.1.1.1.238" class="ltx_text" style="background-color:#FFFFFE;">when</span> <span id="Ax1.F13.1.1.1.239" class="ltx_text" style="background-color:#FFFFFF;">they</span> <span id="Ax1.F13.1.1.1.240" class="ltx_text" style="background-color:#FFFFFE;">tried</span> <span id="Ax1.F13.1.1.1.241" class="ltx_text" style="background-color:#FFFFFE;">to</span> <span id="Ax1.F13.1.1.1.242" class="ltx_text" style="background-color:#FFFFFF;">put</span> <span id="Ax1.F13.1.1.1.243" class="ltx_text" style="background-color:#FFFFFF;">it</span> <span id="Ax1.F13.1.1.1.244" class="ltx_text" style="background-color:#FFFFFF;">back</span> <span id="Ax1.F13.1.1.1.245" class="ltx_text" style="background-color:#FFFFFF;">together</span> <span id="Ax1.F13.1.1.1.246" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F13.1.1.1.247" class="ltx_text" style="background-color:#FFFEFD;">they</span> <span id="Ax1.F13.1.1.1.248" class="ltx_text" style="background-color:#FFFDF8;">hit</span> <span id="Ax1.F13.1.1.1.249" class="ltx_text" style="background-color:#FEFBF0;">trouble</span> <span id="Ax1.F13.1.1.1.250" class="ltx_text" style="background-color:#FFFCF6;">after</span> <span id="Ax1.F13.1.1.1.251" class="ltx_text" style="background-color:#FEFAEF;">it</span> <span id="Ax1.F13.1.1.1.252" class="ltx_text" style="background-color:#FEF9E9;">proved</span> <span id="Ax1.F13.1.1.1.253" class="ltx_text" style="background-color:#FEF9EA;">to</span> <span id="Ax1.F13.1.1.1.254" class="ltx_text" style="background-color:#FEFBF0;">be</span> <span id="Ax1.F13.1.1.1.255" class="ltx_text" style="background-color:#FEF9EB;">more</span> <span id="Ax1.F13.1.1.1.256" class="ltx_text" style="background-color:#FEF9EB;">difficult</span> <span id="Ax1.F13.1.1.1.257" class="ltx_text" style="background-color:#FEF6E1;">than</span> <span id="Ax1.F13.1.1.1.258" class="ltx_text" style="background-color:#FEF6E1;">expected</span> <span id="Ax1.F13.1.1.1.259" class="ltx_text" style="background-color:#FEFAEF;">.</span>

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F13.1.1.1.260" class="ltx_text ltx_font_bold">Reference Summary:</span> 
<br class="ltx_break">david cameron and boris johnson visit nursery to promote childcare .
<br class="ltx_break">duo left <span id="Ax1.F13.1.1.1.261" class="ltx_text ltx_font_italic">flummoxed</span> by jigsaw aimed at teaching children about seasons .
<br class="ltx_break">london mayor resorted to brute force to hammer wrong pieces together .
<br class="ltx_break">prime minister gave his approval , telling boris : ’ if in doubt , wedge it in ’ .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F13.1.1.1.262" class="ltx_text ltx_font_bold">Baseline:</span> 
<br class="ltx_break">david cameron and boris johnson were left mystified by the puzzle . 
<br class="ltx_break">they were at the nursery in UNK to promote a tory pledge to double free childcare for <span id="Ax1.F13.1.1.1.263" class="ltx_text ltx_font_bold" style="color:#DB4437;">25</span> children aged three and four . 
<br class="ltx_break">the pair were at the nursery in UNK to promote a tory pledge to double free childcare for children aged three and four .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F13.1.1.1.264" class="ltx_text ltx_font_bold">Pointer-Generator, No Coverage:</span> 
<br class="ltx_break">london mayor tried to hammer ill-fitting pieces together with his hands . 
<br class="ltx_break">they were supposed to be highlighting tory plans to double free childcare . 
<br class="ltx_break">they were supposed to be highlighting tory plans to double free childcare . 
<br class="ltx_break">they were supposed to be highlighting tory plans to double free childcare .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F13.1.1.1.265" class="ltx_text ltx_font_bold">Pointer-Generator, With Coverage:</span> 
<br class="ltx_break"><span id="Ax1.F13.1.1.1.266" class="ltx_text" style="background-color:#E9F6EF;">london</span> <span id="Ax1.F13.1.1.1.267" class="ltx_text" style="background-color:#EFF8F4;">mayor</span> <span id="Ax1.F13.1.1.1.268" class="ltx_text" style="background-color:#B3E0CA;">tried</span> <span id="Ax1.F13.1.1.1.269" class="ltx_text" style="background-color:#DEF2E8;">to</span> <span id="Ax1.F13.1.1.1.270" class="ltx_text" style="background-color:#EFF8F4;">hammer</span> <span id="Ax1.F13.1.1.1.271" class="ltx_text" style="background-color:#EBF7F1;">ill-fitting</span> <span id="Ax1.F13.1.1.1.272" class="ltx_text" style="background-color:#F2FAF6;">pieces</span> <span id="Ax1.F13.1.1.1.273" class="ltx_text" style="background-color:#F4FAF7;">together</span> <span id="Ax1.F13.1.1.1.274" class="ltx_text" style="background-color:#ECF7F2;">with</span> <span id="Ax1.F13.1.1.1.275" class="ltx_text" style="background-color:#FAFDFB;">his</span> <span id="Ax1.F13.1.1.1.276" class="ltx_text" style="background-color:#FCFEFD;">hands</span> <span id="Ax1.F13.1.1.1.277" class="ltx_text" style="background-color:#AADDC4;">.</span>
<br class="ltx_break"><span id="Ax1.F13.1.1.1.278" class="ltx_text" style="background-color:#E0F2E9;">prime</span> <span id="Ax1.F13.1.1.1.279" class="ltx_text" style="background-color:#FBFDFC;">minister</span> <span id="Ax1.F13.1.1.1.280" class="ltx_text" style="background-color:#CDEBDC;">tried</span> <span id="Ax1.F13.1.1.1.281" class="ltx_text" style="background-color:#EEF8F3;">out</span> <span id="Ax1.F13.1.1.1.282" class="ltx_text" style="background-color:#F0F9F4;">what</span> <span id="Ax1.F13.1.1.1.283" class="ltx_text" style="background-color:#F4FBF8;">could</span> <span id="Ax1.F13.1.1.1.284" class="ltx_text" style="background-color:#F5FBF8;">be</span> <span id="Ax1.F13.1.1.1.285" class="ltx_text" style="background-color:#F8FCFA;">a</span> <span id="Ax1.F13.1.1.1.286" class="ltx_text" style="background-color:#FCFEFD;">new</span> <span id="Ax1.F13.1.1.1.287" class="ltx_text" style="background-color:#FDFEFE;">election</span> <span id="Ax1.F13.1.1.1.288" class="ltx_text" style="background-color:#F9FDFB;">slogan</span> <span id="Ax1.F13.1.1.1.289" class="ltx_text" style="background-color:#A7DBC2;">.</span>
<br class="ltx_break"><span id="Ax1.F13.1.1.1.290" class="ltx_text" style="background-color:#E5F4ED;">but</span> <span id="Ax1.F13.1.1.1.291" class="ltx_text" style="background-color:#D4EEE1;">they</span> <span id="Ax1.F13.1.1.1.292" class="ltx_text" style="background-color:#C2E6D4;">hit</span> <span id="Ax1.F13.1.1.1.293" class="ltx_text" style="background-color:#F5FBF8;">trouble</span> <span id="Ax1.F13.1.1.1.294" class="ltx_text" style="background-color:#C4E7D6;">after</span> <span id="Ax1.F13.1.1.1.295" class="ltx_text" style="background-color:#EAF7F0;">it</span> <span id="Ax1.F13.1.1.1.296" class="ltx_text" style="background-color:#E1F3EA;">proved</span> <span id="Ax1.F13.1.1.1.297" class="ltx_text" style="background-color:#EDF8F3;">to</span> <span id="Ax1.F13.1.1.1.298" class="ltx_text" style="background-color:#F5FBF8;">be</span> <span id="Ax1.F13.1.1.1.299" class="ltx_text" style="background-color:#F6FBF9;">more</span> <span id="Ax1.F13.1.1.1.300" class="ltx_text" style="background-color:#FBFDFC;">difficult</span> <span id="Ax1.F13.1.1.1.301" class="ltx_text" style="background-color:#ECF7F1;">than</span> <span id="Ax1.F13.1.1.1.302" class="ltx_text" style="background-color:#F7FCF9;">expected</span> <span id="Ax1.F13.1.1.1.303" class="ltx_text" style="background-color:#A8DBC2;">.</span>
<br class="ltx_break"></span>
</span></p>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 13: </span>The baseline model appropriately replaces <span id="Ax1.F13.8.1" class="ltx_text ltx_font_italic">stumped</span> with novel word <span id="Ax1.F13.9.2" class="ltx_text ltx_font_italic">mystified</span>.
However, the reference summary chooses <span id="Ax1.F13.10.3" class="ltx_text ltx_font_italic">flummoxed</span> (also novel) so the choice of <span id="Ax1.F13.11.4" class="ltx_text ltx_font_italic">mystified</span> is not rewarded by the ROUGE metric.
The baseline model also incorrectly substitutes <span id="Ax1.F13.12.5" class="ltx_text ltx_font_italic">600,000</span> for <span id="Ax1.F13.13.6" class="ltx_text ltx_font_italic">25</span>.
In the final model’s output we observe that the generation probability is largest at the beginning of sentences (especially the first verb) and on periods.
</figcaption>
</figure>
<figure id="Ax1.F14" class="ltx_figure">
<p id="Ax1.F14.1" class="ltx_p">
<span id="Ax1.F14.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:448.4pt;">
<span id="Ax1.F14.1.1.1" class="ltx_p"><span id="Ax1.F14.1.1.1.1" class="ltx_text ltx_font_bold">Article (truncated):</span> <span id="Ax1.F14.1.1.1.2" class="ltx_text" style="background-color:#FFFFFF;">lagos</span> <span id="Ax1.F14.1.1.1.3" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F14.1.1.1.4" class="ltx_text" style="background-color:#FFFEFC;">nigeria</span> <span id="Ax1.F14.1.1.1.5" class="ltx_text" style="background-color:#FFFFFF;">(</span> <span id="Ax1.F14.1.1.1.6" class="ltx_text" style="background-color:#FFFFFF;">cnn</span> <span id="Ax1.F14.1.1.1.7" class="ltx_text" style="background-color:#FFFFFF;">)</span> <span id="Ax1.F14.1.1.1.8" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F14.1.1.1.9" class="ltx_text" style="background-color:#FFFFFE;">day</span> <span id="Ax1.F14.1.1.1.10" class="ltx_text" style="background-color:#FFFFFF;">after</span> <span id="Ax1.F14.1.1.1.11" class="ltx_text" style="background-color:#FFFDFA;">winning</span> <span id="Ax1.F14.1.1.1.12" class="ltx_text" style="background-color:#FEFBF3;">nigeria</span> <span id="Ax1.F14.1.1.1.13" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F14.1.1.1.14" class="ltx_text" style="background-color:#FFFFFF;">presidency</span> <span id="Ax1.F14.1.1.1.15" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F14.1.1.1.16" class="ltx_text ltx_font_italic" style="background-color:#FDF2D1;">muhammadu</span> <span id="Ax1.F14.1.1.1.17" class="ltx_text ltx_font_italic" style="background-color:#FBE7AC;">buhari</span> <span id="Ax1.F14.1.1.1.18" class="ltx_text" style="background-color:#FCEDC1;">told</span> <span id="Ax1.F14.1.1.1.19" class="ltx_text" style="background-color:#FFFEFD;">cnn</span> <span id="Ax1.F14.1.1.1.20" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F14.1.1.1.21" class="ltx_text" style="background-color:#FFFFFE;">christiane</span> <span id="Ax1.F14.1.1.1.22" class="ltx_text" style="background-color:#FFFFFE;">amanpour</span> <span id="Ax1.F14.1.1.1.23" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F14.1.1.1.24" class="ltx_text" style="background-color:#FDEFCA;">he</span> <span id="Ax1.F14.1.1.1.25" class="ltx_text" style="background-color:#FCE9B5;">plans</span> <span id="Ax1.F14.1.1.1.26" class="ltx_text" style="background-color:#FBE7AC;">to</span> <span id="Ax1.F14.1.1.1.27" class="ltx_text" style="background-color:#FCECBF;">aggressively</span> <span id="Ax1.F14.1.1.1.28" class="ltx_text" style="background-color:#FBE39F;">fight</span> <span id="Ax1.F14.1.1.1.29" class="ltx_text" style="background-color:#FCE8AF;">corruption</span> <span id="Ax1.F14.1.1.1.30" class="ltx_text" style="background-color:#FCEEC5;">that</span> <span id="Ax1.F14.1.1.1.31" class="ltx_text" style="background-color:#FBE199;">has</span> <span id="Ax1.F14.1.1.1.32" class="ltx_text" style="background-color:#FBE7AD;">long</span> <span id="Ax1.F14.1.1.1.33" class="ltx_text" style="background-color:#FBE6AA;">plagued</span> <span id="Ax1.F14.1.1.1.34" class="ltx_text" style="background-color:#FBE5A7;">nigeria</span> <span id="Ax1.F14.1.1.1.35" class="ltx_text" style="background-color:#FCE8B1;">and</span> <span id="Ax1.F14.1.1.1.36" class="ltx_text" style="background-color:#FFFFFF;">go</span> <span id="Ax1.F14.1.1.1.37" class="ltx_text" style="background-color:#FFFFFF;">after</span> <span id="Ax1.F14.1.1.1.38" class="ltx_text" style="background-color:#FFFEFC;">the</span> <span id="Ax1.F14.1.1.1.39" class="ltx_text" style="background-color:#FFFFFE;">root</span> <span id="Ax1.F14.1.1.1.40" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F14.1.1.1.41" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F14.1.1.1.42" class="ltx_text" style="background-color:#FFFFFF;">nation</span> <span id="Ax1.F14.1.1.1.43" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F14.1.1.1.44" class="ltx_text" style="background-color:#FFFFFF;">unrest</span> <span id="Ax1.F14.1.1.1.45" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F14.1.1.1.46" class="ltx_text ltx_font_italic" style="background-color:#FEF5DE;">buhari</span> <span id="Ax1.F14.1.1.1.47" class="ltx_text" style="background-color:#FEFAEE;">said</span> <span id="Ax1.F14.1.1.1.48" class="ltx_text" style="background-color:#FEF6DF;">he</span> <span id="Ax1.F14.1.1.1.49" class="ltx_text" style="background-color:#FFFDF8;">’ll</span> <span id="Ax1.F14.1.1.1.50" class="ltx_text" style="background-color:#FFFEFD;">“</span> <span id="Ax1.F14.1.1.1.51" class="ltx_text" style="background-color:#FFFFFE;">rapidly</span> <span id="Ax1.F14.1.1.1.52" class="ltx_text" style="background-color:#FFFEFD;">give</span> <span id="Ax1.F14.1.1.1.53" class="ltx_text" style="background-color:#FFFFFF;">attention</span> <span id="Ax1.F14.1.1.1.54" class="ltx_text" style="background-color:#FFFFFF;">”</span> <span id="Ax1.F14.1.1.1.55" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F14.1.1.1.56" class="ltx_text" style="background-color:#FFFEFD;">curbing</span> <span id="Ax1.F14.1.1.1.57" class="ltx_text" style="background-color:#FFFEFD;">violence</span> <span id="Ax1.F14.1.1.1.58" class="ltx_text" style="background-color:#FFFEFA;">in</span> <span id="Ax1.F14.1.1.1.59" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F14.1.1.1.60" class="ltx_text" style="background-color:#FFFFFF;">northeast</span> <span id="Ax1.F14.1.1.1.61" class="ltx_text" style="background-color:#FFFFFF;">part</span> <span id="Ax1.F14.1.1.1.62" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F14.1.1.1.63" class="ltx_text" style="background-color:#FFFEFB;">nigeria</span> <span id="Ax1.F14.1.1.1.64" class="ltx_text" style="background-color:#FFFEFD;">,</span> <span id="Ax1.F14.1.1.1.65" class="ltx_text" style="background-color:#FFFFFF;">where</span> <span id="Ax1.F14.1.1.1.66" class="ltx_text" style="background-color:#FFFEFC;">the</span> <span id="Ax1.F14.1.1.1.67" class="ltx_text" style="background-color:#FFFEFD;">terrorist</span> <span id="Ax1.F14.1.1.1.68" class="ltx_text" style="background-color:#FFFFFF;">group</span> <span id="Ax1.F14.1.1.1.69" class="ltx_text" style="background-color:#FFFEFB;">boko</span> <span id="Ax1.F14.1.1.1.70" class="ltx_text" style="background-color:#FFFFFF;">haram</span> <span id="Ax1.F14.1.1.1.71" class="ltx_text" style="background-color:#FFFFFF;">operates</span> <span id="Ax1.F14.1.1.1.72" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F14.1.1.1.73" class="ltx_text" style="background-color:#FFFFFE;">by</span> <span id="Ax1.F14.1.1.1.74" class="ltx_text" style="background-color:#FFFFFF;">cooperating</span> <span id="Ax1.F14.1.1.1.75" class="ltx_text" style="background-color:#FFFFFF;">with</span> <span id="Ax1.F14.1.1.1.76" class="ltx_text" style="background-color:#FFFFFF;">neighboring</span> <span id="Ax1.F14.1.1.1.77" class="ltx_text" style="background-color:#FFFFFF;">nations</span> <span id="Ax1.F14.1.1.1.78" class="ltx_text" style="background-color:#FFFFFF;">chad</span> <span id="Ax1.F14.1.1.1.79" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F14.1.1.1.80" class="ltx_text" style="background-color:#FFFFFE;">cameroon</span> <span id="Ax1.F14.1.1.1.81" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F14.1.1.1.82" class="ltx_text" style="background-color:#FFFFFF;">niger</span> <span id="Ax1.F14.1.1.1.83" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F14.1.1.1.84" class="ltx_text" style="background-color:#FEFBF2;">he</span> <span id="Ax1.F14.1.1.1.85" class="ltx_text" style="background-color:#FEF6DF;">said</span> <span id="Ax1.F14.1.1.1.86" class="ltx_text" style="background-color:#FCEDC0;">his</span> <span id="Ax1.F14.1.1.1.87" class="ltx_text" style="background-color:#FBE6AA;">administration</span> <span id="Ax1.F14.1.1.1.88" class="ltx_text" style="background-color:#FBE7AD;">is</span> <span id="Ax1.F14.1.1.1.89" class="ltx_text" style="background-color:#FBE6AA;">confident</span> <span id="Ax1.F14.1.1.1.90" class="ltx_text" style="background-color:#FBE7AD;">it</span> <span id="Ax1.F14.1.1.1.91" class="ltx_text" style="background-color:#FBE6AB;">will</span> <span id="Ax1.F14.1.1.1.92" class="ltx_text" style="background-color:#FBE7AC;">be</span> <span id="Ax1.F14.1.1.1.93" class="ltx_text" style="background-color:#FBE7AC;">able</span> <span id="Ax1.F14.1.1.1.94" class="ltx_text" style="background-color:#FBE7AB;">to</span> <span id="Ax1.F14.1.1.1.95" class="ltx_text" style="background-color:#FBE7AC;">thwart</span> <span id="Ax1.F14.1.1.1.96" class="ltx_text" style="background-color:#FBE7AD;">criminals</span> <span id="Ax1.F14.1.1.1.97" class="ltx_text" style="background-color:#FBE7AD;">and</span> <span id="Ax1.F14.1.1.1.98" class="ltx_text" style="background-color:#FFFFFF;">others</span> <span id="Ax1.F14.1.1.1.99" class="ltx_text" style="background-color:#FFFFFE;">contributing</span> <span id="Ax1.F14.1.1.1.100" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F14.1.1.1.101" class="ltx_text" style="background-color:#FFFEFB;">nigeria</span> <span id="Ax1.F14.1.1.1.102" class="ltx_text" style="background-color:#FFFEFD;">’s</span> <span id="Ax1.F14.1.1.1.103" class="ltx_text" style="background-color:#FFFFFF;">instability</span> <span id="Ax1.F14.1.1.1.104" class="ltx_text" style="background-color:#FEFBF0;">.</span> <span id="Ax1.F14.1.1.1.105" class="ltx_text" style="background-color:#FFFDFA;">for</span> <span id="Ax1.F14.1.1.1.106" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F14.1.1.1.107" class="ltx_text" style="background-color:#FFFFFF;">first</span> <span id="Ax1.F14.1.1.1.108" class="ltx_text" style="background-color:#FFFFFF;">time</span> <span id="Ax1.F14.1.1.1.109" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F14.1.1.1.110" class="ltx_text" style="background-color:#FFFEFC;">nigeria</span> <span id="Ax1.F14.1.1.1.111" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F14.1.1.1.112" class="ltx_text" style="background-color:#FFFFFF;">history</span> <span id="Ax1.F14.1.1.1.113" class="ltx_text" style="background-color:#FFFFFE;">,</span> <span id="Ax1.F14.1.1.1.114" class="ltx_text" style="background-color:#FEFBF1;">the</span> <span id="Ax1.F14.1.1.1.115" class="ltx_text" style="background-color:#FDF2D2;">opposition</span> <span id="Ax1.F14.1.1.1.116" class="ltx_text" style="background-color:#FFFFFF;">defeated</span> <span id="Ax1.F14.1.1.1.117" class="ltx_text" style="background-color:#FFFFFE;">the</span> <span id="Ax1.F14.1.1.1.118" class="ltx_text" style="background-color:#FFFFFE;">ruling</span> <span id="Ax1.F14.1.1.1.119" class="ltx_text" style="background-color:#FFFFFF;">party</span> <span id="Ax1.F14.1.1.1.120" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F14.1.1.1.121" class="ltx_text" style="background-color:#FFFFFE;">democratic</span> <span id="Ax1.F14.1.1.1.122" class="ltx_text" style="background-color:#FFFFFF;">elections</span> <span id="Ax1.F14.1.1.1.123" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F14.1.1.1.124" class="ltx_text ltx_font_italic" style="background-color:#FEFBF0;">buhari</span> <span id="Ax1.F14.1.1.1.125" class="ltx_text" style="background-color:#FFFCF6;">defeated</span> <span id="Ax1.F14.1.1.1.126" class="ltx_text" style="background-color:#FFFDFA;">incumbent</span> <span id="Ax1.F14.1.1.1.127" class="ltx_text" style="background-color:#FFFEFC;">goodluck</span> <span id="Ax1.F14.1.1.1.128" class="ltx_text" style="background-color:#FFFFFE;">jonathan</span> <span id="Ax1.F14.1.1.1.129" class="ltx_text" style="background-color:#FFFFFF;">by</span> <span id="Ax1.F14.1.1.1.130" class="ltx_text" style="background-color:#FFFFFE;">about</span> <span id="Ax1.F14.1.1.1.131" class="ltx_text" style="background-color:#FFFFFF;">2</span> <span id="Ax1.F14.1.1.1.132" class="ltx_text" style="background-color:#FFFFFF;">million</span> <span id="Ax1.F14.1.1.1.133" class="ltx_text" style="background-color:#FFFFFF;">votes</span> <span id="Ax1.F14.1.1.1.134" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F14.1.1.1.135" class="ltx_text" style="background-color:#FFFFFE;">according</span> <span id="Ax1.F14.1.1.1.136" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F14.1.1.1.137" class="ltx_text" style="background-color:#FFFEFB;">nigeria</span> <span id="Ax1.F14.1.1.1.138" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F14.1.1.1.139" class="ltx_text" style="background-color:#FFFFFF;">independent</span> <span id="Ax1.F14.1.1.1.140" class="ltx_text" style="background-color:#FFFFFF;">national</span> <span id="Ax1.F14.1.1.1.141" class="ltx_text" style="background-color:#FFFFFF;">electoral</span> <span id="Ax1.F14.1.1.1.142" class="ltx_text" style="background-color:#FFFFFF;">commission</span> <span id="Ax1.F14.1.1.1.143" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F14.1.1.1.144" class="ltx_text" style="background-color:#FEFAF0;">the</span> <span id="Ax1.F14.1.1.1.145" class="ltx_text" style="background-color:#FDF0CB;">win</span> <span id="Ax1.F14.1.1.1.146" class="ltx_text" style="background-color:#FBE6AB;">comes</span> <span id="Ax1.F14.1.1.1.147" class="ltx_text" style="background-color:#FBE7AC;">after</span> <span id="Ax1.F14.1.1.1.148" class="ltx_text" style="background-color:#FDEFCA;">a</span> <span id="Ax1.F14.1.1.1.149" class="ltx_text" style="background-color:#FCE9B5;">long</span> <span id="Ax1.F14.1.1.1.150" class="ltx_text" style="background-color:#FBE6AB;">history</span> <span id="Ax1.F14.1.1.1.151" class="ltx_text" style="background-color:#FBE7AC;">of</span> <span id="Ax1.F14.1.1.1.152" class="ltx_text" style="background-color:#FBE7AD;">military</span> <span id="Ax1.F14.1.1.1.153" class="ltx_text" style="background-color:#FBE6AB;">rule</span> <span id="Ax1.F14.1.1.1.154" class="ltx_text" style="background-color:#FBE7AD;">,</span> <span id="Ax1.F14.1.1.1.155" class="ltx_text" style="background-color:#FCE8AF;">coups</span> <span id="Ax1.F14.1.1.1.156" class="ltx_text" style="background-color:#FBE7AC;">and</span> <span id="Ax1.F14.1.1.1.157" class="ltx_text" style="background-color:#FBE7AB;">botched</span> <span id="Ax1.F14.1.1.1.158" class="ltx_text" style="background-color:#FBE6AB;">attempts</span> <span id="Ax1.F14.1.1.1.159" class="ltx_text" style="background-color:#FBE6AB;">at</span> <span id="Ax1.F14.1.1.1.160" class="ltx_text" style="background-color:#FBE7AB;">democracy</span> <span id="Ax1.F14.1.1.1.161" class="ltx_text" style="background-color:#FCEBBB;">in</span> <span id="Ax1.F14.1.1.1.162" class="ltx_text" style="background-color:#FCE8B0;">africa</span> <span id="Ax1.F14.1.1.1.163" class="ltx_text" style="background-color:#FBE7AC;">’s</span> <span id="Ax1.F14.1.1.1.164" class="ltx_text" style="background-color:#FBE7AC;">most</span> <span id="Ax1.F14.1.1.1.165" class="ltx_text" style="background-color:#FBE7AB;">populous</span> <span id="Ax1.F14.1.1.1.166" class="ltx_text" style="background-color:#FBE7AC;">nation</span> <span id="Ax1.F14.1.1.1.167" class="ltx_text" style="background-color:#FBE7AC;">.</span>

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F14.1.1.1.168" class="ltx_text ltx_font_bold">Reference Summary:</span> 
<br class="ltx_break"><span id="Ax1.F14.1.1.1.169" class="ltx_text ltx_font_italic">muhammadu</span> <span id="Ax1.F14.1.1.1.170" class="ltx_text ltx_font_italic">buhari</span> tells cnn ’s christiane amanpour that he will fight corruption in nigeria .
<br class="ltx_break">nigeria is the most populous country in africa and is grappling with violent boko haram extremists .
<br class="ltx_break">nigeria is also africa ’s biggest economy , but up to 70 % of nigerians live on less than a dollar a day .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F14.1.1.1.171" class="ltx_text ltx_font_bold">Baseline:</span> 
<br class="ltx_break">UNK UNK says his administration is confident it will be able to <span id="Ax1.F14.1.1.1.172" class="ltx_text ltx_font_bold" style="color:#DB4437;">destabilize nigeria ’s economy</span> . 
<br class="ltx_break">UNK says his administration is confident it will be able to thwart criminals and other <span id="Ax1.F14.1.1.1.173" class="ltx_text ltx_font_bold" style="color:#DB4437;">nigerians</span> . 
<br class="ltx_break"><span id="Ax1.F14.1.1.1.174" class="ltx_text ltx_font_bold" style="color:#DB4437;">he says the country has long nigeria and nigeria ’s economy .</span>

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F14.1.1.1.175" class="ltx_text ltx_font_bold">Pointer-Generator, No Coverage:</span> 
<br class="ltx_break"><span id="Ax1.F14.1.1.1.176" class="ltx_text ltx_font_italic">muhammadu</span> <span id="Ax1.F14.1.1.1.177" class="ltx_text ltx_font_italic">buhari</span> says he plans to aggressively fight corruption in the northeast part of nigeria . 
<br class="ltx_break">he says he ’ll “ rapidly give attention ” to curbing violence in the northeast part of nigeria . 
<br class="ltx_break">he says his administration is confident it will be able to thwart criminals .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F14.1.1.1.178" class="ltx_text ltx_font_bold">Pointer-Generator, With Coverage:</span> 
<br class="ltx_break"><span id="Ax1.F14.1.1.1.179" class="ltx_text ltx_font_italic" style="background-color:#E3F4EB;">muhammadu</span> <span id="Ax1.F14.1.1.1.180" class="ltx_text ltx_font_italic" style="background-color:#EFF8F4;">buhari</span> <span id="Ax1.F14.1.1.1.181" class="ltx_text" style="background-color:#A5DAC0;">says</span> <span id="Ax1.F14.1.1.1.182" class="ltx_text" style="background-color:#EBF7F1;">he</span> <span id="Ax1.F14.1.1.1.183" class="ltx_text" style="background-color:#D4EEE1;">plans</span> <span id="Ax1.F14.1.1.1.184" class="ltx_text" style="background-color:#CAE9DA;">to</span> <span id="Ax1.F14.1.1.1.185" class="ltx_text" style="background-color:#E8F6EF;">aggressively</span> <span id="Ax1.F14.1.1.1.186" class="ltx_text" style="background-color:#EDF8F2;">fight</span> <span id="Ax1.F14.1.1.1.187" class="ltx_text" style="background-color:#F7FCF9;">corruption</span> <span id="Ax1.F14.1.1.1.188" class="ltx_text" style="background-color:#D1ECDF;">that</span> <span id="Ax1.F14.1.1.1.189" class="ltx_text" style="background-color:#E7F5EE;">has</span> <span id="Ax1.F14.1.1.1.190" class="ltx_text" style="background-color:#EFF9F4;">long</span> <span id="Ax1.F14.1.1.1.191" class="ltx_text" style="background-color:#F7FCF9;">plagued</span> <span id="Ax1.F14.1.1.1.192" class="ltx_text" style="background-color:#F5FBF8;">nigeria</span> <span id="Ax1.F14.1.1.1.193" class="ltx_text" style="background-color:#B1DFC9;">.</span>
<br class="ltx_break"><span id="Ax1.F14.1.1.1.194" class="ltx_text" style="background-color:#DEF1E8;">he</span> <span id="Ax1.F14.1.1.1.195" class="ltx_text" style="background-color:#ACDDC5;">says</span> <span id="Ax1.F14.1.1.1.196" class="ltx_text" style="background-color:#ECF7F2;">his</span> <span id="Ax1.F14.1.1.1.197" class="ltx_text" style="background-color:#F8FCFA;">administration</span> <span id="Ax1.F14.1.1.1.198" class="ltx_text" style="background-color:#E8F6EF;">is</span> <span id="Ax1.F14.1.1.1.199" class="ltx_text" style="background-color:#F7FCFA;">confident</span> <span id="Ax1.F14.1.1.1.200" class="ltx_text" style="background-color:#EBF7F1;">it</span> <span id="Ax1.F14.1.1.1.201" class="ltx_text" style="background-color:#F3FAF7;">will</span> <span id="Ax1.F14.1.1.1.202" class="ltx_text" style="background-color:#F7FCF9;">be</span> <span id="Ax1.F14.1.1.1.203" class="ltx_text" style="background-color:#F8FCFA;">able</span> <span id="Ax1.F14.1.1.1.204" class="ltx_text" style="background-color:#E8F6EF;">to</span> <span id="Ax1.F14.1.1.1.205" class="ltx_text" style="background-color:#F1F9F5;">thwart</span> <span id="Ax1.F14.1.1.1.206" class="ltx_text" style="background-color:#F7FCF9;">criminals</span> <span id="Ax1.F14.1.1.1.207" class="ltx_text" style="background-color:#C2E6D5;">.</span>
<br class="ltx_break"><span id="Ax1.F14.1.1.1.208" class="ltx_text" style="background-color:#D0ECDE;">the</span> <span id="Ax1.F14.1.1.1.209" class="ltx_text" style="background-color:#EFF9F4;">win</span> <span id="Ax1.F14.1.1.1.210" class="ltx_text" style="background-color:#CBEADB;">comes</span> <span id="Ax1.F14.1.1.1.211" class="ltx_text" style="background-color:#E4F4EC;">after</span> <span id="Ax1.F14.1.1.1.212" class="ltx_text" style="background-color:#F8FCFA;">a</span> <span id="Ax1.F14.1.1.1.213" class="ltx_text" style="background-color:#FCFEFD;">long</span> <span id="Ax1.F14.1.1.1.214" class="ltx_text" style="background-color:#FCFEFD;">history</span> <span id="Ax1.F14.1.1.1.215" class="ltx_text" style="background-color:#F4FAF7;">of</span> <span id="Ax1.F14.1.1.1.216" class="ltx_text" style="background-color:#FAFDFC;">military</span> <span id="Ax1.F14.1.1.1.217" class="ltx_text" style="background-color:#FDFEFD;">rule</span> <span id="Ax1.F14.1.1.1.218" class="ltx_text" style="background-color:#E7F5EE;">,</span> <span id="Ax1.F14.1.1.1.219" class="ltx_text" style="background-color:#FCFEFD;">coups</span> <span id="Ax1.F14.1.1.1.220" class="ltx_text" style="background-color:#D5EEE2;">and</span> <span id="Ax1.F14.1.1.1.221" class="ltx_text" style="background-color:#F7FCF9;">botched</span> <span id="Ax1.F14.1.1.1.222" class="ltx_text" style="background-color:#F0F9F4;">attempts</span> <span id="Ax1.F14.1.1.1.223" class="ltx_text" style="background-color:#DBF1E6;">at</span> <span id="Ax1.F14.1.1.1.224" class="ltx_text" style="background-color:#F9FDFB;">democracy</span> <span id="Ax1.F14.1.1.1.225" class="ltx_text" style="background-color:#C7E8D8;">in</span> <span id="Ax1.F14.1.1.1.226" class="ltx_text" style="background-color:#F7FCF9;">africa</span> <span id="Ax1.F14.1.1.1.227" class="ltx_text" style="background-color:#D4EEE1;">’s</span> <span id="Ax1.F14.1.1.1.228" class="ltx_text" style="background-color:#FAFDFB;">most</span> <span id="Ax1.F14.1.1.1.229" class="ltx_text" style="background-color:#FCFEFD;">populous</span> <span id="Ax1.F14.1.1.1.230" class="ltx_text" style="background-color:#FBFDFC;">nation</span> <span id="Ax1.F14.1.1.1.231" class="ltx_text" style="background-color:#B0DFC8;">.</span>
<br class="ltx_break"></span>
</span></p>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 14: </span>The baseline model incorrectly changes <span id="Ax1.F14.6.1" class="ltx_text ltx_font_italic">thwart criminals and others contributing to nigeria’s instability</span> to <span id="Ax1.F14.7.2" class="ltx_text ltx_font_italic">destabilize nigeria’s economy</span> – which has a mostly opposite meaning. It also produces a nonsensical sentence.
Note that our final model produces the novel word <span id="Ax1.F14.8.3" class="ltx_text ltx_font_italic">says</span> to paraphrase <span id="Ax1.F14.9.4" class="ltx_text ltx_font_italic">told cnn ‘s christiane amanpour</span>.
</figcaption>
</figure>
<figure id="Ax1.F15" class="ltx_figure">
<p id="Ax1.F15.1" class="ltx_p">
<span id="Ax1.F15.1.1" class="ltx_inline-block ltx_minipage ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:448.4pt;">
<span id="Ax1.F15.1.1.1" class="ltx_p"><span id="Ax1.F15.1.1.1.1" class="ltx_text ltx_font_bold">Article:</span> <span id="Ax1.F15.1.1.1.2" class="ltx_text" style="background-color:#FFFFFF;">cairo</span> <span id="Ax1.F15.1.1.1.3" class="ltx_text" style="background-color:#FFFFFF;">(</span> <span id="Ax1.F15.1.1.1.4" class="ltx_text" style="background-color:#FFFFFF;">cnn</span> <span id="Ax1.F15.1.1.1.5" class="ltx_text" style="background-color:#FFFFFF;">)</span> <span id="Ax1.F15.1.1.1.6" class="ltx_text" style="background-color:#FFFEFB;">at</span> <span id="Ax1.F15.1.1.1.7" class="ltx_text" style="background-color:#FFFFFF;">least</span> <span id="Ax1.F15.1.1.1.8" class="ltx_text" style="background-color:#FFFEFD;">12</span> <span id="Ax1.F15.1.1.1.9" class="ltx_text" style="background-color:#FFFFFF;">people</span> <span id="Ax1.F15.1.1.1.10" class="ltx_text" style="background-color:#FEF9EC;">were</span> <span id="Ax1.F15.1.1.1.11" class="ltx_text" style="background-color:#FEF9EB;">killed</span> <span id="Ax1.F15.1.1.1.12" class="ltx_text" style="background-color:#FFFCF5;">sunday</span> <span id="Ax1.F15.1.1.1.13" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.14" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F15.1.1.1.15" class="ltx_text" style="background-color:#FFFEFD;">more</span> <span id="Ax1.F15.1.1.1.16" class="ltx_text" style="background-color:#FFFFFE;">injured</span> <span id="Ax1.F15.1.1.1.17" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.18" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F15.1.1.1.19" class="ltx_text" style="background-color:#FFFFFF;">separate</span> <span id="Ax1.F15.1.1.1.20" class="ltx_text" style="background-color:#FFFFFE;">attacks</span> <span id="Ax1.F15.1.1.1.21" class="ltx_text" style="background-color:#FFFFFF;">on</span> <span id="Ax1.F15.1.1.1.22" class="ltx_text" style="background-color:#FFFFFE;">a</span> <span id="Ax1.F15.1.1.1.23" class="ltx_text" style="background-color:#FFFFFD;">police</span> <span id="Ax1.F15.1.1.1.24" class="ltx_text" style="background-color:#FFFDFA;">station</span> <span id="Ax1.F15.1.1.1.25" class="ltx_text" style="background-color:#FEF8E8;">,</span> <span id="Ax1.F15.1.1.1.26" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F15.1.1.1.27" class="ltx_text" style="background-color:#FFFFFE;">checkpoint</span> <span id="Ax1.F15.1.1.1.28" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F15.1.1.1.29" class="ltx_text" style="background-color:#FFFFFF;">along</span> <span id="Ax1.F15.1.1.1.30" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F15.1.1.1.31" class="ltx_text" style="background-color:#FFFFFE;">highway</span> <span id="Ax1.F15.1.1.1.32" class="ltx_text" style="background-color:#FFFFFE;">in</span> <span id="Ax1.F15.1.1.1.33" class="ltx_text" style="background-color:#FFFEFD;">egypt</span> <span id="Ax1.F15.1.1.1.34" class="ltx_text" style="background-color:#FFFFFF;">’s</span> <span id="Ax1.F15.1.1.1.35" class="ltx_text" style="background-color:#FFFFFF;">northern</span> <span id="Ax1.F15.1.1.1.36" class="ltx_text" style="background-color:#FFFFFF;">sinai</span> <span id="Ax1.F15.1.1.1.37" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.38" class="ltx_text" style="background-color:#FEF9EC;">authorities</span> <span id="Ax1.F15.1.1.1.39" class="ltx_text" style="background-color:#FFFFFF;">said</span> <span id="Ax1.F15.1.1.1.40" class="ltx_text" style="background-color:#FFFEFD;">.</span> <span id="Ax1.F15.1.1.1.41" class="ltx_text" style="background-color:#FDEEC6;">six</span> <span id="Ax1.F15.1.1.1.42" class="ltx_text" style="background-color:#FCE8B1;">people</span> <span id="Ax1.F15.1.1.1.43" class="ltx_text" style="background-color:#FDF4D9;">,</span> <span id="Ax1.F15.1.1.1.44" class="ltx_text" style="background-color:#FCE9B3;">including</span> <span id="Ax1.F15.1.1.1.45" class="ltx_text" style="background-color:#FCE7AE;">one</span> <span id="Ax1.F15.1.1.1.46" class="ltx_text" style="background-color:#FCE8B0;">civilian</span> <span id="Ax1.F15.1.1.1.47" class="ltx_text" style="background-color:#FCE8B0;">,</span> <span id="Ax1.F15.1.1.1.48" class="ltx_text" style="background-color:#FCE7AF;">were</span> <span id="Ax1.F15.1.1.1.49" class="ltx_text" style="background-color:#FCEBBB;">killed</span> <span id="Ax1.F15.1.1.1.50" class="ltx_text" style="background-color:#FCEBBC;">when</span> <span id="Ax1.F15.1.1.1.51" class="ltx_text" style="background-color:#FCE8B1;">a</span> <span id="Ax1.F15.1.1.1.52" class="ltx_text" style="background-color:#FDEEC6;">car</span> <span id="Ax1.F15.1.1.1.53" class="ltx_text" style="background-color:#FBE199;">bomb</span> <span id="Ax1.F15.1.1.1.54" class="ltx_text" style="background-color:#FCE8B0;">exploded</span> <span id="Ax1.F15.1.1.1.55" class="ltx_text" style="background-color:#FDEFC8;">near</span> <span id="Ax1.F15.1.1.1.56" class="ltx_text" style="background-color:#FCE9B3;">the</span> <span id="Ax1.F15.1.1.1.57" class="ltx_text" style="background-color:#FCE9B5;">police</span> <span id="Ax1.F15.1.1.1.58" class="ltx_text" style="background-color:#FCE9B4;">station</span> <span id="Ax1.F15.1.1.1.59" class="ltx_text" style="background-color:#FCECBD;">in</span> <span id="Ax1.F15.1.1.1.60" class="ltx_text ltx_font_italic" style="background-color:#FFFFFE;">al-arish</span> <span id="Ax1.F15.1.1.1.61" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.62" class="ltx_text" style="background-color:#FFFFFF;">capital</span> <span id="Ax1.F15.1.1.1.63" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F15.1.1.1.64" class="ltx_text" style="background-color:#FFFFFE;">north</span> <span id="Ax1.F15.1.1.1.65" class="ltx_text" style="background-color:#FFFFFF;">sinai</span> <span id="Ax1.F15.1.1.1.66" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.67" class="ltx_text" style="background-color:#FFFDF9;">health</span> <span id="Ax1.F15.1.1.1.68" class="ltx_text" style="background-color:#FFFFFF;">ministry</span> <span id="Ax1.F15.1.1.1.69" class="ltx_text" style="background-color:#FFFFFF;">spokesman</span> <span id="Ax1.F15.1.1.1.70" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">hossam</span> <span id="Ax1.F15.1.1.1.71" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">abdel-ghafar</span> <span id="Ax1.F15.1.1.1.72" class="ltx_text" style="background-color:#FFFFFF;">told</span> <span id="Ax1.F15.1.1.1.73" class="ltx_text" style="background-color:#FFFFFF;">ahram</span> <span id="Ax1.F15.1.1.1.74" class="ltx_text" style="background-color:#FFFFFF;">online</span> <span id="Ax1.F15.1.1.1.75" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F15.1.1.1.76" class="ltx_text" style="background-color:#FFFDF9;">he</span> <span id="Ax1.F15.1.1.1.77" class="ltx_text" style="background-color:#FFFFFE;">said</span> <span id="Ax1.F15.1.1.1.78" class="ltx_text" style="background-color:#FFFCF6;">40</span> <span id="Ax1.F15.1.1.1.79" class="ltx_text" style="background-color:#FFFFFF;">people</span> <span id="Ax1.F15.1.1.1.80" class="ltx_text" style="background-color:#FFFEFD;">were</span> <span id="Ax1.F15.1.1.1.81" class="ltx_text" style="background-color:#FFFEFC;">injured</span> <span id="Ax1.F15.1.1.1.82" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F15.1.1.1.83" class="ltx_text" style="background-color:#FEFAEF;">ansar</span> <span id="Ax1.F15.1.1.1.84" class="ltx_text" style="background-color:#FDEFC9;">beit</span> <span id="Ax1.F15.1.1.1.85" class="ltx_text ltx_font_italic" style="background-color:#FCECBD;">al-maqdis</span> <span id="Ax1.F15.1.1.1.86" class="ltx_text" style="background-color:#FCEDC2;">,</span> <span id="Ax1.F15.1.1.1.87" class="ltx_text" style="background-color:#FCE9B5;">an</span> <span id="Ax1.F15.1.1.1.88" class="ltx_text" style="background-color:#FBE5A8;">isis</span> <span id="Ax1.F15.1.1.1.89" class="ltx_text" style="background-color:#FCE8B0;">affiliate</span> <span id="Ax1.F15.1.1.1.90" class="ltx_text" style="background-color:#FCE8B1;">,</span> <span id="Ax1.F15.1.1.1.91" class="ltx_text" style="background-color:#FBE6A9;">claimed</span> <span id="Ax1.F15.1.1.1.92" class="ltx_text" style="background-color:#FCE8B1;">responsibility</span> <span id="Ax1.F15.1.1.1.93" class="ltx_text" style="background-color:#FCE8B1;">for</span> <span id="Ax1.F15.1.1.1.94" class="ltx_text" style="background-color:#FCE8B1;">the</span> <span id="Ax1.F15.1.1.1.95" class="ltx_text" style="background-color:#FCE9B3;">attack</span> <span id="Ax1.F15.1.1.1.96" class="ltx_text" style="background-color:#FCE8B0;">,</span> <span id="Ax1.F15.1.1.1.97" class="ltx_text" style="background-color:#FFFFFF;">which</span> <span id="Ax1.F15.1.1.1.98" class="ltx_text" style="background-color:#FFFFFF;">came</span> <span id="Ax1.F15.1.1.1.99" class="ltx_text" style="background-color:#FFFFFF;">hours</span> <span id="Ax1.F15.1.1.1.100" class="ltx_text" style="background-color:#FFFFFF;">after</span> <span id="Ax1.F15.1.1.1.101" class="ltx_text" style="background-color:#FFFFFF;">another</span> <span id="Ax1.F15.1.1.1.102" class="ltx_text" style="background-color:#FFFFFF;">operation</span> <span id="Ax1.F15.1.1.1.103" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F15.1.1.1.104" class="ltx_text" style="background-color:#FFFFFE;">the</span> <span id="Ax1.F15.1.1.1.105" class="ltx_text" style="background-color:#FFFFFF;">group</span> <span id="Ax1.F15.1.1.1.106" class="ltx_text" style="background-color:#FFFFFF;">also</span> <span id="Ax1.F15.1.1.1.107" class="ltx_text" style="background-color:#FFFFFF;">claimed</span> <span id="Ax1.F15.1.1.1.108" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F15.1.1.1.109" class="ltx_text" style="background-color:#FFFEFA;">in</span> <span id="Ax1.F15.1.1.1.110" class="ltx_text" style="background-color:#FFFFFF;">that</span> <span id="Ax1.F15.1.1.1.111" class="ltx_text" style="background-color:#FFFDF9;">earlier</span> <span id="Ax1.F15.1.1.1.112" class="ltx_text" style="background-color:#FFFFFF;">attack</span> <span id="Ax1.F15.1.1.1.113" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.114" class="ltx_text" style="background-color:#FFFEFC;">a</span> <span id="Ax1.F15.1.1.1.115" class="ltx_text" style="background-color:#FFFFFE;">first</span> <span id="Ax1.F15.1.1.1.116" class="ltx_text" style="background-color:#FFFFFF;">lieutenant</span> <span id="Ax1.F15.1.1.1.117" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.118" class="ltx_text" style="background-color:#FFFFFE;">a</span> <span id="Ax1.F15.1.1.1.119" class="ltx_text" style="background-color:#FFFFFE;">sergeant</span> <span id="Ax1.F15.1.1.1.120" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F15.1.1.1.121" class="ltx_text" style="background-color:#FFFFFF;">four</span> <span id="Ax1.F15.1.1.1.122" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">conscripts</span> <span id="Ax1.F15.1.1.1.123" class="ltx_text" style="background-color:#FFFFFF;">were</span> <span id="Ax1.F15.1.1.1.124" class="ltx_text" style="background-color:#FFFEFD;">killed</span> <span id="Ax1.F15.1.1.1.125" class="ltx_text" style="background-color:#FFFFFE;">when</span> <span id="Ax1.F15.1.1.1.126" class="ltx_text" style="background-color:#FFFFFE;">their</span> <span id="Ax1.F15.1.1.1.127" class="ltx_text" style="background-color:#FFFFFF;">armored</span> <span id="Ax1.F15.1.1.1.128" class="ltx_text" style="background-color:#FFFFFF;">vehicle</span> <span id="Ax1.F15.1.1.1.129" class="ltx_text" style="background-color:#FFFFFF;">was</span> <span id="Ax1.F15.1.1.1.130" class="ltx_text" style="background-color:#FFFFFF;">attacked</span> <span id="Ax1.F15.1.1.1.131" class="ltx_text" style="background-color:#FFFFFF;">on</span> <span id="Ax1.F15.1.1.1.132" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F15.1.1.1.133" class="ltx_text" style="background-color:#FFFFFF;">highway</span> <span id="Ax1.F15.1.1.1.134" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F15.1.1.1.135" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">al-arish</span> <span id="Ax1.F15.1.1.1.136" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F15.1.1.1.137" class="ltx_text" style="background-color:#FFFFFF;">sheikh</span> <span id="Ax1.F15.1.1.1.138" class="ltx_text ltx_font_italic" style="background-color:#FFFFFF;">zuweid</span> <span id="Ax1.F15.1.1.1.139" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F15.1.1.1.140" class="ltx_text" style="background-color:#FFFFFF;">northern</span> <span id="Ax1.F15.1.1.1.141" class="ltx_text" style="background-color:#FFFFFF;">sinai</span> <span id="Ax1.F15.1.1.1.142" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.143" class="ltx_text" style="background-color:#FFFFFE;">the</span> <span id="Ax1.F15.1.1.1.144" class="ltx_text" style="background-color:#FFFFFE;">military</span> <span id="Ax1.F15.1.1.1.145" class="ltx_text" style="background-color:#FFFFFF;">said</span> <span id="Ax1.F15.1.1.1.146" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F15.1.1.1.147" class="ltx_text" style="background-color:#FFFDF7;">two</span> <span id="Ax1.F15.1.1.1.148" class="ltx_text" style="background-color:#FFFFFF;">other</span> <span id="Ax1.F15.1.1.1.149" class="ltx_text" style="background-color:#FFFFFE;">soldiers</span> <span id="Ax1.F15.1.1.1.150" class="ltx_text" style="background-color:#FFFFFE;">were</span> <span id="Ax1.F15.1.1.1.151" class="ltx_text" style="background-color:#FFFFFE;">injured</span> <span id="Ax1.F15.1.1.1.152" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F15.1.1.1.153" class="ltx_text" style="background-color:#FFFFFF;">taken</span> <span id="Ax1.F15.1.1.1.154" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F15.1.1.1.155" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F15.1.1.1.156" class="ltx_text" style="background-color:#FFFFFF;">military</span> <span id="Ax1.F15.1.1.1.157" class="ltx_text" style="background-color:#FFFFFF;">hospital</span> <span id="Ax1.F15.1.1.1.158" class="ltx_text" style="background-color:#FFFDF9;">.</span> <span id="Ax1.F15.1.1.1.159" class="ltx_text" style="background-color:#FEF8E7;">ansar</span> <span id="Ax1.F15.1.1.1.160" class="ltx_text" style="background-color:#FEF8E7;">beit</span> <span id="Ax1.F15.1.1.1.161" class="ltx_text ltx_font_italic" style="background-color:#FEFBF2;">al-maqdis</span> <span id="Ax1.F15.1.1.1.162" class="ltx_text" style="background-color:#FFFDF8;">has</span> <span id="Ax1.F15.1.1.1.163" class="ltx_text" style="background-color:#FFFEFC;">claimed</span> <span id="Ax1.F15.1.1.1.164" class="ltx_text" style="background-color:#FFFFFE;">many</span> <span id="Ax1.F15.1.1.1.165" class="ltx_text" style="background-color:#FFFFFF;">attacks</span> <span id="Ax1.F15.1.1.1.166" class="ltx_text" style="background-color:#FFFFFF;">against</span> <span id="Ax1.F15.1.1.1.167" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F15.1.1.1.168" class="ltx_text" style="background-color:#FFFFFF;">army</span> <span id="Ax1.F15.1.1.1.169" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F15.1.1.1.170" class="ltx_text" style="background-color:#FFFFFF;">police</span> <span id="Ax1.F15.1.1.1.171" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F15.1.1.1.172" class="ltx_text" style="background-color:#FFFFFF;">sinai</span> <span id="Ax1.F15.1.1.1.173" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F15.1.1.1.174" class="ltx_text" style="background-color:#FFFDF7;">a</span> <span id="Ax1.F15.1.1.1.175" class="ltx_text" style="background-color:#FFFFFF;">third</span> <span id="Ax1.F15.1.1.1.176" class="ltx_text" style="background-color:#FFFFFF;">attack</span> <span id="Ax1.F15.1.1.1.177" class="ltx_text" style="background-color:#FFFFFE;">sunday</span> <span id="Ax1.F15.1.1.1.178" class="ltx_text" style="background-color:#FFFFFF;">on</span> <span id="Ax1.F15.1.1.1.179" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F15.1.1.1.180" class="ltx_text" style="background-color:#FFFFFF;">checkpoint</span> <span id="Ax1.F15.1.1.1.181" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F15.1.1.1.182" class="ltx_text" style="background-color:#FFFFFF;">rafah</span> <span id="Ax1.F15.1.1.1.183" class="ltx_text" style="background-color:#FFFFFF;">left</span> <span id="Ax1.F15.1.1.1.184" class="ltx_text" style="background-color:#FFFFFF;">three</span> <span id="Ax1.F15.1.1.1.185" class="ltx_text" style="background-color:#FFFFFF;">security</span> <span id="Ax1.F15.1.1.1.186" class="ltx_text" style="background-color:#FFFFFF;">personnel</span> <span id="Ax1.F15.1.1.1.187" class="ltx_text" style="background-color:#FFFFFF;">injured</span> <span id="Ax1.F15.1.1.1.188" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.189" class="ltx_text" style="background-color:#FFFFFF;">after</span> <span id="Ax1.F15.1.1.1.190" class="ltx_text" style="background-color:#FFFFFF;">unknown</span> <span id="Ax1.F15.1.1.1.191" class="ltx_text" style="background-color:#FFFFFF;">assailants</span> <span id="Ax1.F15.1.1.1.192" class="ltx_text" style="background-color:#FFFFFF;">opened</span> <span id="Ax1.F15.1.1.1.193" class="ltx_text" style="background-color:#FFFFFF;">fire</span> <span id="Ax1.F15.1.1.1.194" class="ltx_text" style="background-color:#FFFFFF;">at</span> <span id="Ax1.F15.1.1.1.195" class="ltx_text" style="background-color:#FFFFFF;">them</span> <span id="Ax1.F15.1.1.1.196" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.197" class="ltx_text" style="background-color:#FFFFFF;">according</span> <span id="Ax1.F15.1.1.1.198" class="ltx_text" style="background-color:#FFFFFF;">to</span> <span id="Ax1.F15.1.1.1.199" class="ltx_text" style="background-color:#FFFFFE;">state</span> <span id="Ax1.F15.1.1.1.200" class="ltx_text" style="background-color:#FFFFFF;">media</span> <span id="Ax1.F15.1.1.1.201" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F15.1.1.1.202" class="ltx_text" style="background-color:#FEFBF1;">the</span> <span id="Ax1.F15.1.1.1.203" class="ltx_text" style="background-color:#FFFDF9;">attacks</span> <span id="Ax1.F15.1.1.1.204" class="ltx_text" style="background-color:#FFFFFF;">come</span> <span id="Ax1.F15.1.1.1.205" class="ltx_text" style="background-color:#FFFFFF;">as</span> <span id="Ax1.F15.1.1.1.206" class="ltx_text" style="background-color:#FFFFFE;">the</span> <span id="Ax1.F15.1.1.1.207" class="ltx_text" style="background-color:#FFFFFF;">military</span> <span id="Ax1.F15.1.1.1.208" class="ltx_text" style="background-color:#FFFFFF;">announced</span> <span id="Ax1.F15.1.1.1.209" class="ltx_text" style="background-color:#FFFFFF;">a</span> <span id="Ax1.F15.1.1.1.210" class="ltx_text" style="background-color:#FFFFFF;">reshuffle</span> <span id="Ax1.F15.1.1.1.211" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F15.1.1.1.212" class="ltx_text" style="background-color:#FFFFFF;">several</span> <span id="Ax1.F15.1.1.1.213" class="ltx_text" style="background-color:#FFFFFF;">senior</span> <span id="Ax1.F15.1.1.1.214" class="ltx_text" style="background-color:#FFFFFF;">military</span> <span id="Ax1.F15.1.1.1.215" class="ltx_text" style="background-color:#FFFFFF;">positions</span> <span id="Ax1.F15.1.1.1.216" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.217" class="ltx_text" style="background-color:#FFFEFC;">state</span> <span id="Ax1.F15.1.1.1.218" class="ltx_text" style="background-color:#FFFFFF;">media</span> <span id="Ax1.F15.1.1.1.219" class="ltx_text" style="background-color:#FFFFFF;">reported</span> <span id="Ax1.F15.1.1.1.220" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F15.1.1.1.221" class="ltx_text" style="background-color:#FEFAEE;">among</span> <span id="Ax1.F15.1.1.1.222" class="ltx_text" style="background-color:#FFFFFF;">those</span> <span id="Ax1.F15.1.1.1.223" class="ltx_text" style="background-color:#FFFFFF;">being</span> <span id="Ax1.F15.1.1.1.224" class="ltx_text" style="background-color:#FFFFFF;">replaced</span> <span id="Ax1.F15.1.1.1.225" class="ltx_text" style="background-color:#FFFFFF;">are</span> <span id="Ax1.F15.1.1.1.226" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F15.1.1.1.227" class="ltx_text" style="background-color:#FFFFFF;">generals</span> <span id="Ax1.F15.1.1.1.228" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F15.1.1.1.229" class="ltx_text" style="background-color:#FFFFFF;">charge</span> <span id="Ax1.F15.1.1.1.230" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F15.1.1.1.231" class="ltx_text" style="background-color:#FFFFFF;">military</span> <span id="Ax1.F15.1.1.1.232" class="ltx_text" style="background-color:#FFFFFF;">intelligence</span> <span id="Ax1.F15.1.1.1.233" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F15.1.1.1.234" class="ltx_text" style="background-color:#FFFFFE;">egypt</span> <span id="Ax1.F15.1.1.1.235" class="ltx_text" style="background-color:#FFFEFC;">’s</span> <span id="Ax1.F15.1.1.1.236" class="ltx_text" style="background-color:#FFFFFE;">second</span> <span id="Ax1.F15.1.1.1.237" class="ltx_text" style="background-color:#FFFFFF;">field</span> <span id="Ax1.F15.1.1.1.238" class="ltx_text" style="background-color:#FFFFFF;">army</span> <span id="Ax1.F15.1.1.1.239" class="ltx_text" style="background-color:#FFFFFD;">,</span> <span id="Ax1.F15.1.1.1.240" class="ltx_text" style="background-color:#FFFFFF;">which</span> <span id="Ax1.F15.1.1.1.241" class="ltx_text" style="background-color:#FFFFFF;">is</span> <span id="Ax1.F15.1.1.1.242" class="ltx_text" style="background-color:#FFFFFF;">spearheading</span> <span id="Ax1.F15.1.1.1.243" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F15.1.1.1.244" class="ltx_text" style="background-color:#FFFFFF;">battle</span> <span id="Ax1.F15.1.1.1.245" class="ltx_text" style="background-color:#FFFFFF;">against</span> <span id="Ax1.F15.1.1.1.246" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F15.1.1.1.247" class="ltx_text" style="background-color:#FFFFFF;">insurgents</span> <span id="Ax1.F15.1.1.1.248" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F15.1.1.1.249" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F15.1.1.1.250" class="ltx_text" style="background-color:#FFFFFF;">northern</span> <span id="Ax1.F15.1.1.1.251" class="ltx_text" style="background-color:#FFFFFF;">sinai</span> <span id="Ax1.F15.1.1.1.252" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F15.1.1.1.253" class="ltx_text" style="background-color:#FEF7E5;">egypt</span> <span id="Ax1.F15.1.1.1.254" class="ltx_text" style="background-color:#FCE9B4;">’s</span> <span id="Ax1.F15.1.1.1.255" class="ltx_text" style="background-color:#FCE8B1;">army</span> <span id="Ax1.F15.1.1.1.256" class="ltx_text" style="background-color:#FCE9B2;">has</span> <span id="Ax1.F15.1.1.1.257" class="ltx_text" style="background-color:#FCE9B4;">been</span> <span id="Ax1.F15.1.1.1.258" class="ltx_text" style="background-color:#FCE8B0;">fighting</span> <span id="Ax1.F15.1.1.1.259" class="ltx_text" style="background-color:#FCE9B3;">a</span> <span id="Ax1.F15.1.1.1.260" class="ltx_text" style="background-color:#FCEBBB;">decade-long</span> <span id="Ax1.F15.1.1.1.261" class="ltx_text" style="background-color:#FBE6AB;">militant</span> <span id="Ax1.F15.1.1.1.262" class="ltx_text" style="background-color:#FBE6AB;">islamist</span> <span id="Ax1.F15.1.1.1.263" class="ltx_text" style="background-color:#FCE7AE;">insurgency</span> <span id="Ax1.F15.1.1.1.264" class="ltx_text" style="background-color:#FCE8B1;">,</span> <span id="Ax1.F15.1.1.1.265" class="ltx_text" style="background-color:#FFFFFF;">which</span> <span id="Ax1.F15.1.1.1.266" class="ltx_text" style="background-color:#FFFFFF;">has</span> <span id="Ax1.F15.1.1.1.267" class="ltx_text" style="background-color:#FFFEFB;">spiked</span> <span id="Ax1.F15.1.1.1.268" class="ltx_text" style="background-color:#FFFFFF;">since</span> <span id="Ax1.F15.1.1.1.269" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F15.1.1.1.270" class="ltx_text" style="background-color:#FFFFFF;">ouster</span> <span id="Ax1.F15.1.1.1.271" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F15.1.1.1.272" class="ltx_text" style="background-color:#FFFFFF;">muslim</span> <span id="Ax1.F15.1.1.1.273" class="ltx_text" style="background-color:#FFFFFF;">brotherhood</span> <span id="Ax1.F15.1.1.1.274" class="ltx_text" style="background-color:#FFFFFF;">president</span> <span id="Ax1.F15.1.1.1.275" class="ltx_text" style="background-color:#FFFFFF;">mohamed</span> <span id="Ax1.F15.1.1.1.276" class="ltx_text" style="background-color:#FFFFFF;">morsy</span> <span id="Ax1.F15.1.1.1.277" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F15.1.1.1.278" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F15.1.1.1.279" class="ltx_text" style="background-color:#FFFFFF;">summer</span> <span id="Ax1.F15.1.1.1.280" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F15.1.1.1.281" class="ltx_text" style="background-color:#FFFFFF;">2013</span> <span id="Ax1.F15.1.1.1.282" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F15.1.1.1.283" class="ltx_text" style="background-color:#FFFCF6;">hundreds</span> <span id="Ax1.F15.1.1.1.284" class="ltx_text" style="background-color:#FFFFFF;">of</span> <span id="Ax1.F15.1.1.1.285" class="ltx_text" style="background-color:#FFFFFF;">police</span> <span id="Ax1.F15.1.1.1.286" class="ltx_text" style="background-color:#FFFFFF;">and</span> <span id="Ax1.F15.1.1.1.287" class="ltx_text" style="background-color:#FFFFFF;">soldiers</span> <span id="Ax1.F15.1.1.1.288" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.289" class="ltx_text" style="background-color:#FFFFFF;">as</span> <span id="Ax1.F15.1.1.1.290" class="ltx_text" style="background-color:#FFFFFF;">well</span> <span id="Ax1.F15.1.1.1.291" class="ltx_text" style="background-color:#FFFFFF;">as</span> <span id="Ax1.F15.1.1.1.292" class="ltx_text" style="background-color:#FFFFFF;">civilians</span> <span id="Ax1.F15.1.1.1.293" class="ltx_text" style="background-color:#FFFFFF;">,</span> <span id="Ax1.F15.1.1.1.294" class="ltx_text" style="background-color:#FFFFFF;">have</span> <span id="Ax1.F15.1.1.1.295" class="ltx_text" style="background-color:#FFFFFF;">been</span> <span id="Ax1.F15.1.1.1.296" class="ltx_text" style="background-color:#FFFFFF;">killed</span> <span id="Ax1.F15.1.1.1.297" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F15.1.1.1.298" class="ltx_text" style="background-color:#FFFFFF;">militant</span> <span id="Ax1.F15.1.1.1.299" class="ltx_text" style="background-color:#FFFFFF;">attacks</span> <span id="Ax1.F15.1.1.1.300" class="ltx_text" style="background-color:#FFFFFF;">in</span> <span id="Ax1.F15.1.1.1.301" class="ltx_text" style="background-color:#FFFFFF;">the</span> <span id="Ax1.F15.1.1.1.302" class="ltx_text" style="background-color:#FFFFFF;">past</span> <span id="Ax1.F15.1.1.1.303" class="ltx_text" style="background-color:#FFFFFF;">months</span> <span id="Ax1.F15.1.1.1.304" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F15.1.1.1.305" class="ltx_text" style="background-color:#FDF5DB;">ian</span> <span id="Ax1.F15.1.1.1.306" class="ltx_text" style="background-color:#FFFFFF;">lee</span> <span id="Ax1.F15.1.1.1.307" class="ltx_text" style="background-color:#FFFFFF;">reported</span> <span id="Ax1.F15.1.1.1.308" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F15.1.1.1.309" class="ltx_text" style="background-color:#FFFFFF;">cairo</span> <span id="Ax1.F15.1.1.1.310" class="ltx_text" style="background-color:#FFFFFF;">.</span> <span id="Ax1.F15.1.1.1.311" class="ltx_text" style="background-color:#FFFEFB;">anas</span> <span id="Ax1.F15.1.1.1.312" class="ltx_text" style="background-color:#FFFFFF;">hamdan</span> <span id="Ax1.F15.1.1.1.313" class="ltx_text" style="background-color:#FFFFFF;">reported</span> <span id="Ax1.F15.1.1.1.314" class="ltx_text" style="background-color:#FFFFFF;">from</span> <span id="Ax1.F15.1.1.1.315" class="ltx_text" style="background-color:#FFFFFF;">atlanta</span> <span id="Ax1.F15.1.1.1.316" class="ltx_text" style="background-color:#FFFFFF;">.</span> 
<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span> <span id="Ax1.F15.1.1.1.317" class="ltx_text ltx_font_bold">Reference Summary:</span> 
<br class="ltx_break">six people , including one civilian , are killed when a car bomb explodes near a police station .
<br class="ltx_break">six others are killed when their armored vehicle is attacked on a highway in northern sinai .
<br class="ltx_break">ansar beit <span id="Ax1.F15.1.1.1.318" class="ltx_text ltx_font_italic">al-maqdis</span> , an isis affiliate , claims responsibility .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span> <span id="Ax1.F15.1.1.1.319" class="ltx_text ltx_font_bold">Baseline:</span> 
<br class="ltx_break">he says the attacks come after another operation that killed 12 . 
<br class="ltx_break">the attacks come as military announced a <span id="Ax1.F15.1.1.1.320" class="ltx_text ltx_font_bold" style="color:#DB4437;">u.n. peacekeeping force</span> in northern sinai .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F15.1.1.1.321" class="ltx_text ltx_font_bold">Pointer-Generator, No Coverage:</span> 
<br class="ltx_break">ansar beit <span id="Ax1.F15.1.1.1.322" class="ltx_text ltx_font_italic">al-maqdis</span> , an isis affiliate , claimed responsibility for the attack . 
<br class="ltx_break">ansar beit <span id="Ax1.F15.1.1.1.323" class="ltx_text ltx_font_italic">al-maqdis</span> , an isis affiliate , claimed responsibility for the attack . 
<br class="ltx_break">the attacks come as the military announced a reshuffle of several senior military positions .

<br class="ltx_break"><span class="ltx_rule" style="width:433.6pt;height:0.4pt;background:black;display:inline-block;"> </span>
<span id="Ax1.F15.1.1.1.324" class="ltx_text ltx_font_bold">Pointer-Generator, With Coverage:</span> 
<br class="ltx_break"><span id="Ax1.F15.1.1.1.325" class="ltx_text" style="background-color:#E2F3EA;">six</span> <span id="Ax1.F15.1.1.1.326" class="ltx_text" style="background-color:#F1F9F5;">people</span> <span id="Ax1.F15.1.1.1.327" class="ltx_text" style="background-color:#BDE4D1;">,</span> <span id="Ax1.F15.1.1.1.328" class="ltx_text" style="background-color:#DCF1E7;">including</span> <span id="Ax1.F15.1.1.1.329" class="ltx_text" style="background-color:#F7FCF9;">one</span> <span id="Ax1.F15.1.1.1.330" class="ltx_text" style="background-color:#FAFDFC;">civilian</span> <span id="Ax1.F15.1.1.1.331" class="ltx_text" style="background-color:#B2E0C9;">,</span> <span id="Ax1.F15.1.1.1.332" class="ltx_text" style="background-color:#D4EEE1;">were</span> <span id="Ax1.F15.1.1.1.333" class="ltx_text" style="background-color:#E8F6EF;">killed</span> <span id="Ax1.F15.1.1.1.334" class="ltx_text" style="background-color:#BDE4D1;">when</span> <span id="Ax1.F15.1.1.1.335" class="ltx_text" style="background-color:#EDF8F2;">a</span> <span id="Ax1.F15.1.1.1.336" class="ltx_text" style="background-color:#FDFEFE;">car</span> <span id="Ax1.F15.1.1.1.337" class="ltx_text" style="background-color:#F8FCFA;">bomb</span> <span id="Ax1.F15.1.1.1.338" class="ltx_text" style="background-color:#BFE5D3;">explodes</span> <span id="Ax1.F15.1.1.1.339" class="ltx_text" style="background-color:#BFE5D3;">near</span> <span id="Ax1.F15.1.1.1.340" class="ltx_text" style="background-color:#F7FCFA;">the</span> <span id="Ax1.F15.1.1.1.341" class="ltx_text" style="background-color:#FBFDFC;">police</span> <span id="Ax1.F15.1.1.1.342" class="ltx_text" style="background-color:#F6FBF9;">station</span> <span id="Ax1.F15.1.1.1.343" class="ltx_text" style="background-color:#B8E2CE;">.</span>
<br class="ltx_break"><span id="Ax1.F15.1.1.1.344" class="ltx_text" style="background-color:#ECF7F2;">ansar</span> <span id="Ax1.F15.1.1.1.345" class="ltx_text" style="background-color:#FEFFFE;">beit</span> <span id="Ax1.F15.1.1.1.346" class="ltx_text ltx_font_italic" style="background-color:#FDFEFE;">al-maqdis</span> <span id="Ax1.F15.1.1.1.347" class="ltx_text" style="background-color:#DAF0E5;">,</span> <span id="Ax1.F15.1.1.1.348" class="ltx_text" style="background-color:#EEF8F3;">an</span> <span id="Ax1.F15.1.1.1.349" class="ltx_text" style="background-color:#FEFFFE;">isis</span> <span id="Ax1.F15.1.1.1.350" class="ltx_text" style="background-color:#FEFFFF;">affiliate</span> <span id="Ax1.F15.1.1.1.351" class="ltx_text" style="background-color:#C8E9D9;">,</span> <span id="Ax1.F15.1.1.1.352" class="ltx_text" style="background-color:#EDF8F2;">claimed</span> <span id="Ax1.F15.1.1.1.353" class="ltx_text" style="background-color:#FAFDFB;">responsibility</span> <span id="Ax1.F15.1.1.1.354" class="ltx_text" style="background-color:#D3EDE1;">for</span> <span id="Ax1.F15.1.1.1.355" class="ltx_text" style="background-color:#FCFEFD;">the</span> <span id="Ax1.F15.1.1.1.356" class="ltx_text" style="background-color:#FAFDFC;">attack</span> <span id="Ax1.F15.1.1.1.357" class="ltx_text" style="background-color:#ABDDC4;">.</span>
<br class="ltx_break"><span id="Ax1.F15.1.1.1.358" class="ltx_text" style="background-color:#CCEADB;">egypt</span> <span id="Ax1.F15.1.1.1.359" class="ltx_text" style="background-color:#CEEBDD;">’s</span> <span id="Ax1.F15.1.1.1.360" class="ltx_text" style="background-color:#F5FBF8;">army</span> <span id="Ax1.F15.1.1.1.361" class="ltx_text" style="background-color:#D3EDE1;">has</span> <span id="Ax1.F15.1.1.1.362" class="ltx_text" style="background-color:#F8FCFA;">been</span> <span id="Ax1.F15.1.1.1.363" class="ltx_text" style="background-color:#F9FCFA;">fighting</span> <span id="Ax1.F15.1.1.1.364" class="ltx_text" style="background-color:#F8FCFA;">a</span> <span id="Ax1.F15.1.1.1.365" class="ltx_text" style="background-color:#FCFEFD;">decade-long</span> <span id="Ax1.F15.1.1.1.366" class="ltx_text" style="background-color:#FBFEFC;">militant</span> <span id="Ax1.F15.1.1.1.367" class="ltx_text" style="background-color:#FDFEFD;">islamist</span> <span id="Ax1.F15.1.1.1.368" class="ltx_text" style="background-color:#F6FBF9;">insurgency</span> <span id="Ax1.F15.1.1.1.369" class="ltx_text" style="background-color:#AEDEC6;">.</span>
<br class="ltx_break"></span>
</span></p>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 15: </span>The baseline model fabricates a completely false detail about <span id="Ax1.F15.5.1" class="ltx_text ltx_font_italic">a u.n. peacekeeping force</span> that is not mentioned in the article.
This is most likely inspired by a connection between U.N. peacekeeping forces and <span id="Ax1.F15.6.2" class="ltx_text ltx_font_italic">northern sinai</span> in the training data.
The pointer-generator model is more accurate, correctly reporting the <span id="Ax1.F15.7.3" class="ltx_text ltx_font_italic">reshuffle of several senior military positions</span>.
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/1704.04367" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/1704.04368" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+1704.04368">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/1704.04368" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/1704.04369" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Fri Mar  8 21:28:27 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

1704.04368
Neural sequence-to-sequence models have provided a viable new approach
forabstractivetext summarization (meaning they are not restricted to simply selecting and rearranging passages from the original text) .
However, these models have two shortcomings: they are liable to reproduce factual details inaccurately, and they tend to repeat themselves.
In this work we propose a novel architecture that augments the standard sequence-to-sequence attentional model in two orthogonal ways.
First, we use a hybrid pointer-generator network that can copy words from the source text viapointing, which aids accurate reproduction of information, while retaining the ability to produce novel words through thegenerator.
Second, we usecoverageto keep track of what has been summarized, which discourages repetition.
We apply our model to theCNN / Daily Mailsummarization
task, outperforming the current abstractive state-of-the-art
by at least 2 ROUGE points.
abstract
Get To The Point: Summarization with Pointer-Generator Networks
Abstract
1704.04368
1IntroductionSummarization is the task of condensing a piece of text to a shorter version that contains the main information from the original.
There are two broad approaches to summarization:extractiveandabstractive.Extractive methodsassemble summaries exclusively from
passages (usually whole sentences)  taken directly from the source
text, whileabstractive methodsmay generate novel words and
phrases not featured in the source text – as a human-written
abstract usually does.
The extractive approach is easier, because copying large chunks of text from the source document ensures baseline levels of grammaticality and accuracy.
On the other hand, sophisticated abilities that are crucial to high-quality summarization, such as paraphrasing, generalization, or the incorporation of real-world knowledge, are possible only in an abstractive framework (see Figure5) .Due to the difficulty of abstractive summarization, the great majority of past work has been extractive.
However, the recent success ofsequence-to-sequencemodels, in which recurrent neural networks (RNNs)  both read and freely generate text, has made abstractive summarization viable.
Though these systems are promising, they exhibit undesirable behavior such as inaccurately reproducing factual details, an inability to deal with out-of-vocabulary (OOV)  words, and repeating themselves (see Figure1) .In this paper we present an architecture that addresses these three
issues in the context of multi-sentence summaries.
While most recent abstractive work has focused on headline generation
tasks (reducing one or two sentences to a single headline) , we believe
that longer-text summarization is both more challenging (requiring
higher levels of abstraction while avoiding repetition)  and ultimately more useful.
Therefore we apply our model to the recently-introducedCNN/Daily Maildataset, which contains news articles (39 sentences on average)  paired with multi-sentence summaries, and show that we outperform the state-of-the-art abstractive system by at least 2 ROUGE points.Our hybridpointer-generatornetwork facilitates copying
words from the source text viapointing, which improves accuracy and handling of OOV words, while retaining the ability togeneratenew words.
The network, which can be viewed as a balance between extractive and abstractive approaches, is similar to’sCopyNet and’sForced-Attention Sentence Compression, that were applied to short-text summarization.
We propose a novel variant of thecoverage vectorfrom Neural Machine Translation, which we use to track and control coverage of the source document.
We show that coverage is remarkably effective for eliminating repetition.
introduction
Get To The Point: Summarization with Pointer-Generator Networks
1Introduction
1704.04368
2Our ModelsIn this section we describe (1)  our baseline sequence-to-sequence model, (2)  our pointer-generator model, and (3)  our coverage mechanism that can be added to either of the first two models. The code for our models is available online.111www.github.com/abisee/pointer-generatorFigure 3:Pointer-generator model.
For each decoder timestep a generation probabilityis calculated, which weights the probability ofgeneratingwords from the vocabulary, versuscopyingwords from the source text.
The vocabulary distribution and the attention distribution are weighted and summed to obtain the final distribution, from which we make our prediction.
Note that out-of-vocabulary article words such as2-0are included in the final distribution.
Best viewed in color.2.1Sequence-to-sequence attentional modelOur baseline model is similar to that of, and is depicted in Figure2.
The tokens of the articleare fed one-by-one into the encoder (a single-layer bidirectional LSTM) ,
producing a sequence ofencoder hidden states.
On each step, the decoder (a single-layer unidirectional LSTM)  receives the word embedding of the previous word (while training, this is the previous word of the reference summary; at test time it is the previous word emitted by the decoder) , and hasdecoder state.
Theattention distributionis calculated as in:(1) (2) where,,andare learnable parameters.
The attention distribution can be viewed as a probability distribution over the source words, that tells the decoder where to look to produce the next word.
Next, the attention distribution is used to produce a weighted sum of the encoder hidden states, known as thecontext vector:(3) The context vector, which can be seen as a fixed-size representation of what has been read from the source for this step, is concatenated with the decoder stateand fed through two linear layers to produce the vocabulary distribution:(4) where,,andare learnable parameters.is a probability distribution over all words in the vocabulary, and provides us with our final distribution from which to predict words:(5) During training, the loss for timestepis the negative log likelihood of the target wordfor that timestep:(6) and the overall loss for the whole sequence is:(7) 2.2Pointer-generator networkOur pointer-generator network is a hybrid between our baseline and a pointer network, as it allows both copying words via pointing, and generating words from a fixed vocabulary.
In the pointer-generator model (depicted in Figure3)  the attention distributionand context vectorare calculated as in section2.1.
In addition, thegeneration probabilityfor timestepis calculated from the context vector, the decoder stateand the decoder input:(8) where vectors,,and scalarare learnable parameters andis the sigmoid function.
Next,is used as a soft switch to choose betweengeneratinga word from the vocabulary by sampling from, orcopyinga word from the input sequence by sampling from the attention distribution.
For each document let theextended vocabularydenote the union of the vocabulary, and all words appearing in the source document.
We obtain the following probability distribution over the extended vocabulary:(9) Note that ifis an out-of-vocabulary (OOV)  word, thenis zero;
similarly ifdoes not appear in the source document, thenis zero.
The ability to produce OOV words is one of the primary advantages of pointer-generator models; by contrast models such as our baseline are restricted to their pre-set vocabulary.The loss function is as described in equations (6)  and (7) , but with respect to our modified probability distributiongiven in equation (9) .2.3Coverage mechanismRepetition is a common problem for sequence-to-sequence models, and is especially pronounced when generating multi-sentence text (see Figure1) .
We adapt thecoverage modelofto solve the problem.
In our coverage model, we maintain acoverage vector, which is the sum of attention distributions over all previous decoder timesteps:(10) Intuitively,is a (unnormalized)  distribution over the source document words that represents the degree of coverage that those words have received from the attention mechanism so far.
Note thatis a zero vector, because on the first timestep, none of the source document has been covered.The coverage vector is used as extra input to the attention mechanism, changing equation (1)  to:(11) whereis a learnable parameter vector of same length as.
This ensures that the attention mechanism’s current decision (choosing where to attend next)  is informed by a reminder of its previous decisions (summarized in) .
This should make it easier for the attention mechanism to avoid repeatedly attending to the same locations, and thus avoid generating repetitive text.We find it necessary (see section5)  to additionally define acoverage lossto penalize repeatedly attending to the same locations:(12) Note that the coverage loss is bounded; in particular.
Equation (12)  differs from the coverage loss used in Machine Translation.
In MT, we assume that there should be a roughly one-to-one translation ratio; accordingly the final coverage vector is penalized if it is more or less than 1.
Our loss function is more flexible: because summarization should not require uniform coverage, we only penalize the overlap between each attention distribution and the coverage so far – preventing repeated attention.
Finally, the coverage loss, reweighted by some hyperparameter, is added to the primary loss function to yield a new composite loss function:(13) 
body
Get To The Point: Summarization with Pointer-Generator Networks
2Our Models
1704.04368
3Related WorkNeural abstractive summarization.were the first to apply modern neural networks to abstractive text summarization, achieving state-of-the-art performance on DUC-2004 and Gigaword, two sentence-level summarization datasets.
Their approach, which is centered on the attention mechanism, has been augmented with recurrent decoders, Abstract Meaning Representations, hierarchical networks, variational autoencoders, and direct optimization of the performance metric, further improving performance on those datasets.However, large-scale datasets for summarization oflongertext are rare.adapted the DeepMind question-answering datasetfor summarization, resulting in theCNN/Daily Maildataset, and provided the first abstractive baselines.
The same authors then published a neuralextractiveapproach, which uses hierarchical RNNs to select sentences, and found that it significantly outperformed their abstractive result with respect to the ROUGE metric.
To our knowledge, these are the only two published results on the full dataset.Prior to modern neural methods, abstractive summarization received less attention than extractive summarization, butexplored cutting unimportant parts of sentences to create summaries, andexplore sentence fusion using dependency trees.Pointer-generator networks.The pointer networkis a sequence-to-sequence model that uses the soft attention distribution ofto produce an output sequence consisting of elements from the input sequence.
The pointer network has been used to create hybrid approaches for NMT, language modeling, and summarization.Our approach is close to the Forced-Attention Sentence Compression model ofand the CopyNet model of, with some small differences:
(i)  We calculate an explicit switch probability, whereas Gu et al. induce competition through a shared softmax function.
(ii)  We recycle the attention distribution to serve as the copy distribution, but Gu et al. use two separate distributions.
(iii)  When a word appears multiple times in the source text, we sum probability mass from all corresponding parts of the attention distribution, whereas Miao and Blunsom do not.
Our reasoning is that (i)  calculating an explicitusefully enables us to raise or lower the probability of all generated words or all copy words at once, rather than individually, (ii)  the two distributions serve such similar purposes that we find our simpler approach suffices, and (iii)  we observe that the pointer mechanism often copies a word while attending to multiple occurrences of it in the source text.Our approach is considerably different from that ofand.
Those works train their pointer components to activate only for out-of-vocabulary words or named entities (whereas we allow our model to freely learn when to use the pointer) , and they do not mix the probabilities from the copy distribution and the vocabulary distribution.
We believe the mixture approach described here is better for abstractive summarization – in section6we show that the copy mechanism is vital for accurately reproducing rare but in-vocabulary words, and in section7.2we observe that the mixture model enables the language model and copy mechanism to work together to perform abstractive copying.Coverage.Originating from Statistical Machine Translation, coverage was adapted for NMT byand, who both use a GRU to update the coverage vector each step.
We find that a simpler approach – summing the attention distributions to obtain the coverage vector – suffices.
In this respect our approach is similar to, who apply a coverage-like method to image captioning,
and, who also incorporate a coverage mechanism (which they call ‘distraction’)  as described in equation (11)  into neural summarization of longer text.Temporal attentionis a related technique that has been applied to NMTand summarization.
In this approach, each attention distribution is divided by the sum of the previous, which effectively dampens repeated attention.
We tried this method but found it too destructive, distorting the signal from the attention mechanism and reducing performance.
We hypothesize that an early intervention method such as coverage is preferable to a post hoc method such as temporal attention – it is better toinformthe attention mechanism to help it make better decisions, than tooverrideits decisions altogether.
This theory is supported by the large boost that coverage gives our ROUGE scores (see Table1) , compared to the smaller boost given by temporal attention for the same task.
body
Get To The Point: Summarization with Pointer-Generator Networks
3Related Work
1704.04368
4DatasetWe use theCNN/Daily Maildataset, which contains online news articles (781 tokens on average)  paired with multi-sentence summaries (3.75 sentences or 56 tokens on average) .
We used scripts supplied byto obtain the same version of the the data, which has 287,226 training pairs, 13,368 validation pairs and 11,490 test pairs.
Both the dataset’s published resultsuse theanonymizedversion of the data, which has been pre-processed to replace each named entity, e.g.,The United Nations, with its own unique identifier for the example pair, e.g.,@entity5.
By contrast, we operate directly on the original text (ornon-anonymizedversion of the data) ,222atwww.github.com/abisee/pointer-generatorwhich we believe is the favorable problem to solve because it requires no pre-processing.ROUGEMETEOR12Lexact match+ stem/syn/paraabstractive model*35.4613.3032.65--seq-to-seq + attn baseline (150k vocab) 30.4911.1728.0811.6512.86seq-to-seq + attn baseline (50k vocab) 31.3311.8128.8312.0313.20pointer-generator36.4415.6633.4215.3516.65pointer-generator + coverage39.5317.2836.3817.3218.72lead-3 baseline (ours) 40.3417.7036.5720.4822.21lead-3 baseline*39.215.735.5--extractive model*39.616.235.3--Table 1:ROUGE F1and METEOR scores on the test set.
Models and baselines in the top half are abstractive, while those in the bottom half are extractive.
Those marked with * were trained and evaluated on the anonymized dataset, and so are not strictly comparable to our results on the original text.
All our ROUGE scores have a 95% confidence interval of at mostas reported by the official ROUGE script.
The METEOR improvement from the 50k baseline to the pointer-generator model, and from the pointer-generator to the pointer-generator+coverage model, were both found to be statistically significant using an approximate randomization test with.
body
Get To The Point: Summarization with Pointer-Generator Networks
4Dataset
1704.04368
5ExperimentsFor all experiments, our model has 256-dimensional hidden states and 128-dimensional word embeddings.
For the pointer-generator models, we use a vocabulary of 50k words for both source and target – note that due to the pointer network’s ability to handle OOV words, we can use a smaller vocabulary size than’s150k source and 60k target vocabularies.
For the baseline model, we also try a larger vocabulary size of 150k.Note that the pointer and the coverage mechanism introduce very few additional parameters to the network: for the models with vocabulary size 50k, the baseline model has 21,499,600 parameters, the pointer-generator adds 1153 extra parameters (,,andin equation8) , and coverage adds 512 extra parameters (in equation11) .Unlike, we do not pre-train the word embeddings – they are learned from scratch during training.
We train using Adagradwith learning rate 0.15 and an initial accumulator value of 0.1.
(This was found to work best of Stochastic Gradient Descent, Adadelta, Momentum, Adam and RMSProp) .
We use gradient clipping with a maximum gradient norm of 2, but do not use any form of regularization.
We use loss on the validation set to implement early stopping.During training and at test time we truncate the article to 400 tokens and limit the length of the summary to 100 tokens for training and 120 tokens at test time.333The upper limit of 120 is mostly invisible: the beam search algorithm is self-stopping and almost never reaches the 120th step.This is done to expedite training and testing, but we also found that truncating the article canraisethe performance of the model (see section7.1for more details) .
For training, we found it efficient to start with highly-truncated sequences, then raise the maximum length once converged.
We train on a single Tesla K40m GPU with a batch size of 16.
At test time our summaries are produced using beam search with beam size 4.We trained both our baseline models for about 600,000 iterations (33 epochs)  – this is similar to the 35 epochs required by’sbest model.
Training took 4 days and 14 hours for the 50k vocabulary model, and 8 days 21 hours for the 150k vocabulary model.
We found the pointer-generator model quicker to train, requiring less than 230,000 training iterations (12.8 epochs) ; a total of 3 days and 4 hours.
In particular, the pointer-generator model makes much quicker progress in the early phases of training.
To obtain our final coverage model, we added the coverage mechanism with coverage loss weighted to(as described in equation13) , and trained for a further 3000 iterations (about 2 hours) .
In this time the coverage loss converged to about 0.2, down from an initial value of about 0.5.
We also tried a more aggressive value of; this reduced coverage loss but increased the primary loss function, thus we did not use it.We tried training the coverage model without the loss function, hoping that the attention mechanism may learn by itself not to attend repeatedly to the same locations, but we found this to be ineffective, with no discernible reduction in repetition.
We also tried training with coverage from the first iteration rather than as a separate training phase, but found that in the early phase of training, the coverage objective interfered with the main objective, reducing overall performance.
body
Get To The Point: Summarization with Pointer-Generator Networks
5Experiments
1704.04368
6Results6.1PreliminariesOur results are given in Table1.
We evaluate our models with the standard ROUGE metric, reporting the F1scores for ROUGE-1, ROUGE-2 and ROUGE-L (which respectively measure the word-overlap, bigram-overlap, and longest common sequence between the reference summary and the summary to be evaluated) .
We obtain our ROUGE scores using thepyrougepackage.444pypi.python.org/pypi/pyrouge/0.1.3We also evaluate with the METEOR metric, both in exact match mode (rewarding only exact matches between words)  and full mode (which additionally rewards matching stems, synonyms and paraphrases) .555www.cs.cmu.edu/~alavie/METEORIn addition to our own models, we also report the lead-3 baseline (which uses the first three
sentences of the article as a summary) , and compare to the only existing
abstractiveand extractivemodels on the full dataset.
The output of our models is available online.666www.github.com/abisee/pointer-generatorGiven that we generate plain-text summaries but Nallapati et
al.generate anonymized summaries (see Section4) , our ROUGE scores are not strictly comparable.
There is evidence to suggest that the original-text dataset may result in higher ROUGE scores in general than the anonymized dataset – the lead-3 baseline is higher on the former than the latter.
One possible explanation is that multi-word named entities lead to a higher rate of-gram overlap.
Unfortunately, ROUGE is the only available means of comparison with Nallapati et al.’s work.
Nevertheless, given that the disparity in the lead-3 scores is (+1.1 ROUGE-1, +2.0 ROUGE-2, +1.1 ROUGE-L)  points respectively, and our best model scores exceedby (+4.07 ROUGE-1, +3.98 ROUGE-2, +3.73 ROUGE-L)  points, we may estimate that we outperform the only previous abstractive system by at least 2 ROUGE points all-round.1-grams2-grams3-grams4-gramssentences% that are duplicatespointer-generator, no coveragepointer-generator + coveragereference summariesFigure 4:Coverage eliminates undesirable repetition.
Summaries from ournon-coverage modelcontain many duplicated-grams while ourcoverage modelproduces a similar number as thereference summaries.6.2ObservationsWe find that both our baseline models perform poorly with respect to ROUGE and METEOR, and in fact the larger vocabulary size (150k)  does not seem to help.
Even the better-performing baseline (with 50k vocabulary)  produces summaries with several common problems.
Factual details are frequently reproduced incorrectly, often replacing an uncommon (but in-vocabulary)  word with a more-common alternative.
For example in Figure1, the baseline model appears to struggle with the rare wordthwart, producingdestabilizeinstead, which leads to the fabricated phrasedestabilize nigeria’s economy.
Even more catastrophically, the summaries sometimes devolve into repetitive nonsense, such as the third sentence produced by the baseline model in Figure1.
In addition, the baseline model can’t reproduce out-of-vocabulary words (such asmuhammadu buhariin Figure1) .
Further examples of all these problems are provided in the supplementary material.Our pointer-generator model achieves much better ROUGE and METEOR scores than the baseline, despite many fewer training epochs.
The difference in the summaries is also marked: out-of-vocabulary words are handled easily, factual details are almost always copied correctly, and there are no fabrications (see Figure1) .
However, repetition is still very common.Our pointer-generator model with coverage improves the ROUGE and METEOR scores further, convincingly surpassing the best abstractive model ofby several ROUGE points.
Despite the brevity of the coverage training phase (about 1% of the total training time) , the repetition problem is almost completely eliminated, which can be seen both qualitatively (Figure1)  and quantitatively (Figure4) .
However, our best model does not quite surpass the ROUGE scores of the
lead-3 baseline, nor the current best extractive model.
We discuss this issue in section7.1.
body
Get To The Point: Summarization with Pointer-Generator Networks
6Results
1704.04368
7DiscussionArticle:smugglers lure arab and african migrants by offering discounts to get onto overcrowded ships if people bring more potential passengers, a cnn investigation has revealed. (…) Summary:cnn investigationuncoversthebusiness insideahuman smuggling ring.Article:eyewitness video showing white north charleston police officer michael slager shooting to death an unarmed black man has exposed discrepancies in the reports of the first officers on the scene. (…) Summary:morequestions than answers emergeincontroversial s.c.police shooting.Figure 5:Examples of highly abstractive reference summaries (bolddenotes novel words) .7.1Comparison with extractive systemsIt is clear from Table1that extractive systems tend to achieve higher ROUGE scores than abstractive, and that the extractive lead-3 baseline is extremely strong (even the best extractive system beats it by only a small margin) .
We offer two possible explanations for these observations.Firstly, news articles tend to be structured with the most important information at the start; this partially explains the strength of the lead-3 baseline.
Indeed, we found that using only the first 400 tokens (about 20 sentences)  of the article yielded significantly higher ROUGE scores than using the first 800 tokens.Secondly, the nature of the task and the ROUGE metric make extractive approaches and the lead-3 baseline difficult to beat.
The choice of content for the reference summaries is quite subjective – sometimes the sentences form a self-contained summary; other times they simply showcase a few interesting details from the article.
Given that the articles contain 39 sentences on average, there are many equally valid ways to choose 3 or 4 highlights in this style.
Abstraction introduces even more options (choice of phrasing) , further decreasing the likelihood of matching the reference summary.
For example,smugglers profit from desperate migrantsis a valid alternative abstractive summary for the first example in Figure5, but it scores 0 ROUGE with respect to the reference summary.
This inflexibility of ROUGE is exacerbated by only having one reference summary, which has been shown to lower ROUGE’s reliability compared to multiple reference summaries.Due to the subjectivity of the task and thus the diversity of valid summaries, it seems that ROUGE rewards safe strategies such as selecting the first-appearing content, or preserving original phrasing.
While the reference summariesdosometimes deviate from these techniques, those deviations are unpredictable enough that the safer strategy obtains higher ROUGE scores on average.
This may explain why extractive systems tend to obtain higher ROUGE scores than abstractive, and even extractive systems do not significantly exceed the lead-3 baseline.To explore this issue further, we evaluated our systems with the METEOR metric, which rewards not only exact word matches, but also matching stems, synonyms and paraphrases (from a pre-defined list) .
We observe that all our models receive over 1 METEOR point boost by the inclusion of stem, synonym and paraphrase matching, indicating that they may be performing some abstraction.
However, we again observe that the lead-3 baseline is not surpassed by our models.
It may be that news article style makes the lead-3 baseline very strong with respect to any metric.
We believe that investigating this issue further is an important direction for future work.1-grams2-grams3-grams4-gramssentences% that are novelpointer-generator + coveragesequence-to-sequence + attention baselinereference summariesFigure 6:Although ourbest modelis abstractive, it does not produce novel-grams (i.e.,-grams that don’t appear in the source text)  as often as thereference summaries. Thebaseline modelproduces more novel-grams, but many of these are erroneous (see section7.2) .Article:andy murray (…)  is into the semi-finals of the miami open , but not before getting a scare from 21 year-old austrian dominic thiem, who pushed him to 4-4 in the second set before going down 3-6 6-4, 6-1 in an hour and three quarters. (…) Summary:andy murraydefeateddominic thiem 3-6 6-4, 6-1 in an hour and three quarters.Article:(…)  wayne rooney smashes home during manchester united ’s 3-1 win over aston villa on saturday. (…) Summary:manchester unitedbeataston villa 3-1 at old trafford on saturday.Figure 7:Examples of abstractive summaries produced by our model (bolddenotes novel words) .7.2How abstractive is our model?We have shown that our pointer mechanism makes our abstractive system more reliable, copying factual details correctly more often.
But does the ease of copying make our system any lessabstractive?Figure6shows that our final model’s summaries contain a much lower rate of novel-grams (i.e., those that don’t appear in the article)  than the reference summaries, indicating a lower degree of abstraction.
Note that the baseline model produces novel-grams more frequently – however, this statistic includes all the incorrectly copied words,UNKtokens and fabrications alongside the good instances of abstraction.In particular, Figure6shows that our final model copies whole article sentences 35% of the time; by comparison the reference summaries do so only 1.3% of the time.
This is a main area for improvement, as we would like our model to move beyond simple sentence extraction.
However, we observe that the other 65% encompasses a range of abstractive techniques.
Article sentences are truncated to form grammatically-correct shorter versions, and new sentences are composed by stitching together fragments.
Unnecessary interjections, clauses and parenthesized phrases are sometimes omitted from copied passages.
Some of these abilities are demonstrated in Figure1, and the supplementary material contains more examples.Figure7shows two examples of more impressive abstraction – both with similar structure.
The dataset contains many sports stories whose summaries follow theX beat Yscoreondaytemplate, which may explain why our model is most confidently abstractive on these examples.
In general however, our model does not routinely produce summaries like those in Figure7, and is not close to producing summaries like in Figure5.The value of the generation probabilityalso gives a measure of the abstractiveness of our model.
During training,starts with a value of about 0.30 then increases, converging to about 0.53 by the end of training.
This indicates that the model first learns to mostly copy, then learns to generate about half the time.
However at test time,is heavily skewed towards copying, with a mean value of 0.17.
The disparity is likely due to the fact that during training, the model receives word-by-word supervision in the form of the reference summary, but at test time it does not.
Nonetheless, the generator module is useful even when the model is copying.
We find thatis highest at times of uncertainty such as the beginning of sentences, the join between stitched-together fragments, and when producing periods that truncate a copied sentence.
Our mixture model allows the network to copy while simultaneously consulting the language model – enabling operations like stitching and truncation to be performed with grammaticality.
In any case, encouraging the pointer-generator model to write more abstractively, while retaining the accuracy advantages of the pointer module, is an exciting direction for future work.
body
Get To The Point: Summarization with Pointer-Generator Networks
7Discussion
1704.04368
8ConclusionIn this work we presented a hybrid pointer-generator architecture with coverage, and showed that it reduces inaccuracies and repetition.
We applied our model to a new and challenging long-text dataset, and significantly outperformed the abstractive state-of-the-art result.
Our model exhibits many abstractive abilities, but attaining higher levels of abstraction remains an open research question.
body
Get To The Point: Summarization with Pointer-Generator Networks
8Conclusion
1704.04368
9AcknowledgmentWe thank the ACL reviewers for their helpful comments.
This work was begun while the first author was an intern at Google Brain and continued at Stanford. Stanford University gratefully acknowledges the support of the DARPA DEFT Program AFRL contract no. FA8750-13-2-0040. Any opinions in this material are those of the authors alone.
conclusion
Get To The Point: Summarization with Pointer-Generator Networks
9Acknowledgment