An Editorial Network for Enhanced Document Summarization
Edward Moroshko Guy Feigenblat Haggai Roitman David Konopnicki

On the Generalization of GAN Image Forensics
    Xinsheng Xuan, Bo Peng, Wei Wang and Jing DongThis work is funded by the National Natural Science Foundation of China (Grant No. 61502496, No. 61303262 and No. U1536120) and Beijing Natural Science Foundation (Grant No. 4164102).
    December 30, 2023
=========================================================================================================================================================================================================================================


We suggest a new idea of Editorial Network – a mixed extractive-abstractive summarization approach, which is applied as a post-processing step over a given sequence of extracted sentences. Our network tries to imitate the decision process of a human editor during summarization. Within such a process, each extracted sentence may be either kept untouched, rephrased or completely rejected. We further suggest an effective way for training the “editor" based on a novel soft-labeling approach. Using the CNN/DailyMail dataset we demonstrate the effectiveness of our approach compared to state-of-the-art extractive-only or abstractive-only baseline methods.




§ INTRODUCTION

Automatic text summarizers condense a given piece of text into a shorter version (the summary). This is done while trying to preserve the main essence of the original text and keeping the generated summary as readable as possible.

Existing summarization methods can be classified into two main types, either extractive or abstractive <cit.>. Extractive methods select and order text fragments (e.g., sentences) from the original text source <cit.>. Such methods are relatively simpler to develop and keep the extracted fragments untouched, allowing to preserve important parts, e.g., keyphrases, facts, opinions, etc. Yet, extractive summaries tend to be less fluent, coherent and readable and may include superfluous text.

Abstractive methods apply natural language paraphrasing and/or compression on a given text. A common approach is based on the encoder-decoder (seq-to-seq) paradigm <cit.>, with the original text sequence being encoded while the summary is the decoded sequence. While such methods usually generate summaries with better readability, their quality declines over longer textual inputs, which may lead to higher redundancy <cit.>. Moreover, such methods are sensitive to vocabulary size, making them more difficult to train and generalize <cit.>.

A common approach for handling long text sequences in abstractive settings is through attention mechanisms, which aim to imitate the attentive reading behaviour of humans <cit.>.
Two main types of attention methods may be utilized, either soft or hard. Soft attention methods first locate salient text regions within the input text and then bias the abstraction process to prefer such regions during decoding <cit.>. On the other hand, hard attention methods perform abstraction only on text regions that were initially selected by some extraction process <cit.>.

Compared to previous works, whose final summary is either entirely extracted or generated using an abstractive process, in this work, we suggest a new idea of  “Editorial Network" (EditNet) – a mixed extractive-abstractive summarization approach.  A summary generated by EditNet may include sentences that were either extracted, abstracted or of both types. Moreover, per considered sentence, EditNet may decide not to take either of these decisions and completely reject the sentence. 

Using the CNN/DailyMail dataset we demonstrate that, EditNet's summarization quality transcends that of state-of-the-art abstractive-only baselines. EditNet's summarization quality is also demonstrated to be highly competitive with that of NeuSum <cit.>, which is, to the best of our knowledge, the best performing extractive-only baseline. Yet, while EditNet obtains (more or less) a similar summarization quality to that of NeuSum, compared to the latter which applies only extraction, the former (on average) applies abstraction to the majority of each summary's extracted sentences.





§ EDITORIAL NETWORK



Figure <ref> now depicts the architecture of our proposed Editorial Network-based approach.
We apply this approach as a post-processing step over a given summary whose sentences were selected by some extractor. The key idea is to try to imitate the decision process of a human editor who needs to edit the summary so as to enhance its quality.

Let S denote a summary which was extracted from a given text (document) D.
The editorial process is implemented by iterating over sentences in S according to the selection order of the extractor.
For each sentence in S, the “editor" may make three possible decisions. The first decision is to keep the extracted sentence untouched (represented by label  in Figure <ref>). 
The second alternative is to rephrase the sentence (represented by label  in Figure <ref>). Such a decision, for example, may represent the editor's wish to simplify or compress the original source sentence. The last possible decision is to completely reject the sentence (represented by label  in Figure <ref>). For example, the editor may wish to ignore a superfluous or duplicate information expressed in the current sentence.
An example mixed summary generated by our approach is depicted in Figure <ref>, further emphasizing the various editor's decisions.







 §.§ Implementing the editor's decisions

For a given sentence s∈D, we now denote by s^e and s^a its original (extracted) and paraphrased (abstracted) versions.
To obtain s^a we use an abstractor, whose details will be shortly explained (see Section <ref>). Let e_s∈ℝ^n and a_s∈ℝ^n further denote the corresponding sentence representations of s^e and s^a, respectively. Such representations allow to compare both sentence versions on the same grounds.

Recall that, for each sentence s_i∈S (in order) the editor makes one of the three possible decisions: extract, abstract or reject s_i.
Therefore, the editor may modify summary S by paraphrasing or rejecting some of its sentences, resulting in a mixed extractive-abstractive summary S'.

Let l be the number of sentences in S. In each step i∈{1,2,…,l}, in order to make an educated decision, the editor considers both sentence representations e_s_i and a_s_i as its input, together with two additional auxiliary representations. The first auxiliary representation is that of the whole document D itself, hereinafter denoted d∈ℝ^n. Such a representation provides a global context for decision making. Assuming document D has N sentences, let e̅=1/N∑_s∈D^Ne_s. Following <cit.>, d is then calculated as follows:

    d=tanh(W_de̅+b_d),


where W_d∈ℝ^n×n and b_d∈ℝ^n are learnable parameters.

The second auxiliary representation is that of the summary that was generated by the editor so far, denoted at step i as g_i-1∈ℝ^n, with g_0=0⃗. Such a representation provides a local context for decision making. Given the four representations as an input, the editor's decision for sentence s_i∈S is implemented using two fully-connected layers, as follows:

    softmax(Vtanh(W_c[e_s_i,a_s_i,g_i-1,d]+b_c)+b),


where [·] denotes the vectors concatenation, V∈ℝ^3×m, W_c∈ℝ^m×4n, b_c∈ℝ^m and b∈ℝ^3 are learnable parameters.

In each step i, therefore, the editor chooses the action π_i∈{,,} with the highest likelihood (according to Eq. <ref>), further denoted p(π_i). Upon decision, in case it is either  or , the editor appends the corresponding sentence version (i.e., either s_i^e or s_i^a) to S'; otherwise, the decision is  and sentence s_i is discarded. Depending on its decision, the current summary representation is further updated as follows:

    g_i=g_i-1+tanh(W_gh_i),


where W_g∈ℝ^n×n are learnable parameters, g_i-1 is the summary representation from the previous decision step; and h_i∈{e_s_i,a_s_i,0⃗}, depending on which decision is made.


Such a network architecture allows to capture various complex interactions between the different inputs. 
For example, the network may learn that given the global context, one of the sentence versions may allow to produce a summary with a better coverage. As another example, based on the interaction between both sentence versions with either of the local or global contexts (and possibly among the last two), the network may learn that both sentence versions may only add superfluous or redundant information to the summary, and therefore, decide to reject both.



 §.§ Extractor and Abstractor

As a proof of concept, in this work, we utilize the extractor and abstractor that were previously used in <cit.>, with a slight modification to the latter, motivated by its specific usage within our approach. 
We now only highlight important aspects of these two sub-components and kindly refer the reader to <cit.> for the full implementation details.

The extractor of <cit.> consists of two main sub-components. The first is an encoder which encodes each sentence s∈D into e_s using an hierarchical representation[Such a representation is basically a combination of a temporal convolutional model followed by a biLSTM encoder.]. The second is a sentence selector using a Pointer-Network <cit.>. For the latter, let P(s) be the selection likelihood of sentence s.

The abstractor of <cit.> is basically a standard encoder-aligner-decoder with a copy mechanism <cit.>. Yet, instead of applying it directly only on a single given extracted sentence s_i^e∈S, we apply it on a “chunk" of three consecutive sentences[The first and last chunks would only have two consecutive sentences.]  (s_-^e,s_i^e,s_+^e), where s_-^e and s_+^e denote the sentence that precedes and succeeds s_i^e in D, respectively. This in turn, allows to generate an abstractive version of s_i^e (i.e., s_i^a) that benefits from a wider local context. Inspired by previous soft-attention methods, we further utilize the extractor's sentence selection likelihoods P(·) 
for enhancing the abstractor's attention mechanism, as follows. Let C(w_j) denote the abstractor's original attention value of a given word w_j occurring in (s_-^e,s_i^e,s_+^e); we then recalculate this value to be C'(w_j)=C(w_j)· P(s)/Z, with w_j∈s and s∈{s_-^e,s_i^e,s_+^e}; Z=∑_s'∈{s_-^e,s_i^e,s_+^e}∑_w_j∈s'C(w_j)· P(s') denotes the normalization term.






 §.§ Sentence representation

Recall that, in order to compare s_i^e with s_i^a, we need to represent both sentence versions on as similar grounds as possible.
To achieve that, we first replace s_i^e with s_i^a within the original document D. By doing so, we basically treat sentence s_i^a as if it was an ordinary sentence within D, where the rest of the document remains untouched. We then obtain s_i^a's representation by encoding it using the extractor's encoder in a similar way in which sentence s_i^e was originally supposed to be encoded.
This results in a representation a_s_i that provides a comparable alternative to e_s_i, whose encoding is expected to be effected by similar contextual grounds.




 §.§ Network training

We conclude this section with the description of how we train the editor using a novel soft labeling approach.
Given text S (with l extracted sentences), let π=(π_1,…,π_l) denote its editing decisions sequence. We define the following “soft" cross-entropy loss:

    ℒ(π|S)=-1/l∑_s_i∈S∑_π_i∈{,,}y(π_i)log p(π_i),


where, for a given sentence s_i∈S, y(π_i) denotes its soft-label for decision.  

We next explain how each soft-label y(π_i) is estimated. 
To this end, we utilize a given summary quality metric r(S') which can be used to evaluate the quality of any given summary S' 
(e.g., ROUGE <cit.>). Overall, for a given text input S with l sentences, there are 3^l possible summaries S' to consider. Let π^*=(π_1^*,…,π_l^*) denote the best decision sequence which results in the summary which maximizes r(·). For i∈{1,2,…,l}, let r̅(π_1^*,…,π_i-1^*,π_i) denote the average r(·) value obtained by decision sequences that start with the prefix (π_1^*,…,π_i-1^*,π_i). Based on π^*, the soft label y(π_i) is then calculated[For i=1 we have: r̅(π_1^*,…,π_0^*,π_1)=r̅(π_1).] as follows:

    y(π_i)=r̅(π_1^*,…,π_i-1^*,π_i)/∑_π_j∈{,,}r̅(π_1^*,…,π_i-1^*,π_j)






§ EVALUATION



 §.§ Dataset and Setup

We trained, validated and tested our approach using the non-annonymized version of the CNN/DailyMail dataset <cit.>. Following <cit.>, we used the story highlights associated with each article as its ground truth summary. We further used the F-measure versions of ROUGE-1 (R-1), ROUGE-2 (R-2) and ROUGE-L (R-L) as our evaluation metrics <cit.>.

The extractor and abstractor were trained similarly to <cit.> (including the same hyperparameters).
The Editorial Network (hereinafter denoted EditNet) was trained according to Section <ref>, using the ADAM optimizer with a learning rate of 10^-4 and a batch size of 32. To speedup the training time, we precalculated the soft labels (see Eq. <ref>).
Following <cit.>, we set the reward metric to be r(·) = α R1(·) + β R2(·) + γ  RL(·); with α=0.4, β=1 and γ=0.5, which were further suggested by <cit.>.




We further applied the Teacher-Forcing approach <cit.> during training, where we considered the true-label instead of the editor's decision (including when updating g_i at each step i according to Eq. <ref>).
Following <cit.>, we set m=512 and n=512.
We trained for 20 epochs, which has taken about 72 hours on a single GPU. We chose the best model over the validation set for testing. Finally, all components were implemented in Python 3.6 using the pytorch 0.4.1 package.



 §.§ Results

Table <ref> compares the quality of EditNet with that of several state-of-the-art extractive-only or abstractive-only baselines. This includes the extractor (rnn-ext-RL) and abstractor (rnn-ext-abs-RL) components of <cit.> that we further utilized for implementing[The rnn-ext-RL extractor results reported in Table <ref> are the ones that were reported by <cit.>. Training the public extractor released by these authors, we obtained the following significantly lower results: R-1:38.43, R-2:18.07 and R-L:35.37.] EditNet (see again Section <ref>).

Overall, EditNet provides a highly competitive summary quality, where it outperforms all baselines in the R-2 and R-L metrics. On R-1, EditNet outperforms all abstractive baselines and almost all extractive ones. 
Interestingly, EditNet's summarization quality is quite similar to that of NeuSum <cit.>. Yet, while NeuSum applies an extraction-only approach, summaries generated by EditNet include a mixture of sentences that have been either extracted or abstracted.

On average, 56% and 18% of EditNet's decisions were to abstract () or reject (), respectively. Moreover, on average, per summary, EditNet keeps only 33% of the original (extracted) sentences, while the rest (67%) are abstracted ones. 
This demonstrates that, EditNet has a high capability of utilizing abstraction, while being also able to maintain or reject the original extracted text whenever it is estimated to provide the best benefit for the summary's quality.




§ CONCLUSION AND FUTURE WORK

We have shown that instead of solely applying extraction or abstraction, a better choice would be a mixed one. As future work, we plan to evaluate other alternative extractor+abstractor configurations and try to train the network end-to-end. We further plan to explore reinforcement learning (RL) as an alternative decision making approach.


plain
10

ChenFastAS2018
Yen-Chun Chen and Mohit Bansal.
Fast abstractive summarization with reinforce-selected sentence
  rewriting.
In Proceedings of the 56th Annual Meeting of the Association for
  Computational Linguistics (Volume 1: Long Papers), pages 675–686.
  Association for Computational Linguistics, 2018.

Cheng2016ACL
Jianpeng Cheng and Mirella Lapata.
Neural summarization by extracting sentences and words.
In Proceedings of the 54th Annual Meeting of the Association for
  Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany,
  Volume 1: Long Papers, 2016.

Chopra2016Abs
Sumit Chopra, Michael Auli, and Alexander M. Rush.
Abstractive sentence summarization with attentive recurrent neural
  networks.
In Proceedings of the 2016 Conference of the North American
  Chapter of the Association for Computational Linguistics: Human Language
  Technologies, pages 93–98. Association for Computational Linguistics, 2016.

Cohan2018ADA
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter
  Chang, and Nazli Goharian.
A discourse-aware attention model for abstractive summarization of
  long documents.
In Proceedings of the 2018 Conference of the North American
  Chapter of the Association for Computational Linguistics: Human Language
  Technologies, Volume 2 (Short Papers), pages 615–621. Association for
  Computational Linguistics, 2018.

Dlikman2016UsingML
Alexander Dlikman and Mark Last.
Using machine learning methods and linguistic features in
  single-document extractive summarization.
In DMNLP@PKDD/ECML, 2016.

DBLP:conf/emnlp/DongSCHC18
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit
  Cheung.
Banditsum: Extractive summarization as a contextual bandit.
In Proceedings of the 2018 Conference on Empirical Methods in
  Natural Language Processing, Brussels, Belgium, October 31 - November 4,
  2018, pages 3739–3748, 2018.

summit2017
Guy Feigenblat, Haggai Roitman, Odellia Boni, and David Konopnicki.
Unsupervised query-focused multi-document summarization using the
  cross entropy method.
In Proceedings of the 37th International ACM SIGIR Conference on
  Research & Development in Information Retrieval, SIGIR '17. ACM, 2017.

Gambhir:2017
Mahak Gambhir and Vishal Gupta.
Recent automatic text summarization techniques: A survey.
Artif. Intell. Rev., 47(1):1–66, January 2017.

Gehrmann2018BottomUpAS
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush.
Bottom-up abstractive summarization.
In Proceedings of the 2018 Conference on Empirical Methods in
  Natural Language Processing, pages 4098–4109. Association for Computational
  Linguistics, 2018.

GuoSoftLayer2018
Han Guo, Ramakanth Pasunuru, and Mohit Bansal.
Soft layer-specific multi-task summarization with entailment and
  question generation.
In Proceedings of the 56th Annual Meeting of the Association for
  Computational Linguistics (Volume 1: Long Papers), pages 687–697.
  Association for Computational Linguistics, 2018.

Hermann:2015:TMR:2969239.2969428
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse
  Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.
Teaching machines to read and comprehend.
In Proceedings of the 28th International Conference on Neural
  Information Processing Systems - Volume 1, NIPS'15, pages 1693–1701,
  Cambridge, MA, USA, 2015. MIT Press.

HsuAUM2018
Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun.
A unified model for extractive and abstractive summarization using
  inconsistency loss.
In Proceedings of the 56th Annual Meeting of the Association for
  Computational Linguistics (Volume 1: Long Papers), pages 132–141.
  Association for Computational Linguistics, 2018.

JiangClosedBook2018
Yichen Jiang and Mohit Bansal.
Closed-book training to improve summarization encoder memory.
In Proceedings of the 2018 Conference on Empirical Methods in
  Natural Language Processing, pages 4067–4077. Association for Computational
  Linguistics, 2018.

lamb2016professor
Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang,
  Aaron C Courville, and Yoshua Bengio.
Professor forcing: A new algorithm for training recurrent networks.
In Advances In Neural Information Processing Systems, pages
  4601–4609, 2016.

Li2018KIGN
Chenliang Li, Weiran Xu, Si Li, and Sheng Gao.
Guiding generation for abstractive text summarization based on key
  information guide network.
In Proceedings of the 2018 Conference of the North American
  Chapter of the Association for Computational Linguistics: Human Language
  Technologies, Volume 2 (Short Papers), pages 55–60. Association for
  Computational Linguistics, 2018.

lin2004rouge
Chin-Yew Lin.
Rouge: A package for automatic evaluation of summaries.
In Text summarization branches out: Proceedings of the ACL-04
  workshop, volume 8. Barcelona, Spain, 2004.

liu2018a
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz
  Kaiser, and Noam Shazeer.
Generating wikipedia by summarizing long sequences.
arXiv preprint arXiv:1801.10198, 2018.

Nallapati2017a
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou.
Summarunner: A recurrent neural network based sequence model for
  extractive summarization of documents.
In Proceedings of the Thirty-First AAAI Conference on
  Artificial Intelligence, February 4-9, 2017, San Francisco, California,
  USA., pages 3075–3081, 2017.

DBLP:conf/conll/NallapatiZSGX16
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar
  Gülçehre, and Bing Xiang.
Abstractive text summarization using sequence-to-sequence rnns and
  beyond.
In Proceedings of the 20th SIGNLL Conference on Computational
  Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016,
  pages 280–290, 2016.

DBLP:conf/naacl/NarayanCL18
Shashi Narayan, Shay B. Cohen, and Mirella Lapata.
Ranking sentences for extractive summarization with reinforcement
  learning.
In Proceedings of the 2018 Conference of the North American
  Chapter of the Association for Computational Linguistics: Human Language
  Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018,
  Volume 1 (Long Papers), pages 1747–1759, 2018.

Pasunuru2018a
Ramakanth Pasunuru and Mohit Bansal.
Multi-reward reinforced summarization with saliency and entailment.
In Proceedings of the 2018 Conference of the North American
  Chapter of the Association for Computational Linguistics: Human Language
  Technologies, Volume 2 (Short Papers), pages 646–653. Association for
  Computational Linguistics, 2018.

Paulus2017ADR
Romain Paulus, Caiming Xiong, and Richard Socher.
A deep reinforced model for abstractive summarization.
CoRR, abs/1705.04304, 2017.

See2017GetTT
Abigail See, Peter J. Liu, and Christopher D. Manning.
Get to the point: Summarization with pointer-generator networks.
In Proceedings of the 55th Annual Meeting of the Association for
  Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4,
  Volume 1: Long Papers, pages 1073–1083, 2017.

Sutskever2014Seq
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.
Sequence to sequence learning with neural networks.
In Proceedings of the 27th International Conference on Neural
  Information Processing Systems - Volume 2, NIPS'14, pages 3104–3112,
  Cambridge, MA, USA, 2014. MIT Press.

Tan2017a
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao.
Abstractive document summarization with a graph-based attentional
  neural model.
In Proceedings of the 55th Annual Meeting of the Association for
  Computational Linguistics (Volume 1: Long Papers), pages 1171–1181.
  Association for Computational Linguistics, 2017.

vinyals2015pointer
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly.
Pointer networks.
In Advances in Neural Information Processing Systems, pages
  2692–2700, 2015.

Wu2018LearningTE
Yuxiang Wu and Baotian Hu.
Learning to extract coherent summary via deep reinforcement learning.
In Proceedings of the Thirty-Second AAAI Conference on
  Artificial Intelligence, (AAAI-18), the 30th innovative Applications of
  Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on
  Educational Advances in Artificial Intelligence (EAAI-18), New Orleans,
  Louisiana, USA, February 2-7, 2018, pages 5602–5609, 2018.

DBLP:conf/aaai/WuH18
Yuxiang Wu and Baotian Hu.
Learning to extract coherent summary via deep reinforcement learning.
In Proceedings of the Thirty-Second AAAI Conference on
  Artificial Intelligence, (AAAI-18), the 30th innovative Applications of
  Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on
  Educational Advances in Artificial Intelligence (EAAI-18), New Orleans,
  Louisiana, USA, February 2-7, 2018, pages 5602–5609, 2018.

ZhangLatent2018
Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou.
Neural latent extractive document summarization.
In Proceedings of the 2018 Conference on Empirical Methods in
  Natural Language Processing, pages 779–784. Association for Computational
  Linguistics, 2018.

Qingyu2018
Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao.
Neural document summarization by jointly learning to score and select
  sentences.
In Proceedings of the 56th Annual Meeting of the Association for
  Computational Linguistics (Volume 1: Long Papers), pages 654–663.
  Association for Computational Linguistics, 2018.





