A Hybrid Word-Character Approach to Abstractive Summarization
Chieh-Teng Chang Chi-Chia Huang Chih-Yuan Yang Jane Yung-Jen Hsu

UTF8gbsn

Flavor fluctuations in 3-level quantum dots: Generic SU(3)-Kondo fixed point
  in equilibrium and non-Kondo fixed points in nonequilibrium
    Jan von Delft
    
==========================================================================================================================================



Automatic abstractive text summarization is an important and challenging research topic of natural language processing. Among many widely used languages, the Chinese language has a special property that a Chinese character contains rich information comparable to a word. Existing Chinese text summarization methods, either adopt totally character-based or word-based representations, fail to fully exploit the information carried by both representations. To accurately capture the essence of articles, we propose a hybrid word-character approach (HWC) which preserves the advantages of both word-based and character-based representations. We evaluate the advantage of the proposed HWC approach by applying it to two existing methods, and discover that it generates state-of-the-art performance with a margin of 24 ROUGE points on a widely used dataset LCSTS. In addition, we find an issue contained in the LCSTS dataset and offer a script to remove overlapping pairs (a summary and a short text) to create a clean dataset for the community. The proposed HWC approach also generates the best performance on the new, clean LCSTS dataset.





§ INTRODUCTION


Text summarization aims to create a short, fluent, and effective summary from a long text document. Since the rapid growth of information stored in the textual form in digital documents, summaries greatly help address the amount of text data available online for searching useful information and consuming relevant articles.
Text summarization covers various types. General summarization improves the effectiveness of indexing and reduces the bias caused by humans. Query-focused summarization takes preferences into consideration to satisfy individual needs of information. Multiple-document summarization aims to generate summaries across multiple documents about the same topic. Extractive summarization combines a group of informative pieces of text from the source without changing them, and abstractive summarization generates entirely new sentences by absorbing the information contained in the source text <cit.>.

The recently rapid development of neural networks brings significant advances into text summarization, especially from the family of attentional sequence to sequence (Seq2Seq) architectures <cit.>, which generates promising performance for abstractive summarization <cit.>. They learn an internal language representation from a large number of examples to generate summaries similar to the ones made by humans. 

Since the used language is a critical factor of text summarization, many studies have been done for Chinese due to its large number of users, continuous use in the long history, and widespread influence in East Asia. Chinese is significantly different from any European languages especially in its character representation and word segmentation. As the old Chinese characters were developed thousands of years ago with a monosyllabic structure and used as words, the derived modern Chinese inherits a bank of characters numbering tens of thousands and composing words from either single or multiple characters. Although punctuations are widely used in modern Chinese to separate sentences, there is still no delimiter within a sentence to isolate words. Word segmentation is an error-prone process, and it largely affects the result of automatic summarization <cit.>. Since Chinese characters own semantic meanings although polysemous, many existing studies use character-based representation to simplify the effort and prevent the uncertainty of segmentation <cit.>. 

Although a few existing methods test word-based representation <cit.>, their performances are only slightly improved or even worse. They use word-based representation on both source and target text, but it is doubtful since the approach relies on a prerequisite of sufficient training samples in terms of occurred words. However, the lengths of source and target text of a text summarization sample are definitely asymmetric. Summaries as the targets are surely shorter than their source articles and thus it is questionable whether the used dataset is large enough to provide a satisfactory size of target text. Another issue is the memory limitation. Since a Chinese word is composed of either a single or multiple Chinese characters, representing a given Chinese text dataset using words instead of characters means significantly increasing the vocabulary size. As a text summarization algorithm implemented in an encoder-decoder framework and running on a GPU for fast execution, the size of its vocabulary will be restricted by a GPU's memory capacity. To the best of our knowledge, multi-GPU platforms to expand overall memory capacity for training text summarization models are still being developed. Existing methods which use word-based representation for the decoder have to use a selected subset of the complete vocabulary bank extracted from the target text, usually the high-frequency words. However, once the size of the vocabulary subset is not large enough, many low-frequency words in the text will be replaced by the unknown token and lose their messages, which results in incomplete summaries and low ROUGE scores.

To address the problem, we propose a hybrid word-character (HWC) approach which uses hybrid embedding units for an encoder and a decoder to preserve the advantages of both word-based and character-based representations. Since an encoder does not contain a softmax layer, its computational load and memory requirement are far less than a decoder <cit.>. Thus it is feasible to apply a word-based representation on the encoder and use a large vocabulary bank. Experimental results show this approach works well on two encoder-decoder summarization methods and generates state-of-the-art performance on a widely used Chinese text summarization dataset.



§ RELATED WORK

Extractive and abstractive summarization. Numerous automatic summarization methods have been proposed in the literature, and the formats of the generated summaries categorize existing methods into two classes: extractive and abstractive. While extractive summarization selects keywords or sentences from the original text and arranges them to form a summary <cit.>, abstractive summarization generates a brief version of the original text to preserve its information content ansd overall meaning. Extractive summarization is developed earlier since it highly simplifies text summarization into text partition and selection, and abstractive summarization is heavily studied recently due to its challenge and practicability. Early studies on abstractive approach include statistical machine translation techniques <cit.> and deletion-and-compression methods <cit.>. With the rapid spread of neural networks, many recent studies build their models in an encoder-decoder framework, especially the attentional Seq2Seq model <cit.> for its promising performance.

Chinese text summarization datasaet. In addition to English which is the first language studied for text summarization, the large number of Chinese users motivates the studies to explore its own language features. The first compiled Chinese text dataset available for text summarization is Chinese Gigaword <cit.>, which contains a comprehensive archive of newswire text data acquired from Chinese news sources such as Central News Agency of Taiwan and Xinhua News Agency of Beijing over several years. Although the corpus is impressive for its richness, it is neither free of charge and nor thoroughly categorized. There is a lack of human evaluation on the quality of the summaries (titles of news reports). On the contrary, the LCSTS (Large-scale Chinese Short Text Summarization) dataset <cit.> is created for academic research. Its text sources are still news reports and titles, collected from a Chinese microblogging website. Since they are microblogging articles, their length is restricted under a short text's limit, and make the collected articles consistent. The dataset provides predefined training and test subsets, manually labeled quality indexes on its test summaries, free assess and long-term maintenance.

Open-source implementation.
The rapid development and significant advances of many computer science fields have motivated the trend of publicly available algorithm libraries such as OpenCV <cit.> for computer vision, and OpenMNT <cit.> for neural machine translation. Such platforms provide great convenience to test new ideas, reproduce results, and optimize performance. Since those libraries are well maintained, new methods are soon available and easily called through a unified interface. Researchers benefit from them by reducing the effort of re-implementation, preventing the errors of misunderstanding, and saving the time of conducting experiments. Due to the considerable merits, we validate the proposed HWC approach using the OpenMNT library. 

Language translation.
Text summarization and language translation are two distinct problems in the family of natural language processing. However, they share certain similar properties such as a representation of sequential data and a conversion from text to tokens. Since the two problems are highly close, it is possible to apply models developed for one problem to the other, i.e. replacing
the source and target languages in a translation problem by the source and target text in a summarization problem. Inspired by the significant breakthrough in the translation problem <cit.>, we apply the proposed HWC approach to a state-of-the-art translation model and find that the integrated method generates leading-edge performance.



§ PROPOSED METHOD


As illustrated in Figure <ref>, the proposed HWC approach is used in an encoder-decoder framework that the input articles are represented by words and the output summaries by characters. In this paper, we verify the effectiveness of the proposed HWC approach using a baseline method attentional Seq2Seq and a state-of-the-art one Transformer <cit.>.




Different from existing methods which use a fixed type of the embedding units for both encoder and decoder as shown in Table <ref>, the proposed method uses word embedding to represent input articles and character embedding for output summaries. Such a design is motivated by the observation: In the Chinese language, words are more precise to provide information than characters because Chinese characters are highly polysemous, but characters are shorter and more flexible than words. On the one hand for input data, it is less ambiguous to represent articles using words than characters. In the LCSTS 1.0 dataset we use for experimental validation, there are 960k words made up by mere 10k characters. Using word embedding units expands the limit of a vocabulary bank in terms of size to train an effective encoder to capture the meaning contained in the word relationship. On the other hand for output targets, since summarized sentences are highly concentrated and it is common in Chinese to shorten long phrases for convenience, e.g. 奥运委员会 (the International Olympic Committee) is abbreviated by 奥委会 where the term 奥运 (Olympic Game) is reduced to 奥 and 委员会 (Committee) to 委会, characters are more flexible than words to represent the output sentences because of their definitely shorter length. 




§ EXPERIMENTAL SETUP

All of our experiments are conducted on a machine equipped with a 3.7GHz 8-core CPU, 64G memory, and a high-performance GPU (NVIDIA 1080 Ti). We use an open source implementation and the code released by the original authors to produce results of the Seq2Sqe+attn and Transformer methods respectively. We get the ROUGE scores of RNN and RNN-context methods on the LCSTS 2.0 dataset from the original authors. To train the Seq2Seq+attn and Transformer methods, we follow the existing methods RNN and CopyNet to split the Part I set into two distinct training and validation sets. Since their original authors only report random splitting rather than an explicit splitting mechanism, we use a long-standing random number generator MT19937 with five seeds (0 to 4) to select 1000 articles as the validation sets and use the remaining as training sets, and report the mean ROUGE scores.



 §.§ Dataset



We conduct experiments on the LCSTS dataset <cit.> to evaluate the proposed method. This dataset contains a large number of short Chinese news articles with man-made headlines as the short summaries collected from Sina Weibo[https://www.weibo.com/], a Chinese microblogging website. This dataset is composed of three parts, as shown in Table <ref>. Part I contains a large number of 2,400,591 pairs of articles and headlines but no annotation. Parts II and III contain not only text data but also human-labeled scores, measuring the quality of summaries in terms of their relevance to the source articles. The difference between Parts II and III are the numbers of annotators to create the scores, which is 1 for Part II but 5 for Part III. The relevance scores range from 1 to 5, the larger the more relevant. 


For a fair comparison, we follow the same split setting of existing methods <cit.> to use Part I as the training set and Part III's high-relevance subset (scores equivalent to or greater than 3) as the testing set.


LCSTS 2.0.
By examining the first released version (1.0) of the LCSTS dataset, we found that its Part III contains a high ratio of articles repeated in its Part I. We reported this problem to the authors, and received the response that they released an incorrect dataset which failed to filter out common articles in Parts I and III. To deal with the problem, they re-released the dataset they actually used for their experiments which replaced 329 overlapping articles in Part I with newly collected ones, and assigned it a new version 2.0.


LCSTS 2.0-clean.
After scrutinizing LCSTS 2.0, we find the cleanup is not complete. Many items in LCSTS 2.0's Part I are almost the same to items in Part III in terms of exactly the same summaries and highly similar articles, only differing on a few characters at the end of the articles to show the name of source newspaper. As an example shown in Table <ref>, the name of source newspaper 新闻晨报 (Shanghai Morning Post) does not contribute to the message carried in the article. Since the issue is likely to weaken the dataset, we remove the highly repeated items from Part I and name the amended dataset LCSTS 2.0-clean.



In order to evaluate the proposed method on a well-made dataset, we remove all highly overlapping items from the Part I split of LCSTS 2.0 and name it LCSTS 2.0-clean. The script to generate LCSTS 2.0-clean is in https://github.com/playma/LCSTS2.0-cleanGithub repository for reproducing our experimental results by other researchers.



 §.§ Evaluation Metrics


We adopt ROUGE <cit.> metrics for evaluation, which has been widely used for abstractive summarization. They measure the quality of summaries by computing the overlap of system-generated and reference ones. For a fair comparison, we report ROUGE-1 (1-gram), ROUGE-2 (bigrams) and ROUGE-L (longest common subsequence) F1 scores for all compared methods.



 §.§ Compared Methods


RNN and RNN-context <cit.> are two similar RNN-based methods except for a context generator included. In the simpler architecture without a context generator, its decoder uses the RNN encoder's last state as the input data; in the complexer architecture, a context generator is connected with all gated recurrent units' hidden states, and uses generated context to generate summaries.

CopyNet <cit.> integrates the copying mechanism into the attentional Seq2Seq model in order to combine selected subsequences from the input sequence to generate an output sequence.

Distraction <cit.> is a Seq2Seq framework which distracts a document into different regions by their content in order to better grasp the overall meaning of the input document.

DRGN <cit.> is an attentional Seq2Seq model equipped with a latent structure modeling component.

MRT <cit.> employs the minimum risk training strategy on an attentional Seq2Seq model.

WEAN <cit.> is based on an attentional Seq2Seq model which generates summaries by querying distributed word representations with an attention mechanism in the decoder.

AC-ABS <cit.> employs an actor-critic approach originally developed for reinforcement learning on an attentional Seq2Seq model.

Seq2Seq+attn is the method using the simplest attentional Seq2Seq model without any additional component. We use an implementation available from the OpenMNT system <cit.> and evaluate it as a baseline.

Transformer <cit.> is a newly developed encoder-decoder method which uses attention mechanisms rather than complex recurrent or convolutional neural networks. We adopt its model for Chinese abstractive summarization and first report its performance.



 §.§ Preprocessing and Hyperparameters

We adopt the same approach as RNN-context and CopyNet to segment input articles into words using the jieba segmentation utilities[https://pypi.python.org/pypi/jieba/]. About the hyperparameter of vocabulary size, we do experiments on several numbers as shown in Table <ref> in the same ranges used by RNN-context and CopyNet and some large numbers to assess the extent. The numbers 523506 and 523566 are the amount of words in the training sets whose occurrence frequency is greater than one. About the hyperparameters used the Seq2Seq+attn method, we empirically set the numbers of embedding dimension and hidden layers both as 500. We use Adagrad <cit.> as the optimizer, and set the initial learning rate to 0.15 and dropout rate to 0.3. We set the beam size to 5 for all decoders in our experiments. For the Transformer method, we use the default parameters of an implementation available in the OpenNMT-py repository[https://ppt.cc/ffue9x]. 



















§ RESULTS AND DISCUSSION











By evaluating the proposed model on the LCSTS dataset, we find that the dataset's original version (1.0) contains overlapping items and confirm the issue by its authors. Therefore, we do experiments for fair comparisons with existing methods not only on its original version, but also on two new versions (2.0 and 2.0-clean) to fairly evaluate the performance. The ROUGE scores on the three datasets are shown in Tables <ref>, <ref> and <ref> respectively. On the original version, the HWC+Transformer method generates the best score with a significant margin over existing methods. Using the same method on the LCSTS 2.0-clean dataset, the ROUGE scores decline reasonably and it shows the importance to evaluate methods on a well-made dataset.





Vocabulary size.
As shown in Tables <ref>, <ref> and <ref>, larger vocabulary banks lead to higher ROUGE scores. On the LCTST 1.0 dataset, the char-based Seq2Seq+attn method generates higher ROUGE scores over existing methods merely by enlarging its vocabulary bank. On the LCTST 2.0 and 2.0-clean datasets, the char-based Seq2Seq+attn method also benefits from large vocabulary banks. Adopting the HWC approach, the Seq2Seq+attn method with a word-based encoder uses larger vocabulary banks and generates better ROUGE scores. On both LCSTS 1.0 and 2.0 datasets, the Transformer method generates the top performance by using the HWC approach with a large vocabulary bank.


The effects of the HWC approach.
On all of the three datasets LCSTS 1.0, 2.0, and 2.0-clean, the HWC+Seq2Seq+attn method generates higher ROUGE scores over the Seq2Seq+attn method. Such improvements are also observed about the Transformer method on the LCSTS 1.0 and 2.0-clean dataset. Table <ref> outlines the enhancement and shows that overlapping data result in larger increases of the ROUGE scores. It indicates that the HWC+Transformer model is an aggressive learner which effectively adapts its hidden states to its training data. Thus, it generates very high ROUGE scores as the test data overlap the training ones.




Analysis of generated summaries.
In order to show the reason of the success caused by the HWC approach, we show four example summaries generated by the top two methods Transformer and HWC+Transformer in Table <ref> including their source articles and man-made references. The four examples show that the HWC+Transformer method better catches the major messages contained in the source articles, which are "10 dollars" (10块钱) in Article 1, "private trusts" (私人信托) in Article 2, "23 executive meetings" (23次常务会) and "Li Keqiang" (李克强) in Article 3, and "Xian Ge Qing" (湘鄂情), "Xiaomi" (小米), and "can not compare with us" (和我们没法比) in Article 4. In contrast, the character-based Transformer method tends to copy a full sentence from its source article but miss the point. For example, it copies "I love football, but my family financial status is supported by me." (我热爱足球，但家里真的离不开我。) in Article 1 but misses the key term "10 dollars" (10块钱), "What is the highest level of the rich?" (富人的最高境界是什么?) in Article 2 but misses the key term "private trust" (私人信托), and "Combining Internet boxes such as Xiaomi and routers can not compare with us." (小米等互联网企业那些盒子和路由加一起，都和我们没法比。) but misses the key term "Xian Ge Qing" (湘鄂情). Since the proposed HWC approach helps detect key terms, it leads to higher ROUGE scores.



Training time.
The proposed HWC approach has the advantages of not only improving performance but also increasing the training efficiency. We show the speedup as a chart in Figure <ref> from the experiments shown in Table <ref> conducted on the LCTST 2.0-clean dataset. Their vocabulary sizes are all of the maxima except for the HWC+Transform method due to a memory limitation. Both of the Seq2Seq+attn and Transformer methods use fewer epoches to generate high ROUGE-1 scores by applying the HWC approach. Please note the unit of the horizontal axis is a minute rather than an epoch, which means the actual execution time has been taken into consideration. For the Seq2Seq+attn method, the HWC approach increases the mean training time per epoch from 64 to 85 minutes, but for the Transformer method, it reduces the time from 161 to 110 minutes. Such result is caused by the tension between two factors that a word-based representation shortens the length of input sequences, and thus reduces the level of training difficulty, but it also increases the load of updating vocabulary embedding in the encoder due to a larger vocabulary size.


























§ CONCLUSION AND FUTURE STUDY

In this paper, we propose a hybrid representation approach to improve the performance of text summarization methods in an encoder-decoder framework. Experimental results demonstrate that the proposed approach clearly generates state-of-the-art performance. In addition, we find a few errors in a widely used dataset, and provide a script to polish it. Beyond the improved performance, it may be a better representation by incorporating part-of-speech tagging into the proposed approach. We are also interested in the applicability of the proposed approach to other natural language processing problems such as dialogue generation and machine translation.


    
    acl
	



