title: Query Focused Multi-document Summarisation of Biomedical Texts
authors: Diego Molla Christopher Jones Vincent Nguyen
text: 
  
  Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CLEF 2020, 22-25 September 2020, Thessaloniki, Greece.
  
  
   at BioASQ8b
  
  
  
  
  
  
  
  
  Macquarie University, Australia
  
  <Diego.Molla-Aliod@mq.edu.au>
  
  <Christopher.Jones4@hdr.mq.edu.au>
  Australian National University, Australia
  
  <Vincent.Nguyen@anu.edu.au> 
  
  
  Query Focused Multi-document Summarisation of Biomedical TextsCode associated with this paper is available at <https://github.com/dmollaaliod/bioasq8b-public>
      Diego Mollá0000-0003-4973-09631 Christopher Jones0000-0002-3491-739X
  1 Vincent Nguyen0000-0003-1787-80902
       December 30, 2023
  ==============================================================================================================================================================
  
  
  
  
  This paper presents the participation of  for Task B Phase B of the 2020 BioASQ Challenge (BioASQ8b). Our overall framework implements Query focused multi-document extractive summarisation by applying either a classification or a regression layer to the candidate sentence embeddings and to the comparison between the question and sentence embeddings. We experiment with variants using BERT and BioBERT, Siamese architectures, and reinforcement learning. We observe the best results when BERT is used to obtain the word embeddings, followed by an LSTM layer to obtain sentence embeddings. Variants using Siamese architectures or BioBERT did not improve the results.
  
  
  
  
  § INTRODUCTION
  
  
  Query focused multi-document summarisation aims to generate the answer to a question by combining information from multiple documents <cit.>. This task, therefore, is related to both question answering and text summarisation. There is substantial research in both question answering and text summarisation. In the case of text summarisation, most research focuses on single-document summarisation, and there is also substantial research on multi-document summarisation. However, there is relatively little research on query focused multi-document text summarisation. There are multiple applications where query focused multi-document text summarisation can be useful. A clear example of a useful application is in the domain of biomedicine and clinical medicine, where a doctor or a patient wants to obtain a concise summary of the most relevant evidence related to a particular diagnosis or intervention.
  
  The BioASQ Challenge[<http://www.bioasq.org>] organises shared tasks centered on biomedical texts. The focus of this paper is on the 2020 participation of  in Task B Phase B (BioASQ8b), where the aim is to find the “ideal answer” to a question, given a collection of relevant PubMed abstracts.[<http://www.ncbi.nlm.nih.gov/pubmed>] We approach this task as an instance of query focused multi-document extractive summarisation by scoring each candidate sentence and selecting the top-scoring ones to produce the final summary.  submitted independent runs to BioASQ8b, but we both used the starting code of Macquarie University's BioASQ7b participation <cit.>. Novel contributions of this paper, compared with previous participation at BioASQ, include:
  
  
      
    * The incorporation of BERT and BioBERT in the general architecture of <cit.>.
      
    * The use of Siamese architectures in two main setups: 1) sharing weights in the architecture of <cit.>, and 2) using Sentence-BERT <cit.>.
      
    * The use of Proximal Policy Optimisation (PPO) and BERT in a Reinforcement Learning approach.
  
  
  This paper is structured as follows. Section <ref> describes the general framework and baselines based on previous participation at BioASQ. Section <ref> explains the incorporation of BERT and BioBERT. Section <ref> introduces the incorporation of Siamese architectures. Section <ref> describes the use of Reinforcement Learning. Section <ref> presents and discusses the results of cross-validation evaluations using the BioASQ8b training data. Section <ref> presents and discusses the runs submitted to BioASQ. Finally, Section <ref> concludes this paper.
  
  
  
  
  
  § GENERAL FRAMEWORK AND BASELINES
  
  
  The overall architecture of our systems is based on that of Macquarie University's participation to BioASQ7b <cit.>. This architecture is described in Fig. <ref>.
  
  
  
  The baseline systems re-use the following options from <cit.> to generate the embeddings of the words and sentences.
  
  
      
    * The embedding generator to obtain word embeddings is a matrix of pre-trained embeddings generated by word2vec. We trained word2vec using PubMed documents provided by the organisers of BioASQ. The embeddings had a vector size of 100.
      
    * The embedding reductor to obtain sentence embeddings is a pair of forward and backward LSTM chains. The weights of the embedding reductor for the candidate sentence and the question were not shared.
      
    * The activation function of the final layer is a linear function for the regression setup  and a sigmoid function for the classification setup.
      
    * The similarity between the embeddings of the candidate sentence and the question is the element-wise product.
  
  
  As in <cit.>, we tried a classification setup and a regression setup. In the regression setup (“NNR” in Fig. <ref> Table <ref>), the training data is labelled with the ROUGE-SU4 F1 value of the candidate sentence and the objective function to optimise is the Mean Squared Error. In the classification setup (“NNC” in Fig. <ref> Table <ref>), the 5 candidate sentences with the highest ROUGE-SU4 F1 are labelled as 1, and the rest are labelled as 0. The objective function to optimise in the classification setup is binary cross-entropy.
  
  In both the classification and the regression setup, the summary is produced by scoring each candidate sentence and extracting the top n sentences to generate the summary, where n depended on the question type and was the same as reported in <cit.> (Table <ref>).
  
  
  
  
  
  § EXPERIMENTS WITH BERT AND BIOBERT
  
  
  BERT <cit.> has been used in a wide range of NLP tasks, including text classification <cit.>, and extractive <cit.> and abstractive <cit.> summarisation. We have integrated BERT into our general architecture of Fig. <ref> as described below.
  
  In a first experiment (“BERT untrained” in Table <ref>), we replaced the embedding generator with BERT using the pre-trained model provided by Huggingface.[<https://huggingface.co/>] The resulting word embeddings are now affected by context. The BERT weights were not updated during the training stage. Following the recommendation of the Huggingface library, the embedding reductor of each candidate sentence and the question are the average of the word embeddings.
  
  In a subsequent experiment (“BERT trained” in Table <ref>), the embedding generator and reductors are as in BERT untrained, but we allowed the BERT weights to be fine-tuned during the training process.
  
  We also tried a variant (“BERT LSTM” in Table <ref>) that uses BERT in the embedding generator as in BERT untrained, but the embedding reductor is a bidirectional LSTM chain as in the NNC baseline. This variant is therefore comparable with the NNC baseline, and the only difference being the use of BERT for the embedding generator.
  
  In a final series of experiments (“BioBERT untrained” and “BioBERT LSTM” in Table <ref>), the embedding generator and reductors are as in BERT untrained and BERT LSTM, but the pre-trained model was as provided by the developers of BioBERT[<https://github.com/dmis-lab/biobert>] <cit.>, who used biomedical documents to pre-train BERT.
  
  In all of the experiments in this section, the final layer was a classification layer with a sigmoid activation and the objective function to optimise was binary cross-entropy.
  
  
  
  § EXPERIMENTS WITH SIAMESE NETWORKS
  
  
  Siamese networks have been used in applications that include a comparison between documents, for example to determine semantic similarity <cit.>. The general idea is to use the same processing module for each of the two documents by sharing the weights of the encoding component that generates the embeddings of the documents. We have used this idea in two main kinds of experiments described below.
  
  
  
   §.§ Siamese LSTM
  
  
  A straightforward implementation of Siamese Networks (“Siamese LSTM” in Table <ref>) using the overall architecture of Fig. <ref> shares the weights of the embedding reductors of the candidate sentence and the question. This ensures that the sentence embeddings are generated using exactly the same process.
  
  
  
   §.§ Sentence-BERT
  
  
  The second implementation of Siamese Networks uses Sentence-BERT (SBERT) <cit.> to determine whether a candidate sentence is similar to the question. In particular, the system uses BERT <cit.> in a Siamese setup as described in Fig. <ref>.
  
  
  
  Depending on the settings, the system uses regression, classification or both (multi-task), for prediction. 
  
  
      
    * In the regression setup, the cosine similarity between candidate sentence and question embeddings is computed, and the objective function to optimise is the Mean Squared Error (MSE).
  
      
    * In the classification setup, the candidate sentence and question embeddings are concatenated with the element-wise absolute difference between the sentence and question, and a softmax layer is added. The objective function to optimise is binary cross-entropy.
  
      
    * In the multi-task setup, the classification and regression setups are jointly optimised during training. At prediction time, we use either the classification head or the regression head.
  
  
  In both the regression and the classification setups, the training data was labelled with the classification labels mentioned in Section <ref>.
  
  We also experimented with a BERT and a BioBERT variation[<https://huggingface.co/gsarti/biobert-nli>] of a Sentence Transformer <cit.> to ensure that the sentence embeddings reflect a biomedical word-space. 
  
  Table <ref> shows all the combinations we tried.
  
  
  
  
  
  
  
  
  § EXPERIMENTS WITH REINFORCEMENT LEARNING
  
  
  Reinforcement learning allows the training process to optimise the target evaluation metric (ROUGE-SU4 F1) directly. Whereas <cit.> used the REINFORCE algorithm <cit.>, in our participation to BioASQ8b we used the Proximal Policy Optimisation (PPO) approach.
  We choose PPO for our summarisation task because past research shows that it penalises changes to the policy <cit.>, and because we observe a more consistent learning curve using this approach compared to REINFORCE <cit.>.
  
  
  
   §.§ Approach
  
  As in past submissions, our reinforcement learning system classifies sentences to be either (0) not included in the summary or (1) included in the summary based on a policy, using the ROUGE-SU4 F1 score directly as the reward.
  We use the stable baselines library[<https://stable-baselines.readthedocs.io/>] <cit.> to implement the PPO reinforcement learning approach, and apply this to our BioASQ summarisation task environment.
  
  We perform some hyperparameter tuning, while leaving the PPO code unmodified as much as possible.
  We use a horizon (n steps) of 1000 with 4 mini-batches each, and run for a total of 500,000 timesteps.
  When training our model, we choose any action from the probability distribution of the policy function, but when testing we choose the action out of 100 samples.
  
  The neural network architecture for PPO is shown in Figure <ref>. As in <cit.>,
  the inputs of the neural network consist of:
  
      
    * Candidate sentence
      
    * Question
      
    * Summary generated so far
      
    * Sentences after the candidate sentence
      
    * Entire document
      
    * Length of summary generated so far
  
  
  For each of the first five inputs, we take the mean of the word2vec word embeddings (each of size 100) to generate the sentence embeddings. The sentence embeddings and the length of the summary are concatenated to form a single layer of size 501.
  We then feed the combined layer into a simple Multi-Layer Perceptron with two hidden layers each of size 200.
  The outputs of this neural network become the value function and stochastic policy function for our PPO approach, each of size 2 with linear activations, representing the predicted future rewards and action probabilities respectively for the two actions (classify 0 or 1).
  
  
  
  
  
   §.§ With BERT
  
  We also apply pre-trained BERT embeddings to our reinforcement learning approach for batches 4 and 5 of BioASQ8b.
  Our network architecture is the same as in Figure <ref>, but we change the embedding generator to generate BERT embeddings instead of word2vec embeddings. 
  We use the PyTorch BERT embedding generator provided by Huggingface<ref> and the TensorFlow Multi-Layer Perceptron policy provided by stable baselines<ref>.
  
  We observe a minor improvement in ROUGE-SU4 F1 score as shown in Section <ref> (PPO BERT).
  
  
  
  § RESULTS AND DISCUSSION
  
  
  
  
   §.§ Cross-Validation Results
  
  
  All experiments except those described in Sections <ref> were evaluated using 10-fold cross-validation using the training data provided by the organisers of BioASQ8b. These results are shown in Table <ref>. As described in Section <ref>, cross-validation evaluation was not practical for the experiments with reinforcement learning.
  
  
  
    barchart=[fill=black!20,draw=black]
    errorbar=[very thin,draw=black!75]
    sscale=[very thin,draw=black!75]
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  The baseline runs “firstn”, “NNR” and “NNC” confirm <cit.>'s observation that the classification setup produces better results than the regression setup. The untrained BERT system did not better the classification system, and the trained BERT system produced worse results.
  
  We observed no changes in the evaluation results of the BERT trained system as we changed the number of epochs from 1 to 20 epochs. A detailed look at the changes in the loss during training revealed a very small improvement of the loss as we increase the number of epochs, but not enough to reflect a difference in the final ROUGE-SU4 F1 metric. This suggests that a more elaborate fine-tuning regime with gradual unfreezing as described by <cit.> might lead to improved results.
  
  Given the poor results of the trained BERT system, we kept BERT frozen when we tested the variants using LSTM and using BioBERT.
  
  BERT and BioBERT followed by an LSTM-based sentence reductor did improve results over the version with a mean of embeddings. Unfortunately, the experiments using the LSTM-based reductor were made after the deadlines for submission of results to BioASQ8b.
  
  It was surprising to observe that the BioBERT variants performed worse than the BERT variants. This is not in line with the improvement in the performance of BioBERT for the “exact answers” section of BioASQ7b reported in literature <cit.>.
  
  The Siamese LSTM variant did not improve over the classification system. The reason for this might be, as mentioned in Section <ref>, that questions and candidate sentences are different in nature. We also conducted cross-validation experiments with the SBERT variants but they are not included in Table <ref> as the results are very different from those of the runs submitted to BioASQ (Section <ref>) and we suspect that there might have been an error when running these evaluations.
  
  
  Table <ref> shows and explains the hyperparameters of all systems of Table <ref>.
  
  
  
  
  
  
   §.§ Reinforcement Learning Results
  
  
  The evaluation setup for reinforcement learning did not use cross-validation due to the long time it took to train the system (several days). We therefore used a partition of the training data for the train and test sets.
  
  Table <ref> shows the evaluation results of the PPO <cit.> reinforcement learning approaches discussed in Section <ref>, compared with the REINFORCE <cit.> approach, using the BioASQ7b and BioASQ8b training datasets. 
  
  
  
  
  The BioASQ7b data set consists of 2,747 questions which we divide into a training set (2,289 questions) and a testing set (458 questions) partitioned 5:1 using a random seed to shuffle the data.
  The PPO word2vec and REINFORCE word2vec systems were run 3 times each for 500,000 timesteps on the BioASQ7b data set, and the maximum ROUGE-SU4 F1 score reached in each learning curve was averaged across the 3 runs.
  
  
  
  The BioASQ8b data set consists of 3,243 questions which we divide into a training set (2,702 questions) and a testing set (541 questions) partitioned 5:1 using a random seed to shuffle the data.
  The PPO word2vec, PPO BERT, and REINFORCE word2vec systems were each run once only for 500,000 timesteps on the BioASQ8b data set, and the maximum ROUGE-SU4 F1 score for each run is shown in Table <ref>.
  The models which produced the maximum ROUGE-SU4 F1 score for each system were saved and re-used to generate our PPO submissions to the BioASQ competition in 2020, except for the REINFORCE system which was not included in this year's submission.
  
  Whereas our evaluation results did not show any clear difference between REINFORCE and PPO on word2vec embeddings, the version using PPO and BERT features showed an improvement.
  
  The reinforcement learning experiments of Table <ref>, however, do not outperform the results of the experiments of Table <ref>.
  
  
  
  
  
  
  
  
  § SUBMISSIONS TO BIOASQ8B
  
  
  Table <ref> shows the results of 's submissions to BioASQ8b. We only report ROUGE-SU4 F1 for comparison with our experiments. Also, as observed by <cit.>, F1 scores have a higher correlation with human judgements than recall scores, and similar correlation with human judgements as precision scores.
  
  
  
  All runs in Table <ref> have been described except for First n_b in batch 2. This is the same as First n in all other batches but the data has been pre-processed differently. 
  
  The classifier system (NNC) produces the best results in most runs, and this is consistent with our cross-validation results (Table <ref>). However, in contrast with Table <ref>, in most runs the second best is the regression system (NNR) instead of the untrained BERT system. All systems were re-trained for the BioASQ runs, keeping the same hyperparameters, so that we could use the entire training data, and it is possible that bad luck played a part here, and the BERT system was not trained to its best.
  
  The reinforcement learning experiments show the lowest evaluation results. We should note, however, that the RL runs submitted to BioASQ7b also had lower results than the other runs, but the human evaluation results ranked them higher than our other runs. Also, the reinforcement learning submissions were not re-trained using the entire training data, which may be worth exploring because PPO had a more consistent learning curve for us than REINFORCE. We are waiting for the human evaluation results of BioASQ8b with anticipation. 
  
  Table <ref> shows the results of 's submissions to BioASQ8b.
  
  
  We observe that the SBERT runs perform worse or on par with the BERT untrained model of Table <ref>. This might be because questions and answers may not appear closely in an embedding space, and therefore an identical processing of each of them might not be advantageous. For example, a question includes question words whereas the candidate sentences are not normally questions. We also observe that the BioBERT models seem to produce lower results when compared with the BERT untrained model of Table <ref>. In general, the multitask learning approach performed better than just classification or regression.
  
  
  
  
  
  
  
  § CONCLUSIONS
  
  
  This paper presents our approaches to query focused multi-document extractive summarisation for BioASQ8b. Our experiments include the use of BERT and BioBERT, Siamese architectures and SBERT, and Reinforcement Learning with PPO.
  
  We observed that an approach that uses BERT to obtain the word embeddings, followed by LSTM to map these word embeddings to sentence embeddings, had the most promising results. The variant with BioBERT did not present an improvement, and this conflicts with the overall improvement of the use of BioBERT for question answering on biomedical texts. The approaches with Siamese architectures did not present an improvement over the base versions, presumably because questions and candidate sentences have different sentence styles.
  
  As further research we plan to explore fine-tuning techniques for the BERT and BioBERT variants.
  
  
  
  § ACKNOWLEDGEMENTS
  
  
  Research by Vincent Nguyen is supported by the Australian Research Training Program and the CSIRO Postgraduate Scholarship.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  splncs04
  
  
  