abstract: Software bug reports reported on bug-tracking systems often lack crucial information for the developers to promptly resolve them, costing companies billions of dollars. There has been significant research on effectively eliciting information from bug reporters in bug-tracking systems using different templates that bug reporters need to use. However, the need for asking follow-up questions persists. Recent studies propose techniques to suggest these follow-up questions to help developers obtain the missing details, but there has been little research on answering these follow-up questions, which are often unanswered. In this paper, we propose a novel approach that uses *CodeT5*, a deep learning model in combination with *Lucene*, an information retrieval technique that leverages the relevance of different bug reports, their components and follow-up questions along with *Embedding Similarity* based ranking to recommend answers. These top performing answers along with their bug report serving as additional context apart from the deficient bug report to the deep learning model for generating an answer. We evaluate our recommended answers with the manually annotated answers using similarity metrics like Normalized Smooth BLEU Score, METEOR, Word Mover's Distance and Semantic Similarity. We achieve a BLEU Score of up to $\approx$ 34 and Semantic Similarity of up to $\approx$ 64 which shows that the answers generated are understandable and good according to Google's standard and can outperform multiple baselines.
author:
;
bibliography: ref.bib
title: Answering Follow-up Questions on Bug Reports with Structured Information Retrieval and Deep Learning

# Introduction

Software bugs are points of failure in an application or code, preventing it from working as expected. These bugs are mitigated during the process of bug resolution which is one of the major tasks in software maintenance. These bugs are reported on different bug-tracking systems like GitHub and JIRA which are then investigated and resolved by developers.

Bug reports should contain crucial information like system configuration, expected results, obtained results, and reproducing steps that help a developer reproduce the bug, and hence determine the errors within the code. However, bug reporters often miss crucial information in their reports that are necessary for reproducing and resolving the bug. According to existing findings over 60% of bug reports lack the steps to reproduce a bug and over 40% lack the description of the expected behaviour of a software application. While many software projects on GitHub now require bug reports to adhere to specific templates, there are many instances where reporters fail to comply with these standards and do not provide all the information . In order to retrieve such missing information, developers pose questions to the reporter as follow-up questions soliciting the necessary information.

Recent studies help developers analyze new bugs reports by mining the repositories to obtain relevant historical bug information, such as bug triage (, , bug classification (, ), bug localization (, ), bug prediction , etc. Researchers mainly apply information retrieval (IR) and Deep Learning(DL) techniques with various textual feature extraction methods to formulate the full textual information in a given new bug report (i.e., title and description) as an initial question and automatically search for relevant bug files based on similarity matching (, , , , , , ). However, the support for constituting answers to the follow-up questions has not been addressed before. This paper is the first to propose a technique to recommend answers to such follow-up questions.

Question answering techniques involve automatically analyzing several documents together to generate answer for the question provided. The context provided is crucial for generating answers to such questions. Providing incorrect or insufficient context lead to incorrect or generic answers which are usually false positives.

Traditionally, question answering can be done using information retrieval and deep learning based techniques. Information Retrieval(IR) as a technique isn't sufficient to provide answers to the questions since the answers retrieved can be ambiguous due to poor syntactic overlap between the question and the answer. The corpus of documents being searched, may contain gaps and may not provide a good coverage of the system. This can lead to important information being missed, or irrelevant answers being recommended. As the amount of data grows, it can become more challenging and may be computationally expensive to efficiently retrieve relevant information. Traditional information retrieval techniques may not be sufficient to handle the scale of data. Apart from these challenges textual data often contains noise, such as spelling errors, grammatical mistakes, and irrelevant answers. This can make it more difficult for the system to retrieve relevant answers and filter out irrelevant answers. To overcome these challenges, researchers employ deep learning techniques to improve the accuracy and efficiency of question answering systems.

Deep learning(DL) techniques have shown great promise in improving the accuracy of QA tasks, there are still several challenges that need to be addressed to make these models even more effective. Deep learning models require large amounts of data to train effectively. However, for QA tasks, there is a limited amount of labeled data available, especially for specific domains or languages. This can lead to over-fitting and poor generalization of the models.QA tasks often require specialized knowledge in specific domains. Developing QA models that can effectively handle these domains is challenging, as it requires the model to have a deep understanding of the specific domain and the associated terminology. Natural language is inherently ambiguous and can have multiple interpretations. This can lead to confusion in QA models, as they may not be able to identify the correct answer or may provide an incorrect answer. QA models often require a large amount of computational resources, which can make them difficult to scale up to handle large amounts of data or high-throughput applications.

In this paper, we generate relevant answers to follow-up questions for deficient bug reports that lack important information for triage. To overcome these challenges in our domain we combine IR with DL technique to mitigate the aforementioned challenges in both techniques. The amount of labelled bug-reports, follow-up question and answers wasn't sufficient to fine-tune DL models. For determining the relevant bug report and answer for the context was done using Lucene, an IR based technique. It automatically selects follow-up questions, ranks list of candidate answers from a corpus using Lucene. The recommended answers are re-ranked using semantic similarity measures by using Embedding based Similarity and top retrieved answers along with the bug report and question is used as context along with the original bug report. A CodeT5 model that was pretrained on stack overflow was used to finally generated the top answers for the given follow-up question and context.

We chose the top 20 projects most popular on GitHub and collected $\approx$ 60K bug reports. We evaluate our technique based on different similarity metrics, syntactic similarity metrics like BLEU score and METEOR and semantic similarity metrics like Semantic Similarity Score and WMD of the top-performing answer against the manually annotated answer for a held-out dataset of 550 bug reports.

To summarize, the primary contributions of this paper are:

-   recommending the answers to follow-up questions using information retrieval and deep learning technique.

-   a large corpus of bug reports, follow-up questions and answers from open-source software development platforms like GitHub;

-   A comprehensive evaluation and validation of IR technique and deep learning techniques using automated performance metrics (e.g., Semantic Similarity score)

# Motivating Example

To demonstrate the motivation for our work, let us consider the example in Fig. . The issue is taken from the *tensorflow/tensorflow* repository on GitHub[^1]. The bug report discusses a memory allocation issue and provides minimal steps to reproduce, but it lacks sufficient information about the system and does not follow the issue template provided by GitHub. As a result, the developer requests missing information as a follow-up question. However, the bug reporter or anyone else facing the same issue did not provide an answer to the follow-up question. The bug was initially marked as stale and later closed due to no response.

Follow-up questions that are necessary to reproduce bugs are often unanswered, leading to them becoming stale. Such bugs take longer to be resolved or remain unresolved and are ultimately closed. Memory allocation bugs are considered severe issues that significantly impact software quality, motivating the need for answering these follow-up questions.

# Algorithm

The architecture of our proposed technique is shown in Figure. In this section, we describe the proposed algorithm in detail.

As input, our system requires 1) a bug report of interest; 2) a corpus of already posed follow-up questions extracted from GitHub issues, including their corresponding bug reports and candidate answers as described in Section IV - B2.

## Data Pre-processing

Before calculating the different scores required for ranking, the title, description, follow-up questions and candidate answers need to be pre-processed. The textual noise such as escape sequences, special characters, URLs and stack traces, images etc are removed using NLTK's regular expressions library[^2] keeping just natural language text and code elements. Lemmatization is performed on all of the components of the bug reports in both test bug report and the bug reports present in the corpus.

##  Relevance Score

The first step towards finding the answer to the follow-up question is to determine how relevant the candidate answers are with respect to the question and the bug report of interest. We initially calculate a relevance score by calculating the Lucene Score using the equation given below. Here $S = (t,d,t+d,t+d+q)$ and $t$ is the title, $d$ is the description and $q$ is the follow-up question.

$$CommonRelevanceScore = \sum LuceneScores({{}^{S}\mkern-1muC_{2}})$$

We find the summation of Lucene scores of each component$(t,d,q,t+d,t+d+q)$ of the bug report of interest with respect to the $(t,d,q,t+d,t+d+q)$ each of the bug reports' components and its respective candidate answers in the corpus. The goal here is to determine the similarity of the different components of the bug reports in the corpus to the test bug report.

Initially, a common Relevance score is determined for all the bug reports in the corpus with respect to the test bug report. The Lucene scores of the combination of the title, description and follow-up question of the test bug report with the title, description, and previous follow-up question combinations of the corpus' bug reports are found, called the common relevance score. Then the combination of title description and follow-up question of the test bug report with the three candidate answers of each of the corpus' bug reports are found. The common relevance score of each of the bug reports is summed with the relevance scores found against each of the candidate answers. Each of the candidate answers to the bug reports in the corpus now has a relevance score. The relevance scores for all the candidate answers are ranked in the order of the top scores, to get a ranked list of the top-performing candidate answers.

## Embedding Based Similarity Score

For further improving the ranked-list of the top-performing answers to the follow-up question, we incorporate embedding based similarity to re-rank the top recommended answers. We capture the semantic aspect of the answers with respect to the follow-up question, to overcome the scenario where the answers may not have a lexical overlap but are still semantically similar.

We use Word2Vec word embeddings to encode both the top performing answers and the question that was pretrained on the Stack Overflowand find the cosine similarity between the embeddings. The answers are now re-ranked in the decreasing order of their cosine similarity scores to get a new ranked list of the top recommended answers. To tackle the scenario where the embedding similarity score of multiple answers are similar, the previous rankings of the answers are given the priority and ranked on the basis of their relevance score ranks using the DOI ranking technique explained in Section III - C.

## Ranking

After obtaining the top-performing answer for the bug and its follow-up question, we initially rank them in the order of their relevance scores. However, after calculating the embedding based similarity score for the bug report, often there were multiple answers with similar scores. In order to take into account the relevance score ranking as well, we calculate the degree of interest of the top-performing answers after calculating the relevance scores. We can calculate the Degree of Interest(DOI) using the formula given below :

$$DOI = \frac{I}{N}$$

Here, $I$ is the position of the top-performing answers sorted in the order of their relevance score. N is of the top answers considered, in our case, N = 10,20,25,30,50 for obtaining the top 10, 20, 25, 30, and 50 answers.

## Using Finetuned Code-T5 Question Answering Model

In this step, we employ CodeT5 Generative Question Answering Model, a pre-trained encoder-decoder Transformer model to obtain the final top answers to the follow-up questions. CodeT5 leverages the code semantics conveyed from the developer-assigned identifiers to seamlessly support both code understanding and generation tasks. Studies have shown that CodeT5 significantly outperforms prior and generation tasks and have been found to better capture semantic information from code and text . In our scenario we used a fine-tuned model to perform this step since the amount of labelled data was insufficient in our scenario to train or fine-tune the model.

The model requires two components, a question and a context. We feed the follow-up question as the question to the model and for the context we provide the deficient bug report, the top recommended answer from the previous step along with its associated bug report for each of the top recommended answers obtained in the previous step. Finally, the fine-tuned model generates an answer to the follow-up question.

# Experimental Setup

We curated a large dataset consisting of 60k bug reports and evaluate our technique for the follow-up questions of 550 held-out datasets using four appropriate metrics from relevant literature - BLEU, METEOR , Semantic Similarity and WMD. To place our work in the literature, we compare our technique – with a relevant baseline. Through our experiments, we answer three research questions as follows:

1.  **RQ$\mathbf{_1}$: How does our technique perform in answering follow-up questions in terms of different automatic evaluation metrics on different subject systems?**

2.  **RQ$\mathbf{_2}$: Can our technique outperform the existing baseline in terms of automatic evaluation metrics?**

3.  **RQ$\mathbf{_3}$: How does different components impact the overall performance?**

##  Dataset Collection

### Feasibility Study

In order to investigate the feasibility of answering follow-up questions, we conducted a manual examination of 450 bug reports - the most recent 30 bug reports from each of the top 25 most forked repositories containing high number of issues. To do this, we analyzed the comments associated with each of these bug reports, with a focus on evaluating the quality of the follow-up questions asked and the percentage of these questions that were answered.

During our examination, we found that many of the follow-up questions were fairly generic in nature. These questions often asked for system specifications or software versions, for example,it was not uncommon for there to be existing answers to these questions already present in the comments of previous bug reports. Furthermore, we also observed that there were often duplicate or similar bug reports that contained information that answered these follow-up questions. While the answers found in these other bug reports may not have been phrased exactly the same way as the answer to the follow-up question being asked, they still provided the necessary information.

### Choosing the repositories for the corpus

For performing our experiments, we create a high-quality corpus of bug reports that includes titles, descriptions, follow-up questions, and answers, using a semi-automated approach for dataset collection since completely automating the process would result in a lot of noise in the data, while manually annotating the dataset would be tedious and time-consuming.

To curate the dataset, we followed the approach proposed by Imran et al. . We chose GitHub[^3] platform for our dataset collection, which is a popular open-source platform for reporting bugs and consist of multiple active repositories. The most starred active repositories containing a high number of issues reported as of 2023 were selected using the platform's advanced search.

Using GitHub's REST API[^4], the bug reports were collected from the selected repositories, including their titles, bug descriptions, issue IDs, the author who reported the bug, labels, the creation date (Created At), and the resolution date (Closed At).

To filter out feature requests from bug reports, only those issues labelled as "bug", "crash", or "defect" were included. Additionally, bug reports labelled as "needs more info" were selected to fetch those bug reports that are deficient. Approximately 2,000 bug reports were collected from each repository that were closed within the last 5 years.

##  Dataset Preparation

### Selecting Follow-up Questions

To select follow-up questions for each issue report, the first step was to fetch the issue comments for each bug report of all 15 repositories using Koshuke's Github API[^5] Java Library. For each comment, CommentID, CommentUSER (the author of the comment), Comment, and CommentTime (the time at which the comment was made) were collected. These comments were then merged with the previously fetched bug report.

Since each issue ID had multiple issue comments, the comments were grouped based on the issue ID and sorted based on the comment time for each unique group of issue ID.

The next step was to identify questions from the comments using the NLTK's Naive Bayes Classifier. The comments that began with an interrogative word and ended with a question mark were filtered following Imran et al.'s strategy. Additionally, comments that requested additional information using words like 'please', 'can you', etc., were considered as follow-up questions.

The first most meaningful question submitted to the bug report, that was not written by the author of the bug report, was chosen as the follow-up question for each bug. To verify the validity of the question, it was manually checked if it was relevant to the bug, and any other questions were discarded.

### Selecting Candidate Answers

The next step in building our dataset was choosing the candidate answers. We followed Imran et al's study and chose the comments subsequent to the question as the answer and added futher filtration strategies to determine the appropriate answer. During our feasibility study, we observed that the answers are provided by people who are facing similar issues. So we chose three criteria for selecting our candidate answers.

1.  **Candidate Answer 1:** We selected the first candidate answer as the first comment that was posted right after the follow-up question that is not authored by the person asking the question.

2.  **Candidate Answer 2:** For our second candidate answer, we chose the first comment after the follow-up question that was authored by the person who reported the bug on GitHub.

3.  **Candidate Answer 3:** For deternmining our third candidate answer, we used the BM25 algorithm to obtain the most similar comment to the chosen follow-up question.

Hence, we extracted the following features from each bug report: issueID, Title, Description, Follow-up Question, and the three candidate answers selected using the aforementioned criteria. These features were then merged into a single dataset containing bug reports and their corresponding follow-up questions and candidate answers prior to performing manual annotation.

### Manual Annotation of Candidate Answers for GoldSet

For evaluating our architecture, we require the most relevant answer to the follow-up question to determine to what extent the answer is relevant to the actual answer. First, we manually checked that the bug reports, questions, and answers did not contain stack trace information, were no longer than 10 lines of code, and did not include images or videos. We also discarded questions or candidate answers that required the reporter to implement a bug fix or a workaround or had resolved the issue.

To determine the most relevant answer, we performed manual annotation with six experienced users. A pilot study was conducted to determine the time required for completing the annotation and to identify any issues during annotation. Initially, five bug reports with titles, descriptions, five follow-up questions, and candidate answers were sent to two users for the study. After completing the study, each user reported that each annotation took about 2.5-3 minutes to complete. They found the description to be helpful but cluttered and harder to read. They reported that the bug description and answers were too cluttered and required further cleaning. Therefore, we cleaned the bug description and answers of the noise by removing new line characters and unnecessary special characters and links.

We selected 550 bug reports for manual annotation, with approximately 27 most recent bug reports from each of the 20 repositories. The bug reports were divided into six buckets containing approximately 92 bug reports, questions, and candidate answers pairs. Each user annotated three buckets, i.e., 276 bug reports per person, so each issue report was annotated by three people. The users annotated the most relevant answer to the follow-up question from the three candidate answers, and a majority voting was used to determine the most relevant answers for each of the answers chosen by the users. For scenario where all three users vote for the same answer, that answer is chosen as the accepted answer. When two people vote for one answer and the third person votes for another answer, the answer with two votes is chosen as the accepted answer. When the answers chosen by each of users are mutually exclusive and there is no clear majority, and a decision cannot be made through majority voting, three people discussed their preferences and try to find a compromise or a middle ground that satisfies everyone to some extent and chose the accepted answer, hence forming our gold set for evaluation .

## Evaluation Metrics

For evaluating the top-performing answers with the manually annotated answers, we use three different similarity metrics - BLEU Score, METEOR Score, Word Mover's Distance and semantic similarity metric. We define them as follows:

###  ***BLEU - Bi-Lingual Evaluation of Understanding***

BLEU (Bilingual Evaluation Understudy) score is a commonly used evaluation metric in Natural Language Processing (NLP) for assessing the quality of machine-generated text. It compares a candidate answer to one or more reference texts to determine how similar they are. BLEU score is based on the matching of n-grams between the candidate answer and the reference text. An n-gram is a contiguous sequence of n items from a given sample of text, such as words or characters.

The BLEU score is calculated using the formula: $$BLEU = BP \cdot exp \left ( \sum_{n=1}^{N}w_{n}log(p_{n}) \right )$$ where N is the maximum n-gram order considered, and w~n~ is the weight assigned to the n-gram order. The brevity penalty, BP, is a factor that penalizes the BLEU score when the candidate answer is shorter than the reference text. The term p~n~ is the modified n-gram precision, which measures the ratio of the overlapping n-grams between the candidate answer and the reference text, and the total number of n-grams in the candidate answer. The BLEU score is calculated for different n-gram orders, and their geometric mean is taken as the final score.

In the software engineering domain, BLEU score is commonly used to evaluate the quality of generated bug reports, code comments, and summaries. In this study, a case-insensitive BLEU score with one smoothing is used, which aligns with human judgment. The smoothing factor is added to avoid a zero score when the precision is zero, and the logarithm in the formula produces negative infinity. BLEU score is useful in evaluating how well a generated text captures the important information from the reference text. However, it has some limitations, such as being sensitive to word order and not considering semantic similarity.

###  ***METEOR - Metric for Evaluation of Translation with Explicit ORdering***

The METEOR (Metric for Evaluation of Translation with Explicit ORdering) score is a metric for evaluating the quality of machine translation output. It was introduced in 2005 by Banerjee et al as an alternative to the popular BLEU (Bilingual Evaluation Understudy) score.

The METEOR score measures the similarity between the machine-generated text and a reference text, taking into account both lexical and syntactic information. It is designed to be more robust than BLEU, which primarily focuses on lexical similarity and does not consider word order or grammatical correctness.

The METEOR score is based on a weighted combination of several matching and scoring algorithms. These include exact word matching, stemming, synonym matching, and word reordering. The weights assigned to each algorithm are determined by optimizing the correlation between the METEOR score and human judgments of translation quality.

The score ranges from 0 to 1, with higher values indicating better translation quality. Like other evaluation metrics, the METEOR score is not perfect and should be used in conjunction with other measures and human evaluation to obtain a more comprehensive assessment of translation quality.

### ***WMD - Word Mover's Distance***

Word Mover's Distance (WMD) is a measure of similarity between two pieces of text that takes into account the meaning of words and their relationships with each other. WMD is based on the idea that the distance between two documents is related to the minimum amount of "work" required to transform one document into the other, where "work" is defined as the distance between the word embeddings of the words in the two documents.

To calculate the WMD between two documents, we first represent each document as a bag of words, where each word is represented by its vector embedding. The distance between two words is measured using the Euclidean distance between their vector embeddings. The WMD between the two documents is then computed as the minimum "work" required to transform the bag of words of one document into the other, where "work" is defined as the sum of the distances between the embeddings of each word in one bag and its closest matching word in the other bag.

The WMD metric is particularly useful in situations where the documents being compared are dissimilar in length or structure, or when they use different words to describe the same concept. By taking into account the meaning of words and their relationships with each other, WMD can provide a more accurate measure of similarity between two documents than simpler metrics like cosine similarity or Euclidean distance.

In our scenario we compute the Word Mover's Distance, we obtain the vector representations of the top-ranked answers and the manually annotated answers and performed a head-to-head comparison between them.

### ***Semantic Similarity***

It uses contextualized word embeddings for both reference or accepted answer (ground truth) and candidate answer and computes the cosine similarity between each word from both reference and candidate. Although BLEU score is widely used to evaluate machine translation, it doesn't consider the semantic aspect of the answer. In a human study conducted by Haque et al., a better metric was determined by assessing which metric is more representative of human evaluators' perception. The study found that the Sentence-BERT encoder with cosine similarity had the highest correlation with human evaluated similarity. To generate the embedding for input text, we use the stsb-roberta-large6 pre-trained model, which provides a fixed-length numeric representation for any given text, as suggested by Haque et al . We then compute semantic similarity using the cosine similarity between the numerical representations from Sentence-BERT for, the reference answer (ref), and generated answer from our technqiue(gen).

$SemSim(ref, gen) = \cos(\text{sbert}(ref), \text{sbert}(gen))$

# Study Result

```latex
\begin{table*}[!ht]\centering
\caption{Performance using Hybrid Technique - without using the original bug report(Context1)}
\vspace{.1cm}
\resizebox{6.2in}{!}{
\begin{tabular}{|ccccccccccccc|}
\hline
\multicolumn{13}{|c|}{\textbf{Hybrid Performance - Context   1 (Lucene+Embedding+Codet5)}}  \\ \hline
\multicolumn{1}{|c|}{\multirow{2}{*}{\textbf{Subject System}}} & \multicolumn{4}{c|}{\textbf{Top1}}                                                                                & \multicolumn{4}{c|}{\textbf{Top 3}}                                                                                & \multicolumn{4}{c|}{\textbf{Top 5}}                                                                                \\ \cline{2-13}
\multicolumn{1}{|c|}{}                                         & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC\\    SIMILARITY\end{tabular}}} & \multicolumn{1}{c|}{\textbf{WMD}} & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC \\    SIMILARITY\end{tabular}}} & \multicolumn{1}{c|}{\textbf{WMD}} & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC\\    SIMILARITY\end{tabular}}} & \textbf{WMD} \\ \hline
\multicolumn{1}{|c|}{Python}                                   & \multicolumn{1}{c|}{23.03}         & \multicolumn{1}{c|}{0.205}           & \multicolumn{1}{c|}{42.2}                                                                      & \multicolumn{1}{c|}{5.038}        & \multicolumn{1}{c|}{25.72}         & \multicolumn{1}{c|}{0.314}           & \multicolumn{1}{c|}{57.1}                                                                       & \multicolumn{1}{c|}{4.63}         & \multicolumn{1}{c|}{27.72}         & \multicolumn{1}{c|}{0.336}           & \multicolumn{1}{c|}{61.2}                                                                      & 4.446        \\ \hline
\multicolumn{1}{|c|}{Java}                                     & \multicolumn{1}{c|}{14.29}         & \multicolumn{1}{c|}{0.284}           & \multicolumn{1}{c|}{30.9}                                                                      & \multicolumn{1}{c|}{4.812}        & \multicolumn{1}{c|}{22.36}         & \multicolumn{1}{c|}{0.343}           & \multicolumn{1}{c|}{40.1}                                                                       & \multicolumn{1}{c|}{4.588}        & \multicolumn{1}{c|}{26.3}          & \multicolumn{1}{c|}{0.532}           & \multicolumn{1}{c|}{52.2}                                                                      & 3.403        \\ \hline
\multicolumn{1}{|c|}{JavaScript}                               & \multicolumn{1}{c|}{22.81}         & \multicolumn{1}{c|}{0.241}           & \multicolumn{1}{c|}{39.4}                                                                      & \multicolumn{1}{c|}{5.055}        & \multicolumn{1}{c|}{24.33}         & \multicolumn{1}{c|}{0.278}           & \multicolumn{1}{c|}{45}                                                                         & \multicolumn{1}{c|}{4.941}        & \multicolumn{1}{c|}{27.86}         & \multicolumn{1}{c|}{0.39}            & \multicolumn{1}{c|}{53.8}                                                                      & 4.371        \\ \hline
\multicolumn{1}{|c|}{C++}                                      & \multicolumn{1}{c|}{19.68}         & \multicolumn{1}{c|}{0.345}           & \multicolumn{1}{c|}{45.1}                                                                      & \multicolumn{1}{c|}{4.505}        & \multicolumn{1}{c|}{23.17}         & \multicolumn{1}{c|}{0.37}            & \multicolumn{1}{c|}{48.2}                                                                       & \multicolumn{1}{c|}{4.145}        & \multicolumn{1}{c|}{28.25}         & \multicolumn{1}{c|}{0.444}           & \multicolumn{1}{c|}{53.5}                                                                      & 4.145        \\ \hline
\end{tabular}}
\vspace{.1cm}
\end{table*}
```

```latex
\begin{table*}[!ht]\centering
\caption{Performance using Hybrid Technique - using the original bug report(Context2)}
\vspace{.1cm}
\resizebox{6.2in}{!}{
\begin{tabular}{|ccccccccccccc|}
\hline
\multicolumn{13}{|c|}{\textbf{Hybrid Performance - Context 2 (Lucene+Embedding+Codet5)}}                                                      \\ \hline
\multicolumn{1}{|c|}{\multirow{2}{*}{\textbf{Subject System}}} & \multicolumn{4}{c|}{\textbf{Top1}}                                                                                & \multicolumn{4}{c|}{\textbf{Top 3}}                                                                                & \multicolumn{4}{c|}{\textbf{Top 5}}                                                                                \\ \cline{2-13}
\multicolumn{1}{|c|}{}                                         & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC\\ SIMILARITY\end{tabular}}} & \multicolumn{1}{c|}{\textbf{WMD}} & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC\\ SIMILARITY\end{tabular}}} & \multicolumn{1}{c|}{\textbf{WMD}} & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC\\ SIMILARITY\end{tabular}}} & \textbf{WMD} \\ \hline
\multicolumn{1}{|c|}{Python}                                   & \multicolumn{1}{c|}{26.2}          & \multicolumn{1}{c|}{0.14}            & \multicolumn{1}{c|}{43.2}                                                                         & \multicolumn{1}{c|}{5.004}        & \multicolumn{1}{c|}{29.25}         & \multicolumn{1}{c|}{0.23}            & \multicolumn{1}{c|}{58.8}                                                                         & \multicolumn{1}{c|}{4.89}         & \multicolumn{1}{c|}{34.12}         & \multicolumn{1}{c|}{0.29}            & \multicolumn{1}{c|}{64.5}                                                                         & 4.82         \\ \hline
\multicolumn{1}{|c|}{Java}                                     & \multicolumn{1}{c|}{22.36}         & \multicolumn{1}{c|}{0.24}            & \multicolumn{1}{c|}{42.9}                                                                         & \multicolumn{1}{c|}{4.97}         & \multicolumn{1}{c|}{32.7}          & \multicolumn{1}{c|}{0.26}            & \multicolumn{1}{c|}{42.9}                                                                         & \multicolumn{1}{c|}{4.64}         & \multicolumn{1}{c|}{33.82}         & \multicolumn{1}{c|}{0.57}            & \multicolumn{1}{c|}{54.7}                                                                         & 3.27         \\ \hline
\multicolumn{1}{|c|}{JavaScript}                               & \multicolumn{1}{c|}{26.06}         & \multicolumn{1}{c|}{0.3}             & \multicolumn{1}{c|}{46.3}                                                                         & \multicolumn{1}{c|}{4.91}         & \multicolumn{1}{c|}{31.76}         & \multicolumn{1}{c|}{0.34}            & \multicolumn{1}{c|}{45.5}                                                                         & \multicolumn{1}{c|}{4.75}         & \multicolumn{1}{c|}{32.38}         & \multicolumn{1}{c|}{0.36}            & \multicolumn{1}{c|}{54.2}                                                                         & 4.29         \\ \hline
\multicolumn{1}{|c|}{C++}                                      & \multicolumn{1}{c|}{25.73}         & \multicolumn{1}{c|}{0.21}            & \multicolumn{1}{c|}{46.13}                                                                        & \multicolumn{1}{c|}{5.35}         & \multicolumn{1}{c|}{28.03}         & \multicolumn{1}{c|}{0.26}            & \multicolumn{1}{c|}{54.3}                                                                         & \multicolumn{1}{c|}{4.93}         & \multicolumn{1}{c|}{30.25}         & \multicolumn{1}{c|}{0.29}            & \multicolumn{1}{c|}{56.1}                                                                         & 4.12         \\ \hline
\end{tabular}}
\vspace{.1cm}
\end{table*}
```

```latex
\begin{table*}[!ht]\centering
\caption{ Baseline Performance using IR technique - Lucene}
\vspace{.1cm}
\resizebox{6.2in}{!}{
\begin{tabular}{|ccccccccccccc|}
\hline
\multicolumn{13}{|c|}{\textbf{Lucene Performance}} \\ \hline
\multicolumn{1}{|c|}{\multirow{2}{*}{\textbf{Subject System}}} & \multicolumn{4}{c|}{\textbf{Top1}}                                                                                & \multicolumn{4}{c|}{\textbf{Top 3}}                                                                                & \multicolumn{4}{c|}{\textbf{Top 5}}                                                                                \\ \cline{2-13}
\multicolumn{1}{|c|}{}                                         & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC\\ SIMILARITY\end{tabular}}} & \multicolumn{1}{c|}{\textbf{WMD}} & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC\\  SIMILARITY\end{tabular}}} & \multicolumn{1}{c|}{\textbf{WMD}} & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC\\  SIMILARITY\end{tabular}}} & \textbf{WMD} \\ \hline
\multicolumn{1}{|c|}{Python}                                   & \multicolumn{1}{c|}{6.86}          & \multicolumn{1}{c|}{0.254}           & \multicolumn{1}{c|}{18.9}                                                                         & \multicolumn{1}{c|}{5.985}        & \multicolumn{1}{c|}{8.18}          & \multicolumn{1}{c|}{0.277}           & \multicolumn{1}{c|}{38.8}                                                                         & \multicolumn{1}{c|}{5.23}         & \multicolumn{1}{c|}{19.32}         & \multicolumn{1}{c|}{0.306}           & \multicolumn{1}{c|}{57.5}                                                                         & 4.913        \\ \hline
\multicolumn{1}{|c|}{Java}                                     & \multicolumn{1}{c|}{13.10}         & \multicolumn{1}{c|}{0.08}            & \multicolumn{1}{c|}{27.21}                                                                        & \multicolumn{1}{c|}{5.35}         & \multicolumn{1}{c|}{13.50}         & \multicolumn{1}{c|}{0.08}            & \multicolumn{1}{c|}{37.59}                                                                        & \multicolumn{1}{c|}{5.49}         & \multicolumn{1}{c|}{14.21}         & \multicolumn{1}{c|}{0.07}            & \multicolumn{1}{c|}{39.17}                                                                        & 5.46         \\ \hline
\multicolumn{1}{|c|}{JavaScript}                               & \multicolumn{1}{c|}{9.45}          & \multicolumn{1}{c|}{0.197}           & \multicolumn{1}{c|}{29.8}                                                                         & \multicolumn{1}{c|}{4.432}        & \multicolumn{1}{c|}{11.37}         & \multicolumn{1}{c|}{0.213}           & \multicolumn{1}{c|}{44.6}                                                                         & \multicolumn{1}{c|}{4.079}        & \multicolumn{1}{c|}{12.51}         & \multicolumn{1}{c|}{0.238}           & \multicolumn{1}{c|}{48.91}                                                                        & 4.008        \\ \hline
\multicolumn{1}{|c|}{C++}                                      & \multicolumn{1}{c|}{4.98}          & \multicolumn{1}{c|}{0.176}           & \multicolumn{1}{c|}{17.3}                                                                         & \multicolumn{1}{c|}{4.451}        & \multicolumn{1}{c|}{9.12}          & \multicolumn{1}{c|}{0.201}           & \multicolumn{1}{c|}{31.4}                                                                         & \multicolumn{1}{c|}{4.269}        & \multicolumn{1}{c|}{11.19}         & \multicolumn{1}{c|}{0.231}           & \multicolumn{1}{c|}{45.5}                                                                         & 4.118        \\ \hline
\end{tabular}}
\vspace{.1cm}
\end{table*}
```

```latex
\begin{table*}[!ht]\centering
\caption{ Performance using Lucene+CodeT5}
\vspace{.1cm}
\resizebox{6.2in}{!}{
\begin{tabular}{|ccccccccccccc|}
\hline
\multicolumn{13}{|c|}{\textbf{Lucene+CodeT5 Performance}}  \\ \hline
\multicolumn{1}{|c|}{\multirow{2}{*}{\textbf{Subject System}}} & \multicolumn{4}{c|}{\textbf{Top1}}                                                                                & \multicolumn{4}{c|}{\textbf{Top 3}}                                                                                & \multicolumn{4}{c|}{\textbf{Top 5}}                                                                                \\ \cline{2-13}
\multicolumn{1}{|c|}{}                                         & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC\\ SIMILARITY\end{tabular}}} & \multicolumn{1}{c|}{\textbf{WMD}} & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC\\ SIMILARITY\end{tabular}}} & \multicolumn{1}{c|}{\textbf{WMD}} & \multicolumn{1}{c|}{\textbf{BLEU}} & \multicolumn{1}{c|}{\textbf{METEOR}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}SEMANTIC\\  SIMILARITY\end{tabular}}} & \textbf{WMD} \\ \hline
\multicolumn{1}{|c|}{Python}                                   & \multicolumn{1}{c|}{22.24}         & \multicolumn{1}{c|}{0.141}           & \multicolumn{1}{c|}{21.4}                                                                         & \multicolumn{1}{c|}{5.012}        & \multicolumn{1}{c|}{20.06}         & \multicolumn{1}{c|}{0.283}           & \multicolumn{1}{c|}{41.24}                                                                        & \multicolumn{1}{c|}{4.042}        & \multicolumn{1}{c|}{23.54}         & \multicolumn{1}{c|}{0.299}           & \multicolumn{1}{c|}{58.3}                                                                         & 4.022        \\ \hline
\multicolumn{1}{|c|}{Java}                                     & \multicolumn{1}{c|}{13.86}         & \multicolumn{1}{c|}{0.16}            & \multicolumn{1}{c|}{28.13}                                                                        & \multicolumn{1}{c|}{4.46}         & \multicolumn{1}{c|}{19.39}         & \multicolumn{1}{c|}{0.206}           & \multicolumn{1}{c|}{36.44}                                                                        & \multicolumn{1}{c|}{4.199}        & \multicolumn{1}{c|}{22.39}         & \multicolumn{1}{c|}{0.276}           & \multicolumn{1}{c|}{47.61}                                                                        & 4.082        \\ \hline
\multicolumn{1}{|c|}{JavaScript}                               & \multicolumn{1}{c|}{20.24}         & \multicolumn{1}{c|}{0.206}           & \multicolumn{1}{c|}{32.2}                                                                         & \multicolumn{1}{c|}{4.879}        & \multicolumn{1}{c|}{21.29}         & \multicolumn{1}{c|}{0.308}           & \multicolumn{1}{c|}{46.04}                                                                        & \multicolumn{1}{c|}{4.702}        & \multicolumn{1}{c|}{24.85}         & \multicolumn{1}{c|}{0.3311}          & \multicolumn{1}{c|}{51.74}                                                                        & 4.48         \\ \hline
\multicolumn{1}{|c|}{C++}                                      & \multicolumn{1}{c|}{16.59}         & \multicolumn{1}{c|}{0.188}           & \multicolumn{1}{c|}{28.2}                                                                         & \multicolumn{1}{c|}{5.277}        & \multicolumn{1}{c|}{20.18}         & \multicolumn{1}{c|}{0.228}           & \multicolumn{1}{c|}{38.12}                                                                        & \multicolumn{1}{c|}{4.339}        & \multicolumn{1}{c|}{25.89}         & \multicolumn{1}{c|}{0.313}           & \multicolumn{1}{c|}{51.39}                                                                        & 4.276        \\ \hline
\end{tabular}}
\vspace{.1cm}
\end{table*}
```

## **Answering RQ$\mathbf{_1}$:How does our technique perform in answering follow-up questions in terms of different automatic evaluation metrics on different subject systems?**

Through this experiment, our aim is to analyze the performance of our technique using different evaluation metrics in terms of BLEU score, Semantic Similarity , METEOR and WMD. We categorized our held-out dataset into four different subject systems and summarized the results based on the popular programming language, namely, Python, Java, JavaScript, C++. We then evaluated them on two types of similarity metrics - syntactic similarity techniques like BLEU and METEOR, and semantic similarity techniques like Semantic Similarity and Word Mover's Distance. It is important to note that the WMD metric has a lower-is-better nature, meaning that lower scores indicate better performance, while the other metrics have a higher-is-better nature.

Furthermore, we compared the results of our technique for two different contexts provided to the fine-tuned codeT5 model. The first context - Context 1 - the top answer from the ranked list generated after performing the embedding based similarity, its associated bug report for each of the top answers. The second context - Context 2 - in addition to the context 1, the deficient bug report is also provided for generating the answer. We observed a performance improvement using Context 2 compared to Context 1. The results are summarized in Table I and II.

Our technique achieved average BLEU Score upto 25.33 for Top 1, 31.05 for Top 3 and 33.93 for Top 5 across the four subject systems, which is considered as understandable to good translation according to Google's AutoML Translation documentation[^6].

Our technique also shows an average of 0.20 for Top 1, 0.25 for Top 3 and 0.37 for Top 5 across the subject systems which is considered to be a reasonable. METEOR score relies on string matching between the generated answer and the reference answer. However, question answering systems may generate correct answers that use different wording or sentence structures than the reference answer. In these cases, METEOR score may not capture the semantic similarity between the system-generated answer and the reference answer accurately which is why we also evaluate our technique using Semantic Similarity.

We find that the answers from our technique have an average Semantic Similarity score of 43.90 for Top 1, 47.52 for Top 3, and 57.52 for Top 5 which indicates a major semantic overlap with the explanations from developers. Semantic Similarity Score and WMD have been shown to correlate better with human judgement, and hence, semantic similarity scores are more appropriate in our scenario. This is because there may not be a syntactic overlap between the answers, which is a common problem in question answering as per literature, but they may still be semantically similar to the annotated answer.

.3em **Summary of RQ$\mathbf{_1}$:** Our technique achieved an average BLEU score considered good for translation and a high semantic similarity score, which is more appropriate for the scenario.

## **Answering RQ$\mathbf{_2}$:Can our technique outperform the existing baseline in terms of automatic evaluation metrics?**

The baselines we have identified is ablation, which involves using only one component of our technique,using Lucene output directly, which represents simple retrieval method for recommending the answers to the follow-up question.

Comparing Table II and Table III, we can see that the performance improvement of the hybrid technique using the original bug report (Context 2) is significant compared to the baseline performance using the Lucene IR technique. The two tables compare the performance of two different techniques used for bug localization, namely IR based technique - Lucene Score and Hybrid Technique - using the original bug report (Context 2) that uses Lucene Scoring along with Embedding Similarity and CodeT5 based technique.

Table 1 shows the baseline performance of the IR technique implemented using Lucene. The table presents the performance of the Lucene-based approach on four different subject systems, namely Python, Java, JavaScript, and C++. The performance is evaluated based on four metrics: BLEU, METEOR, Semantic Similarity, and WMD. The results are reported for the top 1, top 3, and top 5 results.

Table 2, on the other hand, presents the performance of the hybrid technique that uses the original bug report (Context 2). Similar to Table 1, Table 2 presents the performance of the hybrid approach on the same four subject systems and evaluates the performance based on the same four metrics. The results are also reported for the top 1, top 3, and top 5 results.

For example, for the Python subject system, the BLEU score for Top 1 improved from 6.86% in Table 1 to 26.20% in Table 2. Similarly, the semantic similarity score for Top 1 improved from 18.9% in Table 1 to 43.20% in Table 2. The WMD score for Top 1 also improved from 5.98 in Table 1 to 5.004 in Table 2, which is an improvement of 16.4%.

Overall, the hybrid technique using the original bug report (Context 2) outperforms the baseline performance using the Lucene IR technique in all metrics and for all subject systems. The exact performance improvement varies depending on the metric and subject system, but it is consistently significant.

.3em **Summary of RQ$\mathbf{_2}$:** The study compares the performance of a hybrid technique with Lucene, a simple retrieval method as baseline, for recommending answers to follow-up questions. The percentage improvement for BLEU, Semantic Similarity, and WMD is reported for the top 1, top 3, and top 5 results. Overall, the study shows that the hybrid technique has better performance than Lucene in most subject systems and evaluation metrics.

## **Answering RQ$\mathbf{_3}$: How does different components impact the overall performance?**

Through this research question, we intend on determining the impact of each of the components in our technique on the quality of the answers generated using the automatic evaluation metrics with the help of an ablation study.

Our technique can be divided into three main components - IR, Embedding Semantics and Deep Learning Model.

-   The first component is Lucene which is an Information Retrieval based technique that is being used to obtain a ranked list of candidate answers for a given deficient bug report.

-   The second component is the Embedding Similarity metric that is uses Word2Vec pretrained on Stack Overflow for generating embeddings and Cosine Similarity for improving the ranked list of answers based on their similarity with the deficient bug report and the follow-up questions.

-   The third component is the Deep Learning model for generating answers. We use Fine-tuned CodeT5 model for generating the answers to the follow-up question by providing the top performing candidate answers, bug reports of those answers and the deficient bug report as the context and the follow-up question as the answer.

We compare the performances of these components individually and by combining them and compare their performances across the four subject systems. The performance of using only Lucene based retrieval of answers is summarized in Table III. The performance of combining Lucene and CodeT5 is summarized in table Table IV and the performance of using all three components, i.e using Lucene, Embedding Similarity and CodeT5 is summarized in Table I and II.

From the four subject systems, Java showed the least improvement in Table IV compared to Table I, II and III. Python and JavaScript showed a significant improvement in terms of BLEU, METEOR, and Semantic Similarity metrics. C++ showed the highest improvement in the WMD metric.

Overall, these results suggest that the performance of our technique in Table I are better than those in Table III and IV at generating high-quality answers across all the subject systems. However, there are varying degrees of improvement among the different subject systems and evaluation metrics, indicating that different subject systems may require different approaches to improve answers generation quality.

We find that, Table I and II, that uses all three components Lucene and Emdedding Similarity and CodeT5 based techniques outperformed Table IV Using Lucene and CodeT5 and Table III using only Lucene in terms of all the evaluation metrics, showing that the final answers generated using our techniques generates better answers when evaluated with the accepted answer.

.3em **Summary of RQ$\mathbf{_3}$:** The study compares the performance of a hybrid technique with Lucene and the importance of adding the embedding similarity to our technique. The percentage improvement for BLEU, Semantic Similarity, and WMD is reported for the top 1, top 3, and top 5 results across all three techniques. Overall, the study shows that each of the components lead to a significant improvement in performance.

# Threats to Validity

We identify a few threats to the validity of our study results. In this section, we will examine these threats and the steps that were taken to mitigate them.

**External Validity:** Threats to external validity refer to the extent to which our results can be generalized to other settings beyond the ones in our study. One of the threats to external validity in our study is the limited number of subject systems used for evaluation. However, it is worth noting that although we evaluated our technique using Python, Java, JavaScript, and C++, the underlying algorithm is not specific to any programming language and can be easily adapted to any traditional programming language.

Another threat to external validity is the small sample size of the manually annotated bug reports used for evaluation. To address this concern, we selected bug reports from four different subject systems using different frameworks to ensure diversity in our evaluation. We also conducted a manual analysis of the top-performing answer retrieved from both versions of the algorithm to provide a more nuanced understanding of our approach's effectiveness.

Finally, the external validity of our developer study is also threatened by the limited subset of bug reports used for evaluation. However, we attempted to minimize bias by selecting bugs with diverse content characteristics from several projects. Despite these efforts, caution should be exercised when generalizing the results of our study beyond the evaluated bug reports.

**Construct Validity :** Construct validity refers to the extent to which the experiment measures what it is intended to measure. In this context, the use of a manually annotated dataset for the held-out dataset evaluation poses a threat to construct validity, as the quality of the annotations may affect the results. To mitigate this issue, the annotators were given training to become familiar with each project, and acceptable Cohen's Kappa values were observed between the annotators.

The pre-processing of bug reports is another threat to the construct validity, as it only considers the text and upto only 10 lines of code of an issue and ignores any additional information such as images or links or stack traces. This approach may lead to missing potentially relevant information in inadequate bug reports. However, the authors made this decision to focus solely on the text-based and limited lines of code in the information, as it is the most widely used form of information in bug reports.

Overall, the authors have attempted to mitigate the threats to construct validity by using appropriate training for annotators, avoiding explicit validation of pattern correctness, and focusing on text-based information and limited lines of code to reduce noise. However, it is important to keep in mind these potential threats when interpreting the results of the study.

# Related Work

Prior research related to this area can be broadly grouped into three main categories including question-answering using information retrieval techniques, using purely deep learning technique and a hybrid technique using information retrieval and deep learning. But most of previous studies dealt with improving the quality of bug reports, approaches for improving deficient bug reports, and techniques in answering questions in domains external to the software engineering field. To the best of our knowledge, the proposed approach is the first effort toward answering the follow-up questions.

# Conclusion and Future Work

This paper describes a technique for answering the posed follow-up questions for deficient bug reports that lack important information for triage. Our technique automatically selects follow-up questions, ranked list of candidate answers from a corpus using Lucene, a popular information retrieval technique, improves the ranked list using embedding based similarity and then uses the ranked list answers along with their bug report an incomplete bug report to generate an answer using fine-tuned CodeT5 model of open source projects on GitHub. To evaluate our technique we curated a held out dataset of 550 bug reports and follow-up questions and corpus of 60k bug reports from 20 unique repositories and implemented a baseline. Our technique outperformed the baselines significantly.

There are several avenues of future work. First line of future work can be on broadening the evaluation, since it is a vital challenge to determine the most relevant answer to follow-up question for different software development contexts and requirements we can develop a tool and integrate our technique with real world platforms like GitHub or JIRA in order to assist developers in the field and gather more developer feedback. Apart from that the size of the labelled dataset can be increased, which can be suitable for fine-tuning CodeT5 for improving the performance of the answers generated.

[^1]: <https://github.com/tensorflow/tensorflow/issues/58280>

[^2]: <https://www.nltk.org/>

[^3]: <https://github.com/>

[^4]: <https://docs.github.com/en/rest?apiVersion=2022-11-28>

[^5]: <https://github-api.kohsuke.org/>

[^6]: <https://cloud.google.com/translate/automl/docs/evaluate>


abstract: Automated text generation has been applied broadly in many domains such as marketing and robotics, and used to create chatbots, product reviews and write poetry. The ability to synthesize text, however, presents many potential risks, while access to the technology required to build generative models is becoming increasingly easy. This work is aligned with the efforts of the United Nations and other civil society organisations to highlight potential political and societal risks arising through the malicious use of text generation software, and their potential impact on human rights. As a case study, we present the findings of an experiment to generate remarks in the style of political leaders by fine-tuning a pretrained AWD-LSTM model on a dataset of speeches made at the UN General Assembly. This work highlights the ease with which this can be accomplished, as well as the threats of combining these techniques with other technologies.
bibliography: example_paper.bib

# Introduction

The rise of Artificial Intelligence (AI) brings many potential benefits to society, as well as significant risks. These risks take a variety of forms, from autonomous weapons and sophisticated cyber-attacks, to the more subtle techniques of societal manipulation. In particular, the threat this technology poses to maintaining peace and political stability is especially relevant to the United Nations (UN) and other international organisations. In terms of applications of AI, several major risks to peace and political stability have been identified, including: the use of automated surveillance platforms to suppress dissent; fake news reports with realistic fabricated video and audio; and the manipulation of information availability .

While research into the field of computer-aided text generation has been ongoing for many years, it is not until more recently that capabilities in data-acquisition, computing and new theory, have come into existence that now enable the generation of highly accurate speech in every major language. Moreover, this availability of resources means that training a customised language model requires minimal investment and can be easily performed by an individual actor. There are also an increasing number of organisations publishing models trained on vast amounts of data (such as OpenAI's GPT2-117M model ), in many cases removing the need to train from scratch what would still be considered highly complex and intensive models.

Language models have many positive applications, including virtual assistants for engaging the elderly or people with disabilities, fraud prevention and hate speech recognition systems, yet the ability of these models to generate text can be used for malicious intent. Being able to synthesize and publish text in a particular style could have detrimental consequences augmenting those witnessed from the dissemination fake news articles and generated videos, or 'deep fakes'. For instance, there have been examples of AI generated videos that depict politicians (Presidents Trump and Putin among others) making statements they did not truly make . The potential harm this technology can cause is clear, and in combination with automatic speech generation presents even greater challenges. Furthermore, by utilising social media platforms, such textual content can now be disseminated widely and rapidly, and used for propaganda, disinformation and personal harm on a large scale.

In this work, we present a case study highlighting the potential for AI models to generate realistic text in an international political context, and the ease with which this can be achieved (Section ). From this, we highlight the implications of these results on the political landscape, and from the point of view of promoting peace and security (Section ). We end with recommendations for the scientific and policy communities to aid in the mitigation of the possible negative consequences of this technology (Section ).

# Case Study

We present a proof-of-concept experiment to understand the complexity, and illustrate the possibilities, of automatic text generation in the international political sphere.

## Overview

In this experiment, we use English language transcripts of speeches given by political leaders at the UN General Assembly (UNGA) between 1970 and 2015 inclusive, as training data . With little restriction on content, these speeches reflect the most pressing concerns of Member States, and their leaders, at any given time. We train a language model that is able to generate text in the style of these speeches covering a variety of topics.

Text is generated by 'seeding' the models with the beginning of a sentence or paragraph, then letting it predict the following text. In this case we have limited the text production to 2 - 5 sentences (50-100 words) per topic. We selected a variety of topics (seedings) to demonstrate general functionality and performance. To demonstrate the performance of the model, paragraphs are generated in a variety of contexts: (1) minimal input - just a simple topic, (2) auto completion of UN Secretary-General remarks, and (3) digressions on sensitive issues (see Section for examples).

## Methodology

Training a language model from scratch is a complex task, requiring access to vast amounts of data and computational power. Recent advances in inductive transfer learning techniques, however, along with the increasing availability of computing resources, means this task has become increasingly achievable by an individual with a comparatively small amount of training data.

### Data

The UNGA speeches dataset, compiled by Baturo *et al.* , contains the text from 7,507 speeches given between 1970-2015 inclusive. Over the course of this period a variety of topics are discussed, with many debated throughout (such as nuclear disarmament). Although the linguistic style has changed over this period, the context of these speeches constrains the variability to the formal domain. Before training the model, the dataset is split into 283,593 paragraphs, cleaned by removing paragraph deliminators and other excess noise, and tokenized using the spaCy tokenizer .

### Model Training

In training the language model we follow the methodology as laid out by Howard and Ruder . We begin with an AWD-LSTM model pretrained on Wikitext-103 , thus giving a breath of understanding across a range of topics and vocabulary. Although it has been shown that pretraining on this dataset still allows for a high degree of generalisability to other tasks , we find the Wikitext-103 dataset particularly advantageous as it also follows a more formal linguistic structure. The language model is then fine-tuned to the cleaned dataset using *discriminative learning rates* and *slanted triangular learning rates* , largely utilising the fastai library . The language model was trained in under 13 hours on NVIDIA K80 GPUs, costing as little as \$7.80 on AWS spot instances.

## Results

Here we show a sample of results generated from the model in an attempt to firstly, construct coherent speech-like examples on topics known to be discussed in the dataset (see [Example 1](#example1)); secondly, demonstrate auto completion of remarks made by a specific leaders, such as the UN Secretary-General, on current issues (see [Example 2](#example2)) and finally, to show some more disturbing generated speech excerpts (see [Example 3](#example3)). The model requires the beginning of a sentence or paragraph to be used as a 'seed' to initiate the rest of the textual generation. The chosen seed is given in bold.

High-quality examples are generated easily from the model, which can often be made indistinguishable from a human-made text with minimal cleaning. Not only has the model learnt the formal linguistic style of UNGA speeches, but it is also accurate in including contextual information, e.g. about nations discussed in the text (see [Example 1](#example1)). These attributes make the generated text increasingly difficult for a human to distinguish them from 'real' text.

Coherent text on subjects regularly discussed in the dataset of a linguistic quality similar to that shown in [Example 1](#example1), were generated $\sim$`<!-- -->`{=html}90% of the time. Due to the relatively benign and diplomatic nature of the dataset, the inflammatory speech examples required several reruns of the model to generate samples of similar characteristics as [Example 3](#example3), with acceptable examples produced $\sim$`<!-- -->`{=html}60% of the time.

The aim of this experiment was to provide a simple proof of concept. Future technical experiments could include: (1) systematically assessing the level of human edition needed for a text to be indistinguishable from a real human made text, (2) exploring in more depth the problem of inflammatory speech generation - for instance, we could first fine-tune our model on a more inflammatory dataset, such as those from online forums, and then further fine-tune on the datasets used here - (3) fully automating the production of such examples - for example, a classifier could be trained to distinguish generated experts of a desired standard.

*Example 1*: 'Regular' speech-style text given generic prompting on current issues.

*Example 2*: UN Secretary-General remark auto completion.

*Example 3*: Hateful and politically inflammatory speech.

# Implications

The above experiment highlights the relative ease with which a language model can be created, and potentially used for malicious purposes. Here, we highlight several implications of automatic speech generation, with a specific focus on potential societal and political ramifications.

**1. Availability.** The results shown by this experiment, and other studies (e.g. ), while not always indistinguishable from human writers, demonstrate a high level of sophistication, including the generation of all punctuation and styling. Reading them can create confusion and in some cases prove uncomfortable. Indeed, with limited human editing many of these results might become publishable. Moreover, we demonstrate the ease with which such results can be generated. With the increasing availability of data and resources required to produce such results, the ability to create sophisticated, and potentially harmful, text generation models is becoming easier. Indeed, organisations such as OpenAI have refused to release advanced text generation models and training code for fear of malicious use , yet within a few months this technology will likely have been replicated and open sourced by other individuals.
**2. Easier disinformation and fake news dissemination**. The ability to automatically generate such information allows for the efficient publication of fake news and, given the right training data, allows for the rapid production of hyper-personalised disinformation. Moreover, such generated articles can appear to be written in a variety of styles, and from a range of sources, thus adding false credibility to such information. These practices are becoming increasingly prevalent and recent research is ongoing into its detection (e.g. ).
**3. Automated generation of hate speech** (see [Example 3](#example3)) presents a critical challenge to human rights. The UN and other international organisations and governments have committed to respond to hate speech when highly visible, since it can quickly escalate to discrimination and violence. This is of particular importance in situations where groups are targeted on the basis of discrimination or to incite political instability. Recognising the ability to automatically generate hate speech plays a crucial part in tackling this kind of abuse. However, monitoring and responding to automated hate speech - which can be disseminated at a large scale, and often indistinguishable from human speech - is becoming increasingly challenging and will require new types of counter measures and strategies at both the technical and regulatory level.
**4. Impersonation.** Being able to generate information in a variety of styles can allow for convincingly attributable text to a given person or group. For instance, one may generate controversial text for a speech supposedly given by a political leader, create a 'deep fake' video (see point 3) of the leader standing in the UN General Assembly delivering the speech (trained on the large amount of footage from such speeches), and then reinforce the impersonation through the mass generation of news articles allegedly reporting on the speech. Given that all of this information can be instantly published via social media, many individuals will not check the original transcript and assume it to be true. Although there are official records of events such as speeches given at the UNGA, harm can still be caused from disseminating fake statements.

# Conclusion

In this paper we have presented a proof-of-concept experiment to illustrate the ease with which a highly-accurate model that generates politically sensitive text can be created and highlighted the potential dangers of automated text generation, specifically in the context of peace and political stability. Based on this experiment, we put forward a series suggestions for the scientific and policy communities that we believe could help to address and mitigate these dangers:
1. **Mapping the potential human rights impacts of these technologies** - while there has been important work in this field more broadly (e.g. ), we must continue to assess these impacts in specific contexts to enhance mitigation efforts and better understand the struggles of potential victims. Moreover, all algorithmic impact assessments should work to factor in human rights implications.
2. **Development of tools for systematically and continuously monitoring AI generated content** - such measures are being implemented by many social media platforms, however, there also needs to be greater awareness and ownership across institutions outside of the technology sector. Public and private institutions should work together to implement relevant monitoring systems, adapting them to the different evolving cultural and societal contexts.
3. **Setting up strategies for countermeasures and scenario planning for critical situations** - although adversarially generated text will not be completely eliminated, preemptive strategies, e.g. better societal education on identifying fake reports, along with countermeasures, can help lessen the impact of disinformation attacks.
4. **Building alliances including civil society, international organisations and governments with technology providers, platforms and researchers for a coherent and proactive global strategy** - ecosystems built around AI technologies should be treated as complex systems and the necessity for a multidisciplinary approach to tackling the risks should be recognised (see e.g. ).
The increasing convergence and ubiquity of AI technologies magnify the complexity of the challenges they present, and too often these complexities create a sense of detachment from their potentially negative implications. We must, however, ensure on a human level that these risks are assessed. Laws and regulations aimed at the AI space are urgently required and should be designed to limit the likelihood of those risks (and harms). With this in mind, the intent of this work is to raise awareness about the dangers of AI text generation to peace and political stability, and to suggest recommendations relevant to those in both the scientific and policy spheres that aim to address these challenges.

# Acknowledgements

JB and MLO are with the United Nations Global Pulse innovation initiative supported by the Governments of Sweden, Netherlands and Germany and the William and Flora Hewlett Foundation. JB also is supported by the UK Science and Technology Facilities Council (STFC) grant number ST/P006744/1.