Question: Represent the article for finding a single-sentence TLDR

Question: Noise injection is a fundamental tool for data augmentation, and yet there is no widely accepted procedure to incorporate it with learning frameworks. This study analyzes the effects of adding or applying different noise models of varying magnitudes to Convolutional Neural Network (CNN) architectures. Noise models that are distributed with different density functions are given common magnitude levels via Structural Similarity (SSIM) metric in order to create an appropriate ground for comparison. The basic are conforming with the most of the common notions in machine learning, and also introduces some novel heuristics and recommendations on noise injection. The new approaches will provide better understanding on optimal learning procedures for image classification. Convolutional Neural Networks (CNNs) find an ever-growing field of application throughout image and sound processing tasks, since the success of AlexNet in the 2012 ImageNet competition. Yet, training these networks still keeps the need of an "artistic" touch: even the most cited state-of-the-art studies employ wildly varying set of solvers, augmentation and regularization techniques . In this study, one of the crucial data augmentation techniques, noise injection, will be thoroughly analysed to determine the correct way of application on image processing tasks. Adding noise to the training data is not a procedure that is unique to the training of neural architectures: additive and multiplicative noise has long been used in signal processing for regression-based methods, in order to create more robust models . The technique is also one of the oldest data augmentation methods employed in the training of feed forward networks, as analysed by , yet it is also pointed out in the same study that while using additive Gaussian noise is helpful, the magnitude of the noise cannot be selected blindly, as a badly-chosen variance may actually harm the performance of the ing network (see and for more examples). The main reasons for noise injection to the training data can be listed as such in a non-excluding manner: first of all, injection of any noise type makes the model more robust against the occurrence of that particular noise over the input data (see and for further reference), such as the cases of Gaussian additive noise in photographs, and Gaussian-Poisson noise on low-light charge coupled devices . Furthermore, it is shown that the neural networks optimize on the noise magnitude they are trained on . Therefore, it is important to choose the correct type and level of the noise to augment the data during training. Another reason for noise addition is to encourage the model to learn the various aspects of each class by occluding random features. Generally, stochastic regularization techniques embedded inside the neural network architectures are used for this purpose, such as Dropout layers, yet it is also possible to augment the input data for such purposes as in the example of "cutout" regularization proposed by. The improvement of the generalization capacity of a network is highly correlated with its performance, which can be scored by the accuracy over a predetermined test set. There has been similar studies conducted on the topic, with the example of which focuses on the effects of noise injection on the training of deep networks and the possible denoising methods, yet they fail to provide a proper methodology to determine the level of noise to be injected into the training data, and use PSNR as the comparison metric between different noise types which is highly impractical (see Section 3). To resolve these issues, this study focuses on the ways to determine which noise types to combine the training data with, and which levels, in addition to the validity of active noise injection techniques while experimenting on a larger set of noise models. In the structure of this work, the effect of injecting different types of noises into images for varying CNN architectures is assessed based on their performance and noise robustness. Their interaction and relationship with each other are analyzed over (also noise-injected) validation sets. Finally as a follow-up study, proper ways on adding or applying noise to a CNN for image classification tasks are discussed. Noise can be -somewhat broadly-defined as unwanted component of the image . It can be sourced from the environment at which the image is taken, the device utilized to take the image, or the medium of communication that is used to convey the information of image from source to the receiver. According to its properties and nature, noise and image can be analytically decomposed as additive or multiplicative, but some of the noise types cannot be described by neither of these classes. Let f (x) denote an image signal. This signal can be decomposed into two components in an additive manner as f (x) = g(x) + n(x) where g(x) denoting the desired component of the image, and n(x) standing for the unwanted noise component. Most commonly encountered variant of this noise class is Gaussian noise, whose multivariate probability density function can be written as: where m and Σ denoting the n-dimensional mean vector and symmetric covariance matrix with rank n, respectively. In images, the mean vector m is generally zero, therefore the distribution is centered and the magnitude is controlled by the variance. This study also follows these assumptions. Again, let f (x) denote an image signal. This signal can also be decomposed into respective desired and noise components as f (x) = g(x)(1 + n(x)). The noise component in this model is called multiplicative noise. The most common variant in this case is called speckle noise, which may have different density functions, and in this study Gaussian is assumed. Similar with the additive noise, the mean is assumed to be 0 and the magnitude refers to the variance. Speckle noise can be encountered in coherent light imaging such as in the cases of SAR images and images with laser-based illumination, but they may also be observed in other digital images . There exists many other noise instances that cannot be modeled by additive or multiplicative decompositions. Most common of these types are listed below, whose effects on the performances of CNNs are also analysed in this study. Salt and pepper (S&P) noise. This noise manifests itself as a basic image degradation, for which only a few pixels in an image are noisy, but they are extremely noisy in a way that pixels are either completely black or white. The magnitude of this noise is the probability of a pixel to be completely black (i.e. pepper), completely white (i.e. salt), or stay unchanged. The probabilities for pepper and salt cases are assumed to be equal, and total probability of the degradation of a pixel is referred as the the magnitude of the noise. Poisson noise. Also referred as photon counting noise or shot noise, this noise type has a particular probability density function: where λ stands for both the variance and the mean of the distribution. As Poisson noise is signaldependant, it does not have a direct magnitude parameter similar to other noise types, therefore a magnitude factor c is used that divides the intensity values of all pixels from which the distribution is sampled, and returned to the original range by multiplying to the same factor. Occlusion noise. Although it is not generally referred as a noise type, occlusion of important features can happen for a number of reasons in an image: image may be cropped, damaged or a particular object in it may be hidden by an obstacle such as a tree. This noise is realized with zerointensity squares appearing on the image, and the magnitude is determined according to the size of the square as the shape of the occluding object does not have a large impact on the final performance of the model . As listed in this section, five different types of noise and their various combinations are added or applied to the images, with varying magnitudes. Robustness against each of these noise types is also assessed. In general, Peak Signal-to-Noise Ratio (PSNR), and similarly Mean Squared Error (MSE) are the most commonly used quality metrics in the image processing field. For an 8-bit two-dimensional MxN imagef (n 1, n 2) and its noise-free counterpart f (n 1, n 2), the MSE is defined as From above definition, PSNR (in dB) can be derived. P SN R = 10 log 10 255 There are several limitations of using PSNR as the image quality metric of a data set: it is shown that PSNR loses its validity as a quality metric when the content and/or codec of the images are different as in that case the correlation between subjective quality and PSNR is highly reduced . Also, even though the sensitivity of PSNR to Gaussian noise is very high, the metric is unable to present similar performance for different types of perturbation (Horé &). There exists another widely accepted image quality metric called Structural Similarity (SSIM), which resolves or alleviates some of the above-listed problems. The Structural Similarity between two non-negative image signals x and y, whose means, standard deviations and covariance are denoted by µ x, µ y, σ x, σ y and σ xy respectively, can be expressed as: where C 1 and C 2 are constants to avoid instability . This metric combines luminance, contrast and structure information in order to provide an assessment of similarity in the range from 0 to 1, where 1 stands for highest similarity. In image classification tasks, the objective is mostly to classify the depicted objects or beings according to the human perception. Therefore, it is crucial for the used metric to be consistent with human opinions: it is shown in several studies that SSIM provides a quality metric closer to average opinion of public than PSNR (. Furthermore, when sampled from the same noise distributions over identical images of a data set, distribution of PSNR values of the noisy images have significantly high kurtosis than their SSIM counterparts in every noise type except S&P. This behavior is demonstrated over a subset of ImageNet dataset called Imagewoof (can be accessed via github.com/fastai/imagenette), with 10 classes and 12454 total number of images on the training set. Each type of noise listed in Section 2 is applied to each image, sampling from the same distribution, and quality metrics are recorded. The utilized noise magnitudes are 0.2 variance for Gaussian noise, 0.2 variance for speckle noise, 0.05 total probability for S&P noise, 0.2 scaling factor for Poisson noise, and 0.3 relative side length for occlusion noise; which are all chosen to provide sensible values from the metrics. The distribution of the metrics can be seen at Figure 1 Table 1. This is interpreted as PSNR having propensity to produce more outliers than SSIM for the same levels of noise . For the reasons listed above, SSIM will be used as the primary metric throughout this study. Effects of injecting different noise types into training data are evaluated for different magnitudes and types of the noise as listed in Section 2. The chosen datasets are two different subsets of ImageNet dataset, namely Imagenette and Imagewoof, each consisting of 10 different classes with 12894 and 12454 training samples respectively and 500 test samples (both can be accessed via github.com/fastai/imagenette). Former dataset contains ten easily classified classes of the original set, while the latter task is the classification of ten different dog breeds and require the network to successfully learn the particularly small features of the classes. Image data range is henceforth from 0 to 1. In order to select the magnitudes of each noise component, a sweep for mean SSIM (MSSIM) of an array of noise magnitudes for each noise type is conducted over 200 images of Imagewoof dataset. Resulting graph can be seen in Figure 2. Very similar are also observed when the same procedure is conducted on Imagenette dataset. According to the shapes of the curves, a quadratic polynomial is fitted to the relative side length of occlusion noise, and logarithmic polynomials are fitted to the rest. Look-up table for the fittings can be seen at Table 2, which are also the degradations applied in the experiments. Exemplary application of the noise can be seen at Figure 3. As the training model, a 18-layer deep residual network (ResNet18V2) as proposed by is chosen, because of the fact that it is a well-known architecture and also sufficiently deep: demonstrate that performance of the deep networks are more sensitive to data augmentation than their more shallow counterparts. The residual connections and layering structure of ResNet18V2 exhibit similar architectural properties of the most often utilized CNNs in the field. Adam solver with learning rate 1e-4 is preferred for training. Models are trained for 20 epochs. No dropout or weight decay is used for regularization purposes, and Batch Normalization layers are used in accordance with. Chosen CNN architecture is trained with noise injected to the training data for all noise models described in Section 2, for magnitudes corresponding to respective MSSIM values from Table 2. Total number of 52 networks are trained (25 on noisy data and 1 on original data for each dataset). The categorical accuracy of each trained network on the validation set can be seen at Figures 4 and 5, and the accuracies of the CNNs depending on the noise they are trained with for noisy test set can be observed at Figure 8. The latter can be considered as the robustness test of the trained networks. One of the most important features of these heatmaps, robustness of the models against the noise injected to their input, is also plotted for each dataset individually in Figures 6 and 7. There are several confirmations to acquire from this set of for the literature: first of all, there exists a trade-off between noise robustness and clean set accuracy. Yet contrary to the common notion, we believe that the data presents a highly valid optimum for this exchange in our study. As it can be seen from Figures 6 and 7; in order to create a robust model against particular kind of noise while maintaining the performance of the model, one must apply a level of degradation that in 0.8 MSSIM over training data. We believe that as long as the noise or perturbation is somewhat homogeneously distributed, this rule of thumb will hold for all image classification tasks. However, the same thing cannot be said for non-homogeneously distributed noise models, as SSIM (and also PSNR as demonstrated in Section 3) fails to capture the level of degradation appropriately for such a verdict (see Occlusion in Figures 6 and 7). A second confirmation of the current literature is the fact that the neural networks optimize on the noise level they are trained with, as seen again at Figures 6 and 7, and also the diagonals of Figure 8. Yet, the level of this optimization is quite small after 0.5 MSSIM, featuring similar robustness for each trained model. Therefore, it is not particularly necessary to determine the noise level of a dataset, or sample the noise from a predetermined interval, as long as the MSSIM does not drop below 0.5, in which case noise removal techniques need to be considered for better models. As noted above, occlusion noise type will not be thoroughly analyzed in this section because of the fact that the quality metric has failed to provide sufficient comparative data for this discussion. Yet, the performance data and the lack of robustness the other models exhibit towards this particular noise type shows that "cutout" regularization as presented by is a crucial part of data augmentation in addition to any other perturbation or noise injection technique. A way to further extend the contribution of this method would be to alternate the intensity level of the patches from 0 to 255 for 8-bit images, which can be a topic of another research. For the rest of the noise types; Gaussian, speckle and Poisson noises are observed to increase the performance of the model while boosting the robustness, and their effects exhibit the possibility of interchangeable usage. For image classification tasks involving RGB images of daily objects, injection of only one of these noise types with above-mentioned level is believed to be sufficient as repetition of the clusters can be observed in Figure 8. Among these three, Gaussian noise is recom-mended considering the of model performance. S&P noise contamination, on the other hand, may not be resolved by injection of the former noise types as the other models are not sufficiently robust against it. Therefore, at this point one of the two methodologies are suggested: either S&P noise can be removed by simple filtering techniques, or S&P noise can be applied in an alternating manner with Gaussian noise during data augmentation. Former approach is recommended for the simplicity of the training procedure. The constant behaviour of the models towards occlusion noise in Figures 6, 7 and 8 unfortunately does not have a satisfactory explanation, despite several diagnostics of the training procedure. A longer training procedure, which was not feasible in our experiment because of the model count, may resolve these undesirable . In this study, an extensive analysis of noise injection to training data has conducted. The confirmed some of the notions in the literature, while also providing new rule of thumbs for CNN training. As further targets of research, extension of "cutout" regularization as described in the above paragraphs, and the distribution behavior of the SSIM and PSNR metrics in Figure 2 with regards to the work of Horé & may be pursued.

represent the short summary for finding the corresponding scientific article (long).

Ideal methodology to inject noise to input data during CNN training

represent this text to find a much much longer article about it with say hundreds-thousands of words E.g.:
Graph-based recurrent retriever that learns to retrieve reasoning paths over Wikipedia Graph outperforms the most recent state of the art on HotpotQA by more than 14 points. == Answering questions that require multi-hop reasoning at web-scale necessitates retrieving multiple evidence documents, one of which often has little lexical or semantic relationship to the question. This paper introduces a new graph-based recurrent retrieval approach that learns to retrieve reasoning paths over the Wikipedia graph to answer multi-hop open-domain questions. Our retriever model trains a recurrent neural network that learns to sequentially retrieve evidence paragraphs in the reasoning path by conditioning on the previously retrieved documents. Our reader model ranks the reasoning paths and extracts the answer span included in the best reasoning path. Experimental show state-of-the-art in three open-domain QA datasets, showcasing the effectiveness and robustness of our method. Notably, our method achieves significant improvement in HotpotQA, outperforming the previous best model by more than 14 points. Open-domain Question Answering (QA) is the task of answering a question given a large collection of text documents (e.g., Wikipedia). Most state-of-the-art approaches for open-domain QA (; a; ;) leverage non-parameterized models (e.g., TF-IDF or BM25) to retrieve a fixed set of documents, where an answer span is extracted by a neural reading comprehension model. Despite the success of these pipeline methods in singlehop QA, whose questions can be answered based on a single paragraph, they often fail to retrieve the required evidence for answering multi-hop questions, e.g., the question in Figure 1. Multi-hop QA usually requires finding more than one evidence document, one of which often consists of little lexical overlap or semantic relationship to the original question. However, retrieving a fixed list of documents independently does not capture relationships between evidence documents through bridge entities that are required for multi-hop reasoning. Recent open-domain QA methods learn end-to-end models to jointly retrieve and read documents . These methods, however, face challenges for entity-centric questions since compressing the necessary information into an embedding space does not capture lexical information in entities. Cognitive Graph incorporates entity links between documents for multi-hop QA to extend the list of retrieved documents. This method, however, compiles a fixed list of documents independently and expects the reader to find the reasoning paths. In this paper, we introduce a new recurrent graph-based retrieval method that learns to retrieve evidence documents as reasoning paths for answering complex questions. Our method sequentially retrieves each evidence document, given the history of previously retrieved documents to form several reasoning paths in a graph of entities. Our method then leverages an existing reading comprehension model to answer questions by ranking the retrieved reasoning paths. The strong interplay between the retriever model and reader model enables our entire method to answer complex questions by exploring more accurate reasoning paths compared to other methods. structure of the documents during the iterative retrieval process. In addition, all of these multi-step retrieval methods do not accommodate arbitrary steps of reasoning and the termination condition is hard-coded. In contrast, our method leverages the Wikipedia graph to retrieve documents that are lexically or semantically distant to questions, and is adaptive to any reasoning path lengths, which leads to significant improvement over the previous work in HotpotQA and SQuAD Open. Overview This paper introduces a new graph-based recurrent retrieval method (Section 3.1) that learns to find evidence documents as reasoning paths for answering complex questions. We then extend an existing reading comprehension model (Section 3.2) to answer questions given a collection of reasoning paths. Our method uses a strong interplay between retrieving and reading steps such that the retrieval method learns to retrieve a set of reasoning paths to narrow down the search space for our reader model, for robust pipeline process. Figure 2 sketches the overview of our QA model. We use Wikipedia for open-domain QA, where each article is divided into paragraphs, ing in millions of paragraphs in total. Each paragraph p is considered as our retrieval target. Given a question q, our framework aims at deriving its answer a by retrieving and reading reasoning paths, each of which is represented with a sequence of paragraphs: E = [p i, . . ., p k]. We formulate the task by decomposing the objective into the retriever objective S retr (q, E) that selects reasoning paths E relevant to the question, and the reader objective S read (q, E, a) that finds the answer a in E: arg max E,a S(q, E, a) s.t. S(q, E, a) = S retr (q, E) + S read (q, E, a). Our method learns to retrieve reasoning paths across a graph structure. Evidence paragraphs for a complex question do not necessarily have lexical overlaps with the question, but one of them is likely to be retrieved, and its entity mentions and the question often entail another paragraph (e.g., Figure 1). To perform such multi-hop reasoning, we first construct a graph of paragraphs, covering all the Wikipedia paragraphs. Each node of the Wikipedia graph G represents a single paragraph p i. Constructing the Wikipedia graph Hyperlinks are commonly used to construct relationships between articles on the web, usually maintained by article writers, and are thus useful knowledge resources. Wikipedia consists of its internal hyperlinks to connect articles. We use the hyperlinks to construct the direct edges in G. We also consider symmetric within-document links, allowing a paragraph to hop to other paragraphs in the same article. The Wikipedia graph G is densely connected and covers a wide range of topics that provide useful evidence for open-domain questions. This graph is constructed offline and is reused throughout training and inference for any question. General formulation with a recurrent retriever We use a Recurrent Neural Network (RNN) to model the reasoning paths for the question q. At the t-th time step (t ≥ 1) our model selects a paragraph p i among candidate paragraphs C t given the current hidden state h t of the RNN. The initial hidden state h 1 is independent of any questions or paragraphs, and based on a parameterized vector. We use BERT's [CLS] token representation to independently encode each candidate paragraph p i along with q. 2 We then compute the probability P (p i |h t) that p i is selected. The RNN selection procedure captures relationships between paragraphs in the reasoning path by conditioning on the selection history. The process is terminated when [EOE], the end-ofevidence symbol, is selected, to allow it to capture reasoning paths with arbitrary length given each question. More specifically, the process of selecting p i at the t-th step is formulated as follows: where b ∈ R 1 is a bias term. Motivated by , we normalize the RNN states to control the scale of logits in Equation and allow the model to learn multiple reasoning paths. The details of Equation are described in Appendix A.1. The next candidate set C t+1 is constructed to include paragraphs that are linked from the selected paragraph p i in the graph. To allow our model to flexibly retrieve multiple paragraphs within C t, we also add K-best paragraphs other than p i (from C t) to C t+1, based on the probabilities. We typically set K = 1 in this paper. Beam search for candidate paragraphs It is computationally expensive to compute Equation over millions of the possible paragraphs. Moreover, a fully trainable retriever often performs poorly for entity-centric questions such as SQuAD, since it does not explicitly maintain lexical information . To navigate our retriever in the large-scale graph effectively, we initialize candidate paragraphs with a TF-IDF-based retrieval and guide the search over the Wikipedia graph. In particular, the initial candidate set C 1 includes F paragraphs with the highest TF-IDF scores with respect to the question. We expand C t (t ≥ 2) by appending the [EOE] symbol. We additionally use a beam search to explore paths in the directed graph. We define the score of a reasoning path E = [p i, . . ., p k] by multiplying the probabilities of selecting the paragraphs: P (p i |h 1)... P (p k |h |E|). The beam search outputs the top B reasoning paths E = {E 1, . . ., E B} with the highest scores to pass to the reader model i.e., S(q, E, a) = S read (q, E, a) for E ∈ E. In terms of the computational cost, the number of the paragraphs processed by Equation is bounded by O(|C 1 | + B t≥2 |C t |), where B is the beam size and |C t | is the average size of C t over the B hypothesises. Data augmentation We train our retriever in a supervised fashion using evidence paragraphs annotated for each question. For multi-hop QA, we have multiple paragraphs for each question, and single paragraph for single-hop QA. We first derive a ground-truth reasoning path g = [p 1, . . ., p |g|] using the available annotated data in each dataset. p |g| is set to [EOE] for the termination condition. To relax and stabilize the training process, we augment the training data with additional reasoning paths -not necessarily the shortest paths -that can derive the answer. In particular, we add a new training path g r = [p r, p 1, . . ., p |g|] by adding a paragraph p r ∈ C 1 that has a high TF-IDF score and is linked to the first paragraph p 1 in the ground-truth path g. Adding these new training paths helps at the test time when the first paragraph in the reasoning path does not necessarily appear among the paragraphs that initialize the Wikipedia search using the heuristic TF-IDF retrieval. Negative examples for robustness Our graph-based recurrent retriever needs to be trained to discriminate between relevant and irrelevant paragraphs at each step. We therefore use negative examples along with the ground-truth paragraphs; to be more specific, we use two types of negative examples: TF-IDF-based and hyperlink-based ones. For single-hop QA, we only use the type. For multi-hop QA, we use both types, and the type is especially important to prevent our retriever from being distracted by reasoning paths without correct answer spans. We typically set the number of the negative examples to 50. Loss function For the sequential prediction task, we estimate P (p i |h t) independently in Equation and use the binary cross-entropy loss to maximize probability values of all the possible paths. Note that using the widely-used cross-entropy loss with the softmax normalization over C t is not desirable here; maximizing the probabilities of g and g r contradict with each other. More specifically, the loss function of g at the t-th step is defined as follows: whereC t is a set of the negative examples described above, and includes [EOE] for t < |g|. We exclude p r fromC 1 for the sake of our multi-path learning. The loss is also defined with respect to g r in the same way. All the model parameters, including those in BERT, are jointly optimized. Our reader model first verifies each reasoning path in E, and finally outputs an answer span a from the most plausible reasoning path. This interplay is effective in making our framework robust; this is further discussed in Appendix A.3. We model the reader as a multi-task learning of reading comprehension, that extracts an answer span from a reasoning path E using a standard approach (; ;, and reasoning path re-ranking, that re-ranks the retrieved reasoning paths by computing the probability that the path includes the answer. For the reading comprehension task, we use BERT, where the input is the concatenation of the question text and the text of all the paragraphs in E. This lets our reader to fully leverage the self-attention mechanism across the concatenated paragraphs in the retrieved reasoning paths; this paragraph interaction is crucial for multi-hop reasoning (a). We share the same model for re-ranking, and use the BERT's [CLS] representation to estimate the probability of selecting E to answer the question: where w n ∈ R D is a weight vector. At the inference time, we select the best evidence E best ∈ E by P (E|q), and output the answer span by S read: where P start i, P end j denote the probability that the i-th and j-th tokens in E best are the start and end positions, respectively, of the answer span, and are calculated by following. Training examples To train the multi-task reader model, we use the ground-truth evidence paragraphs used for training our retriever. It is known to be effective in open-domain QA to use distantly supervised examples, which are not originally associated with the questions but include expected answer strings (; a;). These distantly supervised examples are also effective to simulate the inference time process. Therefore, we combine distantly supervised examples from a TF-IDF retriever with the original supervised examples. Following the procedures in , we add up to one distantly supervised example for each supervised example. We set the answer span as the string that matches a and appears first. To train our reader model to discriminate between relevant and irrelevant reasoning paths, we augment the original training data with additional negative examples to simulate incomplete evidence. In particular, we add paragraphs that appear to be relevant to the given question but actually do not contain the answer. For multi-hop QA, we select one ground-truth paragraph including the answer span, and swap it with one of the TF-IDF top ranked paragraphs. For single-hop QA, we simply replace the single ground-truth paragraph with TF-IDF-based negative examples which do not include the expected answer string. For the distorted evidenceẼ, we aim at minimizing P (Ẽ|q). The objective is the sum of cross entropy losses for the span prediction and re-ranking tasks. The loss for the question q and its evidence candidate E is as follows: where y start and y end are the ground-truth start and end indices, respectively. L no answer corresponds to the loss of the re-ranking model, to discriminate the distorted reasoning paths with no answers. P r is P (E|q) if E is the ground-truth evidence; otherwise P r = 1 − P (E|q). We mask the span losses for negative examples, in order to avoid unexpected effects to the span predictions. We evaluate our method in three open-domain Wikipedia-sourced datasets: HotpotQA, SQuAD Open and Natural Questions Open. We target all the English Wikipedia paragraphs for SQuAD Open and Natural Questions Open, and the first paragraph (introductory paragraph) of each article for HotpotQA following previous studies. More details can be found in Appendix B. HotpotQA HotpotQA is a human-annotated large-scale multi-hop QA dataset. Each answer can be extracted from a collection of 10 paragraphs in the distractor setting, and from the entire Wikipedia in the full wiki setting. Two evidence paragraphs are associated with each question for training. Our primary target is the full wiki setting due to its open-domain scenario, and we use the distractor setting to evaluate how well our method works in a closed scenario where the two evidence paragraphs are always included. The dataset also provides annotations to evaluate the prediction of supporting sentences, and we adapt our retriever to the supporting fact prediction. Note that this subtask is specific to HotpotQA. More details are described in Appendix A.5. SQuAD Open SQuAD Open is composed of questions from the original SQuAD dataset . This is a single-hop QA task, and a single paragraph is associated with each question in the training data. Natural Questions Open Natural Questions Open is composed of questions from the Natural Questions dataset , 3 which is based on Google Search queries independently from the existing articles. A single paragraph is associated with each question, but our preliminary analysis showed that some questions benefit from multi-hop reasoning. Metrics We report standard F1 and EM scores for HotpotQA and SQuAD Open, and EM score for Natural Questions Open to evaluate the overall QA accuracy to find the correct answers. For HotpotQA, we also report Supporting Fact F1 (SP F1) and Supporting Fact EM (SP EM) to evaluate the sentence-level supporting fact retrieval accuracy. To evaluate the paragraph-level retrieval accuracy for the multi-hop reasoning, we use the following metrics: Answer Recall (AR), which evaluates the recall of the answer string among top paragraphs (a;, Paragraph Recall (PR), which evaluates if at least one of the ground-truth paragraphs is included among the retrieved paragraphs, and Paragraph Exact Match (P EM), which evaluates if both of the ground-truth paragraphs for multi-hop reasoning are included among the retrieved paragraphs. Evidence Corpus and the Wikipedia graph We use English Wikipedia as the evidence corpus and do not use other data such as Google search snippets or external structured knowledge bases. We use the several versions of Wikipedia dumps for the three datasets (See Appendix B.5). To construct the Wikipedia graph, the hyperlinks are automatically extracted from the raw HTML source files. Directed edges are added between a paragraph p i and all of the paragraphs included in the target article. The constructed graph consists of 32.7M nodes and 205.4M edges. For HotpotQA we only use the introductory paragraphs in the graph that includes about 5.2M nodes and 23.4M edges. We use the pre-trained BERT models using the uncased base configuration (d = 768) for our retriever and the whole word masking uncased large (wwm) configuration (d = 1024) for our readers. We follow for the TF-IDF-based retrieval model and use the same hyper-parameters. We tuned the most important hyper-parameters, F, the number of the initial TF-IDF-based paragraphs, and B, the beam size, by mainly using the HotpotQA development set (the effects of increasing F are shown in Figure 5 in Appendix C.3 along with the with B = 1). If not specified, we set B = 8 for all the datasets, F = 500 for HotpotQA full wiki and SQuAD Open, and F = 100 for Natural Questions Open. 49.4 37.6 58.5 23.1 ----DecompRC (c) 43.3 ---70.6 ---MUPPET 40. Table 1 compares our method with previous published methods on the HotpotQA development set. Our method significantly outperforms all the previous across the evaluation metrics under both the full wiki and distractor settings. Notably, our method achieves 14.5 F1 and 14.0 EM gains compared to state-of-the-art Semantic Retrieval and 10.9 F1 gains over the concurrent Transformer-XH model on full wiki. We can see that our method, even with the BERT base configuration for our reader, significantly outperforms all the previous QA scores. Moreover, our method shows significant improvement in predicting supporting facts in the full wiki setting. We compare the performance of our approach to other models on the HotpotQA full wiki official hidden test set in Table 2. We outperform all the published and unpublished models including up-to-date work (marked with ♣) by large margins in terms of QA performance. On SQuAD Open, our model outperforms the concurrent state-of-the-art model (b) by 2.9 F1 and 3.5 EM scores as shown in Table 3. Due to the fewer lexical overlap between questions and paragraphs on Natural Questions, pipelined approaches using term-based retrievers often face difficulties finding associated articles. Nevertheless, our approach matches the performance of the best end-to-end retriever (ORQA), as shown in Table 4. In addition to its competitive performance, our retriever can be handled on a single GPU machine, while a fully end-to-end retriever in general requires industry-scale computational resources for training . More on these two datasets are discussed in Appendix D. We compare our retriever with competitive retrieval methods for HotpotQA full wiki, with F = 20. , the widely used retrieval method that scores paragraphs according to the TF-IDF scores of the question-paragraph pairs. We simply select the top-2 paragraphs. Re-rank that learns to retrieve paragraphs by fine-tuning BERT to re-rank the top F TF-IDF paragraphs. We select the top-2 paragraphs after re-ranking. Re-rank 2hop which extends Re-rank to accommodate two-hop reasoning. It first adds paragraphs linked from the top TF-IDF paragraphs. It then uses the same BERT model to select the paragraphs. Entity-centric IR is our re-implementation of that is related to Re-rank 2hop, but instead of simply selecting the top two paragraphs, they re-rank the possible combinations of the paragraphs that are linked to each other. Cognitive Graph Retrieval Table 5 shows that our recurrent retriever yields 8.8 P EM and 9. Re-rank2hop to Entity-centric IR demonstrates that exploring entity links from the initially retrieved documents helps to retrieve the paragraphs with fewer lexical overlaps. On the other hand, comparing our retriever with Entity-centric IR and Semantic Retrieval shows the importance of learning to sequentially retrieve reasoning paths in the Wikipedia graph. It should be noted that our method with F = 20 outperforms all the QA EM scores in Table 1. We conduct detailed analysis of our framework on the HotpotQA full wiki development set. Ablation study of our framework To study the effectiveness of our modeling choices, we compare the performance of variants of our framework. We ablate the retriever with 1) No recurrent module, which removes the recurrence from our retriever, and computes the probability of each paragraph to be included in reasoning paths independently and selects the path with the highest joint probability path on the graph; 2) No beam search, which uses a greedy search (B = 1) in our recurrent retriever; 3) No link-based negative examples, which trains the retriever model without adding hyperlink-based negative examples besides TF-IDF-based negative examples. We ablate the reader model with 1) No reasoning path re-ranking, which outputs the answer only with the best reasoning path from the retriever model, and 2) No negative examples, which trains the model only with the gold paragraphs, removing L no answer from L read. During inference,"No negative examples" reads all the paths and outputs an answer with the highest answer probability. Table 7: Performance with different link structures: comparing our on the Hotpot QA full wiki development set when we use an off-the-shelf entity linking system instead of the Wikipedia hyperlinks. Ablation Table 6 shows that removing any of the listed components gives notable performance drop. The most critical component in our retriever model is the recurrent module, dropping the EM by 17.4 points. As shown in Figure 1, multi-step retrieval often relies on information mentioned in another paragraph. Therefore, without conditioning on the previous time steps, the model fails to retrieve the complete evidence. Training without hyperlink-based negative examples in the second largest performance drop, indicating that the model can be easily distracted by reasoning paths without a correct answer and the importance of negative sampling for training. Replacing the beam search with the greedy search gives a performance drop of about 4 points on EM, which demonstrates that being aware of the graph structure is helpful in finding the best reasoning paths. Performance drop by removing the reasoning path re-ranking indicates the importance of verifying the reasoning paths in our reader. Not using negative examples to train the reader degrades EM more than 16 points, due to the over-confident predictions as discussed in. The performance with an off-the-shelf entity linking system Although the existence of the hyperlinks is not special on the web, one question is how well our method works without the Wikipedia hyperlinks. We evaluate our method on the development set of HotpotQA full wiki with an off-theshelf entity linking system to construct the document graph in our method. More details about this experimental setup can be found in Appendix B.7. Table 7 shows that our approach with the entity linking system shows only 2.3 F1 and 2.2 EM lower scores than those with the hyperlinks, still achieving the state of the art. This suggests that our approach is not restricted to the existence of the hyperlink information, and using hyperlinks is promising. The effectiveness of arbitrary-step retrieval The existing iterative retrieval methods fix the number of reasoning steps (; ;), while our approach accommodates arbitrary steps of reasoning. We also evaluate our method by fixing the length of the reasoning path (L = {1, 2, 3, 4}). Table 8 shows that out adaptive retrieval performs the best, although the length of all the annotated reasoning paths in HotpotQA is two. As discussed in Min et al. (2019b), we also observe that some questions are answerable based on a single paragraph, where our model flexibly selects a single paragraph and then terminates retrieval. The effectiveness of the interplay between retriever and reader Table 6 shows that the interplay between our retriever and reader models is effective. To understand this, we investigate the length of reasoning paths selected by our retriever and reader, and their final QA performance. Table 9 shows that the average length selected by our reader is notably longer than that by our retriever. (L = {1, 2, 3}). We observe that our framework performs the best when it selects the reasoning paths with L = 3, showing 63.0 EM score. Based on these observations, we expect the retriever favors a shorter path, while the reader tends to select a longer and more convincing multi-hop reasoning path to derive an answer string. Qualitative examples of retrieved reasoning paths Finally, we show two examples from HotpotQA full wiki, and Appendix C.5 presents more qualitative examples. In Figure 3, our approach successfully retrieves the correct reasoning path and answers correctly, while Re-rank fails. The top two paragraphs next to the graph are the introductory paragraphs of the two entities on the reasoning path, and the paragraph at the bottom shows the wrong paragraph selected by Re-rank. The "Millwall F.C." has fewer lexical overlaps and the bridge entity "Millwall" is not stated in the given question. Thus, Re-rank chooses a wrong paragraph with high lexical overlaps to the given question. In Figure 4, we compare the reasoning paths ranked highest by our retriever and reader. Although the gold path is included among the top 8 paths selected by the beam search, our retriever model selects a wrong paragraph as the best reasoning path. By re-ranking the reasoning paths, the reader eventually selects the correct reasoning path ("2017-18 Wigan Athletic F.C. season" → "EFL Cup"). This example shows the effectiveness of the strong interplay of our retriever and reader. This paper introduces a new graph-based recurrent retrieval approach, which retrieves reasoning paths over the Wikipedia graph to answer multi-hop open-domain questions. Our retriever model learns to sequentially retrieve evidence paragraphs to form the reasoning path. Subsequently, our reader model re-ranks the reasoning paths, and it determines the final answer as the one extracted from the best reasoning path. Our experimental significantly advance the state of the art on HotpotQA by more than 14 points absolute gain on the full wiki setting. Our approach also achieves the state-of-the-art performance on SQuAD Open and Natural Questions Open without any architectural changes, demonstrating the robustness of our method. Our method provides insights into the underlying entity relationships, and the discrete reasoning paths are helpful in interpreting our framework's reasoning process. Future work involves end-to-end training of our graph-based recurrent retriever and reader for improving upon our current two-stage training. where W r ∈ R d×2d is a weight matrix, b r ∈ R d is a bias vector, and α ∈ R 1 is a scalar parameter (initialized with 1.0). We set the global initial state a 1 to a parameterized vector s ∈ R d, and we also parameterize an [EOE] vector w [EOE] ∈ R d for the [EOE] symbol. The use of w i for both the input and output layers is inspired by;. In addition, we align the norm of w [EOE] with those of w i, by applying layer normalization of the last layer in BERT because w [EOE] is used along with the BERT outputs. Without the layer normalization, the L2-norms of w i and w [EOE] can be quite different, and the model can easily discriminate between them by the difference of the norms. Equation shows that we compute each paragraph representation w i conditioned on the question q. An alternative approach is separately encoding the paragraphs and the question, to directly retrieve paragraphs (; ; . However, due to the lack of explicit interactions between the paragraphs and the question, such a neural retriever using questionindependent paragraph encodings suffers from compressing the necessary information into fixeddimensional vectors, ing in low performance on entity-centric questions . It has been shown that attention-based paragraph-question interactions improve the retrieval accuracy if the retrieval scale is tractable (a;). There is a trade-off between the scalability and the accuracy, and this work aims at striking the balance by jointly using the lexical matching retrieval and the graphs, followed by the rich question-paragraph encodings. A question-independent variant We can also formulate our retriever model by using a questionindependent approach. There are only two simple modifications. First, we reformulate Equation as follows: where we no longer input the question q together with the paragraphs. Next, we condition the initial RNN state h 1 on the question information. More specifically, we compute h 1 by using Equation as follows: where w q is computed by using the same BERT encoder as in Equation, and h 1 is the original h 1 used in our question-dependent approach as described in Appendix A.1. The remaining parts are exactly the same, and we can perform the reasoning path retrieval in the same manner. Our retriever model learns to predict plausibility of the reasoning paths by capturing the paragraph interactions through the BERT's [CLS] representations, after independently encoding the paragraphs along with the question; this makes our retriever scalable to the open-domain scenario. By contrast, our reader jointly learns to predict the plausibility and answer the question, and moreover, fully leverages the self-attention mechanism across the concatenated paragraphs in the retrieved reasoning paths; this paragraph interaction is crucial for multi-hop reasoning (a). In summary, our retriever is scalable, but the top-1 prediction is not always enough to fully capture multi-hop reasoning to answer the question. Therefore, the additional re-ranking process mitigates the uncertainty and makes our framework more robust. In the HotpotQA dataset, we need to handle yes-no questions as well as extracting answer spans from the paragraphs. We treat the two special types of the answers, yes and no, by extending the re-ranking model in Equation. In particular, we extend the binary classification to a multi-class classification task, where the positive "answerable" class is decomposed into the following three classes: span, yes, and no. If the probability of "yes" or "no" is the largest among the three classes, our reader directly outputs the label as the answer, without any span extraction. Otherwise, our reader uses the span extraction model to output the answer. We adapt our recurrent retriever to the subtask of the supporting fact prediction in HotpotQA . The task is outputting sentences which support to answer the question. Such supporting sentences are annotated for the two ground-truth paragraphs in the training data. Since our framework outputs the most plausible reasoning path E along with the answer, we can add an additional step to select supporting facts (sentences) from the paragraphs in E. We train our recurrent retriever by using the training examples for the supporting fact prediction task, where the model parameters are not shared with those of our paragraph retriever. We replace the question-paragraph encoding in Equation with question-answer-sentence encoding for the task, where a question string is concatenated with its answer string. The answer string is the ground-truth one during the training time. We then maximize the probability of selecting the ground-truth sequence of the supporting fact sentences, while setting the other sentences as negative examples. At test time, we use the best reasoning path and its predicted answer string from our retriever and reader models to finally output the supporting facts for each question. The supporting fact prediction task is performed after finalizing the reasoning path and the answer for each question, and hence this additional task does not affect the QA accuracy. HotpotQA The HotpotQA training, development, and test datasets contain 90,564, 7,405 and 7,405 questions, respectively. To train our retriever model for the distractor setting, we use the distractor training data, where only the original ten paragraphs are associated with each question. The retriever model trained with this setting is also used in our ablation study as "retriever, no linkbased negatives" in Table 6. For the full wiki setting, we train our retriever model with the data augmentation technique and the additional negative examples described in Section 3.1.2. We use the same reader model, for both the settings, trained with the augmented additional references and the negative examples described in Section 3.2. SQuAD Open and Natural Questions Open For SQuAD Open, we use the original training set (containing 78,713 questions) as our training data, and the original development set (containing 10,570 questions) as our test data. For Natural Questions Open, we follow the dataset splits provided by Min et al. (2019a) To train our reader model for SQuAD Open, in addition to the TF-IDF top-ranked paragraphs, we add two types of additional negative examples: (i) paragraphs, which do not include the answer string, from the originally annotated articles, and (ii) "unanswerable" questions from SQuAD 2.0 . For Natural Questions Open, we add negative examples of the type (i). To use the pre-trained BERT models, we used the public code base, pytorch-transformers, 4 written in PyTorch. 5 For optimization, we used the code base's implementation of the Adam optimizer , with a weight-decay coefficient of 0.01 for non-bias parameters. A warm-up strategy in the code base was also used, with a warm-up rate of 0.1. Most of the settings follow the default settings. To train our recurrent retriever, we set the learning rate to 3 · 10 −5, and the maximum number of the training epochs to three. The mini-batch size is four; a mini-batch example consists of a question with its corresponding paragraphs. To train our reader model, we set the learning rate to 3 · 10 −5, and the maximum number of training epochs to two. Empirically we observe better performance with a larger batch size as discussed in previous work , and thus we set the mini-batch size to 120. A mini-batch example consists of a question with its evidence paragraphs. We will release our code to follow our experiments. For HotpotQA full wiki, we use the pre-processed English Wikipedia dump from , provided by the HotpotQA authors. 6 For Natural Questions Open, we use the English Wikipedia dump from December 20, 2018, following and Min et al. (2019a). For SQuAD Open, we use the Wikipedia dump provided by. Although using a single dump for different open-domain QA datasets is a common practice (; a;), this potentially causes inconsistent or even unfair evaluation across different experimental settings, due to the temporal inconsistency of the Wikipedia articles. More concretely, every Wikipedia article is editable and and as a , a fact can be rephrased or could be removed. For instance, a question from the SQuAD development set, "Where does Kenya rank on the CPI scale?" is originally paired with a paragraph from the article of Kenya. Based on a single sentence "Kenya ranks low on Transparency International's Corruption Perception Index (CPI)" from the paragraph, an annotated answer span is "low." However, this sentence has been rewritten as "Kenya has a high degree of corruption according to Transparency International's Corruption Perception Index (CPI)" in a later version of the same article. 7 This is problematic considering the major evaluation metrics based on string matching. Another problem exists especially in Natural Questions Open. The dataset contains real Google search queries, and some of them reflect temporal trends at the time when the queries were executed. If a query is related to a TV show broadcasted in 2018, we can hardly expect to extract the answer from a dump in 2017. Like this, although Wikipedia is a useful knowledge source for open-domain QA research, its rapidly evolving nature should be considered more carefully for the reproducibility. We will make all of the data including pre-processed Wikipedia articles for each experiment available for future research. B.6 DETAILS ABOUT INITIAL CANDIDATES C 1 SELECTION To retrieve the initial candidates C 1 for each question, we use a TF-IDF based retriever with the bi-gram hashing . For HotpotQA full wiki, we retrieve top F introductory paragraphs, for each question, from a corpus including all the introductory paragraphs. For SQuAD Open and Natural Questions Open, we first retrieve 50 Wikipedia articles through the same TF-IDF retriever, and further run another TF-IDF-based paragraph retriever (; a) to retrieve F paragraphs in total. We experiment with a variant of our approach, where we incorporate an entity linking system with our framework, in place of the Wikipedia hyperlinks. In this experiment, we first retrieve seed paragraphs using TF-IDF (F = 100), and run an off-the-shelf entity linker (TagMe by) over the paragraphs. If the entity linker detects some entities, we retrieve their corresponding Wikipedia articles, and add edges from the seed paragraphs to the entity-linked paragraphs. Once we build the graph, then we re-run all of the experiments while the other components are exactly the same. We use the TagMe official Python wrapper. For scalability and computational efficiency, we bootstrap our retrieval module with TF-IDF retrieval; we first retrieval F paragraphs using TF-IDF with the method described in Section B.6 and initialize C 1 with these TF-IDF paragraphs. Although we expand our candidate paragraphs at each time step using the Wikipedia graph, if our method failed to retrieve paragraphs a few-hops away from the answer paragraphs, it is likely to fail to reach the answer paragraphs. To estimate the paragraph EM upper-bound, we have checked if two gold paragraphs are included in the top 20 TF-IDF paragraphs and their hyperlinked paragraphs in the HotpotQA full wiki setting. We found that for 75.4% of the questions, all of the gold paragraphs are included in the collections of the TF-IDF paragraphs and the hyperlinked paragraphs. Also, it should be noted when we only consider the TF-IDF retrieval , the upper-bound drops to 35.1%, which suggests that the TF-IDF-based retrieval cannot effectively discover the paragraphs multi-hop away due to the few lexical overlap. When we increase the number of F to 100 and 500, the upper-bound reaches 84.1% and 89.2%, respectively. HOTPOTQA FULL WIKI In HotpotQA, there are two types of questions, bridge and comparison. While comparison-type questions explicitly mention the two entities related to the given questions, in bridge-type questions, the bridge entities are rarely explicitly stated. This makes it hard for a retrieval system to discover the paragraphs entailed by the bridge entities only. gain and 15.1 EM gain over Semantic Retrieval for the challenging bridge-type questions. For the comparison-type questions, our method achieves almost 10 point higher QA EM than Semantic Retrieval. We observed that some of the comparison-type questions can be answered based on single paragraph, and thus our model selects only one paragraph for some of these comparisontype questions, ing in lower P EM scores on the comparison-type questions. We show several examples of the questions where we can answer based on single paragraph in Section C.5. As we discussed in 3.1.1, we aim at significantly reducing the search space and thus scaling the number of initial TF-IDF candidates. Increasing the number of the initial retrieved paragraphs often improves the recall of the evidence paragraphs of the datasets. On the other hand, increasing the candidate paragraphs introduces additional noises, may distract models, and eventually hurt the performance . We compare the performance of three different approaches: (i) ours, (ii) ours (greedy, without reasoning path re-ranking), and (iii) Re-rank. We increase the number of the TF-IDF-based retrieved paragraphs from 10 to 500 (For Re-rank, we compare the performance up to 200 paragraphs). Figure 5 clearly shows that our approach is robust towards the increase of the initial candidate paragraphs, and thus can constantly yield performance gains with more candidate paragraphs. Our approach with the greedy search also shows performance improvements; however, after a certain number, the greedy approach stops improving the performance. Re-rank starts suffering from the noises caused by many distracting paragraphs included in the initial candidate paragraphs at F = 200. To show the importance of the question-paragraph encoding in our retriever model, we conduct an experiment on the development set of HotpotQA, by replacing it with the question-independent encoding described in Appendix A.2. For a fair comparison, we use the same initial TF-IDF-based retrieval (only for the full wiki setting), hyperlink-based Wikipedia graph, beam search, and reader model (BERT wwm). We train the alternative model without using the data augmentation technique (described in Section 3. Table 11 : Effects of the question-dependent paragraph encoding: Comparing our retriever model with and without the query-dependent encoding. For our question-dependent approach, the full wiki correspond to "retriever, no link-based negatives" in Table 6, and the distractor correspond to "Ours (Reader: BERT wwm)" Table 1, to make the comparable. Table 11 shows the in both the full wiki and distractor settings. As seen in this table, the QA F1 and EM performance significantly deteriorates on the full wiki setting, which demonstrates the importance of the question-dependent encoding for complex and entity-centric open-domain question answering. We can also see that the performance drop on the distractor setting is much smaller than that on the full wiki setting. This is due to its closed nature; for each question, we are given only ten paragraphs and the two gold paragraphs are always included, which significantly narrows the searching space down and makes the retrieval task much easier than that in the full wiki setting. Therefore, our recurrent retriever model is likely to discover the gold reasoning paths by the beam search, and our reader model can select the gold paths by the robust re-ranking approach. To verify this hypothesis, we checked the P EM score as a retrieval accuracy in the distractor setting. If we only consider the top-1 path from the beam search, the P EM score of the question-independent model is 12% lower than that of our question-dependent model. However, if we consider all the reasoning paths produced by the beam search, the coverage of the gold paths is almost the same. As a , our reader model can perform similarly with both the question-dependent/independent approaches. This additionally shows the robustness of our re-ranking approach. In this section, we conduct more qualitative analysis on the reasoning paths predicted by our model. Explicitly retrieving plausible reasoning paths and re-ranking the paths provide us interpretable insights into the underlying entity relationships used for multi-hop reasoning. As shown in Table 9, our model flexibly selects one or more paragraphs for each question. To understand these behaviors, we conduct qualitative analysis on these examples whose reasoning paths are shorter or longer than the original gold reasoning paths. Reasoning path only with single paragraph First, we show two examples (one is a bridge-type question and the other is a comparison-type question), where our retriever selects single paragraph and terminates without selecting any additional paragraphs. The bridge-type question in Table 12 shows that, while originally this question requires a system to read two paragraphs, Before I Go to Sleep (film) and Nicole Kidman, our retriever and reader eventually choose Nicole Kidman only. The second paragraph has a lot of lexical overlaps to the given question, and thus, a system may not need to read both of the paragraphs to answer. The comparison-type question in Table 12 also shows that even comparison-type questions do not always require two paragraphs to answer the questions, and our model only selects one paragraph necessary to answer the given example question. In this example, the question has large lexical overlap with one of the ground-truth paragraph (The Bears and I), ing in allowing our model to answer the question based on the single paragraph. Min et al. (2019b) also observed that some of the questions do not necessarily require multi-hop reasoning, while HotpotQA is designed to require multi-hop reasoning . In that sense, we can say that our method automatical detects potentially single-hop questions. Reasoning path with three paragraphs All of the HotpotQA questions are authored by annotators who are shown two relevant paragraphs, and thus, originally the length of ground-truth reason- Table 12: Two examples of the questions that our model retrieves a reasoning path with only one paragraph. We partly remove sentences irrelevant to the questions. Words in red correspond to the answer strings. ing paths is always two. On the other hand, as our model accommodates arbitrary steps of reasoning, it often selects reasoning paths longer than the original annotations as shown in Table 9. When our model selects a longer reasoning path for a HotpotQA question, does it contain paragraphs that provide additional evidence? We show an example in Table 13, so as to answer this question. Our model selects an additional paragraph, Blue Jeans (Lana Del Rey song) at the first step, and then selects the two annotated gold paragraphs. This first paragraph is strongly relevant to the given question, but does not contain the answer. This additional evidence might help the reader to find the correct bridge entity ("Back to December"). Although the main focus in this paper is on open-domain QA, we show the state-of-the-art performance on the HotpotQA distractor setting as well with the exactly same architecture. We conduct qualitative analysis to understand our model's behavior in the closed setting. In this setting, the two ground-truth paragraphs are always given for each question. Table 14 shows two examples from the HotpotQA distractor setting. In the first example, P1 and P2 are its corresponding ground-truth paragraphs. At the first time step, our retriever does not expect that P2 is related to the evidence to answer the question, as the retriever is not aware of the bridge entity, "Pasek & Paul". If we simply adopt the Re-rank strategy, P3 with the second highest probability is selected, ing in a wrong paragraph selection. In our framework, our retriever is conditioned on the previous retrieval history and thus, at the second time step, it chooses the correct paragraph, P2, lowering the probability of P3. This clearly shows the effectiveness of our multi-step retrieval method in the closed setting as well. At the third step, our model stops the prediction by Table 15: Statistics of the reasoning paths for SQuAD Open and Natural Questions Open: the average length and the distribution of length of the reasoning paths selected by our retriever and reader for SQuAD Open and Natural Questions Open. outputting [EOS]. In 588 examples (7.9%) of the entire distractor development dataset, the paragraph selection by our graph-based recurrent retriever differs from the top-2 strategy. We present another example, where only the graph-based recurrent retrieval model succeeds in finding the correct paragraph pair, (P1, P2). The second question in Table 14 shows that at the first time step our retriever successfully selects P1, but does not pay attention to P2 at all, as the retriever is not aware of the bridge entity, "the Russian Civil War". Again, once it is conditioned on P1, which includes the bridge entity, it can select P2 at the second time step. Like this, we can see how our model successfully learns to model relationships between paragraphs for multi-hop reasoning. Although the main focus of this work is on multi-hop open-domain QA, our framework shows competitive performance on the two open-domain QA datasets, SQuAD Open and Natural Questions Open. Both of the two dataets are originally created by assigning a single ground-truth paragraph for each question, and in that sense, our framework is not specific to multi-hop reasoning tasks. In this section, we further analyze our experimental on the two datasets. Table 15 shows statistics of the lengths of the selected reasoning paths on our SQuAD Open experiment. This table is analogous to Table 9 on our HotpotQA experiments. We can clearly see that our recurrent retriever always outputs a single paragraph for each question, if we only use the top-1 predictions. This is because our retriever model for this dataset is trained with the single-paragraph annotations. Our beam search can find longer reasoning paths, and as a , the re-ranking process in our reader model somtimes selects the reasoning paths including two paragraphs. The trend is consistent with that in Table 9. However, the effects of selecting more than one paragraph do not have a big impact; we observed only 0.1% F1/EM improvement over our method with restricting the path length to one (based on the same experiment with L = 1 in Table 8). Considering that SQuAD is a single-hop QA dataset, the matches our intuition. Table 15 also shows the on Natural Questions Open, where we see the same trend again. Thanks to the ground-truth path augmentation technique, our recurrent retriever model prefers longer reasoning paths than those on SQuAD Open. We observed 1% EM improvement over the L = 1 baseline on Natural Questions Open, and next we show an example to discuss why our reasoning path approach can be effective on this dataset. Table 16 shows one example where our model finds a multi-hop reasoning path effectively in Natural Questions Open (development set). The question "who sang the original version of killing me so" has relatively fewer lexical overlap with the originally annotated paragraph (Killing Me Softly with His Song (V) in Table 16 ). Moreover, there are several entities named as "killing me softly" in Wikipedia, because many artists cover the song. To answer this question correctly, our retriever first selects Roberta Flack (I), and then hops to the originally annotated paragraph, Killing Me Softly with His Song (V). Our reader further verifies this reasoning path and extracts the correct answer from Killing Me Softly with His Song (V). This example shows that even without gold reasoning paths annotations, our model trained on the augmented examples learns to retrieve multihop reasoning paths from the entire Wikipedia. These detailed experimental on the two other open-domain QA datasets demonstrate that our framework learns to retrieve reasoning paths flexibly with evidence sufficient to answer a given question, according to each dataset's nature.


Question: Represent the scientific article to find a short TLDR

Question: State-of-the-art Unsupervised Domain Adaptation (UDA) methods learn transferable features by minimizing the feature distribution discrepancy between the source and target domains. Different from these methods which do not model the feature distributions explicitly, in this paper, we explore explicit feature distribution modeling for UDA. In particular, we propose Distribution Matching Prototypical Network (DMPN) to model the deep features from each domain as Gaussian mixture distributions. With explicit feature distribution modeling, we can easily measure the discrepancy between the two domains. In DMPN, we propose two new domain discrepancy losses with probabilistic interpretations. The first one minimizes the distances between the corresponding Gaussian component means of the source and target data. The second one minimizes the pseudo negative log likelihood of generating the target features from source feature distribution. To learn both discriminative and domain invariant features, DMPN is trained by minimizing the classification loss on the labeled source data and the domain discrepancy losses together. Extensive experiments are conducted over two UDA tasks. Our approach yields a large margin in the Digits Image transfer task over state-of-the-art approaches. More remarkably, DMPN obtains a mean accuracy of 81.4% on VisDA 2017 dataset. The hyper-parameter sensitivity analysis shows that our approach is robust w.r.t hyper-parameter changes. Recent advances in deep learning have significantly improved state-of-the-art performance for a wide range of applications. However, the improvement comes with the requirement of a massive amount of labeled data for each task domain to supervise the deep model. Since manual labeling is expensive and time-consuming, it is therefore desirable to leverage or reuse rich labeled data from a related domain. This process is called domain adaptation, which transfers knowledge from a label rich source domain to a label scarce target domain . Domain adaptation is an important research problem with diverse applications in machine learning, computer vision (; ;) and natural language processing . Traditional methods try to solve this problem via learning domain invariant features by minimizing certain distance metric measuring the domain discrepancy, for example Maximum Mean Discrepancy (MMD) (; ; and correlation distance . Then labeled source data is used to learn a model for the target domain. Recent studies have shown that deep neural networks can learn more transferable features for domain adaptation . Consequently, adaptation layers have been embedded in the pipeline of deep feature learning to learn concurrently from the source domain supervision and some specially designed domain discrepancy losses;; ). However, none of these methods explicitly model the feature distributions of the source and target data to measure the discrepancy. Inspired from the recent works by and , which have shown that modeling feature distribution of a training set improves classification performance, we explore explicit distribution modeling for UDA. We model the feature distributions as Gaussin mixture distributions, which facilitates us to measure the discrepancy between the source and target domains. Our proposed method, i.e., DMPN, works as follows. We train a deep network over the source domain data to generate features following a Gaussian mixture distribution. The network is then used to assign pseudo labels to the unlabeled target data. To learn both discriminative and domain invariant features, we fine-tune the network to minimize the cross-entropy loss on the labeled source data and domain discrepancy losses. Specifically, we propose two new domain discrepancy losses by exploiting the explicit Gaussian mixture distributions of the deep features. The first one minimizes the distances between the corresponding Gaussian component means between the source and target data. We call it Gaussian Component Mean Matching (GCMM). The second one minimizes the negative log likelihood of generating the target features from the source feature distribution. We call it Pseudo Distribution Matching (PDM). Extensive experiments on Digits Image transfer tasks and synthetic-to-real image transfer task demonstrate our approach can provide superior than state-of-the-art approaches. We present our proposed method in Section 3, extensive experiment and analysis in Section 4 and in Section 5. Domain adaptation is an important research problem with diverse applications in machine learning, computer vision (; ;) and natural language processing . According to the survey , traditional domain adaptation methods can be organized into two categories: feature matching and instance re-weighting. Feature matching aims to reduce the domain discrepancy via learning domain invariant features by minimizing certain distance metric, for example Maximum Mean Discrepancy (MMD) (; ;, correlation distance , Central Moment Discrepancy (CMD) and et al. Then labeled source data is used to learn a model for the target domain. Instance reweighting aims to reduce the domain discrepancy via re-weighting the source instances according to their importance weights with respect to the target distribution . In the era of deep learning, studies have shown that deep neural networks can learn more transferable features for domain adaptation , therefore, domain adaptation layers have been embedded in the pipeline of deep feature learning to learn concurrently from the source domain supervision and some specially designed domain discrepancy losses;; ). Some recent works , , add a domain discriminator into the deep feature learning pipeline, where a feature generator and a domain discriminator are learned adversarially to generate domain invariant features. All these works can be categorized as the feature matching type of domain adaptation method. However, none of them models the feature distributions of the source and target data for distribution matching. In this paper, we show that explicitly modeling the feature distributions enables us to measure the domain discrepancy more easily and helps us to propose new domain discrepancy losses. Prototypical network (PN) was first proposed in for few shot learning, which shows that learning PN is equivalent to performing mixture density estimation on the deep features with an exponential density. Recently, in's and's works, it has been shown that modeling the deep feature distribution of a training set as Gaussian mixture distribution improves classification performance. As Gaussian density belongs to one type of exponential density, the models proposed in's and's works are variants of PN. However, the two works study the classification problem in a single domain, which is different from our work on the problem of domain adaptation. , prototypical networks are first applied for domain adaptation. Multi-granular domain discrepancy minimization at both class-level and sample-level are employed in to reduce the domain difference and achieves state-of-the-art in various domain adaptation tasks. However, in's work, the deep feature distribution is modeled implicitly when they apply PN for UDA, in our work, we explicitly model the deep feature distribution as Gaussian mixture distribution for UDA. In Unsupervised Domain Adaptation (UDA), we are given N s labeled samples in the source domain and N t unlabeled samples in the target domain. The source and target samples share the same set of labels and are sampled from probability distributions P s and P t respectively with P s = P t. The goal is to transfer knowledge learnt from the labeled source domain to the unlabeled target domain. We model the deep embedded features of the source data as a Gaussian mixture distribution where the Gaussian component means act as the prototypes for each class. Let {µ be the Gaussian component means and covariance matrices of the Gaussian mixture distribution, then the posterior distribution of a class y given the embedded feature f can be expressed as in Eqn. 1 where f = F (x, θ), F: X → R d is the embedding function with parameter θ and d is the dimension of the embedded feature, p(c) is the prior probability of class c and C is the total number of classes. With labeled source data, a classification loss L cls can be computed as the cross-entropy between the posterior probability distribution and the one-hot class label as shown in Eq. 2 and following , a log likelihood regularization term L lkd can be defined as in Eq. 3, where f The final loss function L GM for training a network with Gaussian mixture feature distribution is defined as, where ϕ is a non-negative weighting coefficient. Notice, the distribution parameters {µ are learned automatically from data. To match the deep feature distributions between the source and target data, we propose to match the corresponding Gaussian component means between them. We utilize the network learnt on the labeled source data to assign pseudo labels to target samples. As such, we denote the target samples with pseudo labels asD t = {( . We empirically estimate the Gaussian component means {µ where D s c andD t c denote the sets of source/target samples from class c, f where || · || is the L 2 norm between two vectors. Intuitively, if the source features and target features follow the same Gaussian mixture distribution, then the Gaussian component means of the same class from the two domains will be the same. Thus minimizing L GCM M helps to reduce the domain discrepancy. Better illustrated in Fig. 1 1 {µ, as the latter are learned directly from data and are used to assign pseudo labels for target data. Figure 1: Illustration of the overall training objective. This figure displays the model after we finish pre-training it with the labeled source data on L GM . Different colors represent different classes. Dotted ellipses represent Gaussian mixture distribution of the source embedded features. The amorphous shapes represent pseudo labeled target feature distribution before we optimize the network further on the overall objective function in Eqn. 7. GCMM loss tries to bring the corresponding Gaussian component means between the source data and pseudo labeled target data closer, represented by the black two-way arrows. Minimizing GCMM brings the feature distributions of the source and target domains closer, thus reducing the domain discrepancy. PDM loss tries to match the pseudo target feature distribution to the source Gaussian mixture distribution, represented by the colored one-way arrow. Minimizing PDM increases the likelihood of target features on the source feature distribution, thus reducing the domain discrepancy. Best viewed in color. On the pseudo labeled target dataD t, we further propose to match the embedded target feature distribution with the source Gaussian mixture feature distribution via minimizing the following pseudo negative log likelihood loss, which we denoted as L P DM : Minimizing L P DM 2 maximizes the likelihood of the pseudo labeled target features on the source Gaussian mixture feature distribution. To achieve that, the network is enforced to learn an embedding function which produces similar embedded feature distributions between the source data and target data. Otherwise, this term will induce a large loss value and dominate the overall objective function to be minimized. Therefore, minimizing L P DM helps to reduce the domain discrepancy. As we are using pseudo labeled target data to calculate this domain discrepancy loss function, we term it as Pseudo Distribution Matching (PDM) loss. Furthermore, while minimizing GCMM loss brings the source and target feature distribution closer, minimizing PDM loss shapes the target feature distribution to be similar as the source Gaussian mixture distribution. Thus, these two loss functions complement each other to reduce the distribution discrepancy. Better illustrated in Fig. 1. The overall training objective of DMPN can be written as follows: where minimizing the first two terms of the objective function helps the model to learn discriminative features with the supervision from the labeled source data, and minimizing the last two terms helps to match the embedded feature distributions between the source and target domains so that the learned classifier from the labeled source data can be directly applied in the target domain. The whole model is illustrated in Fig. 1. Training Procedure. To train DMPN, we first pre-train a network with labeled source data on L GM. Then mini-batch gradient descent algorithm is adopted for further optimization of the network on 2 Notice, gradient from LP DM does not back-propagate to update {µ . We learn source distribution parameters only from labeled source data. Eqn. 7, where half of the samples in the mini-batch are from labeled source data D s and the other half are from unlabeled target data D t . To obtain pseudo labels for the unlabeled target data, we use the learned source distribution parameters to calculate the class probabilities for each target data point as in Eqn. 1 and assign the class with the largest probability as the pseudo label. To remedy the error of the self-labeling, we took similar approach as in and to filter unlabeled target data points whose maximum predicted class probability is smaller than some threshold. Apart from that, we also propose to weight the contribution of each sample to the discrepancy loss based on the predicted probability. In this way, less confidently predicted target samples will make smaller contributions in the training process. Inference. For inference, we first apply the learned embedding function F on the target data, then we will use the learned distribution parameters to calculate the class probabilities for each target data point as in Eqn. 1. Finally, we output the class with the largest probability for each target data point as our prediction. There is another type of domain adaptation problem, called Supervised Domain Adaptation (SDA) in the literature. In SDA, we are provided with a large amount of labeled source data and a small amount of labeled target data, the goal is to find a hypothesis that works well in the target domain. By employing pseudo labeled target data in the training process, our method can be considered as working on a generalized problem of SDA, where the labeled target data is noisy. has proved that we can bound the target error of a domain adaptation algorithm that minimizes a convex combination of empirical source and target error in SDA as follows: where is the convex combination of the source and target error with γ, f s and f t are the labeling function in the source and target domains respectively, h is a hypothesis in class H, | measures the domain discrepsancy in the hypothesis space H and λ = s (h *) + t (h *) is the combined error in two domains of the joint ideal hypothesis h * = arg min h∈H s (h) + t (h). Denote the noise ratio of the target labeling function to be ρ, the convex combination of the source and noisy target error as˜ γ (h) = γ t (h) + (1 − γ) s (h), where t (h) is the target error on the noisy target labeling function, then we can bound the target error as follows: In summary, this bound is decomposed into three parts: the domain discrepancy d H∆H, the error λ of the ideal joint hypothesis and the noise ratio ρ of the pseudo labels. In DMPN, we minimize the first term through minimizing the domain discrepancy losses, as d H∆H is small when the source features and target features have similar distribution and minimizing the domain discrepancy losses makes the source and target feature to distribute similarly. The second term is assumed to be small, as otherwise there is no classifier that performs well on both domains. Finally, during training, as we continuously improve the accuracy of the classifier for target data, we get more and more accurate predictions, thus reducing the noise ratio ρ. We empirically verify that ρ is decreasing in Section 4.2. digits from'0' to'9'. The MNIST dataset consists of 70k images and the USPS dataset has 9.3k images. Unlike MNIST and USPS, the SVHN (S) dataset is a real-world Digits dataset of house numbers in Google street view images and contains 100k cropped Digits images. We follow the standard evaluation protocol . We consider three directions of adaptation: M → U, U → M and S → M. For the transfer between MNIST and USPS, we sample 2k images from MNIST training set and 1.8k images from USPS training set for adaptation and evaluation is reported on the standard test sets: MNIST, USPS. For S → M, we use the whole training set SVHN and MNIST for adaptation and evaluation is reported on the standard test set MNIST. In addition, we use the same CNN architecture, namely a simple modified version of to be diagonal and the prior probability to be p(c) = 1/C when pre-training the network on the labeled source data. The three trade-off parameters ϕ, α and β in Eqn. 7 are simply set to be 0.1, 1, 0.1. We strictly follow and set the embedding dimension d as 10/512 for Digits/synthetic-to-real image transfer. We implement DMPN with Pytorch. We use ADAM with 0.0005 weight decay and 0.9/0.999 momentum for training and set the mini-batch size to be 128/120 in Digits/synthetic-to-real image transfer. We train the network for 350 epochs for the Digits Image transfer tasks. The learning rate is initially set to be 1e-5 for the covariance matrices and 1e-3 for the other parameters 4 and is decayed by 0.1 at epoch 150 and 250. For the synthetic-to-real image transfer, we fix the learning rate to be 1e-6 and train the network for 100 epochs 4. Finally, for the Digits Image transfer tasks, we apply weighted PDM loss to remedy the labeling error, where each sample is weighted by the maximum predicted class probability. For the synthetic-to-real image transfer task, we apply filtering to remedy the labeling error, where only target examples with maximum predicted probability over 0.8 is used for training. Following the standard, for Digits Image transfer tasks, we adopt the classification accuracy on target domain as evaluation metric and for synthetic-to-real image transfer, we use the average per class accuracy for evaluation metric. We will publish our code upon acceptance. Compared Methods. To demonstrate the benefits of our proposed method, we compare it with the following approaches: Source-only directly exploits the classification model trained on source domain to classify target samples. separates the source feature learning and target feature learning using different networks and use a domain discriminator to learn domain invariant features. JAN aligns the joint distribution of the network activation of multiple layers across domains. MCD employs task-specific decision boundaries to align the distributions of source and target domains. CDAN+E adds a conditional adversarial classifier on the deep feature learning pipeline to learn domain invariant features. S-En+Mini-aug modifies the mean teacher variant of temporal ensembling for UDA. TPN is the first work to apply PN for UDA. TPN gen is the variant trained only with general-purpose domain discrepancy loss. DMPN is our proposed method. DMPN GCM M and DMPN P DM are trained only with GCMM loss and PDM loss respectively. Train-on-target is an oracle that trained on labeled target samples. Table 1 shows the of all methods for the two tasks. Overall, our proposed method achieves superior than all the existing methods. For the Digits Image transfer tasks, DMPN has improved the accuracy for M → U, U → M and S → M by 2.6%, 0.7% and 3.8% respectively compared to the second best. We have made great advancement considering the second best accuracy are already quite high. For the task S → M, due to convergence reasons, we have added batch normalization layers to the original CNN architectures. For fair comparison, we have re-run some experiments on other methods by adding batch normalization layers to them. For methods whose public code are not available, we simply report the accuracy with the original CNN architecture. For ADDA, adding batch normalization layer has improved its accuracy from 76.0% to 83.6%, which has an increase of 7.6% of accuracy. However, we doubt adding batch normalization layers will have the same effect on TPN, as TPN already has a quite high accuracy. Nonetheless, we think our accuracy of 96.8% on this task will be difficult for the other methods to surpass even with batch normalization layers. For the Synthetic-to-real image transfer task, we only compare with methods without extensive data augmentations and our method has increased the state-of-the-art single model mean accuracy by 1.0%. TPN gen reduces the domain discrepancy via minimizing the pairwise Reproducing Kernel Hilbert Space (RKHS) distances among the corresponding prototypes of the source-only, target-only and source-target data. In DMPN GCMM we minimize the L 2 distance between the corresponding Gaussian component means of the source and target data. The L 2 distance can be viewed as the distance in a Linear RKHS space. The calculation of our proposed GCMM loss is much simpler than the general purpose loss, yet with explicitly modeling of the feature distributions, DMPN GCMM has a gain of accuracy of 3.0%, 1.1%, 6.3% and 6.6% Ablation Analysis. In Table 1, combining GCMM loss and PDM loss helps to increase the accuracy , showing that the two domain discrepancy losses are compatible to each other. DMPN GCMM performs better than or similar to almost all other domain adaptation methods and DMPN PDM performs better than most of them. Convergence Analysis. Figure 2 (a) shows the training progress of DMPN. The GCMM loss and PDM loss keep decreasing with more training epochs. The prediction accuracy on the unlabeled target data keeps increasing. And the noise ratio ρ decreases along the training process, from the initial value of 38.6% decreases to 22.9%, which supports our theoretical analysis in Section 3.5. Figure 3 shows the t-SNE visualizations of the source and target embedded features during training, which shows that target classes are becoming increasingly well discriminated by the source classifier. shows the sensitivity analysis on the hyper-parameters α, β and ϕ with the other hyper-parameters fixed. Overall, the experiment show that we can get similar accuracy or even better when changing the hyper-parameters in a certain range, demonstrating that our method is robust against hyper-parameter changes. The sensitivity analysis on the confidence threshold is in the Appendix A.2, which shows our method is robust against threshold value. In this paper, we propose Distribution Matching Prototypical Network (DMPN) for Unsupervised Domain Adaptation (UDA) where we explicitly model and match the deep feature distribution of the source and target data as Gaussian mixture distributions. Our work fills the gap in UDA where stateof-the-art methods assume the deep feature distributions of the source and target data are unknown when minimizing the discrepancy between them. We propose two new domain discrepancy losses based on the Figure 4: Sensitivity analysis on confidence threshold. Fig. 4 shows the sensitivity analysis of our method on different values of confidence threshold on VisDA 2017 dataset. The experiment show that we can get similar accuracy or even better when changing the confidence threshold in a certain range, demonstrating that our method is robust against hyper-parameter changes. A.3 OFFICE-HOME TRANSFER Table 3 presents experiment of state-of-the-art UDA methods and our method on OfficeHome dataset. Our method gives the best accuracy in all transfer tasks, showing the effectiveness of our method. In this experiment, we train the network for 100 epochs. The learning rate is initially set to be 1e-5 for all the parameters and is decayed by 0.1 at epoch 60 and 80. 1+e −γp respectively, where γ is set to be the default value 10, p is the training process changing from 0 to 1.

represent the short summary for finding the corresponding scientific article (long)

We propose to explicitly model deep feature distributions of source and target data as Gaussian mixture distributions for Unsupervised Domain Adaptation (UDA) and achieve superior results in multiple UDA tasks than state-of-the-art methods.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s) by incorporating a loss function that encourages a vanished correlation between the bias and learned features.