Question: Represent the article for finding a single-sentence TLDR:

Question: Hyperparameter tuning is arguably the most important ingredient for obtaining state of art performance in deep networks. We focus on hyperparameters that are related to the optimization algorithm, e.g. learning rates, which have a large impact on the training speed and the ing accuracy. Typically, fixed learning rate schedules are employed during training. We propose Hyperdyn a dynamic hyperparameter optimization method that selects new learning rates on the fly at the end of each epoch. Our explore-exploit framework combines Bayesian optimization (BO) with a rejection strategy, based on a simple probabilistic wait and watch test. We obtain state of art accuracy on CIFAR and Imagenet datasets, but with significantly faster training, when compared with the best manually tuned networks. Hyperparameter tuning is arguably the most important ingredient for obtaining state of art performance in deep neural networks. Currently, most networks are manually tuned after extensive trial and error, and this is derisively referred to as graduate student descent. Hyperparameter optimization (HPO), on the other hand, attempts to automate the entire process and remove the need for human intervention. Previous works on HPO propose various strategies to either adaptively select good configurations BID12 or to speed up configuration evaluations BID7.One drawback behind existing HPO frameworks is that they do not distinguish between different kinds of hyperparameters and treat them all uniformly. Broadly there are two categories of hyperparameters: those that are fixed throughout the training process and those that need to be varied during training. The former class is mostly structural (e.g. network depth and width), while the latter is mostly related to the optimization algorithm (e.g. learning rates, regularization). The two classes of hyperparameters have very different behaviors. For structural hyperparameters, online evaluation is not possible, and we need to wait for the entire training process to be completed, which is very expensive. On the other hand, for time-varying hyperparameters, it is possible to select parameters on the fly without waiting for training to finish. In this work we keep the structural hyperparameters fixed and focus on optimizing the time-varying hyperparameters to improve efficiency, stability, and accuracy of the training process. Our main contributions are as follows: We propose Hyperdyn an automated approach for dynamically tuning hyperparameters during the training process, based on the past observations. It selects new hyperparameters at the end of each epoch by combining Bayesian optimization (BO) machinery with a simple rejection strategy. It is computationally efficient since it uses Gaussian processes (GP) and simple probabilistic rejection tests. We show state of art performance on image classification benchmarks that match the accuracy of manually tuned networks while significantly improving the training speed. We demonstrate that Hyperdyn is able to automatically decide regions of start, acceleration and slow down for the training process, and can also adapt to different conditions such as batch sizes, network architecture, datasets etc. Although our framework is broadly applicable for any time-varying hyperparameter, we limit ourselves to selecting learning rates in the experiments. We now describe Hyperdyn in this context. A set of learning-rate proposals is randomly initialized along with the weights of the neural network. We choose the best learning rate based on the validation accuracy at the end of the first epoch. For Method 85% 90% 95% CIFAR 10 SGD 1.5x 2x 2x ADAM 1x 1x 2x Imagenet SGD 1.5x 4x 4x Table 1: Speed up in training (in terms of no. of iterations) over manually tuned to reach x% of the best reported top-1 validation accuracy subsequent epochs, we employ standard Bayesian optimization (BO) to obtain new proposals based on the Gaussian process framework BID11. However, we do not always accept the outcomes of BO. We design a simple probabilistic wait and watch test to decide whether to accept the BO outcome or to stick to the previously chosen learning rate, based on the improvement of the validation accuracy over the past few epochs. This rejection test is very crucial for obtaining good performance. Our experiments show that if we naively switch the learning rate to the BO output at the end of each epoch, we have training instability and bad generalization performance. This rejection framework is philosophically similar to the hyperband framework BID7 where more time is spent exploring the more promising choices. Here we require a more sophisticated framework that utilizes the temporal history to assess whether the current choice of learning rate is promising or if one should switch to a new learning rate, as proposed by BO.We investigate performance of Hyperdyn for tuning the learning rates of two most popular optimization algorithms, viz., stochastic gradient descent (SGD) and Adam BID5, on CIFAR-10 and Imagenet datasets. The are summarized in Table. 1. Our method uniformly trains faster and can quickly reach to a significant % of the best validation accuracy, which was previously obtained BID4 ) after extensive manual tuning. In Section 4, we also show that Hyperdyn outperforms other strong baselines such as epoch-based BO (i.e. no rejection) and random 5x, i.e., at every epoch we invest 5x resources more in random search than Hyperdyn. Furthermore, we find that our method is stable and trains quickly even under larger batch sizes. We used a batch size of 1000 for our Imagenet experiments, while the manually tuned baseline was on a much smaller batch size of 256. Larger batches are preferred for distributed training since they reduce the relative communication overhead. However, training on larger batches is generally challenging, and can suffer from poor generalization BID8. The adaptivity of Hyperdyn allows it to overcome this challenge. We conduct detailed empirical analysis of Hyperdyn under a variety of conditions. We find that the learning rates suggested by Hyperdyn for SGD eventually decay, but are not always monotonic in the beginning. This agrees with the previous that using more adaptive algorithms such as Adam is more beneficial in the beginning than at a later stage BID16. We find that learning rates chosen by Hyperdyn generally increase with the batch size, and this rule of thumb has been used for manual tuning BID8. We also verify that the learning rates suggested by Hyperdyn for Adam eventually converge to values that guarantee theoretical convergence. Further, we observe that SGD tuned with Hyperdyn outperforms more sophisticated algorithms (e.g. ones with momentum) that are manually tuned. This suggests the importance of tuning for good learning rates, compared to having more sophisticated optimization algorithms. Bayesian Optimization has been widely used to optimize blackbox functions. The most common frameworks use Gaussian processes (GP) for efficiently selecting good configurations BID11. Recently, BID7 introduced Hyperband, which instead focused on speeding up configuration evaluations based on a simple random search strategy. A key feature of Hyperband is that it adaptively allocates resources using a principled early stopping mechanism. In the context of tuning for learning rates, these methods have been previously employed to tune the schedule of learning rate decay, while keeping the initial learning rate fixed, BID12 ). However, this does not provide full flexibility in finding the best learning rates in each epoch. Moreover, these frameworks require for training to be completed in order to carry out their evaluations, which makes them expensive. On the other hand, Hyperdyn selects new configurations at the end of each epoch, without requiring for training to finish. Also, most previous works only compare across different HPO frameworks, but not with the state of art manually tuned networks. Previous works report around 80% validation accuracy for the hyperband algorithm on CIFAR-10 and worse for other methods such as SMAC and TPE (tree-based BO) and Spearmint (Gaussian process) BID7. None of these previous works report on Imagenet-like large datasets. For the task of finding good learning rates, other methods have been proposed that do not rely on hyperparameter optimization. One strategy is to incorporate more adaptivity into the optimization algorithm, as seen in Adam BID5, which has better performance in certain cases over SGD, but requires even more hyperparameters to be tuned. Another algorithm that automatically controls the learning rate of SGD, known as SALERA, was introduced in BID10. SALERA updates the learning rate by using an exponential moving average and deals with catastrophic events that occur in SGD (that lead to poor training accuracy) by allowing for backtracking. However, the learning rate update rule is fixed unless there is a catastrophic event, so the extent of adaptivity is limited. Additionally, the experiments were only conducted on smaller datasets such as MNIST and CIFAR10, so it is not clear how the algorithm behaves with larger datasets and larger batch sizes. An RL-based approach to learning on the fly was introduced in BID6. However, the RL framework in general requires a large amount of training data and the experiments in that work were not conducted on standard benchmark datasets such as CIFAR10 or MNIST. Recently, BID2, BID17 proposed methods for large batch training on Imagenet dataset that achieve state of art Top-1% accuracy. The work there, however, was restricted to designing a learning rate scheduler for SGD for large batch sizes. Hyperdyn on the other hand is a general framework which can be applied to a wide range of hyperparameter optimization problems and different kinds of training methods. The idea of using information learned in one task to improve on a future task, or meta-learning BID15 ), is also related to the framework of using past information to improve hyperparameter choice, considered here. LSTMs were employed to learn the gradient descent updates in BID0; BID9. However, LSTMs require a large amount training data and it is not clear if the methods can scale to standard benchmark datasets. Currently, these techniques have been shown to work for only small scale problems. The rest of paper is structured as follows -in section 2 we give a brief review of the bayesian optimization algorithm and gaussian processes that are at the center of Hyperdyn. In section 3 we describe its in detail. In section 4 we present the details of the experiments. Bayesian optimization has been widely used to find the minimize a black box function. In general, the black box functions are unknown or difficult to compute and the bayesian optimization algorithm suggests query points based on the optimization of an "easier" acquisition function. So if the blackbox function was f (·) and we had to minimize f (·). Then we would use the bayesian algorithm as described in Algorithm 1. This version of Bayesian Optimization in Algorithm 1 is a one-step sim- DISPLAYFORM0 3: Query objective function to obtain y 0 4: return η 0 plification of the one described in BID11. We next describe the acquisition function α(·; ·) and other statistical details of Algorithm 1. A Gaussian Process is a nonparamteric statistical model that is characterized by µ 0, σ 0, K(·, ·) -initial mean, initial variance and kernel function respectively. Consider the sequence of points, η 1:n (inputs) and y 1:n (noisy observations). We introduce auxillary variables f 1:n, such that f 1:n |η 1:n ∼ N (m, K) where DISPLAYFORM0 Then we have that y 1:n |f 1:n, σ 2 0 ∼ N (f 1:n, σ 2 0 I). Given this GP, we "update" µ n (·), σ n (·) given some observation {(η i, y i)} n i=1, by Algorithm 2 DISPLAYFORM1 A plethora of acquisition functions are discussed in BID11, we specifically use the expected improvement function that we describe now. Consider the improvement function I(η, v, θ) = (v − τ)I(v > τ); this captures the amount of improvement over a given threshold τ. Then expected improvement is just E[I(η, v, θ)] assuming that v is normally distributed with some mean and variance parametrized by η. Then we have DISPLAYFORM0 Typically τ = min n y n, i.e., the minimum of noisy observations. It is not necessary that the kernel function in Algorithm 1 be stationary. In fact there is vast literature on non-stationary kernels for Bayesian Optimization (See Gramacy & Lee FORMULA4). These methods are, in general, very complicated. In the following section we propose a simple compositional nonstationary kernel that works well in practice. The simplicity of Bayesian optimization makes it amenable to blackbox optimizations. However, there are a few severe limitations of Bayesian optimization that prevent it from being applied directly to neural network training. First, as stated before, the framework of Bayesian optimization is such that only one setting of learning rate parameter can be used for the entire duration of training. Second, even if we were somehow able to use multiple learning rate parameter suggestions while training, there is no stationarity, i.e., the same learning rates may produce very different at different points of training. In Hyperdyn we use different temporal estimates of the loss function change and use non stationary kernels to alleviate the two problems mentioned above. By using a simple compositional kernel we avoid the computational burden associated with general non stationary kernels. Any training algorithm can be roughly summarized as follows w t+1 = T (w t, B, Hyperparameters) where B is a batch of data, T denotes the training algorithm and w t are the parameters of some neural network at epoch t of the algorithm. T is run for some predefined number of iterations over the data. For Hyperdyn we define a new set of hyperparameters that includes the epoch number of the algorithm New Hyperparameters = (True Hyperparameters, t) Considering epoch number of the training process as a hyperparameter allows to build a composite kernel over t and the true hyperparameters; we now get kernels that are non-stationary without too much overhead from existing stationary ones. Let (η 1, t 1), (η 2, t 2) be the new hyperparameters where t i is the epoch number, then our kernel is of the form DISPLAYFORM0 Now, the K 1 (·, ·) kernel can be one of RBF kernel or Matern-x kernels over the true hyperparameters, as in standard literature, we will describe K 2 (·, ·), now referred to as the time kernel, in the following section. Unlike the true hyperparameters, we do not need to optimize over the epoch number, i.e., t i s behave in a specific way {1, 2, . . .,} upto end of training process. Such a formulation only helps in introducing non-stationarity in the kernels we use for Bayesian optimization. This is realized by changing the search space for epoch number in the Bayesian optimization to [t, t] for every epoch number t. We use a similar approach to BID14 Section 3 for training curves. The kernel K 2 (·, ·) is of the form DISPLAYFORM0 where the last equality is obtained by choosing ψ(λ) = β α Γ(α) λ α−1 exp (−λβ). This construction is motivated from the observation that SGD decays as O(1/N 1/2) in N iterations, which in our case reduced to α = 1, β = 1/2, and from work in BID14. Hyperdyn is comprised of some crucial moving parts. At the beginning we have no information and employ a purely exploratory approach as described in Algorithm 3. The function Valida- Generate p i ∈ S uniformly at random 6: DISPLAYFORM0 tion Accuracy(·, ·) takes in as input weights (first argument) and hyperparameter values (second argument) and updates weights. The output is top-1 accuracy on the validation set for the updated weights. Function Random Start gives our main algorithm, which we describe in Algorithm 4, some initial information for a more exploitive approach. In Algorithm 4, Update Weights simply updates DISPLAYFORM1 if Check Accuracy Trend(w 1:t−1, ∆) then 10: The function Update Statistical Model has been described in Section 2.1. We might be tempted to switch to the best performing hyperparameter value at every epoch (Lines 9 − 14). As we will show, that such a myopic switching (between hyperparameters) strategy is a hindrance when it comes to good generalization performance. The function Check Accuracy Trend, described in Algorithm 5, prevents us from changing our hyperparameters unnecessarily. The idea is that if a hyperparameter choice has been improving on the validation accuracy, then there is no incentive to change. DISPLAYFORM2 Algorithm 5 Check Accuracy Trend DISPLAYFORM3 return True 6: else 7:return False 8: end if An initial offset is provided so that there is a small probability of changing hyperparameter values even when current values are performing well. The experiments were conducted on Amazon AWS p2x16 EC2 machines. We only use data parallelism for gradient computation in SGD across different GPUs, i.e., if there are k GPUs and a batch size of α then each GPU calculates the gradient on a batch size of α/k and this is summed across GPUs before weight update. The datasets used were CIFAR10 and Imagenet 1K. We employed simple augmentations as described in BID4. We also consider their as the state of art since the structural setting (neural network architecture, augmentation settings etc.) there is closest to the one here. Further for batch sizes not included there we use scaling rules described in BID2. The number of initial search points for Random Start was 5. In our experiments on Hyperdyn we only use learning rate and momentum for hyperparameter optimization. The window size used is 4, temperature is 1 and offset is 0.01. The window size should not be too small ∼ 1 as it leads to very high variance while training, and for the experiments that we run older observations become obsolete fast and a large window size is unnecessary. Comparison to Manually Tuned Networks: In this section we presents when the weight update mechanism is momentum-SGD. We compare the performance of Hyperdyn -tuned momentum-SGD versus the manually tuned momentum-SGD as described in BID4 on a ResNet-20. Each epoch indicates (num points/batch size) number of iterations. As shown in FIG1, Hyperdyn tuning is always faster (at reaching superior generalization performance) than the manually tuned one. Table 1 shows the number of iterations after which a certain top-1 accuracy is reached for a ResNet-20 with batch size 128. Hyperdyn tuned SGD is substantially faster than the fixed schedule manually tuned SGD. There is no additional tuning required for Hyperdyn and as a we circumvent hours of trial and error to get the hyperparameters right. Dynamics of Hyperdyn: For any stable recursive stochastic algorithm (for details see BID1) it must be true that η k → 0 as k → ∞, where η k is the learning rate at iteration k. In FIG0 we observe that Hyperdyn tuning automatically tends to 0 as the learning process progresses. Empirically it has been found that larger learning rates are needed for larger batches in the manual setting BID8 and Hyperdyn is automatically able to achieve this. As can be seen in FIG0 the average learning rate increases as we increase the batch size from 128 to 2k. However, the same trend does not translate, when the batch size is doubled further. A possible explanation to this maybe that we are already in a very large batch regime ∼ 10% of dataset size and the learning rate dynamics behave very differently. Further, we study Hyperdyn in FIG0 when the momentum term is set to 0. We observe that it outperforms manually tuned SGD with momentum, which is a more sophisticated algorithm. This suggests that it is more important to find good learning rates for a simple algorithm, rather than attempt to design a more sophisticated algorithm. Moreover, the wait and watch strategy of Hyperdyn is incorporating past information for stability, similar in principle to momentum. This could also explain why SGD tuned with Hyperdyn can outperform manually tuned SGD with momentum. We compare Hyperdyn to a greedy version of it, where we switch to the best performing hyperparameter in every epoch. This greedy method, or epoch based Bayesian optimization, is obtained by setting the offset i.e., b = ∞ in Check Accuracy Trend sub-routine; which is nothing but the standard Bayesian optimization method with compositional kernel incorporating the temporal dependence. However, as FIG1 shows, the epoch based BO has poor generalization. We set the batch size of 1000 on ResNet-20. This observation necessitates the need for a "wait-and-watch" strategy, as realized by the Check Accuracy Trend module. Although the epoch based BO version outperforms in the initial few epochs, it plateaus quickly and is overtaken by Hyperdyn. The choice of batch size is arbitrary for FIG1 (d) as this observation is consistent across different batch sizes. We also compare Hyperdyn with a version of random search 5x, i.e., at every epoch we invest 5x resources more in random search than in Hyperdyn. We find that random search is susceptible to higher variance than Hyperdyn, especially at larger batch sizes. FIG1 shows a typical comparison of random search 5x and Hyperdyn. We employed a variant of Hyperband where we reduced the search space of learning rate by 0.1 instead of successive halving and did not find any improvement in performance over random search 5x. Using Hyperdyn with Adam: In addition to SGD, Adam is commonly used for optimization. It is adaptive, computationally and memory efficient, but also works for non-stationary objectives and noisy environments BID5. Typically, Adam is employed for the initial part of the training process while SGD is used near the end. We use Hyperdyn to optimize the hyperparameters of Adam, i.e. learning rates β 1, β 2 in BID5. Hyperdyn tuned Adam is much faster than manually tuned Adam (in terms of number of iterations) and also generalizes better on ResNet-20 with batch size 128. As our HPO algorithm proceeds, we observe convergence of learning rates DISPLAYFORM0 Figure 3: Hyperdyn tuned Adam as β 1 → 0 and β 2 is around 1, i.e. the momentum coefficient β 1 becomes very small (Fig. 3(b) ). It turns out that these limits are also needed to establish good theoretical properties for Adam; see Corollary 4.2 in BID5. This also matches the previous observation in BID13 that the momentum coefficient needs to be reduced to obtain good convergence. For Imagenet we used ResNet-50 and a batch size of 1000. This batch size is considerably larger than typical Imagenet experiments, where the batch sizes are ∼ 256. We compare in FIG3 a Hyperdyn tuned Imagenet training process to a manually tuned one. Note that the batch size for the two are different because a manual Imagenet tuning is hard for a batch size of 1000. As a we achieve the same accuracy much faster than a manually tuned training process. Additionally, in BID17 suggest that (upto 40 epochs) Hyperdyn gives the best performance. In this work we introduced a general framework for a certain class of hyperparameter optimization. The algorithm here has the advantage that it is fast, flexible -can be used on any objective function and training method, and stable on larger batch sizes. We demonstrated that our proposed optimizer is faster and at least as accurate as SGD (with momentum) or Adam. We also show that Hyperdyn is stable for larger batch sizes on Imagenet (achieves acceptable accuracy with 4x speed). We demonstrate how too much exploration can be detrimental to generalization accuracy of a training process, and propose a probabilistic "wait-and-watch" strategy. Currently we do not parallelize Hyperdyn; however, computing Validation Accuracy on the suggestion from One Step BO can be easily parallelized. At each epoch we make only suggestion and the validation accuracy on this suggestion can be computed independently of the current hyperparameter setting. In the general case, when we make multiple suggestions we can parallelize in a similar fashion to BID12. We also observe that epoch-based BO in FIG1 outperforms Hyperdyn in the initial epochs. One future direction maybe to have time varying temperature in Check Accuracy Trend based on epoch. We can also exploit the temporal gains obtained by using a backtrack algorithm at the later stages of the algorithm -at a stage when accuracy is sufficiently high but more sophisticated tuning is required to get the best error rates.

Bayesian optimization based online hyperparameter optimization.

represent the short summary for finding the corresponding scientific article (long)

Smaller batch sizes can outperform very large batches on the test set under constant step budgets and with properly tuned learning rate schedules.


Question: Represent the scientific article to find a short TLDR.

Question: Multi-hop text-based question-answering is a current challenge in machine comprehension. This task requires to sequentially integrate facts from multiple passages to answer complex natural language questions. In this paper, we propose a novel architecture, called the Latent Question Reformulation Network (LQR-net), a multi-hop and parallel attentive network designed for question-answering tasks that require reasoning capabilities. LQR-net is composed of an association of \textbf{reading modules} and \textbf{reformulation modules}. The purpose of the reading module is to produce a question-aware representation of the document. From this document representation, the reformulation module extracts essential elements to calculate an updated representation of the question. This updated question is then passed to the following hop. We evaluate our architecture on the \hotpotqa question-answering dataset designed to assess multi-hop reasoning capabilities. Our model achieves competitive on the public leaderboard and outperforms the best current \textit{published} models in terms of Exact Match (EM) and $F_1$ score. Finally, we show that an analysis of the sequential reformulations can provide interpretable reasoning paths. The ability to automatically extract relevant information from large text corpora remains a major challenge. Recently, the task of question-answering has been largely used as a proxy to evaluate the reading capabilities of neural architectures. Most of the current datasets for question-answering focus on the ability to read and extract information from a single piece of text, often composed of few sentences . This has strengthened the emergence of easy questions in the sense of and influenced the recent state-of-the-art models to be good at detecting patterns and named entities (; ;). However they still lack actual reasoning capabilities. The problem of reasoning requires machine comprehension models to gather and compose over different pieces of evidence spread across multiple paragraphs. In this work, we propose an original neural architecture that repeatedly reads from a set of paragraphs to aggregate and reformulate information. In addition to the sequential reading, our model is designed to collect pieces of information in parallel and to aggregate them in its last layer. Throughout the model, the important pieces of the document are highlighted by what we call a reading module and integrated into a representation of the question via our reformulation module. Our contributions can be summarised as follows: • We propose a machine reading architecture, composed of multiple token-level attention modules, that collect information sequentially and in parallel across a document to answer a question, • We propose to use an input-length invariant question representation updated via a dynamic max-pooling layer that compacts information form a variable-length text sequence into a fixed size matrix, • We introduce an extractive reading-based attention mechanism that computes the attention vector from the output layer of a generic extractive machine reading model, • We illustrate the advantages of our model on the HOTPOTQA dataset. The remainder of the paper is organized as follows: Section 2 presents the multi-hop machine reading task, and analyses the required reasoning competencies. In Section 3, we detail our novel reading architecture and present its different building blocks. Section 4 presents the conducted experiments, several ablation studies, and qualitative analysis of the . Finally, Section 5 discusses related work. Our code to reproduce the is publicly available at (removed for review). 2 TEXT-BASED QUESTION-ANSWERING AND MACHINE REASONING Figure 1: Examples of reasoning paths to answer two questions of the HOTPOTQA dataset. In this picture, we do not display the full paragraphs, but only the supporting facts. The task of extractive machine reading can be summarized as follows: given a document D and a question Q, the goal is to extract the span of the document that answers the question. In this work, we consider the explainable multi-hop reasoning task described in and its associated dataset: HOTPOTQA. We focus our experiments on the "distractor" configuration of the dataset. In this task, the input document D is not a single paragraph but a set of ten paragraphs coming from different English Wikipedia articles. Answering each question requires gathering and integrating information from exactly two paragraphs; the eight others are distractors selected among the of a tf-idf retriever . These required paragraphs are called the gold paragraphs. There are two types of questions proposed in this dataset: extractive ones where the answer is a span of text extracted from the document and binary yes/no questions. In addition to the answer, it is required to predict the sentences, also called supporting facts, that are necessary to produce the correct answer. This task can be decomposed in three subtasks: categorize the answer among the three following classes: yes, no, text span, if it is a span, predict the start and end positions of this span in the document, and predict the supporting sentences required to answer the question. In addition to the "distractor" experiments, we show how our proposed approach can be used for opendomain question answering and evaluate the entire reading pipeline on the "fullwiki" configuration of the HotpotQA dataset. In this configuration, no supporting documents are provided, and it is required to answer the question from the entire Wikipedia corpus. Among the competencies that multi-hop machine reading requires, we identify two major reasoning capabilities that human readers naturally exploit to answer these questions: sequential reasoning and parallel reasoning. Sequential reasoning requires reading a document, seeking a piece of information, then reformulating the question and finally extracting the correct answer. This is called multi-hop question-answering and refers to the bridge questions in HOTPOTQA. Another reasoning pattern is parallel reasoning, required to collect pieces of evidence for comparisons or question that required checking multiple properties in the documents. Figure 1 presents two examples from HOTPOTQA that illustrate such required competencies. We hypothesize that these two major reasoning patterns should condition the design of the proposed neural architectures to avoid restricting the model to one or the other reasoning skill. In this section, we describe the Latent Question Reformulation Network (LQR-net), shown in Figure 2. This multi-hop model is designed as an association of four modules: an encoding module, a reading module, a question reformulation module, and an answering module. and are input and output modules, whereas and constitute a hop, and are repeated respectively T and T − 1 times: the answering module does not require a last reformulation step. Figure 2: Overview of LQR-net with K parallel heads and T sequential reading modules. In this architecture, a latent representation of the question is sequentially updated to perform multi-hop reasoning. K independent reading heads collect pieces of information before feeding them to the answering module. Sections 3 present the different building blocks of this end-to-end trainable model. Given a document and a question, the reading module is in charge of computing a question-aware representation of the document. Then, the reformulation module extracts essential elements from this document representation and uses them to update a representation of the question in a latent space. This reformulated question is then passed to the following hop. The model can have multiple heads, as in the Transformer architecture . In this case, the iterative mechanism is performed several times in parallel in order to compute a set of independent reformulations. The final representations of the document produced by the different heads are eventually aggregated before being fed to the answering module. This module predicts the answer and the supporting facts from the document. The following parts of this section describe each module that composes this model. Note: The model is composed of K independent reading heads that process the document and question in parallel. To not overload the notations of the next parts, we do not subscript all the matrices by the index of the head and focus on the description of one. The aggregation process of the multi-head outputs is explained in Section 3.5. We adopt a standard representation of each token by using the pre-trained parametric language model BERT . Let a document D = {p 1, p 2, . . ., p 10} be the set of input paragraphs, of respective lengths {n 1, . . ., n 10}, associated to a question Q of length L. These paragraphs are independently encoded through the pre-trained BERT model. Each token is represented by its associated BERT hidden state from the last layer of the model. The tokens representations are then concatenated to produce a global representation of the set of 10 paragraphs of total length N = 10 i=1 n i. The representations are further passed through a Bidirectional Gated Recurrent Unit (BiGRU) to produce the final representation of the document E D ∈ R N ×2h and question E Q ∈ R L×2h, where h is the hidden state dimension of the BiGRUs. where [;] is the concatenation operation. To compute the first representation of the question U, we use an interpolation layer to map where M is an hyperparameter of the model. Intuitively, R M ×2h corresponds to the space allocated to store the representation of the question and its further reformulations. It does not depend on the length of the original question L. Our model is composed of T hops of reading that sequentially extract relevant information from a document regarding the current reformulation of the question. At step t, given a representation of the reformulated question U (t) ∈ R M ×2h and a representation of the document E D ∈ R N ×2h, this module computes a question-aware representation of the document. This module is a combination of two layers: a document-question attention followed by a document self-attention. We first construct the interaction matrix between the document and the current reformulation of the question S ∈ R N ×M as: where w 1, w 2, w 3 are trainable vectors of R 2h and the element-wise multiplication. Then, we compute the document-to-question attention C q ∈ R N ×2h: And the question-to-document attention q c ∈ R 2h: Finally, we compute the question-aware representation of the document X (t) ∈ R N ×8h: where [;] concatenation operation. Finally, we use a last BiGRU that reduces the dimension of X (t) to N × 2h. This specific attention mechanism was first introduced in the Bidirectional Attention Flow model of. We hypothesize that such token-level attention will produce a finer-grained representation of the document compared to sentence-level attention used in state-of-the-art Memory Network architectures. Document Self-Attention: So far, the contextualization between the ten paragraphs has only be done by the BiGRUs of equation 1. One limitation of the current representation of the document is that each token has very limited knowledge of the other elements of the context. To deal with long-range dependencies, we apply this same attention mechanism between the question-aware representation of the document, X (t), and itself to produce the reading module output V ∈ R N ×2h. This self-contextualization of the document has been found useful in our experiments as presented in the ablation analysis of Section 4.3. A reformulation module t takes as input the output of the previous attention module V (t), the previous representation of the reformulated question U (t), and an encoding of the document E D. It produces an updated reformulation of the question U (t+1). Reading-based Attention: Given V (t) we compute p (t)s ∈ R N and p (t)e ∈ R N using two BiGRUs followed by a linear layer and a softmax operator. They are computed from: where w e and w s are trainable vectors of R h. The two probability vectors p (t)s and p (t)e are not used to predict an answer but to compute a reading-based attention vector a (t) over the document. Intuitively, these probabilities represent the belief of the model at step t of the probability for each word to be the beginning and the end of the answer span. We define the reading-based attention of a token as the probability that the predicted span has started before this token and will end after. It can be computed as follows: Finally, we use these attention values to re-weight each token of the document representation. We Dynamic Max-Pooling: This layer aims at collecting the relevant elements ofẼ (t)D to add to the current representation of dimension M × 2h. We partition the row of the initial sequence into M approximately equal parts. It produces a grid of M × 2h in which we apply a max-pooling operator in each individual window. As a , a matrix of fixed dimension adequately represents the input, preserving the global structure of the document, and focusing on the important elements of each region. This can be seen as an adaptation of the dynamic pooling layer proposed by. (t)D be the input matrix representation, we dynamically compute the kernel size, w, of the max-pooling according to the length of the input sequence and the required output shape: w = N M, · being the ceiling function. Then the output representation of this pooling layer will be Finally, to compute the updated representation of the question U (t+1) ∈ R M ×2h, we sum U (t) and O (t). The answering module is a sequence of four BiGRUs, each of them followed by a fully connected layer. Their respective goal is to supervise the supporting facts p sf, the answer starting and ending probabilities, p e, p s, of each word of the document. The last layer is used as a three-way classifier to predict p c the probability of the answer be classified as yes, no or a span of text. where w s ∈ R h, w e ∈ R h, W c ∈ R h×3 are trainable parameters. To predict the supporting facts, we construct a sentence based representation of the document. Each sentence is represented by the concatenation of its starting and ending supporting fact tokens from Y sf. We compute p sf i,j the probability of sentence j of example i of being a supporting fact with a linear layer followed by a sigmoid function. We define a multi-head version of the model. In this configuration, we use a set of independent parallel heads. All heads are composed of the same number of reading and reformulation modules. Each head produces a representation V (T) k of the document. We finally sum these K matrices to compute the input of the answering block. We jointly optimize the model on the three subtasks (supporting facts, span position, classifier yes/no/span) by minimising a linear combination of the supporting facts loss L sf, the span loss L span and the class loss L class. Let N d be the number of examples in the training dataset. L sf (θ) is defined by: where nbs i corresponds to the number of sentences in the document i. y i,j being 1 if the sentence j of the document i is a supporting fact otherwise 0. Selecting the answer in multi-hop reading datasets is a weakly supervised task. Indeed, similarly to the observations of Min et al. (2019a) for open-domain question-answering and discrete reasoning tasks, it is frequent for a given answer of HOTPOTQA to appear multiple times in its associated document. In our case, we assume that all the mentions of the answer in the supporting facts are related to the question. We tag as a valid solution, the start and end positions of all occurrences of the answer in the given supporting facts. where y ∈ R N are vectors containing the value 1/n i at the start, end positions of all the occurrences of the answer, 0 otherwise; n i being the number of occurrences of the answer in the context. where y i corresponds to the index of the label of the question type {yes, no, span}. We finally define the training loss as follows: where α and β are hyperparameters tuned by cross-validation. In the original HOTPOTQA dataset, the two gold paragraphs required to answer a given question come with eight distractor paragraphs. These eight distractor paragraphs, collected from Wikipedia, are selected among the of a bigram tf-idf retriever using the question as the query. As an augmentation strategy, we created additional "easier" examples by combining the two gold paragraphs with eight other paragraphs randomly selected in the dataset. For each example of the original training set, we generate an additional "easier" example. These examples are shuffled in the dataset. Our model is composed of 3 parallel heads (K = 3) each of them composed of two reading modules and one reformulation module (T = 2). We set the hidden dimension of all the GRUs to d = 80. We use M = 100 to allocate a space of R 100×160 to store the question and its reformulations. We use (b) 55 pre-trained BERT-base-cased model and adapt the implementation of Hugging Face 1 to compute embedding representations of documents and questions. We optimize the network using the Adam optimizer with an initial learning rate of 1e −4. We set α to 1 and β to 10. All these parameters have been defined through cross-validation. Table 1 presents the performance of our LQR-net on the distractor setting of the HOTPOTQA dataset. We compare our model against the published approaches evaluated on the HOTPOTQA dataset. We can see from this table that our model achieves strong performance on the answer prediction task. It outperforms the current best model by 3.9 points of EM and 4.1 points of F 1 score. Our model also achieves competitive performance for the evidence extraction task. The LQR-net achieves state-ofthe-art performance on the joint task improving the best published approaches by 2.9 points on EM and 3.9 points of F 1. To evaluate the impact of the different components of our model, we perform an ablation analysis. Table 2 presents the of this analysis. Impact of sequential and parallel reading: We study the contributions of the sequentiality in the model and of the multiple parallel heads. We compare our model to a similar architecture without the sequential reformulation (T = 1). We find that this sequential association of reading modules and reformulation modules is a critical component. F 1 score decreases by 6.9 points for the answer prediction task and 5.7 points for the evidence extraction task when the model does not have the capability to reformulate the question. The impact of the parallel heads is more limited than the sequentiality but still remains significant. Indeed, the configuration that uses only a single head (K = 1) stands 1 F 1 points below the best model on the joint metric. Weak supervision of the answer: In this work, we propose to label as positive all occurrences of the answer in the supporting facts. We compare this configuration to the standard approach, where only the first occurrence of the answer is labeled as positive and the others as negative. In this last configuration, the span loss corresponds to a cross-entropy loss (CE loss) between the predicted start and end probabilities and the target positions. This decreases the joint F 1 score by 0.8 points. Impact of the self-attention layer: We study the impact of the self-attention layer in the reading module. We found that this self-attention layer is an essential component in the reading process. Indeed, when we omit this layer, the F 1 score decreases by 8.3 points on the joint metric. This outlines the necessity to be able to propagate long-range information between the different paragraphs and not only in the local neighborhood of a token. Compared to previously proposed approaches, this layer does not rely on any handcrafted relationship across words. Question as a single vector: Finally, we study the case where the question representation is reduced to a vector of R 2h (M = 1). This configuration achieves the worst of our analysis, dropping the joint F 1 score by 13.3 points and highlights the importance of preserving a representation of the question as a matrix to maintain its meaning. In this part, we describe how we integrated our model into an entire reading pipeline for opendomain question answering. In this setting, no supporting documents are associated to each question, and it is required to retrieve relevant context from large text corpora such as Wikipedia. We adopt a two-stage process, similar to; , to answer multihop complex questions based on the 5 million documents of Wikipedia. First, we use a paragraph retriever to select a limited amount of relevant paragraphs from a Wikipedia dump, regarding a natural language question. Second, we fed our LQR model with the retrieved paragraphs to extract the predicted answer. We evaluate this approach on the open-domain configuration of the HotpotQA dataset called fullwiki. We use a standard TF-IDF based paragraph retriever to retrieve the paragraphs the most related to the question. In addition to these paragraphs, we consider as relevant their neighbors in the Wikipedia graph, i.e. the documents linked to them by hyperlinks. In our experiments, we considered as relevant, the top 10 paragraphs and their associated neighbors. Table 3 shows the of our approach compared to other published models. Although we are using a very simple retriever, only based on TF-IDF, we report strong on the open-domain question answering task of HotpotQA. The only published approach that outperforms us being a combination of sentence/paragraph retrieval based on BERT encodings. Question Reformulation and Reasoning Chains: Because our model reformulates the question in a latent space, we cannot directly visualize the text of the reformulated question. However, one way to assess the effectiveness of this reformulation is to analyze the evolution of p s and p e across the two hops of the model. We present in Figure 3 an analysis of the evolution of these probabilities on two bridge samples of the development dataset. We display the reading-based attention, that corresponds to the probabilities for each word to be part of the predicted span, computed from p s and p e in Equation 7. These examples show this attention before the first reformulation of the question and in the answering module. From these observations, we can see that the model tends to follow a natural reasoning path to answer bridge questions. Indeed, before the first reformulation module, the attentions tend to focus on the first step of reasoning. For the question "What award did the writer of Never Let Me Go novel win in 1989?", the model tends to focus on the name of the writer at the first step, before jumping the award description in the second step. Similarly, for the question "What is the population according to the 2007 population census of the city in which the National Archives and Library of Ethiopia is SemanticRetrievalMRS 37.60 49.40 23.10 58.5 12.2 35.3 MUPPET 31.07 40.42 17.00 47.71 11.76 27.62 QFE † 28.70 38.10 14.20 44.40 8.69 23.1 Baseline Model 24.68 34.36 5.28 40.98 2.54 17.73 DecompRC (b) N/A 43.26 N/A N/A N/A N/A Table 3: Performance comparison on the development set of HOTPOTQA in the fullwiki setting. We compare our model in terms of Exact Match and F 1 scores against the published models at the time of submission (November 15th). † indicates that the paper does not report the on the development set of the dataset; we display their on the test set. Figure 3: Distribution of the probabilities for each word to be part of the predicted span, before the first reformulation module and in the answering module. We display the reading-based attention computed in Equation 7 and the reading-based attention computed from p s and p e from Equation 10. In these examples, we show only the supporting facts. located? " we can see the model focusing on Addis Ababa at the first step, i.e the name of the city where the National Archives and Library of Ethiopia are located and then jumping to the population of this city in the next hop. We display more visualizations of the sequential evolution of the answer probabilities in Appendix A. Limitations: We manually examine one hundred errors produced by our multi-step reading architecture on the development set of HOTPOTQA. We identify three recurrent cases of model failure: the model stops at the first hop of required reasoning, the model fails at comparing two properties, and the answer does not match all the requirements of the question. We illustrate these three recurrent types of error with examples from the dataset in Appendix B. During this analysis of errors, we found that in only 3% of the cases, the answer is selected among one of the distractor paragraphs instead of a gold one. Our architecture successfully detects the relevant paragraphs regarding a question even among similar documents coming from a tf-idf retriever. Moreover, there are no errors where the model produces a binary yes/no answer instead of extracting a text span and vice versa. Identifying the type of question is not challenging for the model. This might be explained by the question's "patterns" that are generally different between binary yes/no and extractive questions. Multi-hop Machine Comprehension: The question-answering task has recently increased its popularity as a way to assess machine reading comprehension capabilities. The emergence of large scale datasets such as CNN/Daily Mail, , SQuAD or MSMARCO have encouraged the development of multiple machine reading models . These models are mainly composed of multiple attention layers that update the representation of the document conditioned by a representation of the question. However, most of this work focuses on the ability to answer questions from a single paragraph, often limited to a few sentences. Weston et al. (2015a); were the first attempts to introduce the task of multi-documents question-answering. QAngaroo is another dataset designed to evaluate multi-hop reading architectures. However, state-of-the-art architectures on this task ) tend to exploit the structure of the dataset by using the proposed candidate spans as an input of the model. Recently, different approaches have been developed for HOTPOTQA focusing on the multiple challenges of the dataset. focuses on the evidence extraction task and highlight its similarity with the extractive summarization task. Related works also focus on the interpretation of the reasoning chain with an explicit decomposition of the question (b) or a decomposition of the reasoning steps . Other models like aim at integrating a graph reasoning type of attention where the nodes are recognized by a BERT NER model over the document. Moreover, this model leverages on handcrafted relationships between tokens. Related to our approach, different papers have investigated the idea of question reformulation to build multi-hop open-domain question answering models. proposes a framework composed of iterative interaction between a document retriever and a reading model. The question reformulation is performed by a multi-step-reasoner module trained via reinforcement learning. introduces a multi-hop paragraph retriever. They propose a reformulation component integrated into a retrieving pipeline to iteratively retrieve relevant documents. These works are complementary to ours by focusing mostly on the document retrieving part of the problem while we focus on the answer extraction task, and could be combined together. Memory Networks: Memory networks are a generic type of architecture Weston et al. (2015b);; designed to iteratively collect information from memory cells using attention mechanism. They have been used to read from sentences, paragraphs, and knowledge bases. In these models, the answer layer uses the last value of the controller to predict the answer. Two main differences with our architecture are the representation of the controller and the associated attention mechanism. Indeed, in these models, the controller is reduced to a single vector, and the attention mechanism is based on a simple dot-product between each token of the document and the representation of the controller. We utilize a token-level attention mechanism compared to the sentence-level one, classically used in Memory Networks. Transformer Networks: The transformer architecture has been introduced by in the context of machine translation. It is mainly composed of attention layers in both the encoder and the decoder module. The transformer networks introduced the so-called multi-head attention, consisting of several attention layers running in parallel. This multi-head attention allows the model to concurrently access information from different representations of the input vector. Inspired by this work, we designed our multi-head module to read in parallel into different representations of the document while solely accumulate information into the representation of the question. In this paper, we propose a novel multi-hop reading model designed for question-answering tasks that explicitly require reasoning capabilities. We have designed our model to gather information sequentially and in parallel from a given set of paragraphs to answer a natural language question. Our neural architecture, uses a sequence of token-level attention mechanisms to extract relevant information from the paragraphs and update a latent representation of the question. Our proposed model achieves competitive on the HOTPOTQA reasoning task and performs better than the current best published approach in terms of both Exact Match and F 1 score. In addition, we show that an analysis of the sequential attentions can possibly provide human-interpretable reasoning chains. This section includes examples from the HOTPOTQA development set that illustrate the evolution of the probabilities for each word to be part of the predicted span, before the first reformulation module and in the answering module presented in Section 4.5. For each example, we show only the text of the two gold paragraphs. identifies the supporting facts in these visualizations. This section includes examples from the HOTPOTQA development set that illustrate the categories of errors presented in Section 4.5. For each example, we show only the text of the two gold paragraphs. identifies the supporting facts in these visualizations. The model stops at the first hop of required reasoning: The model fails at comparing two properties:

In this paper, we propose the Latent Question Reformulation Network (LQR-net), a multi-hop and parallel attentive network designed for question-answering tasks that require reasoning capabilities.

Represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: We propose a method to automatically compute the importance of features at every observation in time series, by simulating counterfactual trajectories given previous observations. We define the importance of each observation as the change in the model output caused by replacing the observation with a generated one. Our method can be applied to arbitrarily complex time series models. We compare the generated feature importance to existing methods like sensitivity analyses, feature occlusion, and other explanation baselines to show that our approach generates more precise explanations and is less sensitive to noise in the input signals. Multi-variate time series data are ubiquitous in application domains such as healthcare, finance, and others. In such high stakes applications, explaining the model outcome is crucial to build trust among end-users. Finding the features that drive the output of time series models is a challenging task due to complex non-linear temporal dependencies and cross-correlations in the data. The explainability problem is significantly exacerbated when more complex models are used. Most of the current work in time series settings focus on evaluating globally relevant features ). However, often global feature importance represents relevant features for the entire population, that may not characterize local explanations for individual samples. Therefore we focus our work on individualized feature importance in time series settings. In addition, besides identifying relevant features, we also identify the relevant time instances for those specific features, i.e., we identify the most relevant observations. To the best of our knowledge this is the first sample-specific feature importance explanation benchmark at observation level for time series models. In this work, we propose a counterfactual based method to learn the importance of every observation in a multivariate time series model. We assign importance by evaluating the expected change in model prediction had an observation been different. We generate plausible counterfactual observations based on signal history, to asses temporal changes in the underlying dynamics. The choice of the counterfactual distribution affects the quality of the explanation. By generating counterfactuals based on signal history, we ensure samples are realistic under individual dynamics, giving explanations that are more reliable compared to other ad-hoc counterfactual methods. In this section we describe our method, Feed Forward Counterfactual (FFC), for generating explanation for time series models. A feature is considered important if it affects the model output the most. In time series, the dynamics of the features also change over time, which may impact model outcome. As such it is critical to also identify the precise time points of such changes. In order to find important observations for highdimensional time series models, we propose to use a counterfactual based method. Specifically, (a) (b) FFC procedure: CounterfactualXt is generated using signal history. We look into the difference of the original output yt andŷ (i,t) where observation x (i,t) is replaced with a counterfactual. we assign importance to an observation x i,t (feature i at time t) based on its effect on the model output at time T (> t). We replace observation x i,t with a counterfactualx i,t, to evaluate this effect. Figure 1 demonstrates how importance of a feature is characterized by replacing an observation with a counterfactual. Multi-variate time series data is available in the form of X (n) ∈ R d×T (where d is the number of features with T observations over time) for n ∈ [N] samples. We are interested in black-box estimators that receive observations up to time t, x t ∈ R d, and generate output y t at every time point t ∈ [T]. F denotes the target black-box model we wish to explain through the proposed approach called FFC. For exposition, throughout the paper, the index n over samples has been dropped for notational clarity. We index features with subscript i. x −i,t indicates features at time t excluding feature i. The notation used for exposition work is briefly summarized in Table 1. Observed outcome of the black-box model F, at time t Generative Model and Estimator Gi: Latent encoding of history up to t Table 1: Notation used in the paper. We assign importance score to each observation x i,t for t ∈ [T] and i ∈ [d], following the definition: Definition 1. Feature Importance: The importance of the observation i at time t, denoted by Imp(i, t) is defined as E p(xi,t|X0:t−1) [|F(X 0:t) − F(X 0:t−1, x −i,t,x i,t)|], where | · | denotes the absolute value andx i,t is the counterfactual sample. That is, the importance of an observation for feature i at time t is defined as the change in model output when the observation is replaced by a generated counterfactual. The counterfactual observation can come from any distribution, however the quality of the counterfactual random variable directly affects the quality of the explanation. We generate the counterfactual sample conditioned on signal history up to time t by sampling from the distribution p(x i,t |X 0:t−1). Using a conditional generator guarantees that our counterfactuals are sampled not only within domain but also specifically likely under the individual sample X (n), as opposed to having a generator that models data across population. Conditioning on the history also allows us to learn the dynamics of the signal and therefore generate a plausible counterfactual given the past observations. p(x t |X 0:t−1) represents the distribution at time t, if there were no change in the dynamics of the signals. The counterfactualx i,t is sampled from the marginal distribution p(bx i,t |X 0:t−1), obtained from p(x t |X 0:t−1). Let F(X 0:t−1, x −i,t,x i,t) be the output of the model at time T, when x i,t is replaced by the generated counterfactualx i,t. We estimate feature importance Imp(i, t) as E p(xi,t|X0:t−1) [|F(X 0:t) − F(X 0:t−1, x −i,t,x i,t)|], summarized in figure 2(b). Our proposed method has the following compelling properties in explaining the estimator F: Time Importance (TI) For every time series, highlighting relevant time events for different features is important for actionability of the explanations. For instance in a clinical setting, just knowing a feature like heart rate is relevant, is not sufficient to intervene -it is also important to know when a deterioration had happened. With FFC, the most eventful time instances can be obtained as: We can thus rank time instances in order of importance. That is, time t 1 t 2, if Feature Importance (FI) At any time instance t, our method assigns importance to every feature of x i,t. The magnitude of our importance function reflects relative importance. Comparing the importance values across features gives the flexibility to report a subset of important features at each time point t and also reflects the correlation between various features of the time series. We approximate the conditional distribution of p(x t |X 0:t−1) using a recurrent latent variable generator model G, introduced in. The architecture we use is provided in Figure 2(a). The conditional generator G models p(x t |z t−1) where z t−1 ∈ R k is the latent representation of history of the time series up to time t. The latent representation is a continuous random variable, modeling the distribution parameters. We only use past information in the time series to reflect temporal dependencies. Using the recurrent structure allows us to model a non-stationary generative model that can also handle varying length of observations. Implementation details of the generator are in the Appendix. Our counterfactuals are not derived by looking at future values which could be done for reliable imputation. Counterfactuals should represent the past dynamics. Note that our derived feature importance is limited by the quality of imputation that may have been utilized by the black-box risk predictor. For experimental evaluation on the effect of generator specifications on counterfactuals and the quality of explanations, see Section 4.1. The proposed procedure is summarized in Algorithm 1. We assume that we are given a trained block box model F, and the data (without labels) it had been trained on. Using the training data, we first train the generator that generates the conditional distribution, (denoted by G). In our implementation we model x as a multivariate Gaussian with full covariance to model all existing correlation between features. The counterfactualx i,t is then sampled from G and passed to the black-box model to evaluate the effect on the black-box outcome. A common method of explaining model performance, in time-series deep learning, is via visualization of activations of latent layers (; ;) Return Importance_M atrix sensitivity analysis . Understanding latent representations, sensitivity and its relationship to overall model behavior is useful for model debugging. However, these but are too refined to be useful to the end users like clinicians. Attention models (; ;) are the most commonly known explanation mechanisms for sequential data. However, because of the complex mappings to latent space in recurrent models, attention weights cannot be directly attributed to individual observations of the time series . To resolve this issue to some extent, propose an attention model for mortality prediction of ICU patients based on clinical visits. However attention weights may not be consistent as explanations . In vision, prior works tackle explainability from the counterfactual perspective, finding regions of the image that affect model prediction the most. assumes higher importance for inputs that when replaced by an uninformative reference value, maximally change the classifier output. A criticism to such methods is that they may generate out-of-distribution counterfactuals, leading to unreliable explanations. address this issue for images using conditional generative models for inpainting regions with realistic counterfactuals. Evaluating sample based feature importance remains largely unstudied for time series models. While more widely studied for image classification, these methods cannot be directly extended to time series models due to complex time-series dynamics. Most efforts in this domain focus on population level feature importance . is one of the few methods addressing sample based feature importance and use a method similar to , called "feature occlusion". They replace each time series observation by a sample from uniform noise to evaluate its effect on model outcome to attribute feature importance. We argue that carefully choosing the counterfactual selection policy is necessary for derive reliable importances. Specifically, replacing observations with noisy out-of-domain samples can lead to arbitrary changes to model output that are not reflective of systematic behavior in the domain. Even if an observation is sampled from the domain distribution, it does not characterize temporal dynamics and dependencies well enough, potentially highlighting features that only reflect global model behavior, as opposed to sample specific feature importance. We therefore model the data-distribution in order to generate reliable counterfactuals. We demonstrate the implications of the choice of the generator (and hence the counterfactuals) on the quality of explanation. We evaluate our explainability method for finding important features in time series on 2 simulated datasets and 2 real datasets. Our goal is two-fold a) comparison to existing feature importance baselines in time series and b) evaluating the choice of generators on the quality of counterfactuals and explanations. We compare to existing feature importance baselines described below: 1. Feature Occlusion (FO) : Method introduced in. This method is an ad-hoc approach for generating counterfactuals. When replacing x i,t with a random sample from the uniform distribution, the change in model output defines the importance for x i,t. We augment the method introduced in by sampling counterfactuals from the bootstrapped distribution over each feature. This avoids generating out-of-distribution samples. 3. Sensitivity Analysis (SA): This method evaluates the sensitivity of the output to every observation, by taking the derivative of y t with respect to x i,t, at every time point. 4. LIME : One of the most commonly used explainabilty methods that assigns local importance to features. Although LIME does not assign temporal importance, for this baseline, we use LIME at every time point to generate feature importances. Evaluating the quality of explanations is challenging due to the lack of a gold standard/ground truth for the explanations. Additionally, explanations are reflective of model behavior, therefore such evaluations are tightly linked to the reliability of the model itself. Therefore we created the simulated environment in order to test our method. In this experiment, we simulate a time series data such that only one feature determines the outcome. Specifically, the outcome (label) changes to 1 as soon as a spike is observed in the relevant feature. We keep the task fairly simple for two main reasons: 1) to ensure that the black-box classifier can indeed learn the right relation between the important feature and the outcome, which allows us to focus on evaluating the quality of the explanations without worrying about the quality of the classifier. 2) to have a gold standard for the explanations since the exact important event predictive of the outcome are known. We expect the explanations to assign importance only to the one relevant feature, at the exact time of spike, even in the presence of spikes in other non-relevant features. To simulate these data, we generate d = 3 (independent) sequences as a standard non-linear auto-regressive moving average (NARMA) time series of the form:, where the order is 2 and u ∼ Normal(0, 0.01). We add linear trends to the features and introduce random spikes over time for every feature. Note that since spikes are not correlated over time, no of the generators (used in FFC, AFO, FO) will learn to predict it. The important feature in this setting is feature 1. The complete procedure is described in Appendix A.2.1. We train an RNN-based black-box model on this data, ing in AUC= 0.99 on the test set. Figure 7 demonstrates explanations of each of the compared approaches on simulated data for 2 test samples. As shown in Figure 7 (a), Sensitivity analysis does not pick up on the importance of the spike. Feature occlusion gives false positive importance to spikes that happen in non-important signals as well as the important one. Augmented feature occlusion resolves this problem since it samples the counterfactuals from the data distribution, however, it generates noisier as it samples from the bootstrap distribution. The proposed method (FFC) only assigns importance to the first feature at the time of spike. Hence, FFC generates fewer false relevance scores. Note that all baseline methods provide an importance for evry sample at every time point. The true explanation should highlight feature 1 at time points of spike. Using this ground truth, we evaluate the AUROC and AUPRC of the generated explanations denoted by (exp). II, we also show in the third column that the log-probabilities of our counterfactuals are higher under the true distribution. The first simulation does not necessarily evaluate feature importance under complex state dynamics as is common in applications. In this simulation, we create a dataset with complex dynamics with known ground truth explanations. The dataset consists of multivariate time series signals with 3 features. A Hidden Markov Model with 2 latent states, with linear transition matrix and multivariate normal emission probabilities is used to simulate observations. The the outcome y is a random variable, which, in state 1, is only affected by feature 1 and in state 2, only affected by feature 2. Also, we add non-stationarity to the time series by modeling the state transition probabilities as a function of time. The ground truth explanation for output at time T is the observation x i,t where i is the feature that drives the output in the current state and t indicates when feature i became important. In a time series setting, a full explanation for outcome at t = T should include the most important feature variable as well as the time point of importance (here state change). Figure 4 demonstrates assigned importance for a time series sample. The shaded regions indicate the top 5 important observations (x i,t) for each method, the color indicating the corresponding feature i. AFO, FO and FFC are able to learn the state dynamics and are able to find the important feature of each state. However, the top importance values in AFO and FO do not correspond to the important time points. Only in FFC, the top important observations are indicative of state changes. Table 2 shows the performance compared to ground-truth explanations for this data. As mentioned earlier, the quality of explanations rely on the quality of the counterfactuals. The counterfactuals should reflect the underlying latent dynamics for an individual sample. Counterfactuals under the marginal (as used by AFO) need not be likely for a specific sample. The conditional distribution we use, on the other hand, models the underlying characteristic of an individual sample, while the marginal is an average over the population. Counterfactuals unlikely under an individual patient's dynamics can in inaccurate importance assignments since they can potentially overestimate the change in model outcome significantly. We demonstrate this by evaluating the log probability of the counterfactual under the true generator distribution p * (x t |X 0:t−1). Results are summarized in Table 2, Column 3. Since we simulate data using an HMM, we can analytically derive the distribution p * (x i,t |X 0:t−1). Details of the derivation are included in Appendix A.2.1. Following the procedure in Algorithm 1, we train a conditional generators for non-static time series features. We compare across all four existing methods by visualizing importance scores over time. Figure 5 shows an example trajectory of a patient and the predicted outcome. We plot the importance score over time for top 3 signals, selected by each method. Shaded regions in bottom four rows indicate the most important observations, color representing the feature. As shown in Figure 5, counterfactual based methods mostly agree and pick the same signals as important features. We further evaluate this by looking into accordance scores among methods, indicating the percentage of agreement. This analysis is provided in the Appendix A.3, and the heat map in Figure 10 demonstrates the average score across test samples. However, the methods don't agree on the exact time importance. As we observe in Figure 5 and other patient trajectories, FFC assigns importance to observations at the precise times of signal change. This is exactly as expected from the method. The FFC counterfactual is conditioned on patient history and thus the counterfactual represents an observation assuming a patient had not change state. Since evaluation of explanations can be subjective, we also use intervention information present in patient records to evaluate clinical applicability across baselines. Clinicians intervene with a medication or procedure when there is a notable, adverse change in patient trajectory. Therefore, picking up the most relevant features before an intervention onset is indicative of clinical validity of the method. While we cannot directly associate an intervention with a specific cause (observation), we look at the overall distribution of signals considered important by each of the methods, prior to intervention onset. Figure 6 shows these histograms for a number of interventions. We see consistent This experiment evaluates the utility of our method for attributing importance to GHG tracers across different locations in California. The GHG data consists of 15 time series signals from 2921 grid cells. A target time series is a synthetic signal of GHG concentrations. We use an RNN model to estimate GHG concentrations using all tracers. Evaluating which tracers are most useful in reconstructing this synthetic signal can be posed as a feature importance problem for weather modeling over time. In order to quantitatively evaluate the proposed method on real data, we evaluate how well the method performs at selecting globally relevant methods as a proxy. We aggregate the importance of all features over time (and training samples) and retrain the black-box by i) removing top 10 relevant features as indicated by each method ii) using top 3 relevant features only. The performance summary is provided in Table 3 suggesting that among methods that derive instance wise feature importance over time, FFC also generates reasonable global relevance of features. Results for both MIMIC-III and GHG datasets are summarized in We additionally evaluate the quality of the proposed FFC method using the randomization tests proposed as'Sanity Checks' in. Two randomization tests are designed to test for sensitivity of the explanations to i) the black-box model parameters using a model parameter randomization test, and ii) sensitivity to data labels using a using a data randomization test. We conduct this evaluation for Simulation Data II. This test evaluates how different explanations are when the black-box model is trained on permuted labels (breaking the correlation between features and output label). If explanations truly rely on the output labels, as suggested in our definition, then the explanation quality should differ significantly when a model trains on permuted labels. We evaluate the drop in the AUROC and AUPRC of the generated explanations compared to the ground truth. This test evaluates how different explanation quality is when the parameters of the model are arbitrarily shuffled. Significant differences in generated explanations suggests the proposed method is sensitive to black-box model parameters. , these tests are conducted for saliency map methods for images by evaluating the distance between saliency maps for different randomizations. The are included for Simulated Data II, measured with AUROC and AUPRC as ground-truth explanations are available. The of both tests are included in Table 4. They indicate the drops in explanation performance for both randomization tests. The performance of the model used for model randomization test drops to 0.52 AUROC as opposed to 0.95 for the original trained model on this simulated task (Simulation Data II). For data randomization, performance of the model drops to 0.62 from 0.95 in terms of AUROC. AUROCs and AUPRCs for FFC drop the most, suggesting the FFC explanation method is sensitive to perturbations in output labels (as tested by the data randomization test) and to randomization in model parameters. Significant deterioration compared to explanation performance in Table 2 (for Simulation Data II) indicates that the proposed method passes the sanity checks. We propose a new definition for obtaining sample-based feature importance for high-dimensional time series data. We evaluate the importance of each feature at every time point, to locate highly important observations. We define important observations as those that cause the biggest change in model output had they been different from the actual observation. This counterfactual observation is generated by modeling the conditional distribution of the underlying data dynamics. We propose a generative model to sample such counterfactuals. We evaluate and compare the proposed definition and algorithm to several existing approaches. We show that our method is better at localizing important observations over time. This is one of the first methods that provides individual feature importance over time. Future extension to this work will include analysis on real datasets annotated with feature importance explanations. The method will also be extended to evaluate change in risk based on most relevant subsets of observations. A.1 SIMULATED DATA I To simulate these data, we generate d = 3 (independent) sequences as a standard non-linear autoregressive moving average (NARMA) time series. Note also that we add linear trends to features 1 and 2 of the form: x(t + 1) = 0.5x(t) + 0.5x(t) l−1 i=0 x(t − l) + 1.5u(t − (l − 1))u(t) + 0.5 + α d t for t ∈, α > 0 (0.065 for feature 2 and 0.003 for feature 1), and where the order l = 2, u ∼ Normal(0, 0.03). We additionally add linear trends to features. We add spikes to each sample (uniformly at random over time) and for every feature d following the procedure below: where κ > 0 indicates the additive spike. The label y t = 1 ∀t > t 1, where t 1 = min g d, i.e. the label changes to 1 when a spike is encountered in the first feature and is 0 otherwise. We sample our time series using the python TimeSynth 1 package. Number of samples generated: 10000 (80%,20% split). The output y t at every step is assigned using the logit in 3. Depending on the hidden state at time t, only one of the features contribute to the output and is deemed influential to the output. The true conditional distribution can be derived using the forward algorithm as follows: where, where p(s t−1 |X 0:t−1) is estimated using the forward algorithm. Our generator G i is trained using an RNN (GRU). We model the latent state z t with a multivariate Gaussian with diagonal covariance and observations with a multivariate Gaussian with full covariance. The counterfactual for observation i at time t can now be sampled by marginalizing over other features at time t. i.e, x i,t ∼ x−i p(x|X 0:t−1). Feature selection and data processing: For this experiment, we select adult ICU admission data from the MIMIC dataset. We use static patients' static, vital measurements and lab for the analysis. The task is to predict 48 hour mortality based on 48 hours of clinical data, therefor we remove samples with less than 48 hours of data. Parameter Settings for conditional Generator: The recurrent network with specifications show in 8 learns a hidden latent vector h t representing the history. h t is then concatenated with x −i,t and fed into a non-linear 1-layer MLP to model the conditional distribution p(x i, t|X 0:t−1). Additional importance plots are provided in Figure 9. Adam (learning rate = 0.0001, β 1 = 0.9, β 2 = 0.999, weight decay = 0) Accordance testing: For this test we look into how much different baselines agree on important feature assignment. As we observed from the experiments, counterfactual methods mostly agree on the most important features for individual samples. We define accordance score between 2 methods as the percentage of top n signals both identified as important. A score of 80 means on average over the test data, 80 of the assignments were similar. This is depicted in Figure 10. In this section we compare the run-time across multiple baselines. Table 9 shows inference runtime for all the baseline methods on a machine with Quadro 400 GPU and Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz CPU. The runtime for the counterfactual approaches (FFC, FO and AFO) depends only on the length of the time series. It is also the case for FFC since the conditional generator models the joint distribution of all features. This is an advantage, over approaches like LIME, the runtime depends both on the length of the signal as well as the number of features. Overall, FFC performs reasonably compared to ad-hoc counterfactual approaches, since inference on the RNN conditional generator is efficient. This is one of the reasons that the RNN generator model is used to approximate the conditional distribution. Table 9: Run-time for simulated data and MIMIC experiment. Parameter Settings for Generator: The settings are provided in Table 10. Figure 11 shows the training loss of black-box that was used to present feature important in Section 4.4.

Represent the short summary for finding the corresponding scientific article (long):

Explaining Multivariate Time Series Models by finding important observations in time using Counterfactuals

Represent the TLDR for finding the corresponding scientific article.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the article for finding a single-sentence TLDR

Question: This paper addresses unsupervised domain adaptation, the setting where labeled training data is available on a source domain, but the goal is to have good performance on a target domain with only unlabeled data. Like much of previous work, we seek to align the learned representations of the source and target domains while preserving discriminability. The way we accomplish alignment is by learning to perform auxiliary self-supervised task(s) on both domains simultaneously. Each self-supervised task brings the two domains closer together along the direction relevant to that task. Training this jointly with the main task classifier on the source domain is shown to successfully generalize to the unlabeled target domain. The presented objective is straightforward to implement and easy to optimize. We achieve state-of-the-art on four out of seven standard benchmarks, and competitive on segmentation adaptation. We also demonstrate that our method composes well with another popular pixel-level adaptation method.

represent the short summary for finding the corresponding scientific article (long)

We use self-supervision on both domain to align them for unsupervised domain adaptation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Parametric adversarial divergences implicitly define more meaningful task losses for generative modeling, we make parallels with structured prediction to study the properties of these divergences and their ability to encode the task of interest.