Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose Significance-Offset Convolutional Neural Network, a deep convolutional network architecture for regression of multivariate asynchronous time series. The model is inspired by standard autoregressive (AR) models and gating mechanisms used in recurrent neural networks. It involves an AR-like weighting system, where the final predictor is obtained as a weighted sum of adjusted regressors, while the weights are data-dependent functions learnt through a convolutional network. The architecture was designed for applications on asynchronous time series and is evaluated on such datasets: a hedge fund proprietary dataset of over 2 million quotes for a credit derivative index, an artificially generated noisy autoregressive series and household electricity consumption dataset. The pro-posed architecture achieves promising as compared to convolutional and recurrent neural networks. The code for the numerical experiments and the architecture implementation will be shared online to make the research reproducible. Time series forecasting is focused on modeling the predictors of future values of time series given their past. As in many cases the relationship between past and future observations is not deterministic, this amounts to expressing the conditional probability distribution as a function of the past observations: p(X t+d |X t, X t−1, . . .) = f (X t, X t−1, . . .).This forecasting problem has been approached almost independently by econometrics and machine learning communities. In this paper we examine the capabilities of convolutional neural networks (CNNs), BID25 in modeling the conditional mean of the distribution of future observations; in other words, the problem of autoregression. We focus on time series with multivariate and noisy signal. In particular, we work with financial data which has received limited public attention from the deep learning community and for which nonparametric methods are not commonly applied. Financial time series are particularly challenging to predict due to their low signal-to-noise ratio (cf. applications of Random Matrix Theory in econophysics BID24 BID3) and heavy-tailed distributions BID8. Moreover, the predictability of financial market returns remains an open problem and is discussed in many publications (cf. efficient market hypothesis of BID11).A common situation with financial data is that the same signal (e.g. value of an asset) is observed from different sources (e.g. financial news, analysts, portfolio managers in hedge funds, marketmakers in investment banks) in asynchronous moments of time. Each of these sources may have a different bias and noise with respect to the original signal that needs to be recovered (cf. time series in FIG0). Moreover, these sources are usually strongly correlated and lead-lag relationships are possible (e.g. a market-maker with more clients can update its view more frequently and precisely than one with fewer clients). Therefore, the significance of each of the available past observations might be dependent on some other factors that can change in time. Hence, the traditional econometric models such as AR, VAR, VARMA might not be sufficient. Yet their relatively good performance motivates coupling such linear models with deep neural networks that are capable of learning highly nonlinear relationships. Quotes from four different market participants (sources) for the same CDS 1 throughout one day. Each trader displays from time to time the prices for which he offers to buy (bid) and sell (ask) the underlying CDS. The filled area marks the difference between the best sell and buy offers (spread) at each time. For these reasons, we propose SignificanceOffset Convolutional Neural Network, a Convolutional Network extension of standard autoregressive models BID34 BID35 equipped with a nonlinear weighting mechanism and provide empirical evidence on its competitiveness with standard multilayer CNN and recurrent Long-Short Term Memory network BID18. The mechanism is inspired by the gating systems that proved successful in recurrent neural networks BID18 BID6 and highway networks BID37.2 RELATED WORK 2.1 TIME SERIES FORECASTING Literature in time series forecasting is rich and has a long history in the field of econometrics which makes extensive use of linear stochastic models such as AR, ARIMA and GARCH processes to mention a few. Unlike in machine learning, research in econometrics is more focused on explaining variables rather than improving out-of-sample prediction power. In practice, one can notice that these models'over-fit' on financial time series: their parameters are unstable and out-of-sample performance is poor. Reading through recent proceedings of the main machine learning venues (e.g. ICML, NIPS, AIS-TATS, UAI), one can notice that time series are often forecast using Gaussian processes BID31 BID38 BID19, especially when time series are irregularly sampled BID9 BID26. Though still largely independent, researchers have started to "bring together the machine learning and econometrics communities" by building on top of their respective fundamental models yielding to, for example, the Gaussian Copula Process Volatility model BID42. Our paper is in line with this emerging trend by coupling AR models and neural networks. Over the past 5 years, deep neural networks have surpassed from most of the existing literature in many fields BID33: computer vision BID23, audio signal processing and speech recognition BID32, natural language processing (NLP) BID1 BID7 BID14 BID21. Although sequence modeling in NLP, i.e. prediction of the next character or word, is related to our forecasting problem, the nature of the sequences is too dissimilar to allow using the same cost functions and architectures. Same applies to the adversarial training proposed by BID28 for video frame prediciton, as such approach favors most plausible scenarios rather than outputs close to all possible outputs, while the latter is usually required in financial time series due to stochasticity of the considered processes. Literature on deep learning for time series forecasting is still scarce (cf. BID12 for a recent review). Literature on deep learning for financial time series forecasting is even scarcer though interest in using neural networks for financial predictions is not new BID30 BID29. More recent papers include BID36 that used 4-layer perceptrons in modeling price change distributions in Limit Order Books, and BID2 who applied more recent WaveNet architecture of van den BID39 to several short univariate and bivariate time-series (including financial ones). Despite claim of applying deep learning, BID17 use autoencoders with a single hidden layer to compress multivariate financial data. Besides these and claims of secretive hedge funds (it can be marketing surfing on the deep learning hype), no promising or innovative architectures were publicly published so far, to the best of our knowledge. In this paper, we investigate the gold standard architectures' (simple Convolutional Neural Network (CNN), Residual Network, multi-layer LSTM) capabilities on AR-like artificial asynchronous and noisy time series, and on real financial data from the credit default swap market where some inefficiencies may exist, i.e. time series may not be totally random. Gating mechanisms for neural networks were first proposed by BID18 and proved essential in training recurrent architectures BID21 due to their ability to overcome the problem of vanishing gradient. In general, they can be expressed as DISPLAYFORM0 where f is the output function, c is a'candidate output' (usually a nonlinear function of x), ⊗ is an element-wise matrix product and σ: R → is a sigmoid nonlinearity that controls the amount of the output passed to the next layer (or to further operations within a layer). Appropriate compositions of functions of type 2 lead to the popular recurrent architectures such as LSTM BID18 and GRU BID6.A similar idea was recently used in construction of highway networks BID37 which enabled successful training of deeper architectures. van den BID40 and BID10 proposed gating mechanisms (respectively with hyperbolic tangent and linear 'candidate outputs') for training deep convolutional neural networks. The gating system that we propose is aimed at weighting a number of different'candidate predictors' and therefore is most closely related to the softmax gating used in MuFuRU (Multi-Function Recurrent Unit, BID41), i.e. DISPLAYFORM1 where (The idea of weighting outputs of the intermediate layers within a neural networks is also used in attention networks (See e.g. BID4) that proved successful in such tasks as image captioning and machine translation. Our approach is similar as the separate inputs (time series steps) are weighted in accordance with learned functions of these inputs, yet different since we model these functions as multi-layer CNNs (instead of projections on learned directions) and since we do not use recurrent layers. The latter is important in the above mentioned tasks as it enables the network to remember the parts of the sentence/image already translated/described. Time series observed in irregular moments of time make up significant challenges for learning algorithms. Gaussian processes provide useful theoretical framework capable of handling asynchronous data; however, due to assumed Gaussianity they are inappropriate for financial datasets, which often follow fat-tailed distributions BID8 ). On the other hand, prediction of even a simple autoregressive time series such us AR given by X(t) = αX(t − 1) + βX(t − 2) + ε(t) 2 may involve highly nonlinear functions when sampled irregularly. Precisely, it can be shown that the conditional expectation DISPLAYFORM0 where a k and b k are rational functions of α and β (See Appendix A for the proof). This would not be a problem if k was fixed, as then one would be interested in estimating of a k and b k directly; this, however, is not the case with asynchronous sampling. When X is an autoregressive series of higher order and more past observations are available, the analogous expectation E[X(t n)|{X(t n−m), m = 1,..., M }] would involve more complicated functions that in general may not possess closed forms. In real-world applications we often deal with multivariate time series whose dimensions are observed separately and asynchronously. This adds even more difficulty to assigning appropriate weights to the past values, even if the underlying data structure is linear. Furthermore, appropriate representation of such series might be not obvious as aligning such series at fixed frequency may lead to loss of information (if too low frequency is chosen) or prohibitive enlargement of the dataset (especially when durations have varying magnitudes), see Figure 2A. As an alternative, we might consider representing separate dimensions as a single one with dimension and duration indicators as additional features. Figure 2B presents this approach, which is going to be at the core of the proposed architecture. DISPLAYFORM1 Figure 2: (A) Fixed sampling frequency and it's drawbacks; keeping all available information leads to much more datapoints. (B) Proposed data representation for the asynchronous series. Consecutive observations are stored together as a single value series, regardless of which series they belong to; this information, however, is stored in indicator features, alongside durations between observations. A natural model for prediction of such series could be an LSTM, which, given consecutive input values and respective durations (X(t n), t n − t n−1 ) =: x n in each step would memorize the series values and weight them at the output according to the durations. However, the drawback of such approach lies in imbalance between the needs for memory and for nonlinearity: the weights that such network needs to assign to the memorized observations potentially require several layers of nonlinearity to be computed properly, while past observations might just need to be memorized as they are. For these reasons we shall consider a model that combines simple autoregressive approach with neural network in order to allow learning meaningful data-dependent weights DISPLAYFORM2 are modeled using neural network. To allow more flexibility and cover situations when e.g. observed values of x are biased, we should consider the summation over terms α m (x n−m) · f (x n−m), where f is also a neural network. We formalize this idea in Section 4. Suppose that we are given a multivariate time series (x n) n ⊂ R d and we aim to predict the conditional future values of a subset of elements of x n DISPLAYFORM0 where DISPLAYFORM1. We consider the following estimator of y n DISPLAYFORM2 where DISPLAYFORM3 • σ is a normalized activation function independent on each row, i.e. DISPLAYFORM4 for any a 1,..., a d I ∈ R M and σ such that σ(a) DISPLAYFORM5 • ⊗ is Hadamard (element-wise) matrix multiplication. The summation in 7 goes over the columns of the matrix in bracket; hence the i-th element of the output vectorŷ n is a linear combination of the i-th row of the matrix F (x −M n). We are going to consider S to be a fully convolutional network (composed solely of convolutional layers) and F of the form DISPLAYFORM6 where W ∈ R d I ×M and off: R d → R d I is a multilayer perceptron. In that case F can be seen as a sum of projection (x → x I) and a convolutional network with all kernels of length 1. Equation can be rewritten asŷ DISPLAYFORM7 where W m, S m (·) are m-th columns of matrices W and S(·).Figure 3: A scheme of the proposed SOCNN architecture. The network preserves the timedimension up to the top layer, while the number of features per timestep (filters) in the hidden layers is custom. The last convolutional layer, however, has the number of filters equal to dimension of the output. The Weighting frame shows how outputs from offset and significance networks are combined in accordance with Eq. 10.We will call the proposed network a Significance-Offset Convolutional Neural Network (SOCNN), while off and S respectively the offset and significance (sub)networks. The network scheme is shown in Figure 3. Note that when off ≡ 0 and σ ≡ 1 the model simplifies to the collection of d I separate AR(M) models for each dimension. Note that the form of Equation FORMULA0 enforces the separation of temporal dependence (obtained in weights W m), the local significance of observations S m (S as a convolutional network is determined by its filters which capture local dependencies and are independent on the relative position in time) and the predictors off(x n−m) that are completely independent on position in time. This provides some amount of interpretability of the fitted functions and weights. For instance, each of the past observations provides an adjusted single regressor for the target variable through the offset network. Note that due to asynchronous sampling procedure, consecutive values of x might be heterogenous, hence On the other hand, significance network provides data-dependent weights for all regressors and sums them up in autoregressive manner. Figure 7 in Appendix E.2 shows sample significance and offset activations for the trained network. Relation to asynchronous data As mentioned before, one of the common problems with time series are the varying durations between consecutive observations. A simple approach at data-preprocessing level is aligning the observations at some chosen frequency by e.g. duplicating or interpolating observations. This, however, might extend the size of an input and, therefore, model complexity. The other idea is to treat the duration and/or time of the observation as another feature, as presented in Figure 2B. This approach is at the core of the SOCNN architecture: the significance network is aimed at learning the high-level features that indicate the relative importance of past observations, which, as shown in Section 3, could be predominantly dependent on time and duration between observations. Loss function L 2 error is a natural loss function for the estimators of expected value DISPLAYFORM0 As mentioned above, the output of the offset network can be seen as a collection of separate predictors of the changes between corresponding observations x I n−m and the target variable y n off(x n−m) y n − x I n−m.For that reason, we consider the auxiliary loss function equal to mean squared error of such intermediate predictions DISPLAYFORM1 The total loss for the sample (x DISPLAYFORM2 where y n is given by Eq. 10 and α ≥ 0 is a constant. In Section 5.2 we discuss the empirical findings on the impact of positive values of α on the model training and performance, as compared to α = 0 (lack of auxiliary loss). We evaluate the proposed model on a financial dataset of bid/ask quotes sent by several market participants active in the credit derivatives market, artificially generated datasets and household electric power consumption dataset available from UCI repository BID27, comparing its performance with simple CNN, single-and multi-layer LSTM BID18 and 25-layer ResNet BID16.Apart from performance evaluation of SOCNNs, we discuss the impact of the network components, such as auxiliary loss and the depth of the offset sub-network. The details of the training process and hyperparameters used in the proposed architecture as well as in benchmark models are presented in C. We test our network architecture on the artificially generated datasets of multivariate time series. We consider two types of series:1. Synchronous series. The series of K noisy copies ('sources') of the same univariate autoregressive series ('base series'), observed together at random times. The noise of each copy is of different type.2. Asynchronous series. The series of observations of one of the sources in the above dataset. At each time, the source is selected randomly and its value at this time is added to form a new univariate series. The final series is composed of this series, the durations between random times and the indicators of the'available source' at each time. The details of the simulation process are presented in Appendix D. We consider synchronous and asynchronous series X K×N where K ∈ {16, 64} is the number of sources and N = 10, 000, which gives 4 artificial series in total 3. The household electricity dataset 4 contains measurements of 7 different quantities related to electricity consumption in a single household, recorded every minute for 47 months, yielding over 2 million observations. Since we aim to focus on asynchronous time-series, we alter it so that a single observation contains only value of one of the seven features, while durations between consecutive observations range from 1 to 7 minutes 5.The regression aim is to predict all of the features at the next time step. The proposed model was designed primarily for forecasting incoming non-anonymous quotes received from the credit default swap market. The dataset contains 2.1 million quotes from 28 different sources, i.e. market participants. Each quote is characterized by 31 features: the offered price, 28 indicators of the quoting source, the direction indicator (the quote refers to either a buy or a sell offer) and duration from the previous quote. For each source and direction we aim at predicting the next quoted price from this given source and direction considering the last 60 quotes. We formed 15 separate prediction tasks; in each task the model was trained to predict the next quote by one of the fifteen most active market participants 6.This dataset, which is proprietary, motivated the aforementioned construction of artificial asynchronous time series datasets based on its statistical features for reproducible research purpose. TAB0 presents the detailed from the artificial and electricity datasets. The proposed networks outperform significantly the benchmark networks on the asynchronous, electricity and quotes datasets. For the synchronous datasets, on the other hand, SOCNN almost matches the of the benchmarks. This similar performance could have been anticipated -the correct weights of the past values in synchronous artificial datasets are far less nonlinear than in case when separate dimensions are observed asynchronously. For this reason, the significance network's potential is not fully utilized. We can also observe that the depth of the offset network has negligible or negative This means that the significance network has crucial impact on the performance, which is in-line with the potential drawbacks of the LSTM network discussed in Section 3: obtaining proper weights for the past observations is much more challenging than getting good predictors from the single past values. The small positive auxiliary weight helped achieve more stable test error throughout training in many cases. The higher weights of auxiliary loss considerably improved the test error on asynchronous datasets (See TAB1); however for other datasets its impact was negligible. In general, the proposed SOCNN had significantly lower variance of the test and validation errors, especially in the early stage of the training process and for quotes dataset. Figure 4 presents the learning curves for two different artificial datasets. To understand better why SOCNN obtained better than the other networks, we check how these networks react to the presence of additional noise in the input terms 8. Figure 5 presents changes in the mean squared error and significance and offset network outputs with respect to the level of noise. SOCNN is the most robust out of the compared networks and, together with singlelayer LSTM, least prone to overfitting. Despite the use of dropout and cross-validation, the other models tend to overfit the training set and have non-symmetric error curves on test dataset. Figure 5: Experiment comparing robustness of the considered networks for Asynchronous 16 dataset. The plots show how the error would change if an additional noise term was added to the input series. The dotted curves show the total significance and average absolute offset (not to scale) outputs for the noisy observations. Interestingly, significance of the noisy observations increases with the magnitude of noise; i.e. noisy observations are far from being discarded by SOCNN. In this article, we proposed a weighting mechanism that, coupled with convolutional networks, forms a new neural network architecture for time series prediction. The proposed architecture is designed for regression tasks on asynchronous signals in the presence of high amount of noise. This approach has proved to be successful in forecasting financial and artificially generated asynchronous time series outperforming popular convolutional and recurrent networks. The proposed model can be further extended by adding intermediate weighting layers of the same type in the network structure. Another possible generalization that requires further empirical studies can be obtained by leaving the assumption of independent offset values for each past observation, i.e. considering not only 1x1 convolutional kernels in the offset sub-network. Finally, we aim at testing the performance of the proposed architecture on other real-life datasets with relevant characteristics. We observe that there exists a strong need for common'econometric' datasets benchmark and, more generally, for time series (stochastic processes) regression. APPENDIX A NONLINEARITY IN THE ASYNCHRONOUSLY SAMPLED AUTOREGRESSIVE TIME SERIES Lemma 1. Let X(t) be an AR time series given by DISPLAYFORM0 where (ε(t)) t=1,2,... are i.i.d. error terms. Then DISPLAYFORM1 for any t > k ≥ 2, where a k, b k are rational functions of a and b. Proof. The proof follows a simple induction. It is sufficient to show that DISPLAYFORM2 where DISPLAYFORM3 and E k (t) is a linear combination of {ε(t − i), i = 0, 1,..., k − 2}. Basis of the induction is trivially satisfied via 15. In the induction step, we assume that 17 holds for k. For t > k + 1 we have DISPLAYFORM4. Multiplying sides of this equation by b and adding av k X(t − 1) we obtain DISPLAYFORM5 Since aX(t − 1) + bX(t − 2) = X(t) − ε(t) we get DISPLAYFORM6 As DISPLAYFORM7 is a linear combination of {ε(t − i), i = 0, 1,..., k − 1}, the above equation proves 17 for k = k + 1. To see how robust each of the networks is, we add noise terms to part of the input series and evaluate them on such datapoints, assuming unchanged output. We consider varying magnitude of the noise terms, which are added only to the selected 20% of past steps at the value dimension 9. Formally the procedure is following:1. Select randomly N obs = 6000 observations (X n, y n) (half of which coming from training set and half from test set).2. Add noise terms to the observations X n p:= X n + Ξ n · γ p, for {γ p} 128 p=1 evenly distributed on [−6σ, 6σ], where σ is a standard deviation of the differences of the series being predicted and DISPLAYFORM0 3. For each p evaluate each of the trained models on dataset X n p, y n N obs, separately for n's originally coming from training and test sets. To evaluate the model and the significance of its components, we perform a grid search over some of the hyperparameters, more extensively on the artificial and electricity datasets. These include the offset sub-network's depth (we consider depths of 1 and 10 for artificial and electricity datasets; 1 for Quotes data) and the auxiliary weight α (compared values: {0, 0.1, 0.01}). For all networks we have chosen LeakyReLU activation function DISPLAYFORM0 with leak rate a =.1 as an activation function. We compare the performance of the proposed model with CNN, ResNet, multi-layer LSTM networks and linear (VAR) model. The benchmark networks were designed so that they have a comparable number of parameters as the proposed model. Consequently, LeakyReLU activation function with leak rate.1 was used in all layers except the top ones where linear activation was applied. For CNN we provided the same number of layers, same stride and similar kernel size structure. In each trained CNN, we applied max pooling with the pool size of 2 every two convolutional layers 10. TAB3 presents the configurations of the network hyperparameters used in comparison. The training and validation sets were sampled randomly from the first 80% of timesteps in each series, with ratio 3 to 1. The remaining 20% of data was used as a test set. All models were trained using Adam optimizer BID22 which we found much faster than standard Stochastic Gradient Descent in early tests. We used batch size of 128 for artificial data and 256 for quotes dataset. We also applied batch normalization BID20 in between each convolution and the following activation. At the beginning of each epoch, the training samples were shuffled. To prevent overfitting we applied dropout and early stopping 12. Weights were initialized following the normalized uniform procedure proposed by BID13. Experiments were carried out using implementation relying on Tensorflow BID0 and Keras front end BID5. For artificial data we optimized the models using one K20s NVIDIA GPU while for quotes dataset we used 8-core Intel Core i7-6700 CPU machine only. We simulate a multivariate time series composed of K noisy observations of the same autoregressive signal. The simulated series are constructed as follows:Let p 1, p 2,..., p K ∈ and define Figure 6: Simulated synchronous (left) and asynchronous (right) artificial series. Note the different durations between the observations from different sources in the latter plot. For clarity, we present only 6 out of 16 total dimensions. DISPLAYFORM0 DISPLAYFORM1 We call {X t} N t=1 and {X t} N t=1 synchronous and asynchronous time series, respectively. We simulate both of the processes for N = 10, 000 and each K ∈ {16, 64}. The original dataset has 7 features: global active power, global reactive power, voltage, global intensity, sub-metering 1, sub-metering 2 and sub-metering 3, as well as information on date and time. We created asynchronous version of this dataset in two steps:1. Deterministic time step sampling. The durations between the consecutive observations are periodic and follow a scheme [1min, 2min, 3min, 7min, 2min, 2min, 4min, 1min, 2min, 1min]; the original observations in between are discarded. In other words, if the original observations are indexed according to time (in minutes) elapsed since the first observation, we keep the observations at indices n such that n mod 25 ≡ k ∈.2. Random feature sampling. At each remaining time step, we choose one out of seven features that will be available at this step. The probabilities of the features were chosen to be proportional to [1, 1.5, 1.5 2, 1.5 6] and randomly assigned to each feature before sampling (so that each feature has constant probability of its value being available at each time step. At each time step the sub-sampled dataset is 10-dimensional vector that consists information about the time, date, 7 indicator features that imply which feature is available, and the value of this feature. The length of the sub-sampled dataset is above 800 thousand, i.e. 40% of the original dataset's length. The schedule of the sampled timesteps and available features is attached in the data folder in the supplementary material. In Figure 7 we present significance and offset activations for three input series, from the network trained on electricity dataset. Each row represents activations corresponding to past values of a single feature. Figure 7: Activations of the significance and offset sub-networks for the network trained on Electricity dataset. We present 25 most recent out of 60 past values included in the input data, for 3 separate datapoints. Note the log scale on the left graph.

represent the short summary for finding the corresponding scientific article (long)

Convolutional architecture for learning data-dependent weights for autoregressive forecasting of time series.

Represent the short summary for finding the corresponding scientific article (long)

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.


Question: Represent the article for finding a single-sentence TLDR

Question: MixUp is a data augmentation scheme in which pairs of training samples and their corresponding labels are mixed using linear coefficients. Without label mixing, MixUp becomes a more conventional scheme: input samples are moved but their original labels are retained. Because samples are preferentially moved in the direction of other classes \iffalse -- which are typically clustered in input space -- \fi we refer to this method as directional adversarial training, or DAT. We show that under two mild conditions, MixUp asymptotically convergences to a subset of DAT. We define untied MixUp (UMixUp), a superset of MixUp wherein training labels are mixed with different linear coefficients to those of their corresponding samples. We show that under the same mild conditions, untied MixUp converges to the entire class of DAT schemes. Motivated by the understanding that UMixUp is both a generalization of MixUp and a form of adversarial training, we experiment with different datasets and loss functions to show that UMixUp provides improved performance over MixUp. In short, we present a novel interpretation of MixUp as belonging to a class highly analogous to adversarial training, and on this basis we introduce a simple generalization which outperforms MixUp. Deep learning applications often require complex networks with a large number of parameters (; ;). Although neural networks perform so well that their ability to generalize is an area of study in itself (a;), their high complexity nevertheless causes them to overfit their training data . For this reason, effective regularization techniques are in high demand. There are two flavors of regularization: complexity curtailing and data augmentation 1. Complexity curtailing methods constrain models to learning in a subset of parameter space which has a higher probability of generalizing well. Notable examples are weight decay and dropout . Data augmentation methods add transformed versions of training samples to the original training set. Conventionally, transformed samples retain their original label, so that models effectively see a larger set of data-label training pairs. Commonly applied transformations in image applications include flips, crops and rotations. A recently devised family of augmentation schemes called adversarial training has attracted active research interest (; ; ; ; ;). Adversarial training seeks to reduce a model's propensity to misclassify minimally perturbed training samples, or adversarials. While attack algorithms used for testing model robustness may search for adversarials in unbounded regions of input space, adversarial training schemes generally focus on perturbing training samples within a bounded region, while retaining the sample's original label . Another recently proposed data augmentation scheme is MixUp (b), in which new samples are generated by mixing pairs of training samples using linear coefficients. Despite its well established generalization performance (b; ;), the working mechanism of MixUp is not well understood. suggest viewing MixUp as imposing local linearity on the model using points outside of the data manifold. While this perspective is insightful, we do not believe it paints a full picture of how MixUp operates. A recent study provides empirical evidence that MixUp improves adversarial robustness, but does not present MixUp as a form of adversarial training. We build a framework to understand MixUp in a broader context: we argue that adversarial training is a central working principle of MixUp. To support this contention, we connect MixUp to a MixUplike scheme which does not perform label mixing, and we relate this scheme to adversarial training. Without label mixing, MixUp becomes a conventional augmentation scheme: input samples are moved, but their original labels are retained. Because samples are moved in the direction of other samples -which are typically clustered in input space -we describe this method as'directional'. Because this method primarily moves training samples in the direction of adversarial classes, this method is analogous to adversarial training. We thus refer to MixUp without label mixing as directional adversarial training (DAT). We show that MixUp converges to a subset of DAT under mild conditions, and we thereby argue that adversarial training is a working principle of MixUp. Inspired by this new understanding of MixUp as a form of adversarial training, and upon realizing that MixUp is (asymptotically) a subset of DAT, we introduce Untied MixUp (UMixUp), a simple enhancement of MixUp which converges to the entire family of DAT schemes, as depicted in Figure 1. Untied Mixup mixes data-label training pairs in a similar way to MixUp, with the distinction that the label mixing ratio is an arbitrary function of the sample mixing ratio. We perform experiments to show that UMixUp's classification performance improves upon MixUp. In short, this research is motivated by a curiosity to better understand the working of MixUp. In-sodoing we aim to: 1. Establish DAT as analogous to adversarial training. This is discussed in section 4. 2. Establish UMixUp as a superset of MixUp, and as converging to the entire family of DAT schemes. In-so-doing, a) establish MixUp's convergence to a subset of DAT, and thereby that it operates analogously to adversarial training; and b) establish UMixUp as a broader class of MixUp-like schemes that operate analogously to adversarial training. This is discussed in 5. 3. Establish empirically that UMixUp's classification performance improves upon MixUp. This is discussed in section 6. Finally we note that this paper has another contribution. Conventionally, MixUp is only applicable to baseline models that use cross entropy loss. All analytical we develop in this paper are applicable to a wider family of models using any loss function which we term target-linear. We define target-linearity and experiment with a new loss function called negative cosine-loss to show its potential. Regular (non-calligraphic) capitalized letters such as X will denote random variables, and their lowercase counterparts, e.g., x, will denote realizations of a random variable. Any sequence, (a 1, a 2, . . ., a n) will be denoted by a n 1. Likewise (A 1, A 2, . . ., A n) will be denoted by A n 1, and a sequence of sample pairs ((x 1, x 1), (x 2, x 2),..., (x n, x n)) denoted by (x, x) n 1. For any value a ∈, we will use a as a short notation for 1 − a. Classification Setting Consider a standard classification problem, in which one wishes to learn a classifier that predicts the class label for a sample. Formally, let X be a vector space in which the samples of interest live and let Y be the set of all possible labels associated with these samples. The set of training samples will be denoted by D, a subset of X. We will use t(x) to denote the true label of x. Let F be a neural network function, parameterized by θ, which maps X to another vector space Z. Let ϕ: Y → Z be a function that maps a label in Y to an element in Z such that for any y, y ∈ Y, if y = y, then ϕ(y) = ϕ(y). In the space Z, we refer to F (x) as the model's prediction. With slight abuse of language, we will occasionally refer to both t(x) and ϕ(t(x)) as the "label" of x. Let: Z ×Z → R be a loss function, using which one defines an overall loss function as Here we have taken the notational convention that the first argument of represents the model's prediction and the second represents the target label. In this setting, the learning problem is formulated as minimizing L with respect to its characterizing parameters θ. We say that a loss function (z, z) is target-linear if for any scalars α and β, Target-linear loss functions arise naturally in many settings, for which we now provide two examples. For convenience, we define the vectors v = F (x) and y = ϕ(t(x)). Cross-Entropy Loss The conventional cross-entropy loss function, written in our notation, is defined as: where v and y are constrained to being probability vectors. We note that in conventional applications, dim(Z) = |Y|, and the target label v is a one-hot vector where otherwise. Constraining v to being a probability vector is achieved using a softmax output layer. Negative-Cosine Loss The "negative-cosine loss", usually used in its negated version, i.e., as the cosine similarity, can be defined as follows. where v and y are constrained to being unit-length vectors. For v this can be achieved by simple division at the output layer, and for y by limiting the range of ϕ to an orthonormal basis (making it a conventional label embedding function). It is clear that the cross-entropy loss CE and the negative-cosine loss NC are both target-linear, directly following from the definition of target-linearity. Assumptions The theoretical development of this paper relies on two fundamental assumptions, which we call "axioms". Axiom 1 (Target linearity) The loss function used for the classification setting is target-linear. That is, the study of MixUp in this paper is in fact goes beyond the standard MixUp, which uses the cross-entropy loss. Much of the development in this paper concerns drawing sample pairs n 1 is said to be symmetric if for every (a, b) ∈ D × D, the number of occurrences of (a, b) in the sequence is equal to that of (b, a). Axiom 2 (Symmetric pair-sampling distribution) Whenever a sample pair (x, x) is drawn from a distribution Q, Q is assumed to be symmetric. In the standard MixUp, two samples are drawn independently from D to form a pair, making this condition satisfied. 3 MIXUP, DAT, UNTIED MIXUP We first provide a summary of each scheme for the reader's convenience. We then describe each scheme more systematically. For concision of equations to follow, we define and MixUp is a data augmentation scheme in which samples are linearly combined using some mixing ratio λ ∈: where λ ∼ P Mix. A target label is generated using the same mixing ratio λ: DAT and UMixUp use the same method for generating samples, but use different λ distributions (P DAT and P uMix respectively). DAT and UMixUp also differ from MixUp in their target labels. DAT retains the sample's original label: whereas UMixUp's label mixing ratio is a function of λ: In Untied MixUp, the label mixing ratio is "untied" from the sample mixing ratio, and can be any γ(λ). We will refer to γ as the weighting function. An Untied MixUp scheme is specified both by the its mixing policy P uMix and a weighting function γ. To draw comparisons between MixUp, DAT, and Untied MixUp schemes, we establish a framework for characterizing their optimization problems.. We denote the expected value of each scheme's overall loss, L m E, with respect to its mixing ratio Λ. Let n be a positive integer. In every scheme, a sequence (x, x) n 1:= ((x 1, x 1), (x 2, x 2),..., (x n, x n)) of sample pairs are drawn i.i.d. from Q, and a sequence λ In MixUp, we refer to P Mix as the mixing policy. Directional Adversarial Training (DAT) For any x, x ∈ D and any λ ∈, we denote In DAT, we refer to P DAT as the adversarial policy. Let γ be a function mapping to. For any x, x ∈ D and any λ ∈, we denote Let P m be P uMix, and denote the overall and expected overall loss functions At this end, it is apparent that MixUp is a special case of Untied MixUp, where the function γ(λ) takes the simple form γ(λ) = λ. The main theoretical of this paper is the relationship established between DAT and UMixUp, and by extension MixUp. Both MixUp and UMixUp will be shown to converge to DAT as the number of mixed sample pairs, n, tends to infinity. Prior to developing these , we provide insight into DAT, in terms of its similarity to adversarial training and its regularization mechanisms. Conventional adversarial training schemes augment the original training dataset by searching for approximations of true adversarials within bounded regions around each training sample. For a training sample x, a bounded region U known as an L p ball is defined as U = {x + η η η : ||η η η|| p <}. Over this region, the loss function with respect to the true label of x is maximized. A typical loss function for an adversarial scheme is where b is the baseline loss function. Simply put, baseline training serves to learn correct classification over the training data, whereas adversarial training moves the classification boundary to improve generalization. DAT, on the other hand, combines intra-class mixing (mixing two samples of the same class) and inter-class mixing (mixing samples of different classes). Intra-class mixing serves to smooth classification boundaries of inner-class regions, while inter-class mixing perturbs training samples in the direction of adversarial classes, which improves generalization. Inter-class mixing dwarves intraclass mixing by volume of generated samples seen by the learning model in most many-class learning problems (by a 9-1 ratio in balanced 10-class problems for instance). DAT, which primarily consists of inter-class mixing, can therefore be seen as analogous to adversarial training. The key distinction between conventional adversarial training and inter-class mixing is that MixUp movement is determined probabilistically within a bounded region, while adversarial movement is deterministic. Figure 2 illustrates the connection between standard adversarial training and DAT. Consider the problem of classifying the blue points and the black points in Figure 2a), where the dashed curve is a ground-truth classifier and the black curve indicates the classification boundary of F (x), which overfits the training data. In adversarial training, a training sample x is moved to a location within an L p -ball around x while keeping its label to further train the model; the location, denoted by x 1 in Figure 2b ), is chosen to maximize training loss. In DAT, a second sample x governs the direction in which x is perturbed. If x is chosen from a different class as shown in Figure 2c ), then the generated sample x 2 is used to further train the model. If x is chosen from the same class as shown in Figure 2d ), then the sample x 3 is used in further training. Note that the inter-class mixed sample x 2 pushes the model's classification boundary closer to the ground-truth classifier, thus connecting adversarial training and DAT. The intra-class sample x 3, on the other hand, mainly helps to smooth inner parts of the class region. The latter behaviour is an additional feature of DAT and MixUp, which distinguishes these schemes from adversarial training. We now show that Untied MixUp and DAT are equivalent when n tends to infinity. A consequence of this equivalence is that it infuses both MixUp and UMixUp with the intuition of adversarial training. To that end, we relate the Untied MixUp loss function, uMix, with the DAT loss function, DAT. Lemma 1 For any (x, x) ∈ D × D and any λ ∈, This follows directly from the target-linearity of the loss function. The next two lemmas show that as n tends to infinity, the overall loss of both DAT and UMixUp converge in probability to their respective overall expected losses. n 1 in probability. These two lemmas have similar proofs, thus only the proof of Lemma 2 is given in section A.1. Next we show that as n tends to infinity, UMixUp converges in probability to a subset of DAT, and DAT converges in probability to a subset of UMixUp. In other words, we show that as n increases, UMixUp converges to being equivalent to the entire class of DAT schemes. For that purpose, let F denote the space of all functions mapping to. Each configuration in P × F defines an Untied MixUp scheme. We now define U, which maps a DAT scheme to an Untied MixUp scheme. Specifically U is a map from P to P × F such that for any p ∈ P, U(p) is a configuration (p, g) ∈ P × F, where Lemma 4 Let (x, x) n 1 be a sequence of sample pairs on which an Untied MixUp scheme specified by (P uMix, γ) and a DAT scheme with policy P DAT will apply independently. If (x, x) n 1 is symmetric and We now define another map D u that maps an Untied MixUp scheme to a DAT scheme. Specifically D u is a map from P × F to P such that for any (p, g) It is easy to verify that 1 0 p (λ)dλ = 1. Thus p is indeed a distribution in P and D u is well defined. Lemma 5 Let (x, x) n 1 be a sequence of sample pairs on which an Untied MixUp scheme specified by (P uMix, γ) and a DAT scheme with policy P DAT will apply independently. If (x, x) n 1 is symmetric and Lemmas 2, 3, 4 and 5 provide the building blocks for theorem 1, which we state hereafter. As n increases, both DAT and UMixUp converge in probability toward their respective expected loss (lemmas 2 and 3). Since as n increases, the sequence (x, x) n 1 becomes arbitrarily close to the symmetric sampling distribution Q, then by lemma 4 the family of DAT schemes converges in probability to a subset of UMixUp schemes. Lemma 5 proves the converse, i.e. that as n increases the family of UMixUp schemes converges in probability to a subset of DAT schemes. As n n n increases, the family of UMixUp schemes therefore converges in probability to the entire family of DAT schemes. On this sample-pair data, an Untied MixUp scheme specified by (P Mix, γ) and a DAT scheme specified by P DAT will apply. In the Untied MixUp scheme, let Λ ∞ 1 be drawn i.i.d. from P Mix; in the DAT scheme, let Υ The equivalence between the two families of schemes also indicates that there are DAT schemes that do not correspond to a MixUp scheme. These DAT schemes correspond to Untied MixUp scheme beyond the standard MixUp. The relationship between MixUp, DAT and Untied MixUp is shown in Figure 1. We consider an image classification task on the Cifar10, Cifar100, MNIST and Fashion-MNIST datasets. The baseline classifier chosen is PreActResNet18 (see Two target-linear loss functions are essayed: cross-entropy (CE) loss and the negative-cosine (CE) loss as defined earlier. We implement CE loss similarly to previous works, which use CE loss to implement the baseline model. In our implementation of the NC loss model, for each label y, ϕ(y) is mapped to a randomly selected unit-length vector of dimension d and fixed during training; the feature map of the original PreActResNet18 is linearly transformed to a d-dimensional vector. The dimension d is chosen as 300 for Cifar10, MNIST and Fashion-Mnist (which have one black-andwhite channel) and 700 for Cifar100 (which has 3 colored channels). Our implementation of MixUp and Untied MixUp improves upon the published implementation from the original authors of MixUp Zhang et al. (2017b). For example, the original authors' implementation samples only one λ per mini-batch, giving rise to unnecessarily higher stochasticity of the gradient signal. Our implementation samples λ independently for each sample. Additionally, the original code combines inputs by mixing a mini-batch of samples with a shuffled version of itself. This approach introduces a dependency between sampled pairs and again increases the stochasticity of training. Our implementation creates two shuffled copies of the entire training dataset prior to each epoch, pairs them up, and then splits them into mini-batches. This gives a closer approximation to i.i.d. sampling and makes training smoother. While these implementation improvements have merit on their own, they do not provide a theoretical leap in understanding, and so we do not quantify their impact in our analysis. All models examined are trained using mini-batched backpropagation, for 200 epochs. We sweep over the policy space of MixUp and Untied MixUp. For MixUp, it is sufficient to consider distribution P Mix to be symmetric about 0.5. Thus we consider only consider P Mix in the form of B(α, α), and scan through a single parameter α systematically. Since the policy of Untied MixUp is in the form of U(B(α, β)), searching through (α, β) becomes more difficult. Thus our policy search for Untied MixUp is restricted to an ad hoc heuristic search. For this reason, the found best policy for Untied MixUp might be quite far from the true optimal. The main of our experiments are given in tables 1 to 4. As shown in the tables, each setting is run 100 times. For each run, we compute the error rate in a run as the average test error rate over the final 10 epochs. The estimated mean ("MEAN") performance of a setting is computed as the average of the error rates over all runs for the same setting. The 95%-confidence interval ("ConfInt") for the estimated mean performance is also computed and shown in the table. From these , we see that the Untied MixUp schemes each outperform their MixUp counterparts. Specifically, in 6 of the 8 cases (those printed in bold font), the confidence interval of Untied MixUp is completely disjoint from that of the corresponding MixUp scheme; and in some cases, the separation of confidence intervals is by a large margin. Note that the baseline model (PreActResNet18) has been designed with highly focused inductive bias for image classification tasks. Under such an inductive bias, one expects that the room for regularization (or the "amount of overfitting") isn't abundant. As such, we consider the improvement of Untied MixUp over MixUp rather significant. The show empirically that MixUp and Untied MixUp both work on the NC loss models. This validates our generalization of MixUp (and Untied MixUp) to models built with target linear losses. model policy runs MEAN ConfInt baseline-CE − 100 5.476% 0.027% mixUp-CE B(0.9, 0.9) 100 4.199% 0.023% uMixUp-CE U(B(2.2, 0.9)) 100 4.177% 0.025% baseline-NC − 100 5.605% 0.030% mixUp-NC B(1.0, 1.0) 100 4.508% 0.022% uMixUp-NC U(B(1.8, 1.0)) 100 4.455% 0.025% 1.3, 0.9) ) 100 23.819% 0.054% 1.7, 1.0) ) 100 0.609% 0.005% baseline-NC − 100 0.720% 0.007% mixUp-NC B(1.0, 1.0) 100 0.607% 0.004% uMixUp-NC U(B(1.3, 0.9)) 100 0.592% 0.005% ) be defined according to, with the first n elements of (x, x) ∞ 1 and the first n elements of Λ ∞ 1 as input. Define For any given λ E λ∼P Mix γ(λ) where (a) is due to a change of variable in the integration, (b) is due to the symmetry of (x, x) n 1. Note that in equation 5 g(λ) is undefined at values of λ for which the denominator is zero. But the lemma holds true because the denominator is only zero when p(λ) = 0, so those λ for which g(λ) is undefined never get drawn in the DAT scheme. A.3 PROOF OF LEMMA 5: DAT (x k, x k, λ) γ(λ)P Mix (λ) + γ(λ)P Mix (1 − λ) ). where (a) is due to the symmetry of (x, x) n 1, and (b) is by a change of variable in the second term (renaming 1 − λ as λ).

represent this text to find a much much longer article about it with say hundreds-thousands of words.

We present a novel interpretation of MixUp as belonging to a class highly analogous to adversarial training, and on this basis we introduce a simple generalization which outperforms MixUp

represent the tldr for finding the corresponding scientific article

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the scientific article to find a short TLDR

Question: Plan recognition aims to look for target plans to best explain the observed actions based on plan libraries and/or domain models. Despite the success of previous approaches on plan recognition, they mostly rely on correct action observations. Recent advances in visual activity recognition have the potential of enabling applications such as automated video surveillance. Effective approaches for such problems would require the ability to recognize the plans of agents from video information. Traditional plan recognition algorithms rely on access to detailed planning domain models. One recent promising direction involves learning approximate (or shallow) domain models directly from the observed activity sequences. Such plan recognition approaches expect observed action sequences as inputs. However, visual inference are often noisy and uncertain, typically represented as a distribution over possible actions. In this work, we develop a visual plan recognition framework that recognizes plans with an approximate domain model learned from uncertain visual data.

Represent the TLDR for finding the corresponding scientific article

Handling Uncertainty in Visual Perception for Plan Recognition

We investigate a framework for discovery: curating a large collection of predictions, which are used to construct the agent’s representation in partially observable domains.


Question: Represent the scientific article to find a short TLDR

Question: We consider the task of answering complex multi-hop questions using a corpus as a virtual knowledge base (KB). In particular, we describe a neural module, DrKIT, that traverses textual data like a virtual KB, softly following paths of relations between mentions of entities in the corpus. At each step the operation uses a combination of sparse-matrix TFIDF indices and maximum inner product search (MIPS) on a special index of contextual representations. This module is differentiable, so the full system can be trained completely end-to-end using gradient based methods, starting from natural language inputs. We also describe a pretraining scheme for the index mention encoder by generating hard negative examples using existing knowledge bases. We show that DrKIT improves accuracy by 9 points on 3-hop questions in the MetaQA dataset, cutting the gap between text-based and KB-based state-of-the-art by 70%. DrKIT is also very efficient, processing upto 10x more queries per second than existing state-of-the-art QA systems. Large knowledge bases (KBs), such as FreeBase and Wikidata, organize information around entities, which makes it easy to reason over their contents. For example, given a query like "When was Grateful Dead's lead singer born?", one can identify the entity Grateful Dead and the path of relations LeadSinger, BirthDate to efficiently extract the answer-provided that this information is present in the KB. Unfortunately, large KBs are often incomplete . While relation extraction methods can be used to populate KBs, this process is inherently error-prone, and errors in extraction can propagate to downstream tasks. Advances in open-domain QA suggest an alternativeinstead of performing relation extraction, one could treat a large corpus as a virtual KB by answering queries with spans from the corpus. This ensures facts are not lost in the relation extraction process, but also poses challenges. One challenge is that it is relatively expensive to answer questions using QA models which encode each document in a query-dependent fashion -even with modern hardware . The cost of QA is especially problematic for certain complex questions, such as the example question above. If the passages stating that "Jerry Garcia was the lead singer of Grateful Dead" and "Jerry Garcia was born in 1942" are far apart in the corpus, it is difficult for systems that retrieve and read a single passage to find an answer-even though in this example, it might be easy to answer the question after the relations were explicitly extracted into a KB. More generally, complex questions involving sets of entities or paths of relations may require aggregating information from entity mentions in multiple documents, which is expensive. One step towards efficient QA is the recent work of Seo et al. (2018; on phrase-indexed question answering (PIQA), in which spans in the text corpus are associated with question-independent contextual representations and then indexed for fast retrieval. Natural language questions are then answered by converting them into vectors that are used to perform inner product search (MIPS) against the index. This ensures efficiency during inference. However, this approach cannot be directly used to answer complex queries, since by construction, the information stored in the index is about the local context around a span-it can only be used for questions where the answer can be derived by reading a single passage. This paper addresses this limitation of phrase-indexed question answering. We introduce an efficient, end-to-end differentiable framework for doing complex QA over a large text corpus that has been encoded in a query-independent manner. Specifically, we consider "multi-hop" complex queries which can be answered by repeatedly executing a "soft" version of the operation below, defined over a set of entities X and a relation R: Y = X.follow(R) = {x : ∃x ∈ X s.t. R(x, x) holds} In past work soft, differentiable versions of this operation were used to answer multi-hop questions against an explicit KB . Here we propose a more powerful neural module which approximates this operation against an indexed corpus. In our module, the input X is a sparse vector representing a weighted set of entities, and the relation R is a dense feature vector, e.g. a vector derived from a neural network over a natural language query. The output Y is another sparse vector representing the weighted set of entities, aggregated over entity mentions in the top-k spans retrieved from the index. The spans in turn are retrieved using a MIPS query constructed from X and R, and we discuss pretraining schemes for the index in §2.3. For multi-hop queries, the output entities Y can be recursively passed as input to the next iteration of the same module. The weights of the entities in Y are differentiable w.r.t the MIPS queries, which allows end-to-end learning without any intermediate supervision. We discuss an implementation based on sparse matrix-vector products, whose runtime and memory depend only on the number of spans K retrieved from the index. This is crucial for scaling up to large corpora, and leads to upto 15x faster inference than existing state-of-the-art multi-hop and open-domain QA systems. The system we introduce is called DrKIT (for Differentiable Reasoning over a Knowledge base of Indexed Text). We test DrKIT on the MetaQA benchmark for complex question answering, and show that it improves on prior text-based systems by 5 points on 2-hop and 9 points on 3-hop questions, reducing the gap between text-based ad KB-based systems by 30% and 70%, respectively. We also test DrKIT on a new dataset of multi-hop slot-filling over Wikipedia articles, and show that it outperforms DrQA and PIQA adapted to this task. We want to answer a question q using a text corpus as if it were a KB. We start with the set of entities z in the question q and would ideally want to follow relevant outgoing relation edges in the KB to arrive at the answer. To simulate this behaviour on text, we first expand z to set of co-occurring mentions (say using TF-IDF) m. Not all of these co-occurring mentions are relevant for the question q, so we train a neural network which filters the mentions based on a relevance score of q to m. Then we can aggregate the ing set of mentions m to the entities they refer to end up with an ordered set z of entities which are answer candidates, very similar to traversing the KB. Furthermore, if the question requires more than one hop to answer, we can repeat the above procedure starting with z. This is depicted pictorially in Figure 1. We begin by first formalizing this idea in §2.1 in a probabilistic framework. In §2.2, we describe how the expansion of entities to mentions and the filtering of mentions can be performed efficiently, using sparse matrix products, and MIPS algorithms . Lastly we discuss a pretraining scheme for constructing the mention representations in §2.3. We denote the given corpus as is a sequence of tokens. We start by running an entity linker over the corpus to identify mentions of a fixed set of entities E. Each mention m is a tuple (e m, k m, i m, j m) denoting that the text span d im km,..., d jm km in document k m mentions the entity e m ∈ E, and the collection of all mentions in the corpus is denoted as M. Note that typically |M| |E|. We assume a weakly supervised setting where during training we only know the final answer entities a ∈ E for a T -hop question. We denote the latent sequence of entities which answer each of the intermediate hops as z 0, z 1,..., z T ∈ E, where z 0 is mentioned in the question, and z t = a. We can recursively write the probability of an intermediate answer as: Here Pr(z 0 |q) is the output of an entity linking system over the question, and Pr(z t |q, z t−1) corresponds to a single-hop model which answers the t-th hop, given the entity from the previous hop z t−1, by following the appropriate relation. Eq. 1 models reasoning over a chain of latent entities, but when answering questions over a text corpus, we must reason over entity mentions, rather than entities themselves. Hence Pr(z t |q, z t−1) needs to be aggregated over all mentions of z t, which yields The interesting term to model in the above equation is P r(m|q, z t−1), which represents the relevance of mention m given the question about entity z t−1. Following the analogy of a KB, we first expand the entity z t−1 to co-occuring mentions m and use a learnt scoring function to find the relevance of these mentions. Formally, let F (m) denote a TF-IDF vector for the document containing m, G(z t−1) be the TF-IDF vector of the surface form of the entity from the previous hop, and s t (m, z, q) be a learnt scoring function (different for each hop). Thus, we model Pr(m|q, z t−1) as Another equivalent way to look at our model in Eq. 3 is that the second term retrieves mentions of the correct type requested by the question in the t-th hop, and the first term filters these based on co-occurrence with z t−1. When dealing with a large set of mentions m, we will typically retain only the top-k relevant mentions. We will show that this joint modelling of co-occurrence and relevance is important for good performance, which has also been observed in past . The other term left in Eq. 2 is Pr(z|m), which is 1 if mention m matches the entity z else 0, since each mention can only point to a single entity. In general, to compute Eq. 2 the mention scoring of Eq. 3 needs to be evaluated for all latent entity and mention pairs, which is prohibitively expensive. However, by restricting s t to be an inner product we can implement this efficiently (§2.2). To highlight the differentiability of the proposed overall scheme, we can represent the computation in Eq. 2 as matrix operations. We pre-compute the TFIDF term for all entities and mentions into a sparse matrix, which we denote as A E→M [e, m] = 1 (G(e) · F (m) > ). Then entity expansion to co-occuring mentions can be considered to be a sparse-matrix by sparse-vector multiplication between A E→M and z t−1. For the relevance scores, let T K (s t (m, z t−1, q)) denote the top-K relevant mentions encoded as a sparse vector in R |M|. Finally, the aggregation of mentions to entities can be formulated as multiplication with another sparse matrix A M →E, which encodes coreference, i.e. mentions corresponding to the same entity. Putting all these together, using to denote elementwise product, and defining Z t = [Pr(z t = e 1 |q);...; Pr(z t = e |E| |q)], we can observe that for large K (i.e., as K → |M|), eq. becomes equivalent to: Note that every operation in above equation is differentiable and between sparse matrices and vectors: we will discuss efficient implementations in §2.2. Further, the number of non-zero entries in Z t is bounded by K, since we filtered (the multiplication in Eq. 4) to top-k relevant mentions among TF-IDF based expansion and since each mention can only point to a single entity in A M →E. This is important, as it prevents the number of entries in Z t from exploding across hops (which might happen if, for instance, we added the dense and TF-IDF retrievals instead). We can view Z t−1, Z t as weighted multisets of entities, and s t (m, z, q) as implicitly selecting mentions which correspond to a relation R. Then Eq. 4 becomes a differentiable implementation of Z t = Z t−1.follow(R), i.e. mimicking the graph traversal in a traditional KB. We thus call Eq. 4 a textual follow operation. Training and Inference. The model is trained completely end-to-end by optimizing the crossentropy loss between Z T, the weighted set of entities after T hops, and the ground truth answer set A. We use a temperature coefficient λ when computing the softmax in Eq, 4 since the inner product scores of the top-K retrieved mentions are typically high values, which would otherwise in very peaked distributions of Z t. Finally, we also found that taking a maximum over the mention set of an entity M zt in Eq. 2 works better in practice than taking a sum. This corresponds to optimizing only over the most confident mention of each entity, which works for corpora like Wikipedia which do not have much redundancy of information. A similar observation has been made by in weakly supervised settings. Sparse TF-IDF Mention Encoding. To compute the sparse A E→M for entity-mention expansion in Eq. 4, the TF-IDF vectors F (m) and G(z t−1) are constructed over unigrams and bigrams, hashed to a vocabulary of 16M buckets. While F computes the vector from the whole passage around m, G only uses the surface form of z t−1. This corresponds to retrieving all mentions in a document retrieved using z t−1 as the query. We limit the number of retrieved mentions per entity to a maximum of µ, which leads to a |E| × |M| sparse matrix. Efficient Entity-Mention expansion. The expansion from a set of entities to mentions occurring around them can be computed using the sparse-matrix by sparse vector product Z T t−1 A E→M. A simple lower bound for multiplying a sparse |E|×|M| matrix, with maximum µ non-zeros in each row, by a sparse |E| × 1 vector with k nonzeros is Ω(kµ). Note that this lower bound is independent of the size of matrix A E→M or in other words independent of number of entities or mentions. To attain the lower bound, the multiplication algorithm must be vector driven, because any matrix-driven algorithms need to at least iterate over all the rows. Instead we slice out the relevant rows from A E→M. To enable this our solution is to represent the sparse matrix A E→M as two row-wise lists of a variable-sized lists of the non-zero elements index and values. This in a ragged representation of the matrix which can be easily sliced corresponding to the non-zero entries in the vector in O(log E) time. We are now left with k sparse vectors with at most µ non-zero elements in each. We can add these k sparse vectors weighted by corresponding values from vector z in O(k max{k, µ}) time. Moreover, such an implementation is feasible with deep learning frameworks such as TensorFlow (tf.). We quickly test the scalability of our approach by varying the number of entities for a fixed density of mentions µ (from Wikipedia). Figure 2 compared our approach to the default sparse-matrix times dense vector product (no sparse matrix times sparse vector is available in TensorFlow) 1. Efficient top-k mention relevance filtering: To make computation of Eq. 4 feasible, we need an efficient way to get top-k relevant mentions related to an entity in z t−1 for a given question q, without enumerating all possibilities. A key insight is that by restricting the scoring function s t (m, z t−1, q) to an inner product, we can easily approximate a parallel version of this computation, across all mentions m. To do this, let f (m) be a dense encoding of m, and g t (q, z t−1) be a dense encoding of the question q for the t-th hop, both in R p (the details of the dense encoding is provided in next paragraph), then the scoring function s t (m, z t−1, q) becomes which can be computed in parallel by multiplying a matrix f (M) = [f (m 1); f (m 2);...] with g t (q, z t−1). Although this matrix will be very large for a realistic corpus, but since eventually we are only interested in top-k values we can use an approximate algorithm for Maximum Inner Product Search (MIPS) to find the k top-scoring elements. The complexity of this filtering step using MIPS is roughly O(kp polylog|M|). Mention and Question Encoders. Mentions are encoded by passing the passages they are contained in through a BERT-large model (trained as described in § 2.3). Suppose mention m appears in passage d, starting at position i and ending at position j. Then, where H d is the sequence of embeddings output from BERT, and W is a linear projection to size p. The query are encoded with a smaller BERT-like model: specifically, it is tokenized with WordPieces , appended to a special [CLS] token, and then passed through a 4-layer Transformer network with the same architecture as BERT, producing an output sequence H q. The g t functions are defined similarly to the BERT model used for SQuAD-style QA. For each hop t = 1,..., T, we add two addition Transformer layers on top of H q, which will be trained to produce MIPS queries from the [CLS] encoding; the first added layer produces a MIPS query H q st to retrieve a start token, and the second added layer a MIPS query H q en to retrieve an end token. We concatenate the two and definẽ. Finally, to condition on current progress we add the embeddings of z t−1. Specifically, we use entity embeddings E ∈ R |E|×p, to construct an average embedding of the set To avoid a large number of parameters in the model, we compute the entity embeddings as an average over the word embeddings of the tokens in the entity's surface form. The computational cost of the question encoder Thus our total computational complexity to answer a query isÕ(k max{k, µ} + kp + p 2 ) (almost independent to number of entities or mentions!), with O(µ|E| + p|M|) memory to store the precomputed matrices and mention index. Ideally, we would like to train the mention encoder f (m) end-to-end using labeled QA data only. However, this poses a challenge when combined with approximate nearest neighbor search-since after every update to the parameters of f, one would need to recompute the embeddings of all mentions in M. We thus adopt a staged training approach: we first pre-train a mention encoder f (m), then compute compute and index embeddings for all mentions once, keeping these embeddings fixed when training the downstream QA task. Empirically, we observed that using BERT representations "out of the box" do not capture the kind of information our task requires (Appendix §C), and thus, pretraining the encoder to capture better mention understanding is a crucial step. One option adopted by previous researchers is to fine-tune BERT on Squad . However, Squad is limited to only 536 articles from Wikipedia, leading to a very specific distribution of questions, and is not focused on entity-and relation-centric questions. Here we instead train the mention encoder using distant supervision from the KB. Specifically, assume we are given an open-domain KB consisting of facts (e 1, R, e 2) specifying that the relation R holds between the subject e 1 and the object e 2. Then for a corpus of entity-linked text passages {d k}, we automatically identify tuples (d, (e 1, R, e 2)) such that d mentions both e 1 and e 2. Using this data, we learn to answer slot-filling queries in a reading comprehension setup, where the query q is constructed from the surface form of the subject entity e 1 and a natural language description of R (e.g. "Jerry Garcia, birth place, ?"), and the answer e 2 needs to be extracted from the passage d. Using string representations in q ensures our pre-training setup is similar to the downstream task. In pretraining, we use the same scoring function as in previous section, but over all spans m in the passage: For effective transfer to the full corpus setting, we must also provide negative instances during pretraining, i.e. query and passage pairs where the answer is not contained in the passage. We consider three types of hard negatives: Shared-entity negatives, which pair a query (e 1, R, ?) with a passage which mentions e 1 but not the correct tail answer. Shared-relation negative, which pair a query (e 1, R, ?) with a passage mentioning two other entities e 1 and e 2 in the same relation R. Random negatives, which pair queries with random passages from the corpus. For the multi-hop slot-filling experiments below, we used Wikidata (Vrandečić & Krötzsch, 2014) as our KB, Wikipedia as the corpus, and SLING to identify entity mentions. We restrict d be from the Wikipedia article of the subject entity to reduce noise. Overall we collected 950K pairs over 550K articles. For the experiments with MetaQA, we supplemented this data with the corpus and KB provided with MetaQA, and string matching for entity linking. 3.1 METAQA: MULTI-HOP QUESTION ANSWERING WITH TEXT Dataset. We first evaluate DrKIT on the MetaQA benchmark for multi-hop question answering . METAQA consists of around 400K questions ranging from 1 to 3 hops constructed by sampling relation paths from a movies KB and converting them to natural language using templates. The questions cover 8 relations and their inverses, around 43K entities, and are paired with a corpus consisting of 18K Wikipedia passages about those entities. The questions are all designed to be answerable using either the KB or the corpus, which makes it possible to compare the performance of our "virtual KB" QA system to a plausible upper bound system that has access to a complete KB. We used the same version of the data as. Results. Table 1 shows the accuracy of the top-most retrieved entity (Hits@1) for the sub-tasks ranging from 1-3 hops, and compares to the state-of-the-art systems for the text-only setting on these tasks. DrKIT outperforms the prior state-of-the-art by a large margin in the 2-hop and 3-hop cases. The strongest prior method, PullNet (; 2018), uses a graph neural network model with learned iterative retrieval from the corpus to answer multi-hop questions. It uses the MetaQA KB during training to identify shortest paths between the question entity and answer entity, which are used to supervise the text retrieval and reading modules. DrKIT, on the other hand, has strong performance without such supervision, demonstrating its capability for end-to-end learning. (Adding the same intermediate supervision to DrKIT does not even consistently improve performance-it gives DrKIT a small lift on 1-and 2-hop questions but does not help for 3-hop questions.) DrKIT's architecture is driven, in part, by efficiency considerations: unlike PullNet, it is designed to answer questions with minimal processing at query time. Figure 3 compares the tradeoffs between accuracy and inference time of DrKIT with PullNet as we vary K, the number of dense nearest neighbors retrieved. The runtime gains of DrKIT over PullNet range between 5x-15x. Analysis. We perform ablations on DrKIT for the MetaQA data. First, we empirically confirm that taking a sum instead of max over the mentions of an entity hurts performance. So does removing the softmax temperature (by setting λ = 1). Removing the TFIDF component from Eq. 3, leads a large decrease in performance for 2-hop and 3-hop questions. This is because the TFIDF component constrains the end-to-end learning to be along reasonable paths of co-occurring mentions; otherwise the search space becomes too large. The also highlight the importance of pretraining method introduced in §2.3, as DrKIT over an index of BERT representations without pretraining is 23 points worse in the 3-hop case. We also check the performance when the KB used for pre-training is incomplete. Even with only 25% edges retained, we see a high performance, better than PullNet, and far better than state-of-the-art KB-only methods. We analyzed 100 2-hop questions correctly answered by DrKIT and found that for 83 the intermediate answers were also correct. The other 17 cases were all where the second hop asked about genre, e.g. "What are the genres of the films directed by Justin Simien?". We found that in these cases the intermediate answer was the same as the correct final answer-essentially the model learned to answer the question in 1 hop and copy it over for the second hop. Among incorrectly answered questions, the intermediate accuracy was only 47%, so the mistakes were evenly distributed across the two hops. The MetaQA dataset has been fairly well-studied, but has limitations since it is constructed over a small KB. In this section we consider a new task, in a larger scale setting with many more relations, entities and text passages. The new dataset also lets us evaluate performance in a setting where the test set contains documents and entities not seen at training time, an important issue when devising a QA system that will be used in a real-world setting, where the corpus and entities in the discourse change over time, and lets us perform analyses not possible with MetaQA, such as extrapolating from single-hop to multi-hop settings without retraining. Dataset. We sample two subsets of Wikipedia articles, one for pre-training (§2.3) and end-to-end training, and one for testing. For each subset we consider the set of WikiData entities mentioned in the articles, and sample paths of 1-3 hop relations among them, ensuring that any intermediate entino more than 100. Then we construct a semi-structured query by concatenating the surface forms of the head entity with the path of relations (e.g. "Helene Gayle, employer, founded by, ?"). The answer is the tail entity at the end of the path, and the task is to extract it from the Wikipedia articles. Existing slot-filling tasks focus on a single-hop, static corpus setting, whereas our task considers a dynamic setting which requires to travers the corpus. For each setting, we create a dataset with 10K articles, 120K passages, > 200K entities and 1.5M mentions, ing in an index of size about 2gb. We include example queries in Appendix. Baselines. We adapt two publicly available open-domain QA systems for this task -DrQA 3 and PIQA 4 . While DrQA is relatively mature and widely used, PIQA is recent, and similar to our setup since it also answers questions with minimal computation at query time. It is broadly similar to a single textual follow operation in DrKIT, but is not constructed to allow retrieved answers to be converted to entities and then used in subsequent processing, so it is not directly applicable to multi-hop queries. We thus also consider a cascaded architecture which repeatedly applies Eq. 2, using either of PIQA or DrQA to compute Pr(z t |q, z t−1) against the corpus, retaining at most k intermediate answers in each step. We tune k in the range of 1-10, since larger values make the runtime infeasible. Further, since these models were trained on natural language questions, we use the templates released by to convert intermediate questions into natural text. 5 We test off-the-shelf versions of these systems, as well as a version of PIQA re-trained on our our slot-filling data. 6 We compare to a version of DrKIT trained only on single-hop queries (§2.3) and similarly cascaded, and one version trained end-to-end on the multi-hop queries. Results. Table 1 (right) lists the Hits @1 performance on this task. Off-the-shelf open-domain QA systems perform poorly, showing the challenging nature of the task. Re-training PIQA on the slotfilling data improves performance considerably, but DrKIT trained on the same data improves on it. A large improvement is seen on top of these cascaded architectures by end-to-end training, which is made possible by the differentiable operation introduced in this paper. We also list performance of DrKIT when trained against an index of fixed BERT-large mention representations. While this is comparable to the re-trained version of PIQA, it lags behind DrKIT pre-trained using the KB, once again highlighting the importance of the scheme outlined in §2.3. We also plot the Hits @1 against Queries/sec for cascaded versions of PIQA and DrKIT in Figure 3 (middle). We observe gains of 2x-3x to DrKIT, due to the efficient implementation of entity-mention expansion discussed in §2.2. Analysis. In order to understand where the accuracy gains for DrKIT come from, we conduct experiments on the dataset of slot-filling queries released by. We construct an open version of the task by collecting Wikipedia articles of all subject entities in the data. A detailed discussion is in Appendix C, here we note the main findings. PIQA trained on Squad only gets 30% macro-avg accuracy on this data, but this improves to 46% when re-trained on our slot-filling data. Interestingly, a version of DrKIT which selects from all spans in the corpus performs similarly to PIQA (50%), but when using entity linking it significantly improves to 66%. It also has 55% accuracy in answering queries about rare relations, i.e. those observed < 5 times in its training data. We also conduct probing experiments comparing the representations learned using slot-filling to those by vanilla BERT. We found that while the two are comparable in detecting fine-grained entity types, the slot-filling version is significantly better at encoding entity co-occurrence information. Table 2: (Left) Retrieval performance on the HotpotQA benchmark dev set. Q/s denotes the number of queries per second during inference on a single 16-core CPU. Accuracy @k is the fraction where both the correct passages are retrieved in the top k. †: Baselines obtained from Das et al. (2019b). For DrKIT, we report the performance when the index is pretrained using the WikiData KB alone, the HotpotQA training questions alone, or using both. *: Measured on different machines with similar specs. (Right) Overall performance on the HotpotQA task, when passing 10 retrieved passages to a downstream reading comprehension model . Dataset. HotpotQA is a recent dataset of over 100K crowd-sourced multi-hop questions and answers over introductory Wikipedia passages. We focus on the open-domain fullwiki setting where the two gold passages required to answer the question are not known in advance. The answers are free-form spans of text in the passages, not necessarily entities, and hence our model which selects entities is not directly applicable here. Instead, inspired by recent works (b;), we look at the challenging sub-task of retrieving the passages required to answer the questions from a pool of 5.2M. This is a multi-hop IR task, since for many questions at least one passage may be 1-2 hops away from the entities in the question. Further, each passage is about an entity (the title entity of that Wikipedia page), and hence retrieving passages is the same as identifying the title entities of those passages. We apply DrKIT to this task of identifying the two entities for each question, whose passages contain the information needed to answer that question. Then we pass the top 10 passages identified this way to a standard reading comprehension architecture from to select the answer span. Setup. We use the Wikipedia abstracts released by as the text corpus. 7 The total number of entities is the same as the number of abstracts, 5.23M, and we consider hyperlinks in the text as mentions of the entities to whose pages they point to, leading to 22.8M total mentions in an index of size 34GB. For pretraining the mention representations, we compare using the WikiData KB as described in §2.3, to directly using the HotpotQA training questions, with TF-IDF based retrieved passages as negative examples. We set A E→M [e, m] = 1 if either the entity e is mentioned on the page of the entity denoted by m, or vice versa. For entity linking over the questions, we retrieve the top 20 entities based on the match between a bigram based TF-IDF vector of the question with the vector of the surface form of the entity (same as the title of the Wiki article). We found that the gold entities that need to be retrieved are within 2 hops of the entities linked in this manner for 87% dev examples. Unlike the MetaQA and WikiData datasets, however, for HotpotQA we do not know the number of hops required for each question in advance. Instead, we run DrKIT for 2 hops for each question, and then take a weighted average of the distribution over entities after each hop Z * = π 0 Z 0 + π 1 Z 1 + π 2 Z 2. Z 0 consists of the entities linked to the question itself, rescored based on an encoding of the question, since in some cases one or both the entities to be retrieved are in this set. 8 Z 1 and Z 2 are given by Eq. 4. The mixing weights π i are the softmax outputs of a classifier on top of another encoding of the question, learnt end-to-end on the retrieval task. This process can be viewed as soft mixing of different templates ranging from 0 to 2 hops for answering a question, similar to NQL. Results. We compare our retrieval to those presented in (b) in Table 2 (Left). We measure accuracy @k retrievals, since we care about retrieving both the passages required to answer the question to pass to the downstream reading comprehension model. We see an improvement in accuracy across the board, with much higher gains @2 and @5. The main baseline is the entitycentric IR approach which runs a BERT-based re-ranker on 200 pairs of passages for each question. Importantly, DrKIT improves by over 10x in terms of queries per second during inference over it. Note that the inference time is measured using a batch size of 1 for both models for fair comparison. DrKIT can be easily run with batch sizes upto 40, but the entity centric IR baseline cannot due to the large number of runs of BERT for each query. When comparing different datasets for pretraining the index, there is not much difference between using the Wikidata KB, or the HotpotQA questions. The latter has a better accuracy @2, but overall the best performance is using a combination of both. Lastly, we check the performance of the baseline reading comprehension model from , when given the passages retrieved by DrKIT. 9 While there is a significant improvement over the baseline which uses a TF-IDF based retrieval, we see only a small improvement over the passages retrieved by the entity-centric IR baseline, despite the significantly improved accuracy @10 of DrKIT. Among the 33% questions where the top 10 passages do not contain both the correct passages, for around 20% the passage containing the answer is also missing. We conjecture this percentage is lower for the entity-centric IR baseline, and the downstream model is able to answer some of these questions without the other supporting passage. Neural Query Language (NQL) defines differentiable templates for multi-step access to a symbolic KB, in which relations between entities are explicitly enumerated. Here, we focus on the case where the relations are implicit in mention representations derived from text. Knowledge Graph embeddings (; ;) attach continuous representations to discrete symbols which allow them to be incorporated in deep networks . Embeddings often allow generalization to unseen facts using relation patterns, but text corpora are more complete in the information they contain. also examined answering compositional questions by treating a text corpus (in their case the entire web) as a KB. However their approach consists of parsing the query into a computation tree separately, and running a black-box QA model on its leaves separately, which cannot be trained end-to-end. Recent papers have also looked at complex QA using graph neural networks (; ;) or by identifying paths of entities in text (; ; . These approaches rely on identifying a small relevant pool of evidence documents containing the information required for multi-step QA. and , incorporate a dynamic retrieval process to add text about entities identified as relevant in the previous layer of the model. Since the evidence text is processed in a query-dependent manner, the inference speed is slower than when it is pre-processed into an indexed representation (see Figure 3). The same limitation is shared by methods which perform multi-step retrieval interleaved with a reading comprehension model (a; ; . We present DrKIT, a differentiable module that is capable of answering multi-hop questions directly using a large entity-linked text corpus. DrKIT is designed to imitate traversal in KB over the text corpus, providing ability to follow relations in the "virtual" KB over text. We achieve state-of-theart on MetaQA dataset for answering natural language questions, with a 9 point increase in the 3-hop case. We also developed an efficient implementation using sparse operations and inner product search, which led to a 10x increase in QPS over baseline approaches. We use p = 400 dimensional embeddings for the mentions and queries, and 200-dimensional embeddings each for the start and end positions. This in an index of size 750MB. When computing A E→M, the entity to mention co-occurrence matrix, we only retain mentions in the top 50 paragraphs matched with an entity, to ensure sparsity. Further we initialize the first 4 layers of the question encoder with the Transformer network from pre-training. For the first hop, we assign Z 0 as a 1-hot vector for the least frequent entity detected in the question using an exact match. The number of nearest neighbors K and the softmax temperature λ were tuned on the dev set of each task, and we found K = 10000 and λ = 4 to work best. We pretrain the index on a combination of the MetaQA corpus, using the KB provided with MetaQA for distance data, and the Wikidata corpus. Table 3 . Single-hop questions and relation extraction. released a dataset of 1M slotfilling queries of the form (e 1, R, ?) paired with Wikipedia sentences mentioning e 1, which was used for training systems that answered single-step slot-filling questions based on a small set of candidate passages. Here we consider an open version of the same task, where answers to the queries must be extracted from a corpus rather than provided candidates. We construct the corpus by collecting and entity-linking all paragraphs in the Wikipedia articles of all 8K subject entities in the dev and test sets, leading to a total of 109K passages. After constructing the TFIDF A E→M and coreference A M →E matrices for this corpus, we directly use our pre-trained index to answer the test set queries. indexing entity-mentions in single-hop questions over. Note that DrKit-entities has a high Hits@1 performance on the Rare relations subset, showing that there is generalization to less frequent data due to the natural language representations of entities and relations. Probing Experiments Finally, to compare the representations learned by the BERT model finetuned on the Wikidata slot-filling task, we design two probing experiments. In each experiment, we keep the parameters of the BERT model (mention encoders) being probed fixed and only train the query encoders. Similar to , we use a weighted average of the layers of BERT here rather than only the top-most layer, where the weights are learned on the probing task. In the first experiment, we train and test on shared-entity negatives. Good performance here means the BERT model being probed encodes fine-grained entity-type information reliably 10. As shown in Table 4, BERT performs well on this task, suggesting it encodes fine-grained types well. In the second experiment, we train and test only on shared-relation negatives. Good performance here means that the BERT model encodes entity co-occurrence information reliably. In this probe task, we see a large performance drop for Bert, suggesting it does not encode entity co-occurrence information well. The good performance of the DrKIT model on both experiments suggests that fine-tuning on the slot-filling task primarily helps the contextual representations to also encode entity co-occurrence information, in addition to entity type information.

Represent the short summary for finding the corresponding scientific article (long)

Differentiable multi-hop access to a textual knowledge base of indexed contextual representations

Language generation using seq2seq models which produce word embeddings instead of a softmax based distribution over the vocabulary at each step enabling much faster training while maintaining generation quality