Question: Represent the scientific article to find a short TLDR

Question: We study the problem of learning similarity functions over very large corpora using neural network embedding models. These models are typically trained using SGD with random sampling of unobserved pairs, with a sample size that grows quadratically with the corpus size, making it expensive to scale. We propose new efficient methods to train these models without having to sample unobserved pairs. Inspired by matrix factorization, our approach relies on adding a global quadratic penalty and expressing this term as the inner-product of two generalized Gramians. We show that the gradient of this term can be efficiently computed by maintaining estimates of the Gramians, and develop variance reduction schemes to improve the quality of the estimates. We conduct large-scale experiments that show a significant improvement both in training time and generalization performance compared to sampling methods. We consider the problem of learning a similarity function h: X × Y → R, that maps each pair of items, represented by their feature vectors (x, y) ∈ X × Y, to a real number h(x, y), representing their similarity. We will refer to x and y as the left and right feature vectors, respectively. Many problems can be cast in this form: In natural language processing, x represents a context (e.g. a bag of words), y represents a candidate word, and the target similarity measures the likelihood to observe y in context x BID14 BID16 BID13. In recommender systems, x represents a user query, y represents a candidate item, and the target similarity is a measure of relevance of item y to query x, e.g. a movie rating BID0, or the likelihood to watch a given movie BID12 ). Other applications include image similarity, where x and y are pixel-representations of images BID5 BID6 ), and network embedding models BID10 ), where x and y are nodes in a graph and the similarity is whether an edge connects them. A popular approach to learning similarity functions is to train an embedding representation of each item, such that items with high similarity are mapped to vectors that are close in the embedding space. A common property of such problems is that only a small subset of all possible pairs X × Y is present in the training set, and those examples typically have high similarity. Training exclusively on observed examples has been demonstrated to yield poor generalization performance. Intuitively, when trained only on observed pairs, the model places the embedding of a given item close to similar items, but does not learn to place it far from dissimilar ones . Taking into account unobserved pairs is known to improve the embedding quality in many applications, including recommendation BID12 BID1 and word analogy tasks . This is often achieved by adding a low-similarity prior on all pairs, which acts as a repulsive force between all embeddings. But because it involves a number of terms quadratic in the corpus size, this term is computationally intractable (except in the linear case), and it is typically optimized using sampling: for each observed pair in the training set, a set of random unobserved pairs is sampled and used to compute an estimate of the repulsive term. But as the corpus size increases, the quality of the estimates deteriorates unless the sample size is increased, which limits scalability. In this paper, we address this issue by developing new methods to efficiently estimate the repulsive term, without sampling unobserved pairs. Our approach is inspired by matrix factorization models, which correspond to the special case of linear embedding functions. They are typically trained using alternating least squares BID12, or coordinate descent methods BID2, which circumvent the computational burden of the repulsive term by writing it as a matrix-inner-product of two Gramians, and computing the left Gramian before optimizing over the right embeddings, and viceversa. Unfortunately, in non-linear embedding models, each update of the model parameters induces a simulateneous change in all embeddings, making it impractical to recompute the Gramians at each iteration. As a , the Gramian formulation has been largely ignored in the non-linear setting, where models are instead trained using stochastic gradient methods with sampling of unobserved pairs, see BID7. were, to our knowledge, the first to attempt leveraging the Gramian formulation in the non-linear case. They consider a model where only one of the embedding functions is non-linear, and show that the gradient can be computed efficiently in that case. Their is remarkable in that it allows exact gradient computation, but this unfortunately does not generalize to the case where both embedding functions are non-linear. Contributions We propose new methods that leverage the Gramian formulation in the non-linear case, and that, unlike previous approaches, are efficient even when both left and right embeddings are non-linear. Our methods operate by maintaining stochastic estimates of the Gram matrices, and using different variance reduction schemes to improve the quality of the estimates. We perform several experiments that show these methods scale far better than traditional sampling approaches on very large corpora. We start by reviewing preliminaries in Section 2, then derive the Gramian-based methods and analyze them in Section 3. We conduct large-scale experiments on the Wikipedia dataset in Section 4, and provide additional experiments in the appendix. All the proofs are deferred to Appendix A. We consider models that consist of two embedding functions u: R d ×X → R k and v: R d ×Y → R k, which map a parameter vector 1 θ ∈ R d and feature vectors x, y to embeddings u(θ, x), v(θ, y) ∈ R k. The output of the model is the dot product 2 of the embeddings h θ (x, y) = u(θ, x), v(θ, y), where ·, · denotes the usual inner-product on R k. Low-rank matrix factorization is a special case, in which the left and right embedding functions are linear in x and y. Figure 1 illustrates a non-linear model, in which each embedding function is given by a feed-forward neural network. 3 We denote the training set by T = {(x i, y i, s i) ∈ X × Y × R} i∈{1,...,n}, where x i, y i are the feature vectors and s i is the target similarity for example i. To make notation more compact, we will use u i (θ), v i (θ) as a shorthand for u(θ, x i), v(θ, y i), respectively. As discussed in the introduction, we also assume that we are given a low-similarity prior p ij ∈ R for all pairs (i, j) ∈ {1, . . ., n} 2. Given a differentiable scalar loss function: R × R → R, the objective function is given by DISPLAYFORM0 where the first term measures the loss on observed data, the second term penalizes deviations from the prior, and λ is a positive hyper-parameter that trades-off the two terms. To simplify the discussion, we will assume a uniform zero prior p ij as in BID12, the general case is treated in Appendix B.To optimize this objective, existing methods rely on sampling to approximate the second term, and are usually referred to as negative sampling or candidate sampling, see BID7 BID1 for a survey. Due to the double sum in, the quality of the sampling estimates degrades as the corpus size increases, which can significantly increase training times. This can be alleviated by increasing the sample size, but does not scale to very large corpora. DISPLAYFORM1 Figure 1: A dot-product embedding model for a similarity function on X × Y. A different approach to solving, widely popular in matrix factorization, is to rewrite the double sum as the inner product of two Gram matrices. Let us denote by U θ ∈ R n×k the matrix of all left embeddings such that u i (θ) is the i-th row of U θ, and similarly for V θ ∈ R n×k. Then denoting the matrix inner-product by A, B = i,j A ij B ij, we can rewrite the double sum in as: DISPLAYFORM0 Now, using the adjoint property of the inner product, we have DISPLAYFORM1 and if we denote by u ⊗ u the outer product of a vector u by itself, and define the Gram matrices DISPLAYFORM2 the penalty term becomes DISPLAYFORM3 The Gramians are k × k PSD matrices, where k, the dimension of the embedding space, is much smaller than n -typically k is smaller than 1000, while n can be arbitrarily large. Thus, the Gramian formulation has a much lower computational complexity than the double sum formulation, and this reformulation is at the core of alternating least squares and coordinate descent methods BID12 BID2, which operate by computing the exact Gramian for one side, and solving for the embeddings on the other. However, these methods do not apply in the non-linear setting due to the implicit dependence on θ, as a change in the model parameters simultaneously changes all embeddings on both sides, making it intractable to recompute the Gramians at each iteration, so the Gramian formulation has not been used when training non-linear models. In the next section, we show that it can in fact be leveraged in the non-linear case. In order to leverage the Gramian formulation in the non-linear case, we start by rewriting the objective function in terms of the Gramians defined in. Let DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2. Intuitively, for each example i, −∇f i (θ) pulls the embeddings u i and v i close to each other (assuming a high similarity s i), while −∇g i (θ) creates a repulsive force between u i and all embeddings {v j} j∈{1,...,n}, and between v i and all {u j} j∈{1,...,n}, see Appendix D for further discussion, and illustration of the effect of this term. While the Gramians are expensive to recompute at each iteration, we can maintain PSD estimateŝ DISPLAYFORM3. Then the gradient of g(θ) can be approximated by the gradient (w.r.t. θ) of DISPLAYFORM4 as stated in the following proposition. DISPLAYFORM5 is an unbiased estimate of ∇g(θ).In a mini-batch setting, one can further averageĝ i over a batch of examples i ∈ B (which we do in our experiments), but we will omit batches to keep the notation concise. Next, we propose several methods for computing Gramian estimatesĜ u,Ĝ v, and discuss their tradeoffs. Since each Gramian can be written as a sum of rank-one terms, e.g. DISPLAYFORM6, a simple unbiased estimate can be obtained by sampling one term (or a batch) from this sum. We improve on this by using different variance reduction methods, which we discuss in the next two sections. Our first method is inspired by the stochastic average gradient (SAG) method (; BID9), which reduces the variance of the gradient estimates by maintaining a cache of individual gradients, and estimating the full gradient using this cache. Since each Gramian is a sum of outer-products (see equation FORMULA4), we can apply the same technique to estimate Gramians. For all i ∈ {1, . . ., n}, letû i,v i be a cache of the left and right embeddings respectively. We will denote by a superscript (t) the value of a variable at iteration t. LetŜ DISPLAYFORM0 i, which corresponds to the Gramian computed with the current caches. At each iteration t, an example i is drawn uniformly at random and the estimate of the Gramian is given bŷ DISPLAYFORM1 and similarly forĜ DISPLAYFORM2 v. This is summarized in Algorithm 1, where the model parameters are updated using SGD (line 10), but this update can be replaced with any first-order method. Here β can take one of the following values: β = 1 n, following SAG , or β = 1, following SAGA BID9. The choice of β comes with trade-offs that we briefly discuss below. We will denote the cone of positive semi-definite k × k matrices by S DISPLAYFORM3 While taking β = 1 gives an unbiased estimate, note that it does not guarantee that the estimates remain in S k +. In practice, this can cause numerical issues, but can be avoided by projectingĜ u,Ĝ v on S k +, using their eigenvalue decompositions. The per-iteration cost of maintaining the Gramian estimates is O(k) to update the caches, O(k 2) to update the estimatesŜ u,Ŝ v,Ĝ u,Ĝ v, and O(k 3) for projecting on S k +. Given the small size of the embedding dimension k, O(k 3) remains tractable. The memory cost is O(nk), since each embedding needs to be cached (plus a negligible O(k 2) for storing the Gramian estimates). This makes SAGram much less expensive than applying the original SAG(A) methods, which require maintaining caches of the gradients, this would incur a O(nd) memory cost, where d is the number of parameters of the model, and can be orders of magnitude larger than the embedding dimension k. However, O(nk) can still be prohibitively expensive when n is very large. In the next section, we propose a different method which does not incur this additional memory cost. DISPLAYFORM4 Update Gramian estimates (i ∼ Uniform(n)) 8: DISPLAYFORM5 Update model parameters then update caches (i ∼ Uniform(n)) 10: DISPLAYFORM6 To derive the second method, we reformulate problem as a two-player game. The first player optimizes over the parameters of the model θ, the second player optimizes over the Gramian estimateŝ G u,Ĝ v ∈ S k +, and they seek to minimize the respective losses DISPLAYFORM0 whereĝ i is defined in FORMULA10, and · F denotes the Frobenius norm. To justify this reformulation, we can characterize its first-order stationary points, as follows. DISPLAYFORM1 + is a first-order stationary point for if and only if θ is a first-order stationary point for problem FORMULA0 DISPLAYFORM2 Several stochastic first-order dynamics can be applied to problem, and Algorithm 2 gives a simple instance where each player implements SGD with a constant learning rate. In this case, the updates of the Gramian estimates (line 7) have a particularly simple form, since ∇Ĝ DISPLAYFORM3 and similarly forĜ v. One advantage of this form is that each update performs a convex combination between the current estimate and a rank-1 PSD matrix, thus guaranteeing that the estimates remain in S k +, without the need to project. The per-iteration cost of updating the estimates is O(k 2), and the memory cost is O(k 2) for storing the Gramians, which are both negligible. DISPLAYFORM4 Update model parameters (i ∼ Uniform) 8: DISPLAYFORM5 The update can also be interpreted as computing an online estimate of the Gramian by averaging rank-1 terms with decaying weights, thus we call the method Stochastic Online Gramian. Indeed, we have by induction on t,Ĝ DISPLAYFORM6 Intuitively, averaging reduces the variance of the estimator but introduces a bias, and the choice of the hyper-parameter α trades-off bias and variance. The next proposition quantifies this tradeoff under mild assumptions. DISPLAYFORM7 The first assumption simply bounds the variance of single-point estimates, while the second bounds the distance between two consecutive Gramians, a reasonable assumption, since in practice the changes in Gramians vanish as the trajectory θ (τ) converges. In the limiting case α = 1,Ĝu reduces to a single-point estimate, in which case the bias vanishes and the variance is maximal, while smaller values of α decrease variance and increase bias. This is confirmed in our experiments, as discussed in Section 4.2. We conclude this section by showing that candidate sampling methods (see BID7 BID1 for recent surveys) can be reinterpreted in terms of the Gramian formulation. These methods work by approximating the double-sum in using a random sample of pairs. Suppose a batch of pairs (i, j) ∈ B × B is sampled 4, and the double sum is approximated bỹ DISPLAYFORM0 where µ i, ν j are the inverse probabilities of sampling i, j respectively (to guarantee that the estimate is unbiased). Then applying a similar transformation to Section 2.2, one can show that DISPLAYFORM1 which is equivalent to computing two batch-estimates of the Gramians. Implementing existing methods using rather than can decrease their computional complexity in the large batch regime, for the following reason: the double-sum formulation involves a sum of |B||B | dot products of vectors in R k, thus computing its gradient costs O(k|B||B |). On the other hand, the Gramian formulation FORMULA0 is the inner product of two k × k matrices, each involving a sum over the batch, thus computing its gradient costs O(k 2 max(|B|, |B |)), which is cheaper when the batch size is larger than the embedding dimension k, a common situation in practice. With this formulation, the advantage of SOGram and SAGram becomes clear, as they use more embeddings to estimate Gramians (by caching or online averaging) than would be possible using candidate sampling. In this section, we conduct large-scale experiments on the Wikipedia dataset (Wikimedia Foundation). Additional experiments on MovieLens BID11 are given in Appendix F. Datasets We consider the problem of learning the intra-site links between Wikipedia pages. Given a pair of pages (x, y) ∈ X × X, the target similarity is 1 if there is a link from x to y, and 0 otherwise. Here a page is represented by a feature vector x = (x id, x ngrams, x cats), where x id is (a one-hot encoding of) the page URL, x ngrams is a bag-of-words representation of the set of n-grams of the page's title, and x cats is a bag-of-words representation of the categories the page belongs to. Note that the left and right feature spaces coincide in this case, but the target similarity is not necessarily symmetric (the links are directed edges). We carry out experiments on subsets of the Wikipedia graph corresponding to three languages: Simple English, French, and English, denoted respectively by simple, fr, and en. These subgraphs vary in size, and Table 1 shows some basic statistics for each set. Each set is partitioned into training and validation using a (90%, 10%) split. Table 1: Corpus sizes for each training set. Models We train non-linear embedding models consisting of a two-tower neural network as in Figure 1, where the left and right embedding functions map, respectively, the source and destination page features. The two embedding networks have the same structure: the input feature embeddings are concatenated then mapped through two hidden layers with ReLU activations. The input embeddings are shared between the two networks, and their dimensions are 50 for simple, 100 for fr, and 120 for en. Training methods The model is trained using a squared error loss, (s, s) = 1 2 (s − s) 2, optimized using SAGram, SOGram, and as baseline, SGD with candidate sampling, using different sampling strategies. The experiments reported in this section use a learning rate η = 0.01, a penalty coefficient λ = 10, and batch size 1024. These parameters correspond to the best performance of the baseline methods; we report additional with different hyper-parameter settings in Appendix E. For SAGram and SOGram, a batch B is used in the Gramian updates (line 8 in Algorithm 1 and line 6 in Algorithm 2, where we use a sum of rank-1 terms over the batch), and another batch B is used in the model parameter update. For the sampling baselines, the double sum is approximated by all pairs in the cross product (i, j) ∈ B × B, and for efficiency, we implement them using the Gramian formulation as discussed in Section 3.3, since we operate in a regime where the batch size is an order of magnitude larger than the embedding dimension k. In the first baseline method, uniform, items are sampled uniformly from the vocabulary (all pages are sampled with the same probability). The other baseline methods implement importance sampling similarly to BID3; BID14: in linear, the probability is proportional to the number of occurrences of the page in the training set, and in sqrt, the probability is proportional to the square root of the number of occurrences. DISPLAYFORM0 DISPLAYFORM1 In the first set of experiments, we evaluate the quality of the Gramian estimates using each method. In order to have a meaningful comparison, we fix a trajectory of model parameters (θ (t) ) t∈{1,...,T}, and evaluate how well each method tracks the true Gramians G u (θ (t) ), G v (θ (t) ) on that common trajectory. This experiment is done on Wikipedia simple (the smallest of the datasets) so that we can compute the exact Gramians by periodically computing the embeddings u i (θ (t) ), v i (θ (t) ) on the full training set at a given time t. We report in FIG3 the estimation error for each method, measured by the normalized Frobenius distance DISPLAYFORM0. In FIG3, we can observe that both variants of SAGram yield the best estimates, and that SOGram yields better estimates than the baselines. Among the baseline methods, importance sampling (both linear and sqrt) perform better than uniform. We also vary the batch size to evaluate its impact: increasing the batch size from 128 to 1024 improves the quality of all estimates, as expected, but it is worth noting that the estimates of SOGram with |B| = 128 have comparable quality to baseline estimates with |B| = 1024.In Appendix E, we show that a similar effect can be observed for gradient estimates, and we make a formal connection between Gramian and gradient estimation errors. In FIG3, we evaluate the bias-variance tradeoff discussed in Section 3.2, by comparing the estimates of SOGram with different learning rates α. We observe that higher values of α suffer from higher variance which persists throughout the trajectory. A lower α reduces the variance but introduces a bias, which is mostly visible during the early iterations. In order to evaluate the impact of the Gramian estimation quality on training speed and generalization, we compare the validation performance of SOGram to the sampling baselines, on each dataset (we do not use SAGram due to its prohibitive memory cost for corpus sizes of 1M or more). The models are trained with a fixed time budget of 20 hours for simple, 30 hours for fr and 50 hours for en. We estimate the mean average precision (MAP) at 10, by scoring, every 5 minutes, left items in the validation set against 50K random candidates (exhaustively scoring all candidates is prohibitively expensive, but this gives a reasonable approximation). The are reported in FIG4. Compared to the sampling baselines, SOGram exhibits faster training and better validation performance across all sampling strategies. TAB3 The improvement on simple is modest (between 4% and 10%), which can be explained by the relatively small corpus size (85K unique pages), in which case candidate sampling with a large batch size already yields decent estimates. On the larger corpora, we obtain more significant improvements: between 9% and 15% on fr and between 9% and 19% on en. It's interesting to observe that the best performance is consistently achieved by SOGram with linear importance sampling, even though linear performs slightly worse than other strategies in the baseline. SOGram also has a significant impact on training speed: if we measure the time it takes for SOGram to exceed the final validation performance of each baseline method, this time is a small fraction of the total budget. In our experiments, this fraction is between 10% and 17% for simple, between 23% and 30% for fr, and between 16% and 24% for en. Additional numerical are provided in Appendix E, where we evaluate the impact of other parameters, such as the effect of the batch size |B|, the learning rate η, and the Gramian learning rate α. For example, we show that the relative improvement of SOGram compared to the baselines is even larger when using smaller batches, and its generalization performance is more robust to the choice of batch size and learning rate. We showed that the Gramian formulation commonly used in low-rank matrix factorization can be leveraged for training non-linear embedding models, by maintaining estimates of the Gram matrices and using them to estimate the gradient. By applying variance reduction techniques to the Gramians, one can improve the quality of the gradient estimates, without relying on large sample size as is done in traditional sampling methods. This leads to a significant impact on training time and generalization quality, as indicated by our experiments. While we focused on problems with very large vocabulary size, where traditional approaches are inefficient, it will be interesting to evaluate our methods on other applications such as word-analogy tasks BID14. Another direction of future work is to extend this formulation to a larger family of penalty functions, such as the spherical loss family studied in (; BID8 Proof of Proposition 1. Starting from the expression of g(θ) = G u (θ), G v (θ), and applying the chain rule, we have DISPLAYFORM0 where J u (θ) denotes the Jacobian of G u (θ), an order-three tensor given by DISPLAYFORM1 and DISPLAYFORM2 Observing DISPLAYFORM3, and applying the chain rule, we have DISPLAYFORM4 where J u,i (θ) is the Jacobian of u i (θ) ⊗ u i (θ), and DISPLAYFORM5 an similarly for J v,i. We conclude by taking expectations in and using assumption thatĜ u,Ĝ v are independent of i. u, we have, DISPLAYFORM0 which is a sum of matrices in the PSD cone S k +.Proof of Proposition 3. Denoting by (F t) t≥0 the filtration generated by the sequence (θ (t) ) t≥0, and taking conditional expectations in, we have DISPLAYFORM1 + is a first-order stationary point of the game if and only if DISPLAYFORM2 The second and third conditions simply states that ∇Ĝ FORMULA0 is equivalent toĜ u = G u (θ) (and similarly, is equivalent toĜ v = G v (θ)). Using the expression of ∇g, we get that is equivalent to ∇f (θ) + λ∇g(θ) = 0. DISPLAYFORM3 Proof of Proposition 5. We start by proving the first bound. As stated in Section 3.2, we have, by induction on t,Ĝ DISPLAYFORM4 And by definition ofḠ (t), we haveḠ DISPLAYFORM5 ) are zero-mean random variables. Thus, taking the second moment, and using the first assumption (which simply states that the variance of ∆ (τ) u is bounded by σ 2 ), we have DISPLAYFORM6 which proves the first inequality.To prove the second inequality, we start from the definition ofḠ DISPLAYFORM7 u: DISPLAYFORM8 where the first equality uses that fact that DISPLAYFORM9 Focusing on the first term, and bounding G DISPLAYFORM10 u F ≤ (t − τ)δ by the triangle inequality, we get DISPLAYFORM11 Combining FORMULA2 and FORMULA0, we get the desired inequality. So far, we have assumed a uniform zero prior to simplify the notation. In this section, we relax this assumption. Suppose that the prior is given by a low-rank matrix P = QR, where Q, R ∈ R n×k P. In other words, the prior for a given pair (i, j) is given by the dot product of two vectors p ij = q i, r j. In practice, such a low-rank prior can be obtained, for example, by first training a simple low-rank matrix approximation of the target similarity matrix. Given this low-rank prior, the penalty term becomes DISPLAYFORM0 where c = Q Q, R R is a constant that does not depend on θ. Here, we used a superscript P in g P to disambiguate the zero-prior case. Now, if we define weighted embedding matrices DISPLAYFORM1 Finally, if we maintain estimatesĤ u,Ĥ v of H u (θ), H v (θ), respectively (using the methods proposed in Section 3), we can approximate ∇g P (θ) by the gradient of DISPLAYFORM2 Proposition 1 and Algorithms 1 and 2 can be generalized to the low-rank prior case by adding updates forĤ u,Ĥ v, and by using expression ofĝ P i when computing the gradient estimate. DISPLAYFORM3 Proof. Similar to the proof of Proposition 1.The generalized versions of SAGram and SOGram are stated below, where the differences compared to the zero prior case are highlighted. Note that, unlike the Gramian matrices, the weighted embedding matrices H u, H v are not symmetric, thus we do not project their estimates. Algorithm 3 SAGram (Stochastic Average Gramian) with low-rank prior 1: Input: Training data {(x i, y i, s i)} i∈{1,...,n}, low-rank priors {q i, r i} i∈{1,...,n} 2: Initialization phase DISPLAYFORM4 Update weighted embedding estimates 11: DISPLAYFORM5 Update model parameters then update caches (i ∼ Uniform(n)) 14: DISPLAYFORM6 Algorithm 4 SOGram (Stochastic Online Gramian) with low-rank prior 1: Input: Training data {(x i, y i, s i)} i∈{1,...,n}, low-rank priors {q i, r i} i∈{1,...,n} 2: Initialization phase DISPLAYFORM7 Update weighted embedding estimates 9: DISPLAYFORM8 Update model parameters (i ∼ Uniform(n)) 11: DISPLAYFORM9 In additional to using a non-uniform prior, it can also be desirable to use non-uniform weights in the penalty term, for example to balance the contribution of frequent and infrequent items to the penalty term. We discuss how to adapt our algorithms to the non-uniform weights case. Suppose that the penalty function is given by DISPLAYFORM10 where a i, b j are positive left and right weights, respectively. Here we used a superscript W in g W to disambiguate the uniform-weight case. Then using a similar transformation to Section 2.2, we can rewrite g W as follows: DISPLAYFORM11 i.e. g W is the inner-product of two weighted Gramians. Both SAGram and SOGram can be generalized to this case, by maintaining estimates of the weighted Gramians, one simply needs to scale the contribution of each term u i ⊗ u i by the appropriate embedding weight a i (and similarly for the right embedding).Remark Here we discussed the case of a rank-one weight matrix, i.e. when the unobserved weight matrix can be written as W = a ⊗ b for a given left and right weight vectors a, b. The weight matrix cannot be arbitrary (as specifying n 2 individual weights is prohibitively expensive in many applications such as the experiments of this paper), thus one needs a consice description of the weights matrix. One such description is the sum of a sparse and low-rank matrix, and one can generalize SAGram and SOGram to this case: the sparse part of the weight matrix can be optimized explicitly, and the low-rank part can be optimized using weighted Gramians, by generalizing the argument of the previous paragraph. In this section, we briefly discuss different interpretations of the Gramian inner-product g(θ). Starting from the expression of g(θ) and the definition of the Gram matrices, we have DISPLAYFORM0 which is a quadratic form in the left embeddings u i (and similarly for v j, by symmetry). In particular, the partial derivative of the Gramian term with respect to an embedding u i is DISPLAYFORM1. Thus the gradient of g(θ) with respect to u i is an average of scaled projections of u i on each of the right embeddings v j, and moving in the direction of the negative gradient simply moves u i away from regions of the embedding space with a high density of right embeddings. This corresponds to the intuition discussed in the introduction: the purpose of the g(θ) term is precisely to push left and right embeddings away from each other, to avoid placing embeddings of dissimilar items near each other, a phenomenon referred to as folding of the embedding space . In order to illustrate the effect of this term on the embedding distributions, we visualize, in FIG6, the distribution of the inner product u i (θ (t) ), v j (θ (t) ), for random pairs (i, j), and for observed pairs (i = j), and how these distributions change as t increases. The plots are generated for the Wikipedia en model described in Section 4, trained with SOGram (α = 0.01), with two different values of the penalty coefficient, λ = 10 −2 and λ = 10. In both cases, the distribution for observed pairs remains concentrated around values close to 1, as one expects (recall that the target similarity is 1 for observed pairs, i.e. pairs of connected pages in the Wikipedia graph). The distributions for random pairs, however, are very different: with λ = 10, the distribution quickly concentrates around a value close to 0, while with λ = 10 −2 the distribution is more flat, and a large proportion of pairs have a high inner-product. This indicates that with a lower λ, the model is more likely to fold, i.e. place embeddings of unrelated items near each other. This is consistent with the validation MAP, reported in FIG7. With λ = 10 −2, the validation MAP increases very slowly, and remains two orders of magnitude smaller than the model trained with λ = 10. The figure also shows that when λ is too large (λ = 103), the model is over-regularized and the MAP decreases. To conclude this section, we note that our methods also apply to a related regularizer introduced in BID1, called Global Orthogonal Regularization. The authors argue that when learning feature embedding representations, spreading out the embeddings is helpful for generalization, and propose to match the second moment of each embedding distribution with that of the uniform distribution. Formally, and using our notation, they use the penalty term max(g u (θ), 1/k)+max(g v (θ), 1/k), where k is the embedding dimension, g u (θ) = 1 n 2 n i=1 n j=1 u i, u j 2, and similarly for g v. They optimize this term using candidate sampling. We can also apply the same Gramian transformation as in Section 2.2 to write g DISPLAYFORM2, and we can similarly apply SAGram and SOGram to estimate both Gramians. Formally, the difference here is that one would penalize the inner-product of each Gramian with itself, instead of the inner-product of the two. One advantage of this regularizer is that it applies to a broader class of models, as it does not require the output of the model to be the dot-product of two embedding functions. The experiments in Section 4 indicate that our methods give better estimates of the Gramians, and a natural question is how this affects gradient estimation quality. First, one can make a formal connection between the two. Since DISPLAYFORM0 the estimation error of the gradient with respect to the left embeddings u is DISPLAYFORM1 This last expression can be interpreted as a Frobenius norm of the right Gramian estimation error G v −Ĝ v, weighted by the left Gramian G u, thus the gradient error is closely related to the Gramian error. FIG8 shows the gradient estimation quality on Wikipedia simple, measured by the normalized squared norm DISPLAYFORM2. The are similar to the Gramian estimation errors reported in FIG3. Comparing the different baselines methods on simple FIG4, we observe that uniform sampling performs better than sqrt, despite having a worse Gramian estimate according to FIG3. One possible explanation is that the sampling distribution affects both the quality of the Gramian estimates, and the frequency at which the item embeddings are updated, which in turn affects the MAP. In particular, tail items are updated more frequently under uniform than other distributions, and this may have a positive impact on the MAP. In addition to the experiments of Section 4, we also evaluated the effect of the Gramian learning rate α on the quality of the Gramian esimates and generalization performance on Wikipedia en. FIG9 shows the validation MAP of the SOGram method for different values of α (together with the basline for reference). This reflects the bias-variance tradeoff dicussed in Proposition 5: with a lower α, progress is initially slower (due to the bias introduced in the Gramian estimates), but the final performance is better. Given a limited training time budget, this suggests that a higher α can be preferable. We also evaluate the quality of the Gramian estimates, but due to the large vocabulary size in en, computing the exact Gramians is no longer feasible, so we approximate it using a large sample of 1M embeddings. The are reported in FIG10, which shows the normalized Frobenius distance between the Gramian estimatesĜ u and (the large sample approximation of) the true Gramian G u. The are similar to the experiment on simple: with a lower α, the estimation error is initially high, but decays to a lower value as training progresses, which can be explained by the bias-variance tradeoff discussed in Proposition 5. The tradeoff is affected by the trajectory of the true Gramians: smaller changes in the Gramians (captured by the parameter δ in Proposition 5) induce a smaller bias. In particular, changing the learning rate η of the main algorithm can affect the performance of the Gramian estimates by affecting the rate of change of the true Gramians. To investiage this effect, we ran the same experiment with two different learning rates, η = 0.01 as in Section 4, and a lower learning rate η = 0.002. The errors converge to similar values in both cases, but the error decay occurs much faster with smaller η, which is consistent with our analysis. In this section, we explore the effect of the batch size |B| and learning rate η on the performance of SOGram compared to the baselines. We ran the Wikipedia en experiment with different values of these hyperparameters, and report the final validation MAP in TAB6, which correspond to batch size 128 and 512 respectively. We can make several observations. First, the best performance is consistently achieved by SOGram with learning rate α = 0.001. Second, the relative improvement compared to the baseline is, in general, larger for smaller batch sizes. This can be explained intuitively by the fact that because of online averaging, the quality of the Gramian estimates with SOGram suffers less than with the sampling baseline. Finally, we can also observe that the final performance also seems more robust to the choice of batch size and learning rate, compared to the baseline. For example, with the larger learning rate η = 0.02, the performance degrades for all methods, but the drop in performance for the baseline is much more significant than for the SOGram methods. In this section, we report experiments on a regression task on MovieLens. Dataset The MovieLens dataset consists of movie ratings given by a set of users. In our notation, the left features x represent a user, the right features y represent an item, and the target similarity is the rating of movie y by user x. The data is partitioned into a training and a validation set using a (80%-20%) split. Table 5 gives a basic description of the data size. Note that it is comparable to the simple dataset in the Wikipedia experiments. Dataset # users # movies # ratings MovieLens 72K 10K 10M Table 5: Corpus size of the MovieLens dataset. Model We train a two-tower neural network model, as described in Figure 1, where each tower consists of an input layer, a hidden layer, and output embedding dimension k = 35. The left tower takes as input a one-hot encoding of a unique user id, and the right tower takes as input one-hot encodings of a unique movie id, the release year of the movie, and a bag-of-words representation of the genres of the movie. These input embeddings are concatenated and used as input to the right tower. Methods The model is trained using a squared loss (s, s) = 1 2 (s − s) 2, using SOGram with different values of α, and sampling as a baseline. We use a learning rate η = 0.05, and penalty coefficient λ = 1. We measure mean average precision on the trainig set and validation set, following the same procedure described in Section 4. The are given in FIG12.Results The are similar to those reported on the Wikipedia simple dataset, which is comparable in corpus size and number of observations to MovieLens. The best validation mean average precision is achieved by SOGram with α = 0.1 (for an improvement of 2.9% compared to the sampling baseline), despite its poor performance on the training set, which indicates that better estimation of g(θ) induces better regularization. The impact on training speed is also remarkable in this case, SOGram with α = 0.1 achieves a better validation performance in under 1 hour of training than the sampling baseline in 6 hours.

represent the tldr for finding the corresponding scientific article

We develop efficient methods to train neural embedding models with a dot-product structure, by reformulating the objective function in terms of generalized Gram matrices, and maintaining estimates of those matrices.

Represent the TLDR for finding the corresponding scientific article


Question: Represent the scientific article to find a short TLDR

Question: For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks. By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks. GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models. We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task. However, the low absolute performance of our best model indicates the need for improved general NLU systems. The human ability to understand language is general, flexible, and robust. In contrast, most NLU models above the word level are designed for a specific task and struggle with out-of-domain data. If we aspire to develop models with understanding beyond the detection of superficial correspondences between inputs and outputs, then it is critical to develop a more unified model that can learn to execute a range of different linguistic tasks in different domains. To facilitate research in this direction, we present the General Language Understanding Evaluation (GLUE) benchmark: a collection of NLU tasks including question answering, sentiment analysis, and textual entailment, and an associated online platform for model evaluation, comparison, and analysis. GLUE does not place any constraints on model architecture beyond the ability to process single-sentence and sentence-pair inputs and to make corresponding predictions. For some GLUE tasks, training data is plentiful, but for others it is limited or fails to match the genre of the test set. GLUE therefore favors models that can learn to represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge-transfer across tasks. None of the datasets in GLUE were created from scratch for the benchmark; we rely on preexisting datasets because they have been implicitly agreed upon by the NLP community as challenging and interesting. Four of the datasets feature privately-held test data, which will be used to ensure that the benchmark is used fairly. Table 1: Task descriptions and statistics. All tasks are single sentence or sentence pair classification, except STS-B, which is a regression task. MNLI has three classes; all other classification tasks have two. Test sets shown in bold use labels that have never been made public in any form. To better understand the challenged posed by GLUE, we conduct experiments with simple baselines and state-of-the-art sentence representation models. We find that unified multi-task trained models slightly outperform comparable models trained on each task separately. Our best multi-task model makes use of ELMo BID2, a recently proposed pre-training technique. However, this model still achieves a fairly low absolute score. Analysis with our diagnostic dataset reveals that our baseline models deal well with strong lexical signals but struggle with deeper logical structure. In summary, we offer: (i) A suite of nine sentence or sentence-pair NLU tasks, built on established annotated datasets and selected to cover a diverse range of text genres, dataset sizes, and degrees of difficulty.(ii) An online evaluation platform and leaderboard, based primarily on privately-held test data. The platform is model-agnostic, and can evaluate any method capable of producing on all nine tasks. (iii) An expert-constructed diagnostic evaluation dataset. (iv) Baseline for several major existing approaches to sentence representation learning. used a multi-task model with a shared sentence understanding component to jointly learn POS tagging, chunking, named entity recognition, and semantic role labeling. More recent work has explored using labels from core NLP tasks to supervise training of lower levels of deep neural networks BID10 ) and automatically learning cross-task sharing mechanisms for multi-task learning BID6 ).Beyond multi-task learning, much work in developing general NLU systems has focused on sentence-to-vector encoders (; , i.a.), leveraging unlabeled data (; BID2, labeled data , and combinations of these (; BID11 . In this line of work, a standard evaluation practice has emerged, recently codified as SentEval . Like GLUE, SentEval relies on a set of existing classification tasks involving either one or two sentences as inputs. Unlike GLUE, SentEval only evaluates sentenceto-vector encoders, making it well-suited for evaluating models on tasks involving sentences in isolation. However, cross-sentence contextualization and alignment are instrumental in achieving state-of-the-art performance on tasks such as machine translation BID0 BID13, question answering BID8, and natural language inference BID5. GLUE is designed to facilitate the development of these methods: It is model-agnostic, allowing for any kind of representation or contextualization, including models that use no explicit vector or symbolic representations for sentences whatsoever. GLUE also diverges from SentEval in the selection of evaluation tasks that are included in the suite. Many of the SentEval tasks are closely related to sentiment analysis, such as MR , SST BID9, CR , and SUBJ . Other tasks are so close to being solved that evaluation on them is relatively uninformative, such as MPQA and TREC question classification BID14. In GLUE, we attempt to construct a benchmark that is both diverse and difficult. introduce decaNLP, which also scores NLP systems based on their performance on multiple datasets. Their benchmark recasts the ten evaluation tasks as question answering, converting tasks like summarization and text-to-SQL semantic parsing into question answering using automatic transformations. That benchmark lacks the leaderboard and error analysis toolkit of GLUE, but more importantly, we see it as pursuing a more ambitious but less immediately practical goal: While GLUE rewards methods that yield good performance on a circumscribed set of tasks using methods like those that are currently used for those tasks, their benchmark rewards systems that make progress toward their goal of unifying all of NLU under the rubric of question answering. GLUE is centered on nine English sentence understanding tasks, which cover a broad range of domains, data quantities, and difficulties. As the goal of GLUE is to spur development of generalizable NLU systems, we design the benchmark such that good performance should require a model to share substantial knowledge (e.g., trained parameters) across all tasks, while still maintaining some taskspecific components. Though it is possible to train a single model for each task with no pretraining or other outside sources of knowledge and evaluate the ing set of models on this benchmark, we expect that our inclusion of several data-scarce tasks will ultimately render this approach uncompetitive. We describe the tasks below and in Table 1. Appendix A includes additional details. Unless otherwise mentioned, tasks are evaluated on accuracy and are balanced across classes. CoLA The Corpus of Linguistic Acceptability BID15 consists of English acceptability judgments drawn from books and journal articles on linguistic theory. Each example is a sequence of words annotated with whether it is a grammatical English sentence. Following the authors, we use Matthews correlation coefficient as the evaluation metric, which evaluates performance on unbalanced binary classification and ranges from -1 to 1, with 0 being the performance of uninformed guessing. We use the standard test set, for which we obtained private labels from the authors. We report a single performance number on the combination of the in-and out-of-domain sections of the test set. SST-2 The Stanford Sentiment Treebank BID9 consists of sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of a given sentence. We use the two-way (positive/negative) class split, and use only sentence-level labels. MRPC The Microsoft Research Paraphrase Corpus is a corpus of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semantically equivalent. Because the classes are imbalanced (68% positive), we follow common practice and report both accuracy and F1 score. QQP The Quora Question Pairs 2 dataset is a collection of question pairs from the community question-answering website Quora. The task is to determine whether a pair of questions are semantically equivalent. As in MRPC, the class distribution in QQP is unbalanced (63% negative), so we report both accuracy and F1 score. We use the standard test set, for which we obtained private labels from the authors. We observe that the test set has a different label distribution than the training set. The Semantic Textual Similarity Benchmark is a collection of sentence pairs drawn from news headlines, video and image captions, and natural language inference data. Each pair is human-annotated with a similarity score from 1 to 5; the task is to predict these scores. Follow common practice, we evaluate using Pearson and Spearman correlation coefficients. MNLI The Multi-Genre Natural Language Inference Corpus is a crowdsourced collection of sentence pairs with textual entailment annotations. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). The premise sentences are gathered from ten different sources, including transcribed speech, fiction, and government reports. We use the standard test set, for which we obtained private labels from the authors, and evaluate on both the matched (in-domain) and mismatched (cross-domain) sections. We also use and recommend the SNLI corpus as 550k examples of auxiliary training data. QNLI The Stanford Question Answering Dataset BID4 ) is a question-answering dataset consisting of question-paragraph pairs, where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator). We convert the task into sentence pair classification by forming a pair between each question and each sentence in the corresponding context, and filtering out pairs with low lexical overlap between the question and the context sentence. The task is to determine whether the context sentence contains the answer to the question. This modified version of the original task removes the requirement that the model select the exact answer, but also removes the simplifying assumptions that the answer is always present in the input and that lexical overlap is a reliable cue. This process of recasting existing datasets into NLI is similar to methods introduced in BID16 and expanded upon in. We call the converted dataset QNLI (Question-answering NLI). RTE The Recognizing Textual Entailment (RTE) datasets come from a series of annual textual entailment challenges. We combine the data from RTE1 , RTE2 , RTE3 , and RTE5 . 4 Examples are constructed based on news and Wikipedia text. We convert all datasets to a two-class split, where for three-class datasets we collapse neutral and contradiction into not entailment, for consistency. WNLI The Winograd Schema Challenge is a reading comprehension task in which a system must read a sentence with a pronoun and select the referent of that pronoun from a list of choices. The examples are manually constructed to foil simple statistical methods: Each one is contingent on contextual information provided by a single word or phrase in the sentence. To convert the problem into sentence pair classification, we construct sentence pairs by replacing the ambiguous pronoun with each possible referent. The task is to predict if the sentence with the I have never seen a hummingbird not flying. I have never seen a hummingbird. N E Table 3: Examples from the diagnostic set. Fwd (resp. Bwd) denotes the label when sentence 1 (resp. sentence 2) is the premise. Labels are entailment (E), neutral (N), or contradiction (C). Examples are tagged with the phenomena they demonstrate, and each phenomenon belongs to one of four broad categories (in parentheses).pronoun substituted is entailed by the original sentence. We use a small evaluation set consisting of new examples derived from fiction books 5 that was shared privately by the authors of the original corpus. While the included training set is balanced between two classes, the test set is imbalanced between them (65% not entailment). Also, due to a data quirk, the development set is adversarial: hypotheses are sometimes shared between training and development examples, so if a model memorizes the training examples, they will predict the wrong label on corresponding development set example. As with QNLI, each example is evaluated separately, so there is not a systematic correspondence between a model's score on this task and its score on the unconverted original task. We call converted dataset WNLI (Winograd NLI). The GLUE benchmark follows the same evaluation model as SemEval and Kaggle. To evaluate a system on the benchmark, one must run the system on the provided test data for the tasks, then upload the to the website gluebenchmark.com for scoring. The benchmark site shows per-task scores and a macro-average of those scores to determine a system's position on the leaderboard. For tasks with multiple metrics (e.g., accuracy and F1), we use an unweighted average of the metrics as the score for the task when computing the overall macro-average. The website also provides fine-and coarse-grained on the diagnostic dataset. See Appendix D for details. Drawing inspiration from the FraCaS suite and the recent Build-It-Break-It competition , we include a small, manually-curated test set for the analysis of system performance. While the main benchmark mostly reflects an application-driven distribution of examples, our diagnostic dataset highlights a pre-defined set of phenomena that we believe are interesting and important for models to capture. We show the full set of phenomena in TAB1.Each diagnostic example is an NLI sentence pair with tags for the phenomena demonstrated. The NLI task is well-suited to this kind of analysis, as it can easily evaluate the full set of skills involved in (ungrounded) sentence understanding, from resolution of syntactic ambiguity to pragmatic reasoning with world knowledge. We ensure the data is reasonably diverse by producing examples for a variety of linguistic phenomena and basing our examples on naturally-occurring sentences from several domains (news, Reddit, Wikipedia, academic papers). This approaches differs from that of FraCaS, which was designed to test linguistic theories with a minimal and uniform set of examples. A sample from our dataset is shown in Table 3.Annotation Process We begin with a target set of phenomena, based roughly on those used in the FraCaS suite . We construct each example by locating a sentence that can be easily made to demonstrate a target phenomenon, and editing it in two ways to produce an appropriate sentence pair. We make minimal modifications so as to maintain high lexical and structural overlap within each sentence pair and limit superficial cues. We then label the inference relationships between the sentences, considering each sentence alternatively as the premise, producing two labeled examples for each pair (1100 total). Where possible, we produce several pairs with different labels for a single source sentence, to have minimal sets of sentence pairs that are lexically and structurally very similar but correspond to different entailment relationships. The ing labels are 42% entailment, 35% neutral, and 23% contradiction. Evaluation Since the class distribution in the diagnostic set is not balanced, we use R 3 , a three-class generalization of the Matthews correlation coefficient, for evaluation. In light of recent work showing that crowdsourced data often contains artifacts which can be exploited to perform well without solving the intended task BID7 BID3 , i.a.), we audit the data for such artifacts. We reproduce the methodology of , training two fastText classifiers to predict entailment labels on SNLI and MNLI using only the hypothesis as input. The models respectively get near-chance accuracies of 32.7% and 36.4% on our diagnostic data, showing that the data does not suffer from such artifacts. To establish human baseline performance on the diagnostic set, we have six NLP researchers annotate 50 sentence pairs (100 entailment examples) randomly sampled from the diagnostic set. Interannotator agreement is high, with a Fleiss's κ of 0.73. The average R 3 score among the annotators is 0.80, much higher than any of the baseline systems described in Section 5.Intended Use The diagnostic examples are hand-picked to address certain phenomena, and NLI is a task with no natural input distribution, so we do not expect performance on the diagnostic set to reflect overall performance or generalization in downstream applications. Performance on the analysis set should be compared between models but not between categories. The set is provided not as a benchmark, but as an analysis tool for error analysis, qualitative model comparison, and development of adversarial examples. For baselines, we evaluate a multi-task learning model trained on the GLUE tasks, as well as several variants based on recent pre-training methods. We briefly describe them here. See Appendix B for details. We implement our models in the AllenNLP library . Original code for the baselines is available at https://github.com/nyu-mll/GLUE-baselines and a newer version is available at https://github.com/jsalt18-sentence-repl/jiant.Architecture Our simplest baseline architecture is based on sentence-to-vector encoders, and sets aside GLUE's ability to evaluate models with more complex structures. Taking inspiration from , the model uses a two-layer, 1500D (per direction) BiLSTM with max pooling and 300D GloVe word embeddings (840B Common Crawl version; BID1 . For single-sentence tasks, we encode the sentence and pass the ing vector to a classifier. For sentence-pair tasks, we encode sentences independently to produce vectors u, v, and pass [u; v; |u − v|; u * v] to a classifier. The classifier is an MLP with a 512D hidden layer. We also consider a variant of our model which for sentence pair tasks uses an attention mechanism inspired by BID8 between all pairs of words, followed by a second BiLSTM with max pooling. By explicitly modeling the interaction between sentences, these models fall outside the sentence-to-vector paradigm. Pre-Training We augment our base model with two recent methods for pre-training: ELMo and CoVe. We use existing trained models for both. ELMo uses a pair of two-layer neural language models trained on the Billion Word Benchmark (Training We train our models with the BiLSTM sentence encoder and post-attention BiLSTMs shared across tasks, and classifiers trained separately for each task. For each training update, we sample a task to train with a probability proportional to the number of training examples for each task. We train our models with Adam with initial learning rate 10 −4 and batch size 128. We use the macro-average score as the validation metric and stop training when the learning rate drops below 10 −5 or performance does not improve after 5 validation checks. We also train a set of single-task models, which are configured and trained identically, but share no parameters. To allow for fair comparisons with the multi-task analogs, we do not tune parameter or training settings for each task, so these single-task models do not generally represent the state of the art for each task. Sentence Representation Models Finally, we evaluate the following trained sentence-to-vector encoder models using our benchmark: average bag-of-words using GloVe embeddings (CBoW), Skip-Thought , InferSent , DisSent , and GenSen BID11. For these models, we only train task-specific classifiers on the representations they produce. We train three runs of each model and evaluate the run with the best macro-average development set performance (see TAB8 in Appendix C). For single-task and sentence representation models, we evaluate the best run for each individual task. We present performance on the main benchmark tasks in TAB4.We find that multi-task training yields better overall scores over single-task training amongst models using attention or ELMo. Attention generally has negligible or negative aggregate effect in single task training, but helps in multi-task training. We see a consistent improvement in using ELMo embeddings in place of GloVe or CoVe embeddings, particularly for single-sentence tasks. Using CoVe has mixed effects over using only GloVe. Among the pre-trained sentence representation models, we observe fairly consistent gains moving from CBoW to Skip-Thought to Infersent and GenSen. Relative to the models trained directly on the GLUE tasks, InferSent is competitive and GenSen outperforms all but the two best. Looking at per task, we find that the sentence representation models substantially underperform on CoLA compared to the models directly trained on the task. On the other hand, for STS-B, models trained directly on the task lag significantly behind the performance of the best sentence representation model. Finally, there are tasks for which no model does particularly well. On WNLI, no model exceeds most-frequent-class guessing (65.1%) and we substitute the model predictions for the most-frequent baseline. On RTE and in aggregate, even our best baselines leave room for improvement. These early indicate that solving GLUE is beyond the capabilities of current models and methods. We analyze the baselines by evaluating each model's MNLI classifier on the diagnostic set to get a better sense of their linguistic capabilities. Results are presented in TAB6.Coarse Categories Overall performance is low for all models: The highest total score of 28 still denotes poor absolute performance. Performance tends to be higher on Predicate-Argument Structure and lower on Logic, though numbers are not closely comparable across categories. Unlike on the main benchmark, the multi-task models are almost always outperformed by their single-task counterparts. This is perhaps unsurprising, since with our simple multi-task training regime, there is likely some destructive interference between MNLI and the other tasks. The models trained on the GLUE tasks largely outperform the pretrained sentence representation models, with the exception of GenSen. Using attention has a greater influence on diagnostic scores than using ELMo or CoVe, which we take to indicate that attention is especially important for generalization in NLI.Fine-Grained Subcategories Most models handle universal quantification relatively well. Looking at relevant examples, it seems that relying on lexical cues such as "all" often suffices for good performance. Similarly, lexical cues often provide good signal in morphological negation examples. We observe varying weaknesses between models. Double negation is especially difficult for the GLUE-trained models that only use GloVe embeddings. This is ameliorated by ELMo, and to some degree CoVe. Also, attention has mixed effects on overall , and models with attention tend to struggle with downward monotonicity. Examining their predictions, we found that the models are sensitive to hypernym/hyponym substitution and word deletion as a signal of entailment, but predict it in the wrong direction (as if the substituted/deleted word were in an upward monotone context). This is consistent with recent findings by that these systems use the subsequence relation between premise and hypothesis as a heuristic shortcut. Restrictivity examples, which often depend on nuances of quantifier scope, are especially difficult for almost all models. Overall, there is evidence that going beyond sentence-to-vector representations, e.g. with an attention mechanism, might aid performance on out-of-domain data, and that transfer methods like ELMo and CoVe encode linguistic information specific to their supervision signal. However, increased representational capacity may lead to overfitting, such as the failure of attention models in downward monotone contexts. We expect that our platform and diagnostic dataset will be useful for similar analyses in the future, so that model designers can better understand their models' generalization behavior and implicit knowledge. We introduce GLUE, a platform and collection of resources for evaluating and analyzing natural language understanding systems. We find that, in aggregate, models trained jointly on our tasks see better performance than the combined performance of models trained for each task separately. We confirm the utility of attention mechanisms and transfer learning methods such as ELMo in NLU systems, which combine to outperform the best sentence representation models on the GLUE benchmark, but still leave room for improvement. When evaluating these models on our diagnostic dataset, we find that they fail (often spectacularly) on many linguistic phenomena, suggesting possible avenues for future work. In sum, the question of how to design general-purpose NLU models remains unanswered, and we believe that GLUE can provide fertile soil for addressing this challenge. A ADDITIONAL BENCHMARK DETAILS QNLI To construct a balanced dataset, we select all pairs in which the most similar sentence to the question was not the answer sentence, as well as an equal amount of cases in which the correct sentence was the most similar to the question, but another distracting sentence was a close second. Our similarity metric is based on CBoW representations with pre-trained GloVe embeddings. This approach to converting pre-existing datasets into NLI format is closely related to recent work by BID16, as well as to the original motivation for textual entailment presented by. Both argue that many NLP tasks can be productively reduced to textual entailment. We implement our attention mechanism as follows: given two sequences of hidden states u 1, u 2,..., u M and v 1, v 2,..., v N, we first compute matrix H where H ij = u i · v j. For each u i, we get attention weights α i by taking a softmax over the i th row of H, and get the corresponding context vectorṽ i = j α ij v j by taking the attention-weighted sum of the v j. We pass a second BiLSTM with max pooling over the sequence [u 1 ;ṽ 1],... [u M ;ṽ M] to produce u. We process the v j vectors analogously to obtain v. Finally, we feed [u ; v ; |u − v |; u * v] into a classifier. We train our models with the BiLSTM sentence encoder and post-attention BiLSTMs shared across tasks, and classifiers trained separately for each task. For each training update, we sample a task to train with a probability proportional to the number of training examples for each task. We scale each task's loss inversely proportional to the number of examples for that task, which we found to improve overall performance. We train our models with Adam with initial learning rate 10 −3, batch size 128, and gradient clipping. We use macro-average score over all tasks as our validation metric, and perform a validation check every 10k updates. We divide the learning rate by 5 whenever validation performance does not improve. We stop training when the learning rate drops below 10 −5 or performance does not improve after 5 validation checks. We evaluate the following sentence representation models:1. CBoW, the average of the GloVe embeddings of the tokens in the sentence., a BiLSTM with max-pooling trained to predict the discourse marker (because, so, etc.) relating two sentences on data derived from TBC. We use the variant trained for eight-way classification. 5. GenSen BID11, a sequence-to-sequence model trained on a variety of supervised and unsupervised objectives. We use the variant of the model trained on both MNLI and SNLI, the Skip-Thought objective on TBC, and a constituency parsing objective on the Billion Word Benchmark. We train task-specific classifiers on top of frozen sentence encoders, using the default parameters from SentEval. See https://github.com/nyu-mll/SentEval for details and code. The GLUE website limits users to two submissions per day in order to avoid overfitting to the private test data. To provide a reference for future work on GLUE, we present the best development set achieved by our baselines in TAB8. GLUE's online platform is built using React, Redux and TypeScript. We use Google Firebase for data storage and Google Cloud Functions to host and run our grading script when a submission is made. FIG1 shows the visual presentation of our baselines on the leaderboard. Table 7: Diagnostic dataset statistics by coarse-grained category. Note that some examples may be tagged with phenomena belonging to multiple categories. The dataset is designed to allow for analyzing many levels of natural language understanding, from word meaning and sentence structure to high-level reasoning and application of world knowledge. To make this kind of analysis feasible, we first identify four broad categories of phenomena: Lexical Semantics, Predicate-Argument Structure, Logic, and Knowledge. However, since these categories are vague, we divide each into a larger set of fine-grained subcategories. Descriptions of all of the fine-grained categories are given in the remainder of this section. These categories are just one lens that can be used to understand linguistic phenomena and entailment, and there is certainly room to argue about how examples should be categorized, what the categories should be, etc. These categories are not based on any particular linguistic theory, but broadly based on issues that linguists have often identified and modeled in the study of syntax and semantics. The dataset is provided not as a benchmark, but as an analysis tool to paint in broad strokes the kinds of phenomena a model may or may not capture, and to provide a set of examples that can serve for error analysis, qualitative model comparison, and development of adversarial examples that expose a model's weaknesses. Because the distribution of language is somewhat arbitrary, it will not be helpful to compare performance of the same model on different categories. Rather, we recommend comparing performance that different models score on the same category, or using the reported scores as a guide for error analysis. We show coarse-grain category counts and label distributions of the diagnostic set in Table 7. These phenomena center on aspects of word meaning. Lexical Entailment Entailment can be applied not only on the sentence level, but the word level. For example, we say "dog" lexically entails "animal" because anything that is a dog is also an animal, and "dog" lexically contradicts "cat" because it is impossible to be both at once. This relationship applies to many types of words (nouns, adjectives, verbs, many prepositions, etc.) and the relationship between lexical and sentential entailment has been deeply explored, e.g., in systems of natural logic. This connection often hinges on monotonicity in language, so many Lexical Entailment examples will also be tagged with one of the Monotone categories, though we do not do this in every case (see Monotonicity, under Logic).Morphological Negation This is a special case of lexical contradiction where one word is derived from the other: from "affordable" to "unaffordable", "agree" to "disagree", etc. We also include examples like "ever" and "never". We also label these examples with Negation or Double Negation, since they can be viewed as involving a word-level logical negation. Factivity Propositions appearing in a sentence may be in any entailment relation with the sentence as a whole, depending on the context in which they appear. In many cases, this is determined by lexical triggers (usually verbs or adverbs) in the sentence. For example,• "I recognize that X" entails "X".• "I did not recognize that X" entails "X".• "I believe that X" does not entail "X".• "I am refusing to do X" contradicts "I am doing X".• "I am not refusing to do X" does not contradict "I am doing X".• "I almost finished X" contradicts "I finished X".• "I barely finished X" entails "I finished X".Constructions like "I recognize that X" are often called factive, since the entailment (of X above, regarded as a presupposition) persists even under negation. Constructions like "I am refusing to do X" above are often called implicative, and are sensitive to negation. There are also cases where a sentence (non-)entails the existence of an entity mentioned in it, for example "I have found a unicorn" entails "A unicorn exists" while "I am looking for a unicorn" doesn't necessarily entail "A unicorn exists". Readings where the entity does not necessarily exist are often called intensional readings, since they seem to deal with the properties denoted by a description (its intension) rather than being reducible to the set of entities that match the description (its extension, which in cases of non-existence will be empty).We place all examples involving these phenomena under the label of Factivity. While it often depends on context to determine whether a nested proposition or existence of an entity is entailed by the overall statement, very often it relies heavily on lexical triggers, so we place the category under Lexical Semantics. Symmetry/Collectivity Some propositions denote symmetric relations, while others do not. For example, "John married Gary" entails "Gary married John" but "John likes Gary" does not entail "Gary likes John". Symmetric relations can often be rephrased by collecting both arguments into the subject: "John met Gary" entails "John and Gary met". Whether a relation is symmetric, or admits collecting its arguments into the subject, is often determined by its head word (e.g., "like", "marry" or "meet"), so we classify it under Lexical Semantics. Redundancy If a word can be removed from a sentence without changing its meaning, that means the word's meaning was more-or-less adequately expressed by the sentence; so, identifying these cases reflects an understanding of both lexical and sentential semantics. Named Entities Words often name entities that exist in the world. There are many different kinds of understanding we might wish to understand about these names, including their compositional structure (for example, the "Baltimore Police" is the same as the "Police of the City of Baltimore") or their real-world referents and acronym expansions (for example, "SNL" is "Saturday Night Live"). This category is closely related to World Knowledge, but focuses on the semantics of names as lexical items rather than knowledge about their denoted entities. Quantifiers Logical quantification in natural language is often expressed through lexical triggers such as "every", "most", "some", and "no". While we reserve the categories in Quantification and Monotonicity for entailments involving operations on these quantifiers and their arguments, we choose to regard the interchangeability of quantifiers (e.g., in many cases "most" entails "many") as a question of lexical semantics. An important component of understanding the meaning of a sentence is understanding how its parts are composed together into a whole. In this category, we address issues across that spectrum, from syntactic ambiguity to semantic roles and coreference. Syntactic Ambiguity: Relative Clauses, Coordination Scope These two categories deal purely with resolving syntactic ambiguity. Relative clauses and coordination scope are both sources of a great amount of ambiguity in English. Prepositional phrases Prepositional phrase attachment is a particularly difficult problem that syntactic parsers in NLP systems continue to struggle with. We view it as a problem both of syntax and semantics, since prepositional phrases can express a wide variety of semantic roles and often semantically apply beyond their direct syntactic attachment. Core Arguments Verbs select for particular arguments, particularly subjects and objects, which might be interchangeable depending on the context or the surface form. One example is the ergative alternation: "Jake broke the vase" entails "the vase broke" but "Jake broke the vase" does not entail "Jake broke". Other rearrangements of core arguments, such as those seen in Symmetry/Collectivity, also fall under the Core Arguments label. Alternations: Active/Passive, Genitives/Partitives, Nominalization, Datives All four of these categories correspond to syntactic alternations that are known to follow specific patterns in English:• Active/Passive: "I saw him" is equivalent to "He was seen by me" and entails " He was seen".• Genitives/Partitives: "the elephant's foot" is the same thing as "the foot of the elephant".• Nominalization: "I caused him to submit his resignation" entails "I caused the submission of his resignation".• Datives: "I baked him a cake" entails "I baked a cake for him" and "I baked a cake" but not "I baked him".Ellipsis/Implicits Often, the argument of a verb or other predicate is omitted (elided) in the text, with the reader filling in the gap. We can construct entailment examples by explicitly filling in the gap with the correct or incorrect referents. For example, the premise "Putin is so entrenched within Russias ruling system that many of its members can imagine no other leader" entails "Putin is so entrenched within Russias ruling system that many of its members can imagine no other leader than Putin" and contradicts "Putin is so entrenched within Russias ruling system that many of its members can imagine no other leader than themselves." This is often regarded as a special case of anaphora, but we decided to split out these cases from explicit anaphora, which is often also regarded as a case of coreference (and attempted to some degree in modern coreference resolution systems).Anaphora/Coreference Coreference refers to when multiple expressions refer to the same entity or event. It is closely related to Anaphora, where the meaning of an expression depends on another (antecedent) expression in context. These two phenomena have significant overlap; for example, pronouns ("she", "we", "it") are anaphors that are co-referent with their antecedents. However, they also may occur independently, such as coreference between two definite noun phrases (e.g., "Theresa May "and the "British Prime Minister") that refer to the same entity, or anaphora from a word like "other" which requires an antecedent to distinguish something from. In this category we only include cases where there is an explicit phrase (anaphoric or not) that is co-referent with an antecedent or other phrase. We construct examples for these in much the same way as for Ellipsis/Implicits. Intersectivity Many modifiers, especially adjectives, allow non-intersective uses, which affect their entailment behavior. For example:• Intersective: "He is a violinist and an old surgeon" entails "He is an old violinist" and "He is a surgeon".• Non-intersective: "He is a violinist and a skilled surgeon" does not entail "He is a skilled violinist".• Non-intersective: "He is a fake surgeon" does not entail "He is a surgeon".Generally, an intersective use of a modifier, like "old" in "old men", is one which may be interpreted as referring to the set of entities with both properties (they are old and they are men). Linguists often formalize this using set intersection, hence the name. Intersectivity is related to Factivity. For example, "fake" may be regarded as a counter-implicative modifier, and these examples will be labeled as such. However, we choose to categorize intersectivity under predicate-argument structure rather than lexical semantics, because generally the same word will admit both intersective and non-intersective uses, so it may be regarded as an ambiguity of argument structure. Restrictivity Restrictivity is most often used to refer to a property of uses of noun modifiers. In particular, a restrictive use of a modifier is one that serves to identify the entity or entities being described, whereas a non-restrictive use adds extra details to the identified entity. The distinction can often be highlighted by entailments:• Restrictive: "I finished all of my homework due today" does not entail "I finished all of my homework".• Non-restrictive: "I got rid of all those pesky bedbugs" entails "I got rid of all those bedbugs".Modifiers that are commonly used non-restrictively are appositives, relative clauses starting with "which" or "who", and expletives (e.g. "pesky"). Non-restrictive uses can appear in many forms. With an understanding of the structure of a sentence, there is often a baseline set of shallow that can be drawn using logical operators and often modeled using the mathematical tools of logic. Indeed, the development of mathematical logic was initially guided by questions about natural language meaning, from Aristotelian syllogisms to Fregean symbols. The notion of entailment is also borrowed from mathematical logic. Propositional Structure: Negation, Double Negation, Conjunction, Disjunction, Conditionals All of the basic operations of propositional logic appear in natural language, and we tag them where they are relevant to our examples:• Negation: "The cat sat on the mat" contradicts "The cat did not sit on the mat".• Double negation: "The market is not impossible to navigate" entails "The market is possible to navigate".

Represent the TLDR for finding the corresponding scientific article.

We present a multi-task benchmark and analysis platform for evaluating generalization in natural language understanding systems.

Represent the short summary for finding the corresponding scientific article (long)

We propose neural cascades, a simple and trivially parallelizable approach to reading comprehension, consisting only of feed-forward nets and attention that achieves state-of-the-art performance on the TriviaQA dataset.