Question: Represent the article for finding a single-sentence TLDR

Question: GloVe and Skip-gram word embedding methods learn word vectors by decomposing a denoised matrix of word co-occurrences into a product of low-rank matrices. In this work, we propose an iterative algorithm for computing word vectors based on modeling word co-occurrence matrices with Generalized Low Rank Models. Our algorithm generalizes both Skip-gram and GloVe as well as giving rise to other embedding methods based on the specified co-occurrence matrix, distribution of co-occurences, and the number of iterations in the iterative algorithm. For example, using a Tweedie distribution with one iteration in GloVe and using a Multinomial distribution with full-convergence mode in Skip-gram. Experimental demonstrate that multiple iterations of our algorithm improves over the GloVe method on the Google word analogy similarity task. Word embeddings are low dimensional vector representations of words or phrases. They are applied to word analogy tasks and used as feature vectors in numerous tasks within natural language processing, computational linguistics, and machine learning. They are constructed by various methods which rely on the distributional hypothesis popularized by Firth: "words are characterized by the company they keep" BID9. Two seminal methodological approaches to finding word embeddings are Skip-gram [a] and GloVe []. Both methods input a corpus D, process it into a word co-occurence matrix X, then output word vectors with some dimension d. Skip-gram processes a corpus with w words into a count co-occurence matrix X ∈ R w×w, where x ij is the number of times word w i appears in the same context as the word w j. Here, two words being in the same context means that they're within l c tokens of each other. Define this co-occurence matrix to be the count co-occurence matrix. Next, Skip-gram [where u u u T i is the i th row of U, then defines the word vectors to be the rows ofÛ .GloVe processes a corpus with w words into a harmonic co-occurence matrix X ∈ R w×w where x ij is the harmonic sum of the number of tokens between words w i and w j over each co-occurrence. That is, x ij = p1<p2,|p1−p2|≤lc,D(p1)=wi,D(p2)=wj h(x ij) u u u DISPLAYFORM0 where a i and b j are bias terms, h(x ij) = (min{x ij, x max}).75 is the weight, and x max is some prespecified cutoff. GloVe then defines the estimated word vectors to be the rows of 1 2Û + 1 2V. In both Skip-gram and GloVe, a matrix of co-occurences X is introduced by processing the corpus, and an objective function is introduced to find a low rank factorization related to the co-occurences X. In this paper, we derive the objective functions from a model-based perspective. We introduce an iterative algorithm, and show that problem from running the iterative algorithm on full-convergence mode for a Multinomial model and problem is one step of the iterative algorithm for a Tweedie model. This algorithm additionally allows us to introduce methods to "fill in the gaps" between Skip-gram and GloVe and to introduce altogether new methods for finding word vectors. We saw that Skip-gram and GloVe compute a co-occurence matrix X which from processing the corpus D and an objective function J to relate the matrix X to a product of low rank matrices U and V. Many existing approaches for explaining word embedding methods do so by identifying or deriving the co-occurence matrix X or the objective function J. In this section, we review relevant work in this area, which helps frame our approach discussed in Section 4.1.Much of the related work involves using the co-occurence matrix from Skip-gram. For the remainder of this section, let X be the count co-occurence matrix. Early approaches to finding low-dimensional embeddings of words relied on the singular value decomposition [,]. These methods would truncate the singular value decomposition by zeroing out the small singular values. BID7 show that this is equivalent to using an objective function J which is invariant to orthogonal transformation. For simplicity, we specialize to the Frobenius norm and say these early approaches find arg min DISPLAYFORM0 F is the objective function and X is the co-occurence matrix. The co-occurence matrix and the loss function for Skip-gram can be read off from problem: the co-occurence matrix is X and the objective function is written in problem with u u u T i v v v j replaced by m ij. BID4 find a probabilistic interpretation of this loss function related to a Multinomial distribution, but do not take advantage of it and only replace the inner product with a (higher dimensional) variant, somewhat similar to the approach in Tifrea et al.. Mikolov et al. [2013a] introduce Skip-gram with negative sampling (SGNS), a variant of Skip-gram. If we view Skip-gram as maximizing the true positive rate of predicting a word will appear within a context window of another word, we can view SNGS as maximizing the true positive rate plus k times an approximation of the true negative rate. When k = 0, Skip-gram and SGNS coincide. BID19 use a heuristic argument to interpret SGNS as using a co-occurence matrix that is a shifted PMI matrix.2 However, they did not determine the objective function. Later, Li et al. and Landgraf and Bellay explicitly identified both the co-occurence matrix and the objective function. They find a different co-occurence matrix than BID19, one that does not depend on k, while their loss function does depend on k. Surprisingly, they establish that SGNS is finding a low-rank matrix related to X, the same matrix that Skip-gram uses. The loss function is w,w i,j=1 DISPLAYFORM1 2 Define the total number of times word wi appears to be xi· = w j=1 xij, the total number of times context wj appears to be x·j = w i=1 xij, and the total number of words to be x·· = w,w i,j=1 xij. The shifted PMI matrix has entries log DISPLAYFORM2 Landgraf and Bellay explain that this loss function has a probabilistic interpretation, and they use that interpretation to recover the shifted PMI matrix as a prediction from within their model. The approach in this paper will be to view the entries of the co-occurence matrix as random variables and introduce an objective function via the likelihood of that random variable. Our approach is most similar to Landgraf and Bellay and, to a lesser extent, BID4. In order proceed, some in probabilistic modeling and estimation needs to be developed. In this section, we review iteratively reweighted least squares (IRLS) for generalized linear models and review generalized low rank models []. Further (and notation) in exponential dispersion families and generalized linear models is developed in Section A. Generalized linear models (GLMs) are a flexible generalization of linear regression where the mean is a not necessarily linear function of a coefficient β β β and the response has an error distribution which is an exponential dispersion family. The coefficient β β β is unknown and a target of estimation. The standard approach to estimate β β β is maximum likelihood estimation [, Section 7] to produce the maximum likelihood estimator, or MLE,β β β. A computational approach to find the MLE is through Fisher scoring, a variant of Newton's method on the log likelihood which uses the expectation of the Hessian in place of the Hessian [, Section 4.5]. Define (β β β) to be the log likelihood. Specifically, Fisher scoring produces a sequence of estimates {β β β DISPLAYFORM0), where ∇ is the gradient and D 2 is the Hessian. Upon plugging in the gradient and expected Hessian for an exponential dispersion family, a surprising identity emerges: each iteration of Fisher scoring is equivalent to minimizing a weighted least squares objective: DISPLAYFORM1 where the weight H (t) and pseudo-response z (t) at iteration t have DISPLAYFORM2 DISPLAYFORM3, and µ DISPLAYFORM4 Principal components analysis BID13 is one well-known method for finding a low rank matrix related to X ∈ R w×c. In principal components analysis, we model x ij ind. DISPLAYFORM0 A maximum likelihood estimator for u u u i is taken to be a low-dimensional embedding of the i th row of X. The low-dimensional embedding enables interpretability and reduces noise. However, data cannot always be viewed as being drawn from a normal distribution, so it's necessary to extend the method of principal components to non-normal data. The extension can be made in a manner similar to the extension from linear models to generalized linear models: the new model, called a generalized low rank model [] allows us to estimate model-based low-dimensional embeddings of non-normal data. Definition 1 For some exponential dispersion family ED(µ, ϕ) with mean parameter µ and dispersion parameter ϕ, the model for X ∈ R w×c is a generalized low rank model with link function g when DISPLAYFORM1 DISPLAYFORM2 where u u u i, v v v j ∈ R d are the rows of matrices U ∈ R w×d and V ∈ R c×d, respectively, and a a a ∈ R w and b b b ∈ R c are bias (or offset) terms. The difference between the generalized low rank model and the generalized linear model is in the systematic component in equation FORMULA12. Here, the data is modeled as having its link-transformed mean be a matrix with rank at most d. This formalizes the way in which we relate the co-occurence matrix X to a low rank factorization. When the link function g is taken to be canonical, the generalized low rank model is identical to ePCA BID3. The generalization is worthwhile since the canonical link can be inappropriate, as we will see, for instance, in Section 5.1. We now present a method to find word vectors. A key innovation in the method is an iterative algorithm inspired by IRLS to find a maximum likelihood estimator in a generalized low rank model. Our method has three steps:Step 1 Choose a co-occurence matrix X ∈ R w×c to summarize the document. (Note, in many cases c = w so that the "contexts" are just the words.)Step 2 Choose a plausible exponential dispersion family to model the entries of the co-occurence matrix. Choose a corresponding link function. Step 3 Choose a number of iterations r to run IWLRLS (Algorithm) with the input specified above to output word vectors. DISPLAYFORM0; Evaluate the least squares problem arg min DISPLAYFORM1 Algorithm 1: Iteratively weighted low rank least squares (IWLRLS) algorithm for GLRMsThe first step of our method processes the corpus in order to extract the linguistic information. Some co-occurence statistics use more information than others: for instance, the harmonic co-occurence matrix makes use of the number of tokens between words while the count co-occurence matrix does not. A typical tuning parameters here is the length l c of the context window. We view this step as involving a "linguistic" choice. The second step specifies a distribution for the co-occurence matrix. A distribution can be considered as plausibly corresponding to reality if it can be derived by a connection to the corpus. In our Co-occurence: Harmonic Table 1: The rows refers to the number of steps of IWLRLS. A "·" represents no existing work. All filled-in positions in the lowest row were established in previous work. framework, the model is explicit: this is helpful since knowing a model provides interpretation for its output [, Section II.A.]. The choice of distribution will often determine, through convention, the link function, so the link function often does not need to be separately chosen. We view this step as involving a "statistical" choice. DISPLAYFORM2 The third step runs IWLRLS, a generalized version of IRLS. Recall that IRLS is derived by iteratively maximizing a second order Taylor expansion of the likelihood as a function β. The Taylor expansion is centered at the previous iterate. IWLRLS can be derived by iteratively maximizing a second order Taylor expansion of the likelhood as a function of η subject to the constraint 6. We view this as a "computational" choice that we fix in advance. In the following subsections, we run through many examples of our method as it would be used in practice. There are two distinct choices of co-occurence matrices that are made. Various choices of distributions recover common methods for finding word vectors. An altogether new estimator is proposed via an improvement of the assumed distribution in Skip-gram. Casting these estimators in this general framework provides an interpretation and understanding of them: we make explicit their assumptions and therefore know the driver of their behavior. We will apply our proposed method under the choice of the harmonic co-occurence matrix and the Tweedie distribution: one iteration of IWLRLS will recover GloVe. Step 1 The first step of our method is to pick a co-occurence matrix that summarizes the corpus. We choose the harmonic co-occurence matrix X ∈ R w×w.Step 2 Now we must determine a plausible distribution for the co-occurence matrix that is an exponential dispersion family. Recall that the Tweedie distribution has the property mentioned in equation FORMULA0 that it is a sum of Poisson many independent Gamma distributions. An informal way to write this is that DISPLAYFORM0 We argue that the Tweedie distribution is reasonable by connecting the Poisson and Inverse Gamma distributions displayed above to attributes of the corpus. Intuitively, it is reasonable that the number of times word w i and word w j co-occur within the corpus can be modeled as having a Poisson distribution. Another choice of distribution is that of an Inverse Gamma distribution for the number of tokens between word w i and word w j at some co-occurence, although it is an approximation as the number of tokens is an integer while the Inverse Gamma is supported on non-integers. Instead of using the canonical link function, we will take g(µ) = log µ, which is standard []. A problem with the canonical link function preventing its use is that its range is nonpositive. Step 3 Next, we find the form of the weight H and the pseudo-response Z that the Tweedie distribution provides. This amounts to plugging in the cumulant generating function ψ that is given in Section A.1. This in DISPLAYFORM1 When the algorithm is initialized withμ = X, the pseudo-response simplifies to z ij = log x ij.Taking the power p = 1.25, the weight simplifies to x 3/4 ij. In summary, we've shown that: Result 1 Inputting the harmonic co-occurence matrix, the Tweedie distribution with power p = 1.25, the log link, and the number of iterations k = 1 into IWLRLS in GloVe (without the algorithmic regularization induced by truncating the weights.)Given this connection, we can extend GloVe for several iterations rather than one or even use the full likelihood. We experiment with this using real data examples in Section 6. This shows that even though the first iteration does not depend on word pairs where x ij = 0, later iterations do. We now consider an alternative first step: we choose another co-occurence matrix to summarize the corpus. Then, we make multiple possible choices for step 2 to illustrate connections to previous work that step 3 recovers. Various choices for step 2 will recover the SVD BID17 ], Skip-gram [a], a new estimator which is a distributional improvement over those, and Skip-gram with negative sampling [b].Step 1 We choose the count co-occurence matrix. Step 2 A proposed distribution for the entries of X is the Gaussian distribution. This may not be the best choice, since the entries of X are non-negative integers. As is usual, we take the link function to be g(µ) = µ. We restrict the systematic component to not include the bias terms, so that η ij = u u u DISPLAYFORM0 Step 3 We showed in Section A.1 that the cumulant generation function from the normal distribution is ψ(θ) = 1 2 θ 2. This makes it so that DISPLAYFORM1 In other words, the IWLRLS algorithm will always converge in one iteration, so our method recovers the method of computing a truncated SVD of X by BID7.Another choice that could have been made in step 2 is to have the link function g(µ) = log µ. This still may not be the best choice since the normal distribution still has the same problems as before. Step 2 Another proposed distribution for the entries of X is a Multinomial distribution. Specifically, we could propose that the the row of X corresponding to word w i has the distribution x x x i ∼ Multinomial w j=1 x ij, π π π, where π π π ∈ R w is vector of probabilities of word w i appearing within a context window with the other words and w j=1 x ij is the total number of times word w i appears in the corpus. We take the link function to be the multi-logit. 3 Cotterell et al. show that the objective function of Skip-gram coincides with the likelihood of this model when the bias terms are removed, so that the systematic component η ij = u u u Step 3 The Poisson trick BID2 can be used to reduce estimation in a Multinomial model to estimation in a particular Poisson model. LetÛ,V be the maximum likelihood estimators in the Multinomial generalized low rank model described in step 2. Using this trick, it holds thatâ a a, (the same)Û, and (the same)V are maximum likelihood estimators in a Poisson generalized low rank model with independent responses x ij and systematic component DISPLAYFORM0 Notice that there is only one bias term. The weight and pseudo-response are DISPLAYFORM1 In the previous subsubsection, we saw that the choice of Multinomial model implicitly gives rise to a Poisson model with a systematic component given by equation FORMULA20. Since it could be most appropriate to have bias terms for both rows and columns due to the symmetry of the co-occurence matrix, we directly introduce a Poisson estimator with a non-restricted systematic component. Step 2 Another proposed distribution is a Poisson. Due to the "law of rare events" [, Section 3.6 .1], this is a plausible model. We use the canonical link function g(µ) = log µ.Step 3 The cumulant generating function is ψ(θ) = exp(θ) BID0, so that the weight and pseudo-response are given by equations. BID1 propose an estimator which is a close variant of one iteration of IWLRLS. At one point in their derivation, they (using our notation) take η ij = u u u i − v v v j 2 2 + c, where c is an arbitrary constant which does not depend on the word. This is inspired by their theorem 2.2. On the other hand, taking η ij = u u u T i v v v j + a i + b j (as in equation 6) in their derivation recovers one iteration of IWLRLS. The Negative-Binomial distribution is commonly used as an alternative for the Poisson in the presence of over-dispersion, which is the case when the variance is higher than the mean. It produces the same weight and pseudo-response as the Poisson. Step 2 We model x ij ind. DISPLAYFORM0 is an inflated count, η ij = u u u T i v v v j, and k ≥ 0. Landgraf and Bellay showed that a maximum likelihood estimator from this model with canonical link g(π) = log π 1−π is identical to a SGNS estimator. Step 3 The cumulant generating function for the binomial distribution is ψ(θ) = log(1 + exp θ), so the weight and pseudo-response are: DISPLAYFORM1 In Section 4.1 we introduced the IWLRLS algorithm to compute word vectors such as those produced by GloVe or SGNS. We now conduct quantitative evaluation experiments on an English word analogy task, a variety of word similarity tasks [a] to demonstrate the performance of the algorithm. First, in Section 6.1 we introduce the analogy similarity task for evaluating word vectors. In Section 6.2 we present of the algorithm with different distributions according to those presented in Section 5.1 and 5.2. In Section B.1 we provide parameter configurations and training procedures, and in Sections B.2-B.5 we present of IWLRLS in numerous scenarios showcasing improvement through multiple iterations and robustness to other model parameters. We introduce the word analogy task following the presentation of []. The word analogy task is a dataset of 19, 544 statements of the basic form "a is to b as c is to __", which are divided into a semantic and syntactic subsets. The semantic statements are typically analogies relating to people, places, or nouns such as "Athens is to Greece as Berlin is to __", while the syntactic questions relate to verb or adjective forms such as "dance is to dancing as fly is to __". The basic analogy statement is answered by finding the closest vector u u u d to u u u b − u u u a + u u u c 4 in the embedding space via cosine similarity 5. The task has been shown under specific assumptions to be provably solvable by methods such as GloVe and Skip-gram BID8 BID12 ] and as such is closely related to solving the objectives introduced in Sections 1 and 4.1. In this section, of the IWLRLS algorithm are performed for the Tweedie, Multinomial, and Poisson models. Based on the additional experiments in Sections B.2-B.6 we train the Tweedie model with p = 1.25 (Section B.3) and for all models include weight truncation to penalize large co-occurrences (Section B.4), regularization terms (outlined in Section B.5), and include only a single bias term within the systematic component of the Tweedie model (Section B.6).Step To demonstrate the effectiveness of performing multiple iterations of the IWLRLS algorithm, we present for the one-step estimator, an early-stopped estimator, and the full-likelihood estimator. Of particular interest in our are the Tweedie one-step estimator (a variant of the GloVe method), and the full-likelihood estimator for the Multinomial (a variant of the Skip-gram method). For the in TAB1, the full-likelihood is taken to be the iteration which achieves the maximum total accuracy on the analogy task, and the early-stop algorithm is taken to be an iteration between the one-step and full-likelihood iterations which performs best in total accuracy on the analogy task. For both the Tweedie and Multinomials, the full-likelihood is the after 3 iterations and the early-stopped is the after 2 iterations. For the Poisson model, the full-likelihood is the after 9 iterations, and the early-stopped is the after 3 iterations. We find a small difference in total accuracy on the analogy task with the one-step estimator (GloVe) and the full-likelihood differing by roughly 1%. We find a similar relationship in the Poisson estimator and further note that the early-stopped estimator for the Poisson has very similar accuracy to the full-likelihood algorithm. Finally, the Multinomial model yields a difference of 2% between the full-likelihood algorithm (Skip-gram) and the one-step algorithm. The early-stopped algorithm for the Multinomial also performs 1% higher than the one-step algorithm indicating a fair tradeoff between running an additional iteration and stopping after only one iteration. We present a general model-based methodology for finding word vectors from a corpus. This methodology involves choosing the distribution of a chosen co-occurrence matrix to be an exponential dispersion family and choosing the number of iterations to run our algorithm. In Table 1, we see that our methodology unifies the dominant word embedding methods available in the literature and provides new and improved methods. We introduce an extension of Skip-gram that is stopped before full-convergence analagously to GloVe and an extension to GloVe beyond one iteration. Experimental on a small corpus demonstrate our method improves upon GloVe and Skip-gram on the Google word analogy similarity task. It is our hope that this methodology can lead to the development of better, more statistically sound, word embeddings and consequently improve on many other downstream tasks. Further in exponential dispersion families and generalized linear models is developed here. We begin by discussing exponential dispersion families, the distribution of the response in generalized linear models. Definition 2 Let y ∈ R be a random variable. If the density function f (y; θ, ϕ) of y satisfies DISPLAYFORM0 over its support, then the distribution of y is in the exponential dispersion family. The parameter θ is the natural parameter, ϕ is the dispersion parameter, and the function ψ is the cumulant generating function. In many cases, the function δ(ϕ) is very simple, meaning that, for instance, δ(ϕ) = 1 or δ(ϕ) = ϕ.The function c(y; ϕ) can be viewed as the normalizing constant ensuring that the density integrates to one. When y follows a distribution in the exponential dispersion family with natural parameter θ, its mean µ = ψ (θ), so we can equivalently specify the mean µ or the natural parameter θ. Many classical distributions such as the Poisson, Normal, Binomial, and Gamma distribution are exponential dispersion families. For example, when y ∼ Normal(µ, σ 2) is a normal distribution with mean µ and variance σ 2, its log density satisfies DISPLAYFORM1 showing that here the natural parameter θ = µ, the dispersion parameter ϕ = σ 2, the functions ψ(θ) = 1 2 θ 2, δ(ϕ) = ϕ, and c(y; ϕ) = 1 2 log(2πσ 2) + y 2 2σ 2. The Tweedie distribution BID15, of particular importantance to us, also lies within the exponential dispersion family. Instead of defining the Tweedie distribution through the form of its density, we will define it through the relationship between its mean and variance. This relies on a from [Jørgensen, 1987, Theorem 1] that distributions within the exponential dispersion family are defined by the relationship between their mean and variance. Definition 3 A random variable y has a Tweedie distribution with power parameter p ∈ {0} ∪ [1, ∞) when DISPLAYFORM2 and the distribution of y is an exponential dispersion family. In this case, we write y ∼ Tweedie p (µ, ϕ), where µ = E(y) is the mean. The Normal distribution discussed above has a variance that does not depend on the mean. In our new notation, this means that the Normal distribution is a Tweedie distribution with power parameter p = 0. The Poisson distribution has variance equal to the mean and is in the exponential dispersion family, so is a Tweedie distribution with power parameter p = 1 and dispersion parameter ϕ = 1. A Gamma distribution with shape parameter α and rate parameter β is a Tweedie distribution with power p = 2, mean µ = α β, and dispersion parameter ϕ = α −1.We will only consider Tweedie distributions with power parameter p ∈. These distributions are also known as compound Poisson-Gamma distributions due to the representation DISPLAYFORM3 where n ∼ Poisson(λ) and g i DISPLAYFORM4 ∼ Gamma(α, β), and λ = BID15. It is important to note that the Tweedie distribution has positive mass at zero, an important characteristic for capturing the zero-inflation prominent in some co-occurence matrices due to some words never appearing within the same context. Specifically, DISPLAYFORM5 Using other arguments related to representations of the mean and variance in terms of the cumulant generating function ψ, BID15 show that the Tweedie distribution has ψ(θ) DISPLAYFORM6 Exponential dispersion families are defined over real numbers. Now, we generalize their definition to a multivariate setting. Definition 4 Let y y y ∈ R m be a random vector. If the density function f (y y y; θ θ θ, ϕ) of y y y satisfies log f (y y y; θ θ θ, ϕ) = y y y DISPLAYFORM0 over its support, then the distribution of y y y is in the multivariate exponential dispersion family. The parameter θ θ θ ∈ R m is the natural parameter, ϕ ∈ R is the dispersion parameter, and the function ψ: R m → R is the cumulant generating function. A collection of independent draws from the same exponential dispersion family is a multivariate exponential dispersion family. To see this, let y i (i = 1, . . ., m) be i.i.d. from an exponential dispersion family. Then, the density of y y y satisfies log f (y y y; θ θ θ, ϕ) = DISPLAYFORM1 + c(y j ; ϕ), which has cumulant generation function ψ(θ θ θ) = m j=1 ψ(θ j). Another useful example of a multivariate exponential dispersion family is the Multinomial. Let x x x ∈ R c have be distributed as x x x ∼ multinomial(s, π π π), where s ∈ N is the total number of draws and π π π ∈ R c is the probability vector. Introduce a change of parameters where DISPLAYFORM2 showing that the multinomial distribution is in the multivariate exponential dispersion family with ψ(θ θ θ) = s log (c k=1 exp θ k). We start by reviewing the linear model. Given a response y y y ∈ R n comprising n observations, the model for y y y is a linear model with covariates x x x i ∈ R p when y i ind.∼ Normal(x x x T i β β β, σ 2) for all i ∈ {1, . . ., n}. In vector notation, this reads that y y y ∼ Normal(Xβ β β, σ 2 I), where X ∈ R n×p is a matrix with i th row x x x T i. This is one of the more primitive models in the statistical modeling toolbox and isn't always appropriate for the data. Generalized linear models remove the the assumptions of normality and that the mean is a linear function of the coefficients β β β. Definition 5 For some exponential dispersion family ED(µ, ϕ) with mean parameter µ and dispersion parameter ϕ, the model for y y y ∈ R n is a generalized linear model with link function g when DISPLAYFORM0 DISPLAYFORM1 In the first line of the displayed relationships, the distribution of the response y y y is described. In the third line, the systematic component η i expresses the effect of the covariates x x x i. The second line connects the distribution to the covariates through the link function. That is, the covariates effect a link-transformed mean. The canonical link (b) −1 is often chosen as the link function, due to its computational and mathematical properties BID0. Other times, the canonical link is inappropriate and there are alternative default choices. Generalized linear models are used as the default modeling framework in many fields of applied science for non-normal distributions []. When g(µ) = µ is the identity map and ED is the Normal distribution, the generalized linear model is simply the linear model. When g(µ) = logit(µ) = log 1−µ µ and ED is the Binomial distribution, the generalized linear model is logistic regression. Further, a generalized linear model can be viewed as a no-hidden-layer neural network with activation function g. We train our models on the English text8 corpus 6 with approximately 17 million tokens. We filter out word types that occur fewer than 50 times to obtain a vocabulary size of approximately 11, 000; a ratio consistent with other embedding literature experiments 7.The adjustable model configurations in IWLRLS are the choice of power parameter p, penalty tuning parameter λ, and co-occurrence processing step. We experiment with different choices of p ∈ {1.1, 1.25, 1.5, 1.75, 1.9}, different choices of processing including no processing, clamping the weights (as in GloVe) and truncating the outliers in the co-occurrence matrix (elaborated on in Section B.4, and set the penalty tuning parameter λ = 0.002. The estimated word vectors are the rows of DISPLAYFORM0 . For all of our experiments, we set the dimension of the word vectors to d = 150, and the objective function at each iteration is optimized using Adagrad BID5 with a fixed learning rate of 0.1 8 . Models are trained for up to 50 epochs (50 passes through the co-occurrence matrix) with batches of size 512. We evaluate the impact of multiple iterations of the IWLRLS algorithm on all models, but examine different additions to the model only when p = 1.25. We believe the impact of these changes will be present however for any value of p. We present of multiple iterations of our IWLRLS algorithm with different distributions. In particular, we perform multiple iterations of the IWLRLS algorithm with Tweedie distribution and weight truncation to match the GloVe objective function and processing by setting the weight function in our model from h(x) = x 2−p to h(x) = (min{x, x max}).75 with x max = 10 and p = 1.25. We also presents for an early-stopped version of skip-gram, and the new Poisson estimator. The are summarized in FIG4. We remark on a few observations based on these . First, as the number of steps increases, the accuracy on the analogy task increases for the first few iterations. Second, relatively few steps are needed with the accuracy of Tweedie model performing best at the first and second steps of the algorithm, and the Multinomial and model performing best in steps 3-5 but with very similar performance at earlier steps. The Poisson model performs best after 9 iteration, however performs nearly identically to the of an early stopped algorithm at 3 iterations. In , we find that early-stopped and one-step versions of the algorithm can perform comparably to full-likelihood methods. In this section, we examine the effect of the choice of the power p in the tuning parameter when you run a Tweedie generalized low rank model. Table 3: Results for multiple choices of p for one and two iterations. The Results in Table 3 show that values of p which are high perform poorly, while values of p below 1.5 perform similarly. We find that p = 1.25 performs the best, and view this value of p as a good choice as it accounts for zero-inflation present in the co-occurence X. This also agrees with the of [] and BID16 ].An even more interesting and perhaps more appropriate way to estimate the power p of the Tweedie distribution is in a data-driven and model-based way. This approach is taken in BID16. In future work, we plan to use an improved estimating equation relative to BID16 ] to estimate p as part of the algorithm. This would be modeling the marginal distribution of the co-occurences as being Tweedie with the same power. Under a similar assumption, modified likelihood calculations are tractable and so are another possibility. We plan to explore this in future work. We set p = 1.25 in our algorithm with Tweedie distribution, and explore the effect of different strategies in handling large entries in the co-occurrence matrix X. One strategy is to simply input X into step 3 of our method. A second strategy is to clamp the weight h(·) that from step 3 of our method by taking h(x) = (min{x, x max}).75 as in GloVe. A third strategy is to input min{x, x max} for each entry of the matrix X, where x max = 10, into step 3 of our method 9.We find that no adjustment to the weights and GloVe's method of weight truncation both perform similarly with weight truncation slightly outperforming no adjustment. We suspect a more significant improvement will show with larger corpora such as a full Wikipedia corpus. Alternative approaches to alleviating the problem of large co-occurences are to use a more robust distribution or link function. Indeed, the weight truncation in GloVe can be directly mimicked by Table 4: Results for multiple choices of regularizing the large values of the co-occurrence matrix. Our strategies are harmonic matrix, truncation of the weight only, truncation of the co-occurrence matrix to x max = 10.either altering the distribution or the link function. The desired form can be found via the weight and pseudo-response equations in algorithm 1. We leave this to future work. Table 5: Results for including the penalty term in Equation FORMULA0 and not including the diagonal terms. We consider regularizing the word vectors by including the penalty DISPLAYFORM0 with λ =.002 for two reasons. One is to reduce noise in the estimation of the word vectors. Udell et al. [2016, Lemma 7.3] show that penalizing by is equivalent to penalizing by λ U V T *, the nuclear norm of U V T. Since penalizing the nuclear norm U V T shrinks the dimension of the embedding and larger dimensional embeddings tend to be better [], we choose a small tuning parameter to reduce noise while still preserving the dimension. Another reason is to symmetrically distribute the singular values ofÛV T to both matricesÛ andV.Write the singular value decompositionÛV T = U ΣV T, for U and V orthogonal and Σ diagonal. Mu et al. [2018, Theorem 1] shows that using penalty in havingÛ = U Σ 1/2 Q and V = V Σ 1/2 Q for some orthogonal matrix Q. This is desirable since it was argued empirically by Levy et al. that a symmetric distribution of singular values works optimally on semantic tasks. Finally, we experiment with whether the penalty introduced in Equation FORMULA0 improves and accurately reduces noise in the estimate. We also consider not including the diagonal elements of X as a form of regularization and experiment here as well, as these terms are often large (can be considered as outliers) and do not contain a great deal of linguistic information. Table 5 demonstrates the included regularization within the IWLRLS algorithm with Tweedie distribution and p = 1.25 improves . In Experiment 1, we found that the Multinomial model outperforms the Poisson model, although the Poisson model has an additional bias term to model context word frequencies. This was fairly counterintuitive, so we additionally experiment with having only a single bias term a i in the Tweedie model as in the Multinomial model. We find overall that the Tweedie model with a systematic component without the bias term b j performs slightly better than the Tweedie model with systematic component containing both bias terms a i and b j. We hope to further study the impact of bias terms and other systematic components in future work. Table 6: Results for including the bias term on the context word b j in addition to a i.

We present a novel iterative algorithm based on generalized low rank models for computing and interpreting word embedding models.

Represent the TLDR for finding the corresponding scientific article!

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question: Represent the scientific article to find a short TLDR

Question: Deterministic models are approximations of reality that are often easier to build and interpret than stochastic alternatives. Unfortunately, as nature is capricious, observational data can never be fully explained by deterministic models in practice. Observation and process noise need to be added to adapt deterministic models to behave stochastically, such that they are capable of explaining and extrapolating from noisy data. Adding process noise to deterministic simulators can induce a failure in the simulator ing in no return value for certain inputs -- a property we describe as ``brittle.'' We investigate and address the wasted computation that arises from these failures, and the effect of such failures on downstream inference tasks. We show that performing inference in this space can be viewed as rejection sampling, and train a conditional normalizing flow as a proposal over noise values such that there is a low probability that the simulator crashes, increasing computational efficiency and inference fidelity for a fixed sample budget when used as the proposal in an approximate inference algorithm. In order to compensate for epistemic uncertainty due to modelling approximations and unmodeled aleatoric uncertainty, deterministic simulators are often "converted" to "stochastic" simulators by randomly perturbing the state at each time step. In practice, models adapted in this way often provide better inferences (Møller et al., 2011; ; ; ;). State-independent white noise with heuristically tuned variance is often used to perturb the state (; ; ; ; ; ;). However, naively adding noise to the state will, in many applications, render the perturbed input state "invalid," inducing failure (; ;). These failures waste computational resources and reduce sample diversity, worsening inference performance. Examples of failure modes include ordinary differential equation (ODE) solvers not converging to the required tolerance in the allocated time, or, the state crossing into an unhandled configuration, such as solid bodies overlapping. Establishing the cause of failure is non-trivial and hence, the simulation artifact can be sensitive to seemingly inconsequential alterations to the state -a property we describe as "brittle." The principal contribution of this paper is a technique for minimizing this failure rate. We proceed by first framing sampling from brittle simulators as rejection sampling. We then eliminate rejections by learning the state-dependent density over perturbations that do not induce failure, using conditional autoregressive flows . Doing so renders the joint distribution unchanged and retains the interpretability afforded by the simulator, but improves sample efficiency. We show that using the learned proposal increases the fidelity of the inference attainable on a range of examples. We denote the brittle deterministic simulator as f: X → {X, ⊥}, X = R D, where a return value of ⊥ denotes failure. Over the whole support, f defines a many-to-one function, as many states map to ⊥, however we only require that f is one-to-one in the accepted region, a condition satisfied by ODE models. A stochastic, additive perturbation to state, denoted z t ∈ X, is proposed such that x t ← f (x t−1 + z t), z t ∼ p(·|x t−1), although this is often state independent. We include more detailed derivations in Supplementary Materials Section B. The naive approach to iterate the perturbed system is to repeatedly sample from the proposal distribution and evaluate f until the simulator successfully exists. We begin by showing that this process defines a rejection sampler. The use of f and p(z t |·) implicitly specifies a distribution over successfully iterated states, p(x t |x t−1); and consequently a second distribution over accepted perturbations, denoted p(z t |x t−1), which, under the process outlined above, can be written as: p(z t |x t−1) = 1 Mp p(z t |x t−1), if f (x t−1 + z t) = ⊥ 0, otherwise where the normalizing constant M p is the acceptance rate under p. In regions that fail the sample is rejected with certainty. In the accepted region,p ∝ p, which is a sufficient condition for a rejection sampler to be valid, without needing to evaluate M p orp. This represents a rejection sampler with an acceptance rule of I [f (x t−1 + z t) = ⊥] targetinḡ p(x t |x t−1). We now seek to learn a proposal distribution over z t values conditioned on the current state x t−1, denoted q φ, parameterized by φ, to replace p but placing no mass on regions that are rejected, ing in an acceptance rate tending to unity. We denote q φ as the proposal we train, which, coupled with the simulator, implicitly defines a proposal over accepted samples, denoted q φ. We wish to minimize the distance between joint distribution implicitly specified over accepted iterated states using p, f and q φ, amortized across state space : Expanding the Kullback-Leibler divergence (D KL), applying a change of variables yields, and noting that the Jacobian terms can be cancelled yields: However, q φ is defined implicitly after rejection sampling, and so we can adapt for q φ and substitute back into. Differentiation of the acceptance rate (M q φ) is intractable. However, noting that φ * is a maximizer for both q φ andq φ, we can instead optimize q φ (z t |x t−1): By removing the rejection sampler as we have, we have implicitly specified that the proposal distribution must have an acceptance rate of one. This term is differentiable with respect to φ and so we can maximize this quantity using stochastic gradients. Importantly the flow is not explicitly trained to maximize the acceptance rate. The flow is trained to minimize the KL divergence between the implicitly specified distribution over accepted samples and the learned proposal distribution. Accordingly the flow retains the shape of the proposal distribution in regions of state space that do not yield failure (this can be seen by comparing red and green contours in the interior of the dashed lines in Figure 1a) and hence the learned distribution cannot collapse to add trivially small perturbations, as would be the case if we had directly optimized for high acceptance rates. By exploiting change of variables we are able to "project" back through the rejection sampling procedure and hence we can optimize q φ as we do not need to compute the derivative of the rejection rate as we would have otherwise needed to do. Finally, we note that generation of data training data and learning of the autoregressive flow is a computationally intensive procedure. However simulators can take on the order of seconds to iterate and so the intention of this work is to create a technique that maximizes computational efficiency when deployed. Sampling from the flow takes on the order of milliseconds, can be accelerated using GPUs and scale favourably in the number of samples being produced. Furthermore, the training procedure is performed once and hence represents an offline, one-off cost exchanged for higher efficiency deployment. The training data can also be generated using large-scale distributed computing (as the mini-batching process is inherently embarrassingly parallelizable) that may not be available or practical for use at deployment time. Implementation We use a conditional masked autoregressive flow as the structure of q φ, with 5 single-layer MADE blocks , 256 hidden units per layer and batch normalization layers at the input to each intermediate MADE block. The dimensionality of the flow is the number of states perturbed in the original model. To introduce conditioning we use the current state vector, x t−1, as input to a hypernetwork that outputs the parameters for each layer in the flow. The networks are implemented in PyTorch , and optimized using ADAM . We demonstrate on two examples here, and an additional two experiments in the appendix. In these experiments we first aim to demonstrate that learning the required conditional autoregressive flow is tractable and faithfully represents the conditional distribution over accepted perturbations. We then use the learned proposal in a particle-based sequential (c) Figure 1: Results for the annulus problem introduced in Section 3.1. 1a indicates the permissible region as a black dashed band, where p and q φ is shown in red and green respectively. The shape of q φ is the same as p inside the band, with little mass outside of the band. This shows the flow has learnedp effectively with a low rejection rate. 1b confirms q φ all-but eliminates rejection. 1c shows the reduction in the variance of the evidence across 100 independent sequential Monte Carlo sweeps of 100 independent datasets. Monte Carlo state-space inference scheme and show that lower-variance inference can be obtained for a fixed sample budget. In this example, the (unknown) true generative model of the observed data is a constant speed circular orbit around the origin in the x-y plane. We perform inference using a misspecified model that only simulates constant velocity forward motion, such that x t ∈ R 4, with Gaussian perturbations to position and velocity. We impose a failure constraint limiting the change in the distance of the point from the origin to a fixed threshold. This condition mirrors the notion that states in brittle simulators have large allowable covariances in particular directions, but very narrow permissible perturbations in other directions. Figure 1a and Figure 1b shows q φ has effectively learnedp(z t |x t−1), reducing rejection rate under q φ to less than 4% compared to approximately 75% under p. We then use the learned q φ as the proposal in a particle filter , an approximate inference algorithm often applied to posterior inference in time-series models. We use a fixed sample budget and hence failed samples are discarded, without retrying a new sample from the proposal. The in Figure 1c show that we are able to recover lower variance evidence approximations using q φ compared to p, achieving a paired t-test score of < 0.0001. This experiment confirms we are able to learn a proposal that incurs lower rejection, and that reducing the rejection rate increases fidelity of inference (for a fixed computational budget). We now apply our method to the robotics simulator MuJoCo , using the built-in example "tosser." MuJoCo allows some overlap between solid objects to simulate the contact dynamics. This is an example of model misspecification borne out of the requirements of reasonably writing a simulator. We therefore place a hard limit on the amount objects are allowed to overlap. We add Gaussian perturbations to the state. Results of the "tosser" experiment introduced in Section 3.2. 2a shows a typical state evolution. 2b shows the conditional autoregressive flow we learn markedly reduces the number of rejections. 2c shows the of performing sequential Monte Carlo using p and q φ. 2d shows the of performing hypothesis testing, where the correct hypothesis not selected using p, but is using q φ. Figure 2 shows the of this experiment. Collisions are generally rare events and hence the rejection rate of p is just 10%. Figure 2b shows that the autoregressive flow learns a proposal with a significantly lower rejection rate, reaching 3% rejection. However these rejections are concentrated in the critical regions of state-space and so this reduction yields an large reduction in the variance of the evidence approximation, as shown in Figure 2c. We conclude by applying our method to hypothesis testing, selecting the mass of the capsule. Shown in Figure 2d, using p in higher variance evidence approximations than when q φ is used, causing p to select the wrong model, with a reasonable level of significance (p = 0.125), while using q φ selects the correct hypothesis with p = 0.0127. In this paper we have tackled reducing simulator failures caused by naively perturbing the input state. We achieve this by defining these simulators as rejection samplers and learning a conditional autoregressive flow to estimate the state-dependent proposal distribution conditioned on acceptance. We show that using this learned proposal reduces the variance of inference when used as the proposal in a subsequent approximate inference scheme. This work has readily transferable practical contributions in the scientific community where naively modified simulation platforms are widely deployed. Journal of basic Engineering, 82 Appendix A. Background Deterministic simulators are often stochastically perturbed to increase the diversity of the achievable simulations and to fit data more effectively. White noise perturbation to time series systems is common, such as the widely used ARMA models . The most straightfoward example of this however is the widely used Kalman filter . The Kalman filter, at its core, is determinstic transition model which is then perturbed with additive Gaussian noise. The form of the process and noise kernels are chosen such the system has a closed form representation. Without the additive process noise, the Kalman filter is deterministic and would be unable to represent the variability in the real-world. More complex systems cannot be analyzed in closed form like the Kalman filter. Accordingly deterministic simulators of the dynamics with stochastic perturbations and numerical methods are used in practice. Specific examples of such systems that are: stochastic Hodgkin Huxley models of neural dynamics (; ; ;), computational finance analysis of asset prices (; ;), predator-prey dynamics , epidemiology and mobile robotics . As simulators become more complex, guaranteeing robustness is more difficult, and individual function evaluations are more expensive. and establish the sensitivity of earth science models to static input parameter values by building a discriminative classifer for parameters that induce failure. take an alternative approach instead treating simulator failure as an imputation problem, fitting a function regressor to predict the outcome of the failed experiment given the neighbouring experiments that successfully terminated. However these methods are limited by the lack of clear probabilistic interpretation in terms of the originally specified joint distribution in time series models and their ability to scale to high dimensions. Autoregressive flows (AFs) are a flexible class of density estimators. AFs define a density, q φ (x), trainable using stochastic gradient descent to approximate the target distribution p(x), by minimizing the KL-divergence between the target distribution and the approximation: AFs operate by transforming samples from a "base distribution" through a series of learned warpings, interpretable as a change of variables, into samples distributed according to the target distribution. The flow layers are designed such that the required Jacobians and inverses are cheaply computable. A popular flow structure is the masked autoencoder for distribution estimation , or MADE, that facilitates GPU-based parallelization. Multiple MADE blocks are used in masked autoregressive flows (MAF) overcoming the ordering dependency of autoregressive flows. AFs are also capable of learning conditional distributions by making the parameters of the flow dependent on the data using hypernetworks . We include here a more complete derivation of the presented in the main text. The overarching aim of this work is to develop a flexible proposal over perturbations that places minimal mass on perturbations that cause the simulator to not return a value, while not changing the originally specified model. Doing so reduces the wasted computational cost incurred by simulations failing, and also increases the effective sample size for a given sample budget. We consider deterministic models, expressed as simulators, describing the time-evolution of a state x t ∈ X, where we denote application of the simulator iterating the state as x t ← f (x t−1). However, brittle simulators fail for "invalid" inputs, which we denote as a return value of ⊥ (read as "bottom") from the simulator. Hence the complete definition of f is f: X → {X, ⊥}. We denote the region of valid inputs as X A ⊂ X, and the region of invalid inputs as X R ⊂ X, such that X A ∪ X R = X, where the boundary between these regions is unknown. Over the whole support, f defines a many-to-one function, as all X R maps to ⊥. However, the algorithm we go on to derive only requires that f is one-to-one in the accepted region. This is not uncommon in real simulators, and is satisfied by, for example, ODE models. A stochastic, additive perturbation to state, denoted z t ∈ X, is applied to induce a distribution over states. The distribution of this perturbation is denoted p(z t |x t−1), although, in practice, this distribution is often state independent. The iterated state is therefore calculated as x t ← f (x t−1 + z t). We define the random variable A t ∈ {0, 1} to denote whether the perturbation (as x t−1 is being conditioned on) is accepted. The naive approach to sampling from the perturbed system, shown in Algorithm 1, is to repeatedly sample from the proposal distribution and evaluate f until the simulator successfully exists. This procedure defines A t = I [f (x t−1 + z t) = ⊥], z t ∼ p(z t |x t−1), i.e. successfully iterated samples are accepted with certainty. This approach incurs significant wasted computation as the simulator must be called repeatedly, with failed iterations being discarded. Therefore the objective of this work is to derive a more efficient sampling mechanism. We begin by showing that Algorithm 1 defines a rejection sampler, with a specific form, targeting the space of successfully iterated states. This reasoning is illustrated in Figure 3. The behavior of f and the distribution p(z t |·) implicitly define a distribution over successfully iterated states. We denote this "target" distribution as p(x t |x t−1) = p(x t |x t−1, A t = 1), where the bar indicates that the sample was accepted, and hence places no probability mass on failures. Note there is no bar on p(z t |x t−1) above, indicating that it is defined Algorithm 1 Sampling from a brittle simulator. Data: Current state x t−1, brittle simulator f, perturbation proposal p(z t |x t−1). Result: Iterated state x t and perturbation z t. Figure 3: Graphical representation of how a brittle deterministic simulator acts as a rejection sampler, targetingp(z t |x t−1). For clarity here we assume x t=1 = 0 and z t is independent of x t. The simulator, f (z t), returns ⊥ for some unknown input regions, shown in green. The proposal over z t is shown in blue. The target distribution,p(z t), shown in orange, is implicitly defined asp(z t) = 1 Mp p(z t)I [f (z t) = ⊥], where M p is the normalizing constant from p, equal to the acceptance rate. Accordingly the proposal distribution, scaled by M p, is exactly equal top(z t) in the accepted region. Therefore sampling from p until f successfully exits, as in Algorithm 1, can be seen as constructing a rejection sampler with proposal p(z t), and acceptance ratio,p with no knowledge of the accept/reject behaviors of f and hence probability mass may be placed on regions that yield failure. The functional form ofp is unavailable, and the density cannot be evaluated for any input value. Importantly,p is the distribution specified a-priori by the modeler, sampled from by the entire simulation pipeline, and hence any algorithm we develop must also targetp(x t |x t−1). The existence of p(x t |x t−1) implies the existence of a second distribution: the distribution over accepted perturbations, denoted p(z t |x t−1). Note that this distribution is also conditioned on acceptance under the chosen simulator indicated be the presence of a bar. We assume f is one-to-one in the accepted regime, and so the change of variables rule can be applied to directly relate this to p(x t |x t−1). Under our initial algorithm for sampling from a brittle simulator we can therefore write the following identity: where the normalizing constant M p is the acceptance rate under p. By inspecting, accepting with certainty perturbations that exit successfully can be seen as proportionally shifting mass from regions of p where the simulator does not exit to regions where it does. In a rejection sampler, the probability of accepting a proposed sample is proportional to the ratio between the target distribution and the proposal, scaled by a constant such that: As we have already stated, we cannot evaluate the target density, but we can establish if the density is non-zero (indicated by the simulator not failing). A sufficient condition to ensure the correctness of a rejection sampler in this scenario is that the proposal density is proportional to the target density wherever the target density has support. Applying this condition to our scenario implies that if the simulator fails, the density under the target distribution is known to be zero and the sample should be rejected with certainty, regardless of the density under the proposal distribution. In the accepted region, the sample should be accepted with probability p(z t |·)/M p(z t |·), where M is selected to satisfy. However, from, it can be seen p ∝ p hence proposal and target are proportional irrespective of the choice of M, and the value of M p, satisfying the above criteria. The acceptance rule of the rejection sampler is therefore reduced to I [f (x t−1 + z t) = ⊥]. Importantly, we do not need to evaluate M p, M, orp to use Algorithm 1 as a valid rejection sampler. This simple probabilistic interpretation of the behavior of the simulation process enables us to establish as a definition ofp valid across the entire input domain of f -a definition we now exploit to learn an efficient proposal. We now derive how we can learn the proposal distribution, denoted q φ parameterized by φ, to replace p, such that the acceptance rate under q φ (denoted M q φ) tends towards unity, minimizing wasted computation, while also retaining the same joint distribution as the originally specified model. We denote q φ as the proposal we train, which, coupled with the simulator, implicitly define a proposal over accepted samples, denoted q φ. Expressing this mathematically, we wish to minimize the distance between joint distribution implicitly specified over accepted iterated states using the a-priori specified proposal distribution p, and the joint distribution defined implicitly as q φ: where we select the Kullback-Leibler (KL) divergence as the metric of distance between distributions. The outer expectation defines this objective as amortized across state space. As is standard in amortized and compiled inference methods we can generate the samples by directly sampling from the model . We eventually perform this minimization using stochastic gradient descent, and so this expectation defines the distribution from which we sample the minibatches used, and so we drop this expectation for compactness. Expanding the KL term yields: Noting that q φ and p are defined only on accepted samples, where f is one-to-one, we can apply a change of variables defined for q φ as: and likewise for p. This transforms the distribution over x t into a distribution over z t and a Jacobian term: Noting that the same Jacobian terms appear in the numerator and denominator we are able to cancel these: taking care to also apply the change variables in the distribution we are sampling from in. We can now discard the remaining p term as it is independent of φ, and noting that f −1 (x t) = x t−1 + z t we can write: It can now be read off that minimizing the KL stated in can be performed by setting q φ (z t |x t−1) equal to p(z t |x t−1). Had we have discarded p a step earlier, we would have been unable to eliminate the Jacobian terms inside the logarithm. However, this distribution is defined after rejection sampling, and can only be defined as in: denoting M q φ as the acceptance rate under q φ. Note again that q φ is not dependent on the accept/reject characteristics of f. Differentiation of M q φ is intractable. Further, there is an infinite family of q φ proposals that yield p = q φ, that have non-zero rejection rates. However, we observe that φ * is a maximizer for both q φ andq φ in the limit of zero rejection, and so we can instead optimize q φ (z t |x t−1): with no consideration of the rejection behavior. Additionally, by removing the rejection sampler as we have, we have implicitly specified that the proposal distribution must have an acceptance rate of one. This term is differentiable with respect to φ and so we can maximize this quantity using stochastic gradients, where samples from the outer expectation over x t−1 defines the distribution from which we sample minibatches from. This expression shows that we can learn the distribution over accepted x t values by learning the distribution over z t, without needing to calculate the Jacobian or inverse of the transformation defined by f. We can now perform density estimation on the accepted samples from the a-priori specified rejection sampler to learn a proposal for accepted x t samples, thus minimizing wasted computation, targeting the same overall joint distribution, and retaining interpretability by utilizing the simulator. In this section we include additional figures and experimental details for the annulus and MuJoCo experiment presented in the main text, along with an additional two experiments. Our first additional example uses a simulator of balls elastically bouncing in a square enclosure, as shown in Figure 4a. The dimensionality of the state vector, x t, is four times the number of balls -the x-y coordinate and velocity of the centre of mass, per ball. We add a small amount of Gaussian noise at each iteration to the position and velocity of each ball. This perturbation induces the possibility that two balls overlap, or, a ball intersects with the wall. Both of these represent invalid physical arrangements and so the simulator returns ⊥ for such configurations. We note that here, we are conditioning on the state of both balls simultaneously, and proposing the perturbation to the state jointly. Figure 4 displays the of this experiment. Figure 5 shows the distribution over x-y perturbations of a single ball, conditioned on the other ball being static and stationary. Green contours show the perturbations learned by autoregressive flow such that failure is not induced. In Figure 4c we plot the rejection rate under p and q φ as a function of the position of the first ball, with the second ball fixed in the position shown, showing that rejection has been all but eliminated. We again see a reduction in the variance of the evidence approximation when comparing p and q in an SMC scheme, as shown in Figure 4d. This example demonstrates the applicability of the autoregressive flow; but also demonstrates how a seemingly simple simulator becomes brittle when naively perturbed. shows the rejection rate as a function of the position of the first ball, with the second ball in the position shown. The trained proposal (right) has all but eliminated rejection in the permissible space compared to the a-priori specified proposal (left). The rejection rate under p is much higher in the interior as the second ball may also leave the enclosure, whereas q φ has practically eliminated rejection by jointly proposing perturbations. 4d shows the reduction in variance achieved by using q φ. Although the reduction appears more modest compared to, say, the annulus example, it still achieves a paired t-test score of < 0.0001, indicating a strong level of statistical significance. We conclude by applying our algorithm to a simulator for the widely studied Caenorhabditis elegans roundworm. WormSim, presented by , is a simulator of the body of the worm, driven by a surrogate for the true neural architecture of Caenorhabditis elegans, and uses a 510 dimensional state representation. We apply perturbations to the 98 dimensional subspace defining the 49 x-y coordinate pairs physical position of the worm, while conditioning on the full 510 dimensional state vector. The expected rate of failure increases sharply as a function of the scale of the perturbation applied, as shown in Figure 6a, as the integrator used in WormSim is unable to integrate highly perturbed states. We then train a autoregressive flow targetingp, where the rejection rate during training is shown in Figure 6b. We see that we are able to learn an autoregressive flow with lower rejection rates, reaching approximately 53% rejection, for a p with approximately 75% rejection. Although the rejection rate is higher than ultimately desired, we include this example to show how rejections occur in real simulators through integrator failure. We believe larger flows with regularized parameters can reduce the rejection rate further. Figure 5: Shown is the a-priori specified proposal distribution, p, over the perturbation to the position of the first ball specified in the model, and the learned proposal, q φ in green, for the bouncing balls experiment introduced in Section C.1. The edge of the permissible region of the enclosure is shown as a black dashed line. The second ball is fixed at, and the ing invalid region induced shaded. The flow has learned to deflect away from the disallowed regions. x t ← x t−1 + δt ×ẋ t−1 + z xt, z xt ∼ N (0, 0.1), y t ← y t−1 + δt ×ẏ t−1 + z yt, z xt ∼ N (0, 0.1), x t ←ẋ t−1 + zẋ t, z xt ∼ N (0, 0.1), y t ←ẏ t−1 + zẏ t, z yt ∼ N (0, 0.1), Failure is defined as the change in radius being greater than 0.03. To compute the variances of the SMC sweep we generate 100 random traces. We then perform 50 SMC sweeps per trace, using 100 particles, and compute the evidence. We use the configuration "tosser" included in , only modifying it by removing the second unused bucket. We use completely standard simulation configuration. We introduce the limit on overlap leveraging MuJoCos in-built collision detection, rejecting overlaps above 0.005. Typical overlaps in the standard execution of MuJoCo are below this limit. An integration time of 0.002 is used. We observe only the x-y position of the capsule with Gaussian distributed noise, with standard deviation 0.1. We perturb the x-y position and velocity of the capsule with Gaussian distributed noise, with standard deviation 0.005 and 0.1 respectively. We perturb the angle and angular velocity of the capsule with Gaussian distributed noise, with standard deviation 0.05 and 0.05 respectively. These values were chosen to be in line with typical simulated values in the tosser example. We place a prior over the initial position and velocity with standard deviation 0.01 for positions and 0.1 for velocities, and mean equal to their true position. In this, the state input to the normalizing flow is the position and angle, and derivatives, of the capsule, as well as the state of the actuator. The actuators state is unobserved and is not perturbed under the model. We also input time into the normalizing flow as the control dynamics are not constant with time. To compute the variances of the SMC sweep we generate 50 random traces. We then perform 20 SMC sweeps per trace, using 100 particles, and compute the evidence.

We learn a conditional autoregressive flow to propose perturbations that don't induce simulator failure, improving inference performance.

Represent the TLDR for finding the corresponding scientific article.

We propose a framework for modifying the latent space operations such that the distribution mismatch between the resulting outputs and the prior distribution the generative model was trained on is fully eliminated.


Question: Represent the following document

Question: Multi-hop question answering requires models to gather information from different parts of a text to answer a question. Most current approaches learn to address this task in an end-to-end way with neural networks, without maintaining an explicit representation of the reasoning process. We propose a method to extract a discrete reasoning chain over the text, which consists of a series of sentences leading to the answer. We then feed the extracted chains to a BERT-based QA model to do final answer prediction. Critically, we do not rely on gold annotated chains or ``supporting facts:'' at training time, we derive pseudogold reasoning chains using heuristics based on named entity recognition and coreference resolution. Nor do we rely on these annotations at test time, as our model learns to extract chains from raw text alone. We test our approach on two recently proposed large multi-hop question answering datasets: WikiHop and HotpotQA, and achieve state-of-art performance on WikiHop and strong performance on HotpotQA. Our analysis shows the properties of chains that are crucial for high performance: in particular, modeling extraction sequentially is important, as is dealing with each candidate sentence in a context-aware way. Furthermore, human evaluation shows that our extracted chains allow humans to give answers with high confidence, indicating that these are a strong intermediate abstraction for this task.

Represent the TLDR for finding the corresponding scientific article

We improve answering of questions that require multi-hop reasoning extracting an intermediate chain of sentences.

Represent the short summary for finding the corresponding scientific article (long)!

We introduce ReClor, a reading comprehension dataset requiring logical reasoning, and find that current state-of-the-art models struggle with real logical reasoning with poor performance near that of random guess.


Question: represent the article for finding a single-sentence tldr

Question: Normalizing constant (also called partition function, Bayesian evidence, or marginal likelihood) is one of the central goals of Bayesian inference, yet most of the existing methods are both expensive and inaccurate. Here we develop a new approach, starting from posterior samples obtained with a standard Markov Chain Monte Carlo (MCMC). We apply a novel Normalizing Flow (NF) approach to obtain an analytic density estimator from these samples, followed by Optimal Bridge Sampling (OBS) to obtain the normalizing constant. We compare our method which we call Gaussianized Bridge Sampling (GBS) to existing methods such as Nested Sampling (NS) and Annealed Importance Sampling (AIS) on several examples, showing our method is both significantly faster and substantially more accurate than these methods, and comes with a reliable error estimation. Normalizing constant, also called partition function, Bayesian evidence, or marginal likelihood, is the central object of Bayesian methodology. Despite its importance, existing methods are both inaccurate and slow, and may require specialized tuning. One such method is Annealed Importance Sampling (AIS), and its alternative, Reverse AIS (RAIS), which can give stochastic lower and upper bounds to the normalizing constant, bracketing the true value . However, as the tempered distribution may vary substantially with temperature, it can be expensive to obtain good samples at each temperature, which can lead to poor estimates . Nested sampling (NS) is another popular alternative , which can be significantly more expensive than standard sampling methods in higher dimensions but, as we show, can also lead to very inaccurate estimates. Moreover, there is no simple way to know how accurate the estimate is. Here we develop a new approach to the problem, combining Normalizing Flow (NF) density estimators with Optimal Bridge Sampling (OBS). In a typical Bayesian inference application, we first obtain posterior samples using one of the standard Markov Chain Monte Carlo (MCMC) methods. In our approach we use these samples to derive the normalizing constant with relatively few additional likelihood evaluations required, making the additional cost of normalizing constant estimation small compared to posterior sampling. All of our calculations are run on standard CPU platforms, and will be available in the BayesFast Python package. Let p(x) and q(x) be two possibly unnormalized distributions defined on Ω, with normalizing constants Z p and Z q. For any function α(x) on Ω, we have if the integral exists. Suppose that we have samples from both p(x) and q(x), and we know Z q, then Equation gives which is the Bridge Sampling estimation of normalizing constant . It can be shown that many normalizing constant estimators, including Importance Sampling and Harmonic Mean, are special cases with different choices of bridge function α(x) . For a given proposal function q(x), an asymptotically optimal bridge function can be constructed, such that the ratio r = Z p /Z q is given by the root of the following score function equation where n p and n q are the numbers of samples from p(x) and q(x). For r ≥ 0, S(r) is monotonic and has a unique root, so one can easily solve it with e.g. secant method. This estimator is optimal, in the sense that its relative mean-square error is minimized . Choosing a suitable proposal q(x) for Bridge Sampling can be challenging, as it requires a large overlap between q(x) and p(x). One approach is Warp Bridge Sampling (WBS) , which transforms p(x) to a Gaussian with linear shifting, rescaling and symmetrizing. As we will show, this approach can be inaccurate or even fail completely for more complicated probability densities. As stated above, an appropriate proposal q(x) which has large overlap with p(x) is required for OBS to give accurate . In a typical MCMC analysis we have samples from the posterior, so one can obtain an approximate density estimation q(x) from these samples using a bijective NF. In this approach one maps p(x) to an unstructured distribution such as zero mean unit variance Gaussian N (0, I). For density evaluation we must also keep track of the Jacobian of transformation |dΨ/dx|, so that our estimated distribution is q(x) = N (0, I)|dΨ/dx|, where Ψ(x) is the transformation. The probability density q(x) is normalized, so we know Z q = 1. There have been various methods of NF recently proposed in machine learning literature ((Dinh et al.,, 2016), which however failed on several examples we present below. Moreover, we observed that training with these is very expensive and can easily dominate the overall computational cost. For these reasons we instead develop Iterative Neural Transform (INT), a new NF approach, details of which will be presented elsewhere. It is based on combining optimal transport and information theory, repeatedly finding and transforming one dimensional marginals that are the most deviant between the target and proposal (Gaussian) distributions. After computing dual representation of Wasserstein-1 distance to find the maximally non-Gaussian directions, we apply a bijective transformation that maximizes the entropy along these directions. For this we use a non-parametric spline based transformation that matches the 1-d cumulative distribution function (CDF) of the data to a Gaussian CDF, where kernel density estimation (KDE) is used to smooth the probability density marginals. We found that using a fixed number of 5 to 10 iterations is sufficient for evidence estimation, and the computational cost of our NF density estimation is small when compared to the cost of sampling. We propose the following Gaussianized Bridge Sampling (GBS) approach, which combines OBS with NF density estimation. In our typical application, we first run No-U-Turn Sampler (NUTS) to obtain 2n p samples from p(x) if its gradient is available, while affine invariant sampling can be used in the gradient-free case. To avoid underestimation of Z p , these 2n p samples are divided into two batches, and we fit INT with the first batch of n p samples to obtain the proposal q(x). Then we draw n q samples from q(x) and evaluate their corresponding p(x), where n q is determined by an adaptive rule (see Appendix B.4). We solve for the normalizing constant ratio r with Equation, using these n q samples from q(x) and the second batch of n p samples from p(x) (also evaluating their corresponding q(x)), and report the in form of ln Z p, with its error approximated by the relative mean-square error of Z p given in Equation . We used four test problems to compare the performance of various estimators. See Appendix A and B for more details of the examples and algorithms. The 16-d Funnel example is adapted from. The funnel structure is common in Bayesian hierarchical models, and in practice it is recommended to reparameterize the model to overcome the pathology . Here we stick to the original parameterization for test purpose. The 32-d Banana example comes from a popular variant of multidimensional Rosenbrock function , which is composed of 16 uncorrelated 2-d bananas. In addition, we apply a random 32-d rotation to the bananas, which makes all the parameters correlated with each other. The 48-d Cauchy example is adapted from the LogGamma example in;. In contrast to the original example, where the mixture structure only exists in the first two dimensions, we place a mixture of two heavy-tailed Cauchy distributions along every dimension. The 64-d Ring example has strong non-linear correlation between the parameters, as the marginal distribution of every two successive parameters is ring-shaped. See Figure 1 for a comparison of the estimators. For all of the four test examples, the proposed GBS algorithm gives the most accurate and a valid error estimation. We use NS as implemented in dynesty with its default settings. For all other cases, we use NUTS as the MCMC transition operator. We chose to run (R)AIS with equal number of evaluations as our GBS, but as seen from Figure 1 this number is inadequate for (R)AIS, which needs about 10-100 times more evaluations to achieve sufficient accuracy (see Appendix B.3). In contrast, if we run GBS with 4 times fewer evaluations (Gaussianized Bridge Sampling Lite, GBSL), we achieve an unbiased with a larger error than GBS, but still smaller than other estimators. For comparison we also show replacing OBS with IS (GIS) or HM (GHM), while still using INT for q(x). Although GIS and GHM are better than NS or (R)AIS, they are worse than GBS(L), highlighting the importance of OBS. Finally, we also compare to WBS, which uses a very simple proposal distribution q(x), and fails on several examples, highlighting the importance of using a more expressive NF for q(x). For our GBS(L), most of evaluation time is used to get the posterior samples with standard MCMC, which is a typical Bayesian inference goal, and the additional cost to evaluate evidence is small compared to the MCMC (see Appendix B.4). In contrast, Thermodynamic Integration (TI) or (R)AIS is more expensive than posterior sampling, since the chains need to be accurate at every intermediate state . The same comment applies to NS, which is more expensive than the MCMC approaches we use here for posterior analysis, especially when non-informative prior is used. We present a new method to estimate the normalizing constant (Bayesian evidence) in the context of Bayesian analysis. Our starting point are the samples from the posterior using standard MCMC based methods, and we assume that these have converged to the correct probability distribution. In our approach we combine OBS with INT, a novel NF based density estimator, showing on several high dimensional examples that our method outperforms other approaches in terms of accuracy and computational cost, and provides a reliable error estimate. The model likelihood is with flat prior x 1 ∼ U(−4, 4), x 2:n ∼ U(−30, 30). We use ln Z p = −63.4988 as the fiducial value, and the corner plot of the first four dimensions is shown in Figure 2. The model likelihood is with flat prior U(−15, 15) on all the parameters. The rotation matrix A is generated from a random sample of SO(n), and the same A is used for all the simulations. We use ln Z p = −127.364 as the fiducial value, and the corner plot of the first four dimensions, without or with the random rotation, is shown in Figure 3. The strong degeneracy can no longer be identified in the plot once we apply the rotation, however it still exists and hinders most estimators from getting reasonable . The model likelihood is with flat prior U(−100, 100) on all the parameters. We use ln Z p = −254.627 as the fiducial value, and the corner plot of the first four dimensions is shown in Figure 4. The model likelihood is, a = 2, b = 1, n = 64, with flat prior U(−5, 5) on all the parameters. We use ln Z p = −114.492 as the fiducial value, and the corner plot of the first four dimensions is shown in Figure 5. We use dynamic NS implemented in dynesty, which is considered more efficient than static NS. Traditionally, NS does not need the gradient of the likelihood, at the cost of lower sampling efficiency in high dimensions. Since analytic gradient of the four examples is available, we follow dynesty's default setting, which requires the gradient to perform Hamitonian Slice Sampling for dimensions d > 20. While for dimensions 10 ≤ d ≤ 20, random walks sampling is used instead. dynesty also provides an error estimate for the evidence; see for details. For (R)AIS, we divide the warm-up iterations of NUTS into two equal stages, and the (flat) prior is used as the base density. In the first stage, we set β = 0.5 and adapt the mass matrix and step size of NUTS, which acts as a compromise between the possibly broad prior and narrow posterior. In the second stage, we set β = 0 (β = 1) for AIS (RAIS) to get samples from the prior (posterior). After warm-up, we use the following sigmoidal schedule to perform annealing, where σ denotes the logistic sigmoid function and we set δ = 4 . We use 1,000 warm-up iterations for all the four examples, and adjust the number of states T so that it needs roughly the same number of evaluations as GBS in total. The exact numbers are listed in Table 1. We run 16 chains for each case, and average reported ln Z p of different chains, which gives a stochastic lower (upper) bound for AIS (RAIS) according to Jensen's inequality. The uncertainty is estimated from the scatter of different chains, and should be understood as the error of the lower (upper) bound of ln Z p, instead of ln Z p itself. Funnel Banana Cauchy Ring AIS 800 2000 3000 3500 RAIS 700 1500 2500 3000 Table 1: The number of states T used by (R)AIS. Using the mass matrix and step size of NUTS adapted at β = 0.5, and the prior as base density, may account for the phenomenon that RAIS failed to give an upper bound in the Banana example: the density is very broad at high temperatures and very narrow at low temperatures, which is difficult for samplers adapted at a single β. One may remedy this issue by using a better base density that is closer to the posterior, but this will require delicate hand-tuning and is beyond the scope of this paper. While the upper (lower) bounds of (R)AIS are valid in the limit of a very large number of samples, achieving this limit may be extremely costly in practice. The remaining normalizing constant estimators require a sufficient number of samples from p(x), which we obtain with NUTS. For WBS, GBS, GIS and GHM, we run 8 chains with 2,500 iterations for the Funnel and Banana examples, and 5,000 iterations for the Cauchy and Ring examples, including the first 20% warm-up iterations, which are removed from the samples. Then we fit INT using 10 iterations for GBS, GIS and GHM, whose computation cost (a few seconds for the Funnel example) is small or negligible relative to NUTS sampling, and does not depend on the cost of ln p(x) evaluations. For GBSL, the number of NUTS chains, NUTS iterations and INT iterations are all reduced by half, leading to a factor of four decrease in the total computation cost. The relative mean-square error of OBS is minimized and given by where np+nq. Here p (x) = p(x)/Z p and q(x) should be normalized densities. We assume the samples from q(x) are independent, whereas the samples from p(x) may be autocorrelated, and τ f 2 is the integrated autocorrelation time of f 2 (x p) (Frühwirth-), which is estimated by the autocorr module in emcee . Analogous expressions can be derived for IS and HM, The claimed uncertainty in Figure 1 is obtained by assuming that the error is Gaussian distributed. There can be different strategies to allocate samples for BS. In the literature, it is recommended that one draws samples from p(x) and q(x) based on equal-sample-size or equal-time allocation . Since NUTS based sampling usually requires at least hundreds of evaluations to obtain one effective sample from p(x) in high dimensions , which is orders of magnitude more expensive than our NF based sampling for q(x), it could be advantageous to set n q > n p. Throughout this paper, the following adaptive strategy is adopted to determine n q for GBS(L). After obtaining 2n p samples from p(x), we divide them into two equal batches, which will be used for fitting the proposal q(x) and evaluating the evidence, respectively. As an starting point, we draw n q,0 = n p samples from q(x) and estimate the error of OBS using Equation. Note that the right side of Equation is composed of two terms, and only the first term will decrease as one increases n q but fixes n p. Assuming that current samples provide an accurate estimate of the variance and expectation terms in Equation, one can solve for n q such that f err, the fraction of q(x) contributions in Equation, is equal to some specified value, which we set to 0.1. Since the n q,0 samples from q(x) can be reused, if n q < n q,0, no additional samples are required and we set n q = n q,0. On the other hand, we also require that f eva, the fraction of p(x) evaluations that are used for the q(x) samples, is no larger than 0.1, although this constraint is usually not activated in practice. We use 0.1 as the default values of f err and f eva, so that the additional cost of evidence evaluation is small relative to the cost of sampling, while using a larger n q alone can no longer significantly improve the accuracy of normalizing constant. However, if one wants to put more emphasis on posterior sampling (evidence estimation), a larger (smaller) f err and/or smaller (larger) f eva can be used. In principle, it is also possible to use different number of p(x) samples to fit the proposal and evaluate the evidence, in contrast to equal split used in , which we leave for feature research. For GIS and WBS, we use the same n q as solved for GBS(L). No samples from p(x) are required to estimate normalizing constant for GIS, so in this case all the 2n p samples will be used to fit INT. While for GHM, no samples from q(x) are required. Note that for WBS, the additional p(x) evaluations required for evidence estimation is n p + 2n q instead of n q, which comes from the symmetrization of ln p(x).

We develop a new method for normalization constant (Bayesian evidence) estimation using Optimal Bridge Sampling and a novel Normalizing Flow, which is shown to outperform existing methods in terms of accuracy and computational time.

Represent the short summary for finding the corresponding scientific article (long).

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.