Question: Represent the scientific article to find a short TLDR

Question: Recurrent Neural Networks have long been the dominating choice for sequence modeling. However, it severely suffers from two issues: impotent in capturing very long-term dependencies and unable to parallelize the sequential computation procedure. Therefore, many non-recurrent sequence models that are built on convolution and attention operations have been proposed recently. Notably, models with multi-head attention such as Transformer have demonstrated extreme effectiveness in capturing long-term dependencies in a variety of sequence modeling tasks. Despite their success, however, these models lack necessary components to model local structures in sequences and heavily rely on position embeddings that have limited effects and require a considerable amount of design efforts. In this paper, we propose the R-Transformer which enjoys the advantages of both RNNs and the multi-head attention mechanism while avoids their respective drawbacks. The proposed model can effectively capture both local structures and global long-term dependencies in sequences without any use of position embeddings. We evaluate R-Transformer through extensive experiments with data from a wide range of domains and the empirical show that R-Transformer outperforms the state-of-the-art methods by a large margin in most of the tasks. Recurrent Neural Networks (RNNs) especially its variants such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have achieved great success in a wide range of sequence learning tasks including language modeling, speech recognition, recommendation, etc (; ; ; ;). Despite their success, however, the recurrent structure is often troubled by two notorious issues. First, it easily suffers from gradient vanishing and exploding problems, which largely limits their ability to learn very long-term dependencies . Second, the sequential nature of both forward and backward passes makes it extremely difficult, if not impossible, to parallelize the computation, which dramatically increases the time complexity in both training and testing procedure. Therefore, many recently developed sequence learning models have completely jettisoned the recurrent structure and only rely on convolution operation or attention mechanism that are easy to parallelize and allow the information flow at an arbitrary length. Two representative models that have drawn great attention are Temporal Convolution Networks(TCN) and Transformer . In a variety of sequence learning tasks, they have demonstrated comparable or even better performance than that of RNNs (; ;). The remarkable performance achieved by such models largely comes from their ability to capture long-term dependencies in sequences. In particular, the multi-head attention mechanism in Transformer allows every position to be directly connected to any other positions in a sequence. Thus, the information can flow across positions without any intermediate loss. Nevertheless, there are two issues that can harm the effectiveness of multi-head attention mechanism for sequence learning. The first comes from the loss of sequential information of positions as it treats every position identically. To mitigate this problem, Transformer introduces position embeddings, whose effects, The illustration of one layer of R-Transformer. There are three different networks that are arranged hierarchically. In particular, the lower-level is localRNNs that process positions in a local window sequentially (This figure shows an example of local window of size 3); The middle-level is multi-head attention networks which capture the global long-term dependencies; The upper-level is Position-wise feedforward networks that conduct non-linear feature transformation. These three networks are connected by a residual and layer normalization operation. The circles with dash line are the paddings of the input sequence however, have been shown to be limited . In addition, it requires considerable amount of efforts to design more effective position embeddings or different ways to incorporate them in the learning process . Second, while multi-head attention mechanism is able to learn the global dependencies, we argue that it ignores the local structures that are inherently important in sequences such as natural languages. Even with the help of position embeddings, the signals at local positions can still be very weak as the number of other positions is significantly more. To address the aforementioned limitations of the standard Transformer, in this paper, we propose a novel sequence learning model, termed as R-Transformer. It is a multi-layer architecture built on RNNs and the standard Transformer, and enjoys the advantages of both worlds while naturally avoids their respective drawbacks. More specifically, before computing global dependencies of positions with the multi-head attention mechanism, we firstly refine the representation of each position such that the sequential and local information within its neighborhood can be compressed in the representation. To do this, we introduce a local recurrent neural network, referred to as LocalRNN, to process signals within a local window ending at a given position. In addition, the LocalRNN operates on local windows of all the positions identically and independently and produces a latent representation for each of them. In this way, the locality in the sequence is explicitly captured. In addition, as the local window is sliding along the sequence one position by one position, the global sequential information is also incorporated. More importantly, because the localRNN is only applied to local windows, the aforementioned two drawbacks of RNNs can be naturally mitigated. We evaluate the effectiveness of R-Transformer with a various of sequence learning tasks from different domains and the empirical demonstrate that R-Transformer achieves much stronger performance than both TCN and standard Transformer as well as other state-of-the-art sequence models. The rest of the paper is organized as follows: Section 2 discusses the sequence modeling problem we aim to solve; The proposed R-Transformer model is presented in Section 3. In Section 4, we describe the experimental details and discuss the . The related work is briefly reviewed in Section 5. Section 6 concludes this work. Before introducing the proposed R-Transformer model, we formally describe the sequence modeling problem. Given a sequence of length N: x 1, x 2, · · ·, x N, we aim to learn a function that maps the input sequence into a label space Y: where y ∈ Y is the label of the input sequence. Depending on the definition of label y, many tasks can be formatted as the sequence modeling problem defined above. For example, in language modeling task, x t is the character/word in a textual sentence and y is the character/word at next position ; in session-based recommendation, x t is the user-item interaction in a session and y is the future item that users will interact with ; when x t is a nucleotide in a DNA sequence and y is its function, this problem becomes a DNA function prediction task . Note that, in this paper, we do not consider the sequence-to-sequence learning problems. However, the proposed model can be easily extended to solve these problems and we will leave it as one future work. The proposed R-Transformer consists of a stack of identical layers. Each layer has 3 components that are organized hierarchically and the architecture of the layer structure is shown in Figure 1. As shown in the figure, the lower level is the local recurrent neural networks that are designed to model local structures in a sequence; the middle level is a multi-head attention that is able to capture global long-term dependencies; and the upper level is a position-wise feedforward networks which conducts a non-linear feature transformation. Next, we describe each level in detail. Sequential data such as natural language inherently exhibits strong local structures. Thus, it is desirable and necessary to design components to model such locality. In this subsection, we propose to take the advantage of RNNs to achieve this. Unlike previous works where RNNs are often applied to the whole sequence, we instead reorganize the original long sequence into many short sequences which only contain local information and are processed by a shared RNN independently and identically. In particular, we construct a local window of size M for each target position such that the local window includes M consecutive positions and ends at the target position. Thus, positions in each local window form a local short sequence, from which the shared RNN will learn a latent representation. In this way, the local structure information of each local region of the sequence is explicitly incorporated in the learned latent representations. We refer to the shared RNN as LocalRNN. Comparing to original RNN operation, LocalRNN only focuses on local short-term dependencies without considering any long-term dependencies. Figure 2 shows the different between original RNN and LocalRNN operations. Concretely, given the positions x t−M +1, x t−M +2, · · ·, x t of a local short sequence of length M, the LocalRNN processes them sequentially and outputs M hidden states, the last of which is used as the representation of the local short sequences: where RNN denotes any RNN cell such as Vanilla RNN cell, LSTM, GRU, etc. To enable the model to process the sequence in an auto-regressive manner and take care that no future information is available when processing one position, we pad the input sequence by (M − 1) positions before the start of a sequence. Thus, from sequence perspective, the LocalRNN takes an input sequence and outputs a sequence of hidden representations that incorporate information of local regions: The localRNN is analogous to 1-D Convolution Neural Networks where each local window is processed by convolution operations. However, the convolution operation completely ignores the sequential information of positions within the local window. Although the position embeddings have been proposed to mitigate this problem, a major deficiency of this approach is that the effectiveness of the position embedding could be limited; thus it requires considerable amount of extra efforts . On the other hand, the LocalRNN is able to fully capture the sequential information within each window. In addition, the one-by-one sliding operation also naturally incorporates the global sequential information. Discussion: RNNs have long been a dominating choice for sequence modeling but it severely suffers from two problems -The first one is its limited ability to capture the long-term dependencies and the second one is the time complexity, which is linear to the sequence length. However, in LocalRNN, these problems are naturally mitigated. Because the LocalRNN is applied to a short sequence within a local window of fixed size, where no long-term dependency is needed to capture. In addition, the computation procedures for processing the short sequences are independent of each other. Therefore, it is very straightforward for the parallel implementation (e.g., using GPUs), which can greatly improve the computation efficiency. The RNNs at the lower level introduced in the previous subsection will refine representation of each positions such that it incorporates its local information. In this subsection, we build a sub-layer on top of the LocalRNN to capture the global long-term dependencies. We term it as pooling sublayer because it functions similarly to the pooling operation in CNNs. Recent works have shown that the multi-head attention mechanism is extremely effective to learn the long-term dependencies, as it allows a direct connection between every pair of positions. More specifically, in the multihead attention mechanism, each position will attend to all the positions in the past and obtains a set of attention scores that are used to refine its representation. Mathematically, given current representations h 1, h 2, · · ·, h t, the refined new representations u t are calculated as: where head k (h t) is the of k th attention pooling and W o is a linear projection matrix. Considering both efficiency and effectiveness, the scaled dot product is used as the attention function . Specifically, head i (h t) is the weighted sum of all value vectors and the weights are calculated by applying attention function to all the query, key pairs: where q, k i, and v i are the query, key, and value vectors and d k is the dimension of k i. Moreover, q, k i, and v i are obtained by projecting the input vectors into query, key and value spaces, respectively . They are formally defined as: where W q, W k and W v are the projection matrices and each attention pooling head i has its own projection matrices. As shown in Eq., each head i is obtained by letting h t attending to all the "past" positions, thus any long-term dependencies between h t and h i can be captured. In addition, different heads will focus on dependencies in different aspects. After obtaining the refined representation of each position by the multi-head attention mechanism, we add a position-wise fully connected feed-forward network sub-layer, which is applied to each position independently and identically. This feedforward network transforms the features non-linearly and is defined as follows: Following , We add a residual and layernorm connection between all the sub-layers. With all the aforementioned model components, we can now give a formal description of the overall architecture of an N -layer R-Transformer. For the i th layer (i ∈ {1, 2, · · · N}): where T is the length of the input sequence and x i t is the input position of the layer i at time step t. Comparing with TCN: R-Transformer is partly motivated by the hierarchical structure in , thus, we make a detailed comparison here. In TCN, the locality in sequences in captured by convolution filters. However, the sequential information within each receptive field is ignored by convolution operations. In contrast, the LocalRNN structure in R-Transformer can fully incorporate it by the sequential nature of RNNs. For modeling global long-term dependencies, TCN achieves it with dilated convolutions that operate on nonconsecutive positions. Although such operation leads to larger receptive fields in lower-level layers, it misses considerable amount of information from a large portion of positions in each layer. On the other hand, the multi-head attention pooling in R-Transformer considers every past positions and takes much more information into consideration than TCN. The proposed R-Transformer and standard Transformer enjoys similar long-term memorization capacities thanks to the multi-head attention mechanism . Nevertheless, two important features distinguish R-Transformer from the standard Transformer. First, R-Transformer explicitly and effectively captures the locality in sequences with the novel LocalRNN structure while standard Transformer models it very vaguely with multi-head attention that operates on all of the positions. Second, R-Transformer does not rely on any position embeddings as Transformer does. In fact, the benefits of simple position embeddings are very . In the next section, we will empirically demonstrate the advantages of R-Transformer over both TCN and the standard Transformer. Since the R-Transformer is a general sequential learning framework, we evaluate it with sequential data from various domains including images, audios and natural languages. We mainly compare it with canonical recurrent architectures (Vanilla RNN, GRU, LSTM) and two of the most popular generic sequence models that do not have any recurrent structures, namely, TCN and Transformer. However, since the majority of existing efforts to enhance Transformer are for natural languages, in the natural language evaluation, we also include one recent advanced Transformer, i.e., Transformer-XL. For all the tasks, Transformer and R-Transformer were implemented with Pytorch and the for canonical recurrent architectures and TCN were directly copied from as we follow the same experimental settings. In addition, to make the comparison fair, we use the same set of hyperparameters (i.e, hidden size, number of layers, number of heads) for R-Transformer and Transformer. Moreover, unless specified otherwise, for training, all models are trained with same optimizer and learning rate is chosen from the same set of values according to validation performance. In addition, the learning rate annealed such that it is reduced when validation performance reaches plateau. This task is designed to test model ability to memorize long-term dependencies. It was firstly proposed by and has been used by many previous works (; ;). Following previous settings, we rescale each 28 × 28 image in MNIST dataset into a 784 × 1 sequence, which will be classified into ten categories (each image corresponds to one of the digits from 0 to 9) by the sequence models. Since the rescaling could make pixels that are connected in the origin images far apart from each other, it requires the sequence models to learn very long-term dependencies to understand the content of each sequence. The dataset is split into training and testing sets as same as the default ones in Pytorch(version 1.0.0) 2. The model hyperparameters and classification accuracy are reported in Table 1. From the table, it can be observed that firstly, RNNs based methods generally perform worse than others. This is because the input sequences exhibit very long-term dependencies and it is extremely difficult for RNNs to memorize them. On the other hand, methods that build direct connections among positions, i.e., Transformer, TCN, achieve much better . It is also interesting to see that TCN is slightly better than Transformer, we argue that this is because the standard Transformer cannot model the locality very well. However, our proposed R-Transformer that leverages LocalRNN to incorporate local information, has achieved better performance than TCN. -3.46 LSTM -3.29 TCN 4 /150 3.07 Transformer 3/160 3.34 R-Transformer 3/160 2.37 Next, we evaluate R-Transformer on the task of polyphonic music modeling with Nottingham dataset . This dataset collects British and American folk tunes and has been commonly used in previous works to investigate the model's ability for polyphonic music modeling (; ;). Following the same setting in , we split the data into training, validation, and testing sets which contains 694, 173 and 170 tunes, respectively. The learning rate is chosen from {5e −4, 5e −5, 5e −6} and dropout with probability of 0.1 is used to avoid overfitting. Moreover, gradient clipping is used during the training process. We choose negative log-likelihood (NLL) as the evaluation metrics and lower value indicates better performance. The experimental are shown in Table 2. Both LTSM and TCN outperform Transformer in this task. We suspect this is because these music tunes exhibit strong local structures. While Transformer is equipped with multi-head attention mechanism that is effective to capture long-term dependencies, it fails to capture local structures in sequences that could provide strong signals. On the other hand, R-Transformer enhanced by LocalRNN has achieved much better than Transformer. In addition, it also outperforms TCN by a large margin. This is expected because TCN tends to ignore the sequential information in the local structure, which can play an important role as suggested by . In this subsection, we further evaluate R-Transformer's ability on both character-level and wordlevel language modeling tasks. The dataset we use is PennTreebank(PTB) that contains 1 million words and has been extensively used by previous works to investigate sequence models (; ; ;). For character-level language modeling task, the model is required to predict the next character given a context. Following the experimental settings in , we split the dataset into train-ing, validation and testing sets that contains 5059K, 396K and 446K characters, respectively. For Transformer and R-Transformer, the learning rate is chosen from {1, 2, 3} and dropout rate is 0.15. Gradient clipping is also used during the training process. The bpc is used to measure the predicting performance. For word-level language modeling, the models are required to predict the next word given the contextual words. Similarly, we follow previous works and split PTB into training, validation, and testing sets with 888K, 70K and 79K words, respectively. The vocabulary size of PTB is 10K. As with character-level language modeling,the learning rate is chosen from {1, 2, 3} for Transformer and R-Transformer and dropout rate is 0.35. In this task, we also add Transformer-XL as one baseline, which has been particularly designed for language modeling tasks and has achieved state-of-the-art performance. Note that to make the comparison fair, we apply the same model configuration, i.e., number of layers, to Transformer-XL. All other settings such as optimizer are the same as its original ones. The learning rate is chosen from {0.01, 0.001, 0.0001} and its best validation performance is achieved with 0.001. Note that, except dropout, no other regularization tricks such as variational dropout and weight dropout are applied. The prediction performance is evaluated with perplexity, the lower value of which denotes better performance. The experimental of character-level and word-level language modeling tasks are shown in Table 3 and Table 4, respectively. Several observations can be made from the Table 3. First, Transformer performs only slightly better than RNNs while much worse than other models. The reason for this observation is similar to the case of polyphonic music modeling task that language exhibits strong local structures and standard Transformer can not fully capture them. Second, TCN achieves better than all of the RNNs, which is attributed to its ability to capture both local structures and long-term dependencies in languages. Notably, for both local structures and longterm dependencies, R-Transformer has more powerful components than TCN, i.e., LocalRNN and Multi-head attention. Therefore, it is not surprising to see that R-Transformer achieves significantly better . Table 4 presents the for word-level language modeling. Similar trends are observed, with the only exception that LSTM achieves the best among all the methods. In addition, the of Transformer-XL is only slightly better than R-transformer. Considering the fact that Transformer-XL is specifically designed for language modeling and employs the recurrent connection of segments , this suggests the limited contribution of engineered positional embeddings. In summary, experimental have shown that the standard Transformer can achieve better than RNNs when sequences exhibit very long-term dependencies, i.e., sequential MNIST while its performance can drop dramatically when strong locality exists in sequences, i.e., polyphonic music and language. Meanwhile, TCN is a very strong sequence model that can effectively learn both local structures and long-term dependencies and has very stable performance in different tasks. More importantly, the proposed R-Transformer that combines a lower level LocalRNN and a higher level multi-head attention, outperforms both TCN and Transformer by a large margin consistently in most of the tasks. The experiments are conducted on various sequential learning tasks with datasets from different domains. Moreover, all experimental settings are fair to all baselines. Thus, the observations from the experiments are reliable with the current experimental settings. However, due to the computational limitations, we are currently restricted our evaluation settings to moderate model and dataset sizes. Thus, more evaluations on big models and large datasets can make the more convincing. We would like to leave this as one future work. Recurrent Neural Networks including its variants such LSTM and GRU have long been the default choices for generic sequence modeling. A RNN sequentially processes each position in a sequence and maintains an internal hidden state to compresses information of positions that have been seen. While its design is appealing and it has been successfully applied in various tasks, several problems caused by its recursive structures including low computation efficiency and gradient exploding or vanishing make it ineffective when learning long sequences. Therefore, in recent years, a lot of efforts has been made to develop models 3 /700 78.93 TCN 4 without recursive structures and they can be roughly divided into two categories depending whether they rely on convolutions operations or not. The first category includes models that mainly built on convolution operations. For example, van den Oord et al. have designed an autoregressive WaveNet that is based on causal filters and dilated convolution to capture both global and local information in raw audios . Ghring et al. has successfully replace traditional RNN based encoder and decoder with convolutional ones and outperforms LSTM setup in neural machine translation tasks (; . Moreover, researchers introduced gate mechanism into convolutions structures to model sequential dependencies in languages . Most recently, a generic architecture for sequence modeling, termed as Temporal Convolutional Networks (TCN), that combines components from previous works has been proposed in . Authors in have systematically compared TCN with canonical recurrent networks in a wide range of tasks and TCN is able achieve better performance in most cases. Our R-transformer is motivated by works in this group in a sense that we firstly models local information and then focus on global ones. The most popular works in second category are those based on multi-head attention mechanism. The multi-head attention mechanism was firstly proposed in , where impressive performance in machine translation task has been achieved with Transformer. It was then frequently used in other sequence learning models (; ;). The success of multi-head attention largely comes from its ability to learn long-term dependencies through direct connections between any pair of positions. However, it heavily relies on position embeddings that have limited effects and require a fair amount of effort to design effective ones. In addition, our empirical shown that the local information could easily to be ignored by multi-head attention even with the existence of position embeddings. Unlike previously proposed Transformer-like models, R-Transformer in this work leverages the strength of RNN and is able model the local structures effectively without the need of any position embeddings. In this paper, we propose a novel generic sequence model that enjoys the advantages of both RNN and the multi-head attention while mitigating their disadvantages. Specifically, it consists of a LocalRNN that learns the local structures without suffering from any of the weaknesses of RNN and a multi-head attention pooling that effectively captures long-term dependencies without any help of position embeddings. In addition, the model can be easily implemented with full parallelization over the positions in a sequence. The empirical on sequence modeling tasks from a wide range of domains have demonstrated the remarkable advantages of R-Transformer over state-of-the-art nonrecurrent sequence models such as TCN and standard Transformer as well as canonical recurrent architectures.

Represent the short summary for finding the corresponding scientific article (long)

This paper proposes an effective generic sequence model which leverages the strengths of both RNNs and Multi-head attention.

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Many tasks in natural language processing and related domains require high precision output that obeys dataset-specific constraints. This level of fine-grained control can be difficult to obtain in large-scale neural network models. In this work, we propose a structured latent-variable approach that adds discrete control states within a standard autoregressive neural paradigm. Under this formulation, we can include a range of rich, posterior constraints to enforce task-specific knowledge that is effectively trained into the neural model. This approach allows us to provide arbitrary grounding of internal model decisions, without sacrificing any representational power of neural models. Experiments consider applications of this approach for text generation and part-of-speech induction. For natural language generation, we find that this method improves over standard benchmarks, while also providing fine-grained control.

represent the short summary for finding the corresponding scientific article (long)

The provided query could be "We present a novel iterative algorithm based on generalized low rank models for computing and interpreting word embedding models." and the positive "GloVe and Skip-gram word embedding methods learn word vectors by decomposing a denoised matrix of word co-occurrences into a product of low-rank matrices. In this work, we propose an iterative algorithm for computing word vectors based on modeling word co-occurrence matrices with Generalized Low Rank Models. Our algorithm generalizes both Skip-gram and GloVe as well as giving rise to other embedding methods based on the specified co-occurrence matrix, distribution of co-occurences, and the number of iterations in the iterative algorithm. For example, using a Tweedie distribution with one iteration in GloVe and using a Multinomial distribution with full-convergence mode in Skip-gram. Experimental demonstrate that multiple iterations of our algorithm improves over the GloVe method on the Google word analogy similarity task. Word embeddings are low dimensional vector representations of words or phrases. They are applied to word analogy tasks and used as feature vectors in numerous tasks within natural language processing, computational linguistics, and machine learning. They are constructed by various methods which rely on the distributional hypothesis popularized by Firth: "words are characterized by the company they keep" BID9. Two seminal methodological approaches to finding word embeddings are Skip-gram [a] and GloVe []. Both methods input a corpus D, process it into a word co-occurence matrix X, then output word vectors with some dimension d. Skip-gram processes a corpus with w words into a count co-occurence matrix X ∈ R w×w, where x ij is the number of times word w i appears in the same context as the word w j. Here, two words being in the same context means that they're within l c tokens of each other. Define this co-occurence matrix to be the count co-occurence matrix. Next, Skip-gram [where u u u T i is the i th row of U, then defines the word vectors to be the rows ofÛ .GloVe processes a corpus with w words into a harmonic co-occurence matrix X ∈ R w×w where x ij is the harmonic sum of the number of tokens between words w i and w j over each co-occurrence. That is, x ij = p1<p2,|p1−p2|≤lc,D(p1)=wi,D(p2)=wj h(x ij) u u u DISPLAYFORM0 where a i and b j are bias terms, h(x ij) = (min{x ij, x max}).75 is the weight, and x max is some prespecified cutoff. GloVe then defines the estimated word vectors to be the rows of 1 2Û + 1 2V. In both Skip-gram and GloVe, a matrix of co-occurences X is introduced by processing the corpus, and an objective function is introduced to find a low rank factorization related to the co-occurences X. In this paper, we derive the objective functions from a model-based perspective. We introduce an iterative algorithm, and show that problem from running the iterative algorithm on full-convergence mode for a Multinomial model and problem is one step of the iterative algorithm for a Tweedie model. This algorithm additionally allows us to introduce methods to "fill in the gaps" between Skip-gram and GloVe and to introduce altogether new methods for finding word vectors. We saw that Skip-gram and GloVe compute a co-occurence matrix X which from processing the corpus D and an objective function J to relate the matrix X to a product of low rank matrices U and V. Many existing approaches for explaining word embedding methods do so by identifying or deriving the co-occurence matrix X or the objective function J. In this section, we review relevant work in this area, which helps frame our approach discussed in Section 4.1.Much of the related work involves using the co-occurence matrix from Skip-gram. For the remainder of this section, let X be the count co-occurence matrix. Early approaches to finding low-dimensional embeddings of words relied on the singular value decomposition [,]. These methods would truncate the singular value decomposition by zeroing out the small singular values. BID7 show that this is equivalent to using an objective function J which is invariant to orthogonal transformation. For simplicity, we specialize to the Frobenius norm and say these early approaches find arg min DISPLAYFORM0 F is the objective function and X is the co-occurence matrix. The co-occurence matrix and the loss function for Skip-gram can be read off from problem: the co-occurence matrix is X and the objective function is written in problem with u u u T i v v v j replaced by m ij. BID4 find a probabilistic interpretation of this loss function related to a Multinomial distribution, but do not take advantage of it and only replace the inner product with a (higher dimensional) variant, somewhat similar to the approach in Tifrea et al.. Mikolov et al. [2013a] introduce Skip-gram with negative sampling (SGNS), a variant of Skip-gram. If we view Skip-gram as maximizing the true positive rate of predicting a word will appear within a context window of another word, we can view SNGS as maximizing the true positive rate plus k times an approximation of the true negative rate. When k = 0, Skip-gram and SGNS coincide. BID19 use a heuristic argument to interpret SGNS as using a co-occurence matrix that is a shifted PMI matrix.2 However, they did not determine the objective function. Later, Li et al. and Landgraf and Bellay explicitly identified both the co-occurence matrix and the objective function. They find a different co-occurence matrix than BID19, one that does not depend on k, while their loss function does depend on k. Surprisingly, they establish that SGNS is finding a low-rank matrix related to X, the same matrix that Skip-gram uses. The loss function is w,w i,j=1 DISPLAYFORM1 2 Define the total number of times word wi appears to be xi· = w j=1 xij, the total number of times context wj appears to be x·j = w i=1 xij, and the total number of words to be x·· = w,w i,j=1 xij. The shifted PMI matrix has entries log DISPLAYFORM2 Landgraf and Bellay explain that this loss function has a probabilistic interpretation, and they use that interpretation to recover the shifted PMI matrix as a prediction from within their model. The approach in this paper will be to view the entries of the co-occurence matrix as random variables and introduce an objective function via the likelihood of that random variable. Our approach is most similar to Landgraf and Bellay and, to a lesser extent, BID4. In order proceed, some in probabilistic modeling and estimation needs to be developed. In this section, we review iteratively reweighted least squares (IRLS) for generalized linear models and review generalized low rank models []. Further (and notation) in exponential dispersion families and generalized linear models is developed in Section A. Generalized linear models (GLMs) are a flexible generalization of linear regression where the mean is a not necessarily linear function of a coefficient β β β and the response has an error distribution which is an exponential dispersion family. The coefficient β β β is unknown and a target of estimation. The standard approach to estimate β β β is maximum likelihood estimation [, Section 7] to produce the maximum likelihood estimator, or MLE,β β β. A computational approach to find the MLE is through Fisher scoring, a variant of Newton's method on the log likelihood which uses the expectation of the Hessian in place of the Hessian [, Section 4.5]. Define (β β β) to be the log likelihood. Specifically, Fisher scoring produces a sequence of estimates {β β β DISPLAYFORM0), where ∇ is the gradient and D 2 is the Hessian. Upon plugging in the gradient and expected Hessian for an exponential dispersion family, a surprising identity emerges: each iteration of Fisher scoring is equivalent to minimizing a weighted least squares objective: DISPLAYFORM1 where the weight H (t) and pseudo-response z (t) at iteration t have DISPLAYFORM2 DISPLAYFORM3, and µ DISPLAYFORM4 Principal components analysis BID13 is one well-known method for finding a low rank matrix related to X ∈ R w×c. In principal components analysis, we model x ij ind. DISPLAYFORM0 A maximum likelihood estimator for u u u i is taken to be a low-dimensional embedding of the i th row of X. The low-dimensional embedding enables interpretability and reduces noise. However, data cannot always be viewed as being drawn from a normal distribution, so it's necessary to extend the method of principal components to non-normal data. The extension can be made in a manner similar to the extension from linear models to generalized linear models: the new model, called a generalized low rank model [] allows us to estimate model-based low-dimensional embeddings of non-normal data. Definition 1 For some exponential dispersion family ED(µ, ϕ) with mean parameter µ and dispersion parameter ϕ, the model for X ∈ R w×c is a generalized low rank model with link function g when DISPLAYFORM1 DISPLAYFORM2 where u u u i, v v v j ∈ R d are the rows of matrices U ∈ R w×d and V ∈ R c×d, respectively, and a a a ∈ R w and b b b ∈ R c are bias (or offset) terms. The difference between the generalized low rank model and the generalized linear model is in the systematic component in equation FORMULA12. Here, the data is modeled as having its link-transformed mean be a matrix with rank at most d. This formalizes the way in which we relate the co-occurence matrix X to a low rank factorization. When the link function g is taken to be canonical, the generalized low rank model is identical to ePCA BID3. The generalization is worthwhile since the canonical link can be inappropriate, as we will see, for instance, in Section 5.1. We now present a method to find word vectors. A key innovation in the method is an iterative algorithm inspired by IRLS to find a maximum likelihood estimator in a generalized low rank model. Our method has three steps:Step 1 Choose a co-occurence matrix X ∈ R w×c to summarize the document. (Note, in many cases c = w so that the "contexts" are just the words.)Step 2 Choose a plausible exponential dispersion family to model the entries of the co-occurence matrix. Choose a corresponding link function. Step 3 Choose a number of iterations r to run IWLRLS (Algorithm) with the input specified above to output word vectors. DISPLAYFORM0; Evaluate the least squares problem arg min DISPLAYFORM1 Algorithm 1: Iteratively weighted low rank least squares (IWLRLS) algorithm for GLRMsThe first step of our method processes the corpus in order to extract the linguistic information. Some co-occurence statistics use more information than others: for instance, the harmonic co-occurence matrix makes use of the number of tokens between words while the count co-occurence matrix does not. A typical tuning parameters here is the length l c of the context window. We view this step as involving a "linguistic" choice. The second step specifies a distribution for the co-occurence matrix. A distribution can be considered as plausibly corresponding to reality if it can be derived by a connection to the corpus. In our Co-occurence: Harmonic Table 1: The rows refers to the number of steps of IWLRLS. A "·" represents no existing work. All filled-in positions in the lowest row were established in previous work. framework, the model is explicit: this is helpful since knowing a model provides interpretation for its output [, Section II.A.]. The choice of distribution will often determine, through convention, the link function, so the link function often does not need to be separately chosen. We view this step as involving a "statistical" choice. DISPLAYFORM2 The third step runs IWLRLS, a generalized version of IRLS. Recall that IRLS is derived by iteratively maximizing a second order Taylor expansion of the likelihood as a function β. The Taylor expansion is centered at the previous iterate. IWLRLS can be derived by iteratively maximizing a second order Taylor expansion of the likelhood as a function of η subject to the constraint 6. We view this as a "computational" choice that we fix in advance. In the following subsections, we run through many examples of our method as it would be used in practice. There are two distinct choices of co-occurence matrices that are made. Various choices of distributions recover common methods for finding word vectors. An altogether new estimator is proposed via an improvement of the assumed distribution in Skip-gram. Casting these estimators in this general framework provides an interpretation and understanding of them: we make explicit their assumptions and therefore know the driver of their behavior. We will apply our proposed method under the choice of the harmonic co-occurence matrix and the Tweedie distribution: one iteration of IWLRLS will recover GloVe. Step 1 The first step of our method is to pick a co-occurence matrix that summarizes the corpus. We choose the harmonic co-occurence matrix X ∈ R w×w.Step 2 Now we must determine a plausible distribution for the co-occurence matrix that is an exponential dispersion family. Recall that the Tweedie distribution has the property mentioned in equation FORMULA0 that it is a sum of Poisson many independent Gamma distributions. An informal way to write this is that DISPLAYFORM0 We argue that the Tweedie distribution is reasonable by connecting the Poisson and Inverse Gamma distributions displayed above to attributes of the corpus. Intuitively, it is reasonable that the number of times word w i and word w j co-occur within the corpus can be modeled as having a Poisson distribution. Another choice of distribution is that of an Inverse Gamma distribution for the number of tokens between word w i and word w j at some co-occurence, although it is an approximation as the number of tokens is an integer while the Inverse Gamma is supported on non-integers. Instead of using the canonical link function, we will take g(µ) = log µ, which is standard []. A problem with the canonical link function preventing its use is that its range is nonpositive. Step 3 Next, we find the form of the weight H and the pseudo-response Z that the Tweedie distribution provides. This amounts to plugging in the cumulant generating function ψ that is given in Section A.1. This in DISPLAYFORM1 When the algorithm is initialized withμ = X, the pseudo-response simplifies to z ij = log x ij.Taking the power p = 1.25, the weight simplifies to x 3/4 ij. In summary, we've shown that: Result 1 Inputting the harmonic co-occurence matrix, the Tweedie distribution with power p = 1.25, the log link, and the number of iterations k = 1 into IWLRLS in GloVe (without the algorithmic regularization induced by truncating the weights.)Given this connection, we can extend GloVe for several iterations rather than one or even use the full likelihood. We experiment with this using real data examples in Section 6. This shows that even though the first iteration does not depend on word pairs where x ij = 0, later iterations do. We now consider an alternative first step: we choose another co-occurence matrix to summarize the corpus. Then, we make multiple possible choices for step 2 to illustrate connections to previous work that step 3 recovers. Various choices for step 2 will recover the SVD BID17 ], Skip-gram [a], a new estimator which is a distributional improvement over those, and Skip-gram with negative sampling [b].Step 1 We choose the count co-occurence matrix. Step 2 A proposed distribution for the entries of X is the Gaussian distribution. This may not be the best choice, since the entries of X are non-negative integers. As is usual, we take the link function to be g(µ) = µ. We restrict the systematic component to not include the bias terms, so that η ij = u u u DISPLAYFORM0 Step 3 We showed in Section A.1 that the cumulant generation function from the normal distribution is ψ(θ) = 1 2 θ 2. This makes it so that DISPLAYFORM1 In other words, the IWLRLS algorithm will always converge in one iteration, so our method recovers the method of computing a truncated SVD of X by BID7.Another choice that could have been made in step 2 is to have the link function g(µ) = log µ. This still may not be the best choice since the normal distribution still has the same problems as before. Step 2 Another proposed distribution for the entries of X is a Multinomial distribution. Specifically, we could propose that the the row of X corresponding to word w i has the distribution x x x i ∼ Multinomial w j=1 x ij, π π π, where π π π ∈ R w is vector of probabilities of word w i appearing within a context window with the other words and w j=1 x ij is the total number of times word w i appears in the corpus. We take the link function to be the multi-logit. 3 Cotterell et al. show that the objective function of Skip-gram coincides with the likelihood of this model when the bias terms are removed, so that the systematic component η ij = u u u Step 3 The Poisson trick BID2 can be used to reduce estimation in a Multinomial model to estimation in a particular Poisson model. LetÛ,V be the maximum likelihood estimators in the Multinomial generalized low rank model described in step 2. Using this trick, it holds thatâ a a, (the same)Û, and (the same)V are maximum likelihood estimators in a Poisson generalized low rank model with independent responses x ij and systematic component DISPLAYFORM0 Notice that there is only one bias term. The weight and pseudo-response are DISPLAYFORM1 In the previous subsubsection, we saw that the choice of Multinomial model implicitly gives rise to a Poisson model with a systematic component given by equation FORMULA20. Since it could be most appropriate to have bias terms for both rows and columns due to the symmetry of the co-occurence matrix, we directly introduce a Poisson estimator with a non-restricted systematic component. Step 2 Another proposed distribution is a Poisson. Due to the "law of rare events" [, Section 3.6 .1], this is a plausible model. We use the canonical link function g(µ) = log µ.Step 3 The cumulant generating function is ψ(θ) = exp(θ) BID0, so that the weight and pseudo-response are given by equations. BID1 propose an estimator which is a close variant of one iteration of IWLRLS. At one point in their derivation, they (using our notation) take η ij = u u u i − v v v j 2 2 + c, where c is an arbitrary constant which does not depend on the word. This is inspired by their theorem 2.2. On the other hand, taking η ij = u u u T i v v v j + a i + b j (as in equation 6) in their derivation recovers one iteration of IWLRLS. The Negative-Binomial distribution is commonly used as an alternative for the Poisson in the presence of over-dispersion, which is the case when the variance is higher than the mean. It produces the same weight and pseudo-response as the Poisson. Step 2 We model x ij ind. DISPLAYFORM0 is an inflated count, η ij = u u u T i v v v j, and k ≥ 0. Landgraf and Bellay showed that a maximum likelihood estimator from this model with canonical link g(π) = log π 1−π is identical to a SGNS estimator. Step 3 The cumulant generating function for the binomial distribution is ψ(θ) = log(1 + exp θ), so the weight and pseudo-response are: DISPLAYFORM1 In Section 4.1 we introduced the IWLRLS algorithm to compute word vectors such as those produced by GloVe or SGNS. We now conduct quantitative evaluation experiments on an English word analogy task, a variety of word similarity tasks [a] to demonstrate the performance of the algorithm. First, in Section 6.1 we introduce the analogy similarity task for evaluating word vectors. In Section 6.2 we present of the algorithm with different distributions according to those presented in Section 5.1 and 5.2. In Section B.1 we provide parameter configurations and training procedures, and in Sections B.2-B.5 we present of IWLRLS in numerous scenarios showcasing improvement through multiple iterations and robustness to other model parameters. We introduce the word analogy task following the presentation of []. The word analogy task is a dataset of 19, 544 statements of the basic form "a is to b as c is to __", which are divided into a semantic and syntactic subsets. The semantic statements are typically analogies relating to people, places, or nouns such as "Athens is to Greece as Berlin is to __", while the syntactic questions relate to verb or adjective forms such as "dance is to dancing as fly is to __". The basic analogy statement is answered by finding the closest vector u u u d to u u u b − u u u a + u u u c 4 in the embedding space via cosine similarity 5. The task has been shown under specific assumptions to be provably solvable by methods such as GloVe and Skip-gram BID8 BID12 ] and as such is closely related to solving the objectives introduced in Sections 1 and 4.1. In this section, of the IWLRLS algorithm are performed for the Tweedie, Multinomial, and Poisson models. Based on the additional experiments in Sections B.2-B.6 we train the Tweedie model with p = 1.25 (Section B.3) and for all models include weight truncation to penalize large co-occurrences (Section B.4), regularization terms (outlined in Section B.5), and include only a single bias term within the systematic component of the Tweedie model (Section B.6).Step To demonstrate the effectiveness of performing multiple iterations of the IWLRLS algorithm, we present for the one-step estimator, an early-stopped estimator, and the full-likelihood estimator. Of particular interest in our are the Tweedie one-step estimator (a variant of the GloVe method), and the full-likelihood estimator for the Multinomial (a variant of the Skip-gram method). For the in TAB1, the full-likelihood is taken to be the iteration which achieves the maximum total accuracy on the analogy task, and the early-stop algorithm is taken to be an iteration between the one-step and full-likelihood iterations which performs best in total accuracy on the analogy task. For both the Tweedie and Multinomials, the full-likelihood is the after 3 iterations and the early-stopped is the after 2 iterations. For the Poisson model, the full-likelihood is the after 9 iterations, and the early-stopped is the after 3 iterations. We find a small difference in total accuracy on the analogy task with the one-step estimator (GloVe) and the full-likelihood differing by roughly 1%. We find a similar relationship in the Poisson estimator and further note that the early-stopped estimator for the Poisson has very similar accuracy to the full-likelihood algorithm. Finally, the Multinomial model yields a difference of 2% between the full-likelihood algorithm (Skip-gram) and the one-step algorithm. The early-stopped algorithm for the Multinomial also performs 1% higher than the one-step algorithm indicating a fair tradeoff between running an additional iteration and stopping after only one iteration. We present a general model-based methodology for finding word vectors from a corpus. This methodology involves choosing the distribution of a chosen co-occurrence matrix to be an exponential dispersion family and choosing the number of iterations to run our algorithm. In Table 1, we see that our methodology unifies the dominant word embedding methods available in the literature and provides new and improved methods. We introduce an extension of Skip-gram that is stopped before full-convergence analagously to GloVe and an extension to GloVe beyond one iteration. Experimental on a small corpus demonstrate our method improves upon GloVe and Skip-gram on the Google word analogy similarity task. It is our hope that this methodology can lead to the development of better, more statistically sound, word embeddings and consequently improve on many other downstream tasks. Further in exponential dispersion families and generalized linear models is developed here. We begin by discussing exponential dispersion families, the distribution of the response in generalized linear models. Definition 2 Let y ∈ R be a random variable. If the density function f (y; θ, ϕ) of y satisfies DISPLAYFORM0 over its support, then the distribution of y is in the exponential dispersion family. The parameter θ is the natural parameter, ϕ is the dispersion parameter, and the function ψ is the cumulant generating function. In many cases, the function δ(ϕ) is very simple, meaning that, for instance, δ(ϕ) = 1 or δ(ϕ) = ϕ.The function c(y; ϕ) can be viewed as the normalizing constant ensuring that the density integrates to one. When y follows a distribution in the exponential dispersion family with natural parameter θ, its mean µ = ψ (θ), so we can equivalently specify the mean µ or the natural parameter θ. Many classical distributions such as the Poisson, Normal, Binomial, and Gamma distribution are exponential dispersion families. For example, when y ∼ Normal(µ, σ 2) is a normal distribution with mean µ and variance σ 2, its log density satisfies DISPLAYFORM1 showing that here the natural parameter θ = µ, the dispersion parameter ϕ = σ 2, the functions ψ(θ) = 1 2 θ 2, δ(ϕ) = ϕ, and c(y; ϕ) = 1 2 log(2πσ 2) + y 2 2σ 2. The Tweedie distribution BID15, of particular importantance to us, also lies within the exponential dispersion family. Instead of defining the Tweedie distribution through the form of its density, we will define it through the relationship between its mean and variance. This relies on a from [Jørgensen, 1987, Theorem 1] that distributions within the exponential dispersion family are defined by the relationship between their mean and variance. Definition 3 A random variable y has a Tweedie distribution with power parameter p ∈ {0} ∪ [1, ∞) when DISPLAYFORM2 and the distribution of y is an exponential dispersion family. In this case, we write y ∼ Tweedie p (µ, ϕ), where µ = E(y) is the mean. The Normal distribution discussed above has a variance that does not depend on the mean. In our new notation, this means that the Normal distribution is a Tweedie distribution with power parameter p = 0. The Poisson distribution has variance equal to the mean and is in the exponential dispersion family, so is a Tweedie distribution with power parameter p = 1 and dispersion parameter ϕ = 1. A Gamma distribution with shape parameter α and rate parameter β is a Tweedie distribution with power p = 2, mean µ = α β, and dispersion parameter ϕ = α −1.We will only consider Tweedie distributions with power parameter p ∈. These distributions are also known as compound Poisson-Gamma distributions due to the representation DISPLAYFORM3 where n ∼ Poisson(λ) and g i DISPLAYFORM4 ∼ Gamma(α, β), and λ = BID15. It is important to note that the Tweedie distribution has positive mass at zero, an important characteristic for capturing the zero-inflation prominent in some co-occurence matrices due to some words never appearing within the same context. Specifically, DISPLAYFORM5 Using other arguments related to representations of the mean and variance in terms of the cumulant generating function ψ, BID15 show that the Tweedie distribution has ψ(θ) DISPLAYFORM6 Exponential dispersion families are defined over real numbers. Now, we generalize their definition to a multivariate setting. Definition 4 Let y y y ∈ R m be a random vector. If the density function f (y y y; θ θ θ, ϕ) of y y y satisfies log f (y y y; θ θ θ, ϕ) = y y y DISPLAYFORM0 over its support, then the distribution of y y y is in the multivariate exponential dispersion family. The parameter θ θ θ ∈ R m is the natural parameter, ϕ ∈ R is the dispersion parameter, and the function ψ: R m → R is the cumulant generating function. A collection of independent draws from the same exponential dispersion family is a multivariate exponential dispersion family. To see this, let y i (i = 1, . . ., m) be i.i.d. from an exponential dispersion family. Then, the density of y y y satisfies log f (y y y; θ θ θ, ϕ) = DISPLAYFORM1 + c(y j ; ϕ), which has cumulant generation function ψ(θ θ θ) = m j=1 ψ(θ j). Another useful example of a multivariate exponential dispersion family is the Multinomial. Let x x x ∈ R c have be distributed as x x x ∼ multinomial(s, π π π), where s ∈ N is the total number of draws and π π π ∈ R c is the probability vector. Introduce a change of parameters where DISPLAYFORM2 showing that the multinomial distribution is in the multivariate exponential dispersion family with ψ(θ θ θ) = s log (c k=1 exp θ k). We start by reviewing the linear model. Given a response y y y ∈ R n comprising n observations, the model for y y y is a linear model with covariates x x x i ∈ R p when y i ind.∼ Normal(x x x T i β β β, σ 2) for all i ∈ {1, . . ., n}. In vector notation, this reads that y y y ∼ Normal(Xβ β β, σ 2 I), where X ∈ R n×p is a matrix with i th row x x x T i. This is one of the more primitive models in the statistical modeling toolbox and isn't always appropriate for the data. Generalized linear models remove the the assumptions of normality and that the mean is a linear function of the coefficients β β β. Definition 5 For some exponential dispersion family ED(µ, ϕ) with mean parameter µ and dispersion parameter ϕ, the model for y y y ∈ R n is a generalized linear model with link function g when DISPLAYFORM0 DISPLAYFORM1 In the first line of the displayed relationships, the distribution of the response y y y is described. In the third line, the systematic component η i expresses the effect of the covariates x x x i. The second line connects the distribution to the covariates through the link function. That is, the covariates effect a link-transformed mean. The canonical link (b) −1 is often chosen as the link function, due to its computational and mathematical properties BID0. Other times, the canonical link is inappropriate and there are alternative default choices. Generalized linear models are used as the default modeling framework in many fields of applied science for non-normal distributions []. When g(µ) = µ is the identity map and ED is the Normal distribution, the generalized linear model is simply the linear model. When g(µ) = logit(µ) = log 1−µ µ and ED is the Binomial distribution, the generalized linear model is logistic regression. Further, a generalized linear model can be viewed as a no-hidden-layer neural network with activation function g. We train our models on the English text8 corpus 6 with approximately 17 million tokens. We filter out word types that occur fewer than 50 times to obtain a vocabulary size of approximately 11, 000; a ratio consistent with other embedding literature experiments 7.The adjustable model configurations in IWLRLS are the choice of power parameter p, penalty tuning parameter λ, and co-occurrence processing step. We experiment with different choices of p ∈ {1.1, 1.25, 1.5, 1.75, 1.9}, different choices of processing including no processing, clamping the weights (as in GloVe) and truncating the outliers in the co-occurrence matrix (elaborated on in Section B.4, and set the penalty tuning parameter λ = 0.002. The estimated word vectors are the rows of DISPLAYFORM0 . For all of our experiments, we set the dimension of the word vectors to d = 150, and the objective function at each iteration is optimized using Adagrad BID5 with a fixed learning rate of 0.1 8 . Models are trained for up to 50 epochs (50 passes through the co-occurrence matrix) with batches of size 512. We evaluate the impact of multiple iterations of the IWLRLS algorithm on all models, but examine different additions to the model only when p = 1.25. We believe the impact of these changes will be present however for any value of p. We present of multiple iterations of our IWLRLS algorithm with different distributions. In particular, we perform multiple iterations of the IWLRLS algorithm with Tweedie distribution and weight truncation to match the GloVe objective function and processing by setting the weight function in our model from h(x) = x 2−p to h(x) = (min{x, x max}).75 with x max = 10 and p = 1.25. We also presents for an early-stopped version of skip-gram, and the new Poisson estimator. The are summarized in FIG4. We remark on a few observations based on these . First, as the number of steps increases, the accuracy on the analogy task increases for the first few iterations. Second, relatively few steps are needed with the accuracy of Tweedie model performing best at the first and second steps of the algorithm, and the Multinomial and model performing best in steps 3-5 but with very similar performance at earlier steps. The Poisson model performs best after 9 iteration, however performs nearly identically to the of an early stopped algorithm at 3 iterations. In , we find that early-stopped and one-step versions of the algorithm can perform comparably to full-likelihood methods. In this section, we examine the effect of the choice of the power p in the tuning parameter when you run a Tweedie generalized low rank model. Table 3: Results for multiple choices of p for one and two iterations. The Results in Table 3 show that values of p which are high perform poorly, while values of p below 1.5 perform similarly. We find that p = 1.25 performs the best, and view this value of p as a good choice as it accounts for zero-inflation present in the co-occurence X. This also agrees with the of [] and BID16 ].An even more interesting and perhaps more appropriate way to estimate the power p of the Tweedie distribution is in a data-driven and model-based way. This approach is taken in BID16. In future work, we plan to use an improved estimating equation relative to BID16 ] to estimate p as part of the algorithm. This would be modeling the marginal distribution of the co-occurences as being Tweedie with the same power. Under a similar assumption, modified likelihood calculations are tractable and so are another possibility. We plan to explore this in future work. We set p = 1.25 in our algorithm with Tweedie distribution, and explore the effect of different strategies in handling large entries in the co-occurrence matrix X. One strategy is to simply input X into step 3 of our method. A second strategy is to clamp the weight h(·) that from step 3 of our method by taking h(x) = (min{x, x max}).75 as in GloVe. A third strategy is to input min{x, x max} for each entry of the matrix X, where x max = 10, into step 3 of our method 9.We find that no adjustment to the weights and GloVe's method of weight truncation both perform similarly with weight truncation slightly outperforming no adjustment. We suspect a more significant improvement will show with larger corpora such as a full Wikipedia corpus. Alternative approaches to alleviating the problem of large co-occurences are to use a more robust distribution or link function. Indeed, the weight truncation in GloVe can be directly mimicked by Table 4: Results for multiple choices of regularizing the large values of the co-occurrence matrix. Our strategies are harmonic matrix, truncation of the weight only, truncation of the co-occurrence matrix to x max = 10.either altering the distribution or the link function. The desired form can be found via the weight and pseudo-response equations in algorithm 1. We leave this to future work. Table 5: Results for including the penalty term in Equation FORMULA0 and not including the diagonal terms. We consider regularizing the word vectors by including the penalty DISPLAYFORM0 with λ =.002 for two reasons. One is to reduce noise in the estimation of the word vectors. Udell et al. [2016, Lemma 7.3] show that penalizing by is equivalent to penalizing by λ U V T *, the nuclear norm of U V T. Since penalizing the nuclear norm U V T shrinks the dimension of the embedding and larger dimensional embeddings tend to be better [], we choose a small tuning parameter to reduce noise while still preserving the dimension. Another reason is to symmetrically distribute the singular values ofÛV T to both matricesÛ andV.Write the singular value decompositionÛV T = U ΣV T, for U and V orthogonal and Σ diagonal. Mu et al. [2018, Theorem 1] shows that using penalty in havingÛ = U Σ 1/2 Q and V = V Σ 1/2 Q for some orthogonal matrix Q. This is desirable since it was argued empirically by Levy et al. that a symmetric distribution of singular values works optimally on semantic tasks. Finally, we experiment with whether the penalty introduced in Equation FORMULA0 improves and accurately reduces noise in the estimate. We also consider not including the diagonal elements of X as a form of regularization and experiment here as well, as these terms are often large (can be considered as outliers) and do not contain a great deal of linguistic information. Table 5 demonstrates the included regularization within the IWLRLS algorithm with Tweedie distribution and p = 1.25 improves . In Experiment 1, we found that the Multinomial model outperforms the Poisson model, although the Poisson model has an additional bias term to model context word frequencies. This was fairly counterintuitive, so we additionally experiment with having only a single bias term a i in the Tweedie model as in the Multinomial model. We find overall that the Tweedie model with a systematic component without the bias term b j performs slightly better than the Tweedie model with systematic component containing both bias terms a i and b j. We hope to further study the impact of bias terms and other systematic components in future work. Table 6: Results for including the bias term on the context word b j in addition to a i."

A structured latent-variable approach that adds discrete control states within a standard autoregressive neural paradigm to provide arbitrary grounding of internal model decisions, without sacrificing any representational power of neural models.

Represent the TLDR for finding the corresponding scientific article!

A fully connected architecture is used to produce word embeddings from character representations, outperforms traditional embeddings and provides insight into sparsity and dropout.