Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Graph Neural Networks as a combination of Graph Signal Processing and Deep Convolutional Networks shows great power in pattern recognition in non-Euclidean domains. In this paper, we propose a new method to deploy two pipelines based on the duality of a graph to improve accuracy. By exploring the primal graph and its dual graph where nodes and edges can be treated as one another, we have exploited the benefits of both vertex features and edge features. As a , we have arrived at a framework that has great potential in both semisupervised and unsupervised learning. Convolutional Neural Networks (CNNs) has been very successfully used for automated feature extraction in Euclidean domains, especially for computer vision, such as 2D image classification, object detection, etc. However, many real-life data has a non-Euclidean graph structure in nature, from which we want to investigate the underlying relations among different objects by utilizing the representation of nodes and edges. Recently, research on applying the generalization of Convolutional Neural Networks to the non-Euclidean domains has attracted growing attention. As a , a branch of research on Geometric Deep Learning based on that has been ignited. Previous works including ChebNet and GCN have demonstrated strong in solving problems in semi-supervised learning where the labels of only a few objects are given, and we want to find out the labels of other objects through their inner connections. Current methods generalizing convolution operations include both spatial and spectral domains . The spatial one deals with each node directly in the vertex domain while the spectral one takes a further step in converting signals via graph Fourier transform into the spectral domain. However, one critical weakness would be the fact that the interchangeable and complementary nature between nodes and edges are generally ignored in previous research. As a , the duality of the graph is not fully utilized. If we treat those edges in the original, or known as the primal graph, as the nodes in the new graph, and original nodes as edges, we can arrive at a new graph that further exploits the benefits of edge features. In such a way, we are able to get both the primal graph and the dual graph . By combining both the vertex features and the edge features, we will be able to solve wider range of problems and achieve better performance. In this paper, we propose a new approach to transform the primal graph into its dual form and have implemented two pipelines based on these two forms of graph to improve the accuracy and the performance. With two pipelines, we also exploited a path to make the model wider instead of merely deeper. Meanwhile, we have developed a new framework that can be applied later on both semi-supervised learning and unsupervised learning. Graph-based semi-supervised learning aims to annotate data from a small amount of label data on a graph. To learn the vectors that can recover the labels of the training data as well as distinguish data with different labels, conventionally, graph Laplacian regularizer gives penalty between sampling based on graph Laplacian matrix (; ;). Sample-based method takes random walk to get samples from the context of data points in order to propagate information (; ;). Graph Convolutional Networks generalize the operation of convolution from grid data to graph data . After the emergence of the spectral-based convolutional networks on graph , ChebNet approximate the filters by Chebyshev polynomials according to the Laplacian eigendecomposition. GCN simplifies ChebNet by introducing its first-order approximation and can be viewed as a spatial-based perspective, which requires vertices in the graph to propagate their information to the neighbors. MoNet is a spatial-based method, of which convolution is defined as a Gaussian mixture of the candidates. GAT (Veličković et al. ) applies the attention mechanism to the graph network. DGI (Veličković et al. ) proposes a framework to learn the unsupervised representations on graph-structured data by maximizing location mutual information. We refer to;;; as a more comprehensive and thorough review on graph neural networks. Dual approaches on graph networks usually unlike the above mono-methods, apply mixed methods to study graph networks. DGCN makes a balance between the spatialbased domain and spectral-based domain by regularizing their mutual information. GIN proposes a dual-path from graph convolution on texts and another network on images to gather cross-modal information into a common semantic space. DPGCNN extends the classification on vertices to edges by considering the attention mechanism on both. Our study follows this path, which classifies vertices from the relationship between them (edges) and regularization from the mutual information between classification on both vertices and edges. 3.1 PRELIMINARIES Let G = {V, E, A} denote a graph, where V = {1, . . ., N} is the set of nodes with |V| = N, E is the set of edges, and A = (A (i,j)∈V = 0) ∈ R N ×N is the adjacency matrix. When G is undirected then A is symmetric with A i,j = A j,i, G is an undirected graph, otherwise a directed graph. The Laplacian matrix, also acts a propagation matrix, has the combinatorial form as L = D − A ∈ R N ×N, and its normalized form is N ×N is the degree matrix of graph G with d(i) = j∈V A i,j and I ∈ R N ×N is the identity matrix. In some literature, the random walk Laplacian L rw = I − D −1 A is employed to directed graph G. Let L = U ΛU T be the eigendecomposition, where U ∈ R N ×N is composed of orthonormal eigenbasis and Λ = diag(λ 0, λ 1, . . ., λ N −1) is a diagonal matrix of eigenvalues which denotes frequencies of graph G, and λ i and u i form an eigenpair. The convolutional operator * G on the graph signal x is defined by wheref = U T x andĝ = U T g are regarded as the graph Fourier transform of graph signal x and graph filter g, respectively; f = U (·) is the inverse graph Fourier transform, and is the Hadamard product. Ĝ = diag(ĝ 0, · · ·,ĝ N −1) behaves as spectral filter coefficients. Graph convolution can be approximated by polynomial filters, the k-th order form is where Based on the above approximation, ChebNet further introduces Chebyshev polynomials into graph filters of the convolutional layers for the sake of computational efficiency. Chebyshev polynomials are recursively expressed as T i (x) = 2xT i−1 (x) − T i−2 (x) with T 0 (x) = 1 and T 1 (x) = x. The graph filter then becomes whereL = 2/λ max L − I denotes the scaled normalized Laplacian for all eigenvalues λ i ∈ [−1, 1] and θ i is trainable parameter. Graph Convolutional Network (GCN) is a variant of ChebNet which only takes first two terms of Equation. By setting the coefficients θ 0 and θ 1 as θ = θ 0 = −θ 1 and with λ max = 2, the convolution operator in convolution layer of GCN is induced as g In graph theory, The definition of the dual varies according to the choice of embedding of the graph G. For planar graphs generally, there may be multiple dual graphs, depending on the choice of planar embedding of the graph. In this work, we follow the most common definition. Given a plane graph G = {V, E A}, which is designated as the primal graph, the dual graphĜ = {Ṽ = E,Ẽ,Ã} is a graph that has a vertex (or node) for each edge of G. The dual graphĜ has an edge whenever two edges of G share at least one common vertex. To be clarified, the vertices (i, j) and (j, i) of dual graphĜ converted from a undirected graph are regarded as the same. Fig.1 shows the conversion from primal graph to its dual counterpart. When vertices of the primal graph embed features (or signals in terminology of spectral graph theory), the features of a dual node can be obtained by applying a specified functions to its corresponding primal nodes' features, i.e. the simplest applicable function is to calculate the distance between the features of two nodes. In addition, if the edges of primal graph possess features or attributes, we also take them into account as the their inherited features of dual nodes. Take node of dual graph in Fig.1b) as an example, its feature is obtained by performing the element-wise subtraction to the feature vectors of nodes 0 and 3 of primal graph in Fig.1a), i.e. T − T = [0, −1, 1] T. The Twin Graph Convolutional Networks (TwinGCN) proposed in this work consists of two pipelines. Both pipelines are built with the same architecture as GCN, and contain two convolution layers in each pipeline, as shown in Fig.2. The upper pipeline acts exactly as GCN; however, the lower one takes the dual featuresX derived from primal features X as its inputs (as described in section 3.3), the predictions or outputs in dual vertex domain (i.e. edge domain in primal) is then aggregated to primal vertex domain. The goal of introducing a dual pipeline into the model is that we desire to utilize the predictions on the dual node (edges in primal graph) to affect the predictions on primal nodes since the knowledge about those neighbors of a node can be propagated through edges. For the purpose of training the dual pipeline, we also need to get the labels of dual nodes. Let us take an example, given a dual node (i, j) (corresponds to an edge in primal graph), primal node i has label α and j has label β, then dual node (i, j) is assigned with a label (α, β). One thing worth mentioned is that TwinGCN's convolution layers are not limited to those used in GCN, they can be replaced with other types of convolution layer, such as ChebNet, GWNN , etc. The convolution layers in the pipelines perform graph convolution operations with shared weights as learnable parameters, mathematically expressed as where H (l) is the activation in l-th layer, W (l) is learnable weights in that layer. σ represents nonlinear activation function, e.g. ReLU. For the task of semi-supervised node classification, the loss function is defined as where Y L is set of node labels for L ∈ V labeled node set, F denotes the number of labels of the nodes, and Z is predicted outcome, a softmax of the output of the network. In order to take effect of dual pipeline on the prediction of primal pipeline, we adopt KullbackLeibler Divergence (D KL) as a regularization term in training. Suppose that P (Y |X) is predictions by primal pipeline and P (Ŷ |X) = P (Ŷ |X) is the derived predictions obtained through an aggregation from the predictions on dual labels by dual pipeline to primal label predictions. X is derived from X as aforementioned (Section 3.3). We first calculate the joint probability matrix P (Y,Ŷ) of two matrices P (Y |X) and P (Ŷ |X) we further get the marginal probabilities of P (Y) and P (Ŷ) from P (Y,Ŷ). KullbackLeibler Divergence D KL is evaluated by finally, we attains the loss function as 3 illustrates a fast algorithm deriving primal predictions from predictions of dual pipeline. It is conducted by introducing two special incidence matrices. The matrix at the left hand side (N × M, N = |V| and M = |E|) is an incidence matrix in which the rows represent primal nodes, each column depicts whether a primal node in a row has an incidence in the dual node represented by this column. The rightmost matrix is the incidence matrix of primal labels presenting in dual labels with dimension of L 2 × L. Although these two matrices are extremely sparse when node number is very large (we store them in compressed form), by taking advantage of GPU's powerful computing capability, the delicate sparse matrix multiplication subroutine, e.g. Nvidia's cuSARSE, runs much faster than codes with loops for lumping the incidences. In this section, we evaluate the performance of TwinGCN, we mainly focus on semi-supervised node classification in current work. Actually, TwinGCN also support unsupervised learning by changing the loss functions which we will fulfill in future work. We conduct experiments on three benchmark datasets and follow existing studies (; ; etc.) The datasets include Cora, Citeseer, and Pubmed . All these three datasets are collected from their corresponding citation networks, the nodes represent documents and edges are the citations. Table 4.1 shows details of these datasets. Label rate indicates the portion of the available labeled nodes used for training. The training process takes 20 labeled samples for each class for every dataset. Since both pipelines of our proposed architecture work with graph convolution based on spectral graph theory, we use recent works, such as ChebNet GCN , and GWNN , etc. These models maintain the same graph Lapalacian base structure, unlike some other methods take partial graph structure, e.g. FastGCN applies Monte Carlo importance sampling on edges. however, this kind of method only guarantees the convergence as the sample size goes to infinity. For the sake of consistency for comparison, the hyper-parameters for training are kept the same for primal pipeline as other models. The primal are composed with two graph convolution layers with 16 hidden units and applied with ReLU non-linear activations. Loss is evaluated with the softmax function. Dropout of primal pipeline is set to p = 0.5 for the primal. We use the Adam optimizer for optimizing the weights with an initial learning rate lr = 0.01. As the dual graph is normally much bigger than the counterpart primal graph, its adjacency/Laplacian matrix and the number of dual nodes becomes quadratically larger, e.g. N nodes with N × (N − 1) edges in a fully-connected graph. Therefore, to avoid overfitting on dual pipeline, we set its dropout rate higher than 70%. We also introduce a sampling rate to extract a small fraction from the total dual node labels. Having a large number of edges in the primal graph also means a large number of dual nodes. In such situation, the performance will be degraded severely. The quantitative comparison among different models is given in Table 4.4. For node classification, TwinGCN achieves similar or outperforms with some datasets. The performance gain comes from the aggregation of knowledge propagated through edges (or dual nodes) trained by dual pipeline. However, primal pipeline only will ignore the dependency between labels of nodes. Fig.4a ) illustrate that when compared to the GCN, TwinGCN bearing two pipelines converges slower but achieves a higher accuracy as the number of epoch increases. This is because that we have two pipelines through mutual interaction. In Fig.4b ), we observe that two loss curves of traditional GCN and TwinGCN have very similar decreasing trends. However, the loss curve of TwinGCN is slightly above GCN because the loss of TwinGCN is the summation of both primal and dual pipelines. To test whether the introduced dual pipeline and regularization improve the basic GCN pipeline, we conducted controlled experiments to make comparison among GCN, GCNs with pipelines on original graph and dual graph and TwinGCN(GCNs with both pipelines and regularization by KLdivergence). Method Cora Citeseer Pubmed GCN 81.5% ± 0.3% 70.8% ± 0.1% 78.8% ± 0.1% GCN(double-pipeline) 81.6% ± 0.4% 72.5% ± 1.0% 79.8% ± 2.3% TwinGCN 83.0% ± 1.3% 72.5 ± 0.8% 79.5% ± 1.2% In this work, we propose the TwinGCN with parallel pipelines working on both the primal graph and its dual graph, respectively. TwinGCN achieves the state-of-the-art performance in semisupervised learning tasks. Moreover, TwinGCN's ability is not limited to this, we can extend its power/utilization into unsupervised learning by altering its loss functions. Use unnumbered third level headings for the acknowledgments. All acknowledgments, including those to funding agencies, go at the end of the paper.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A primal dual graph neural network model for semi-supervised learning

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: represent the scientific article to find a short tldr

Question: We describe two end-to-end autoencoding models for semi-supervised graph-based dependency parsing. The first model is a Local Autoencoding Parser (LAP) encoding the input using continuous latent variables in a sequential manner; The second model is a Global Autoencoding Parser (GAP) encoding the input into dependency trees as latent variables, with exact inference. Both models consist of two parts: an encoder enhanced by deep neural networks (DNN) that can utilize the contextual information to encode the input into latent variables, and a decoder which is a generative model able to reconstruct the input. Both LAP and GAP admit a unified structure with different loss functions for labeled and unlabeled data with shared parameters. We conducted experiments on WSJ and UD dependency parsing data sets, showing that our models can exploit the unlabeled data to boost the performance given a limited amount of labeled data. Dependency parsing captures bi-lexical relationships by constructing directional arcs between words, defining a head-modifier syntactic structure for sentences, as shown in Figure 1. Dependency trees are fundamental for many downstream tasks such as semantic parsing (;, machine translation , information extraction and question answering . As a , efficient parsers have been developed using various neural architectures. While supervised approaches have been very successful, they require large amounts of labeled data, particularly when neural architectures are used. Syntactic annotation is notoriously difficult and requires specialized linguistic expertise, posing a serious challenge for low-resource languages. Semisupervised parsing aims to alleviate this problem by combining a small amount of labeled data and a large amount of unlabeled data, to improve parsing performance over labeled data alone. Traditional semi-supervised parsers use unlabeled data to generate additional features, assisting the learning process , together with different variants of self-training (Søgaard & Rishøj, 2010). However, these approaches are usually pipe-lined and error-propagation may occur. In this paper, we propose two end-to-end semi-supervised parsers based on probabilistic autoencoder models illustrated in Figure 3, Locally Autoencoding Parser (LAP) and Globally Autoencoding Parser (GAP). In LAP, continuous latent variables are used to support tree inference by providing a better representation, while in GAP, the latent information forms a probability distribution over dependency trees corresponding to the input sentence. A similar idea has been proposed by , but our GAP model differs fundamentally from their parser, as GAP does not sample from the posterior of the latent tree structure to approximate the Evidence Lower Bound (ELBO). Instead it relies on a tractable algorithm to directly compute the posterior to calculate the ELBO. We summarize our contributions as follows: 1. We proposed two autoencoding parsers for semi-supervised dependency parsing, with complementary strengths, trading off speed vs. accuracy; 2. We propose a tractable inference algorithm to compute the expectation and marginalization of the latent dependency tree posterior analytically for GAP, avoiding sampling from the posterior to approximate the expectation ; 3. We show improved performance of both LAP and GAP with unlabeled data on WSJ and UD data sets empirically, and improved of GAP comparing to a recently proposed semi-supervised parser . Most dependency parsing studies fall into two major groups: graph-based and transition-based . Graph-based parsers regard parsing as a structured prediction problem to find the most probable tree, while transition-based parsers (; 2008) treat parsing as a sequence of actions at different stages leading to a dependency tree. While earlier works relied on manual feature engineering, in recent years the hand-crafted features were replaced by embeddings and deep neural architectures, leading to improved performance in both graph-based parsing and transition-based parsing (; ;). More recent works rely on neural architectures for learning a representation for scoring structural decisions;;. The annotation difficulty for this task, has also motivated work on unsupervised (grammar induction) and semi-supervised approaches to parsing (; ; ; ; ; ;). Similar to other structured prediction tasks, directly optimizing the objective is difficult when the underlying probabilistic model requires marginalizing over the dependency trees. Variational approaches are a natural way for alleviating this problem, as they try to improve the lower bound of the original objective, and were applied in several recent NLP works (; b; ; b; a). Variational Autoencoder (VAE) is particularly useful for latent representation learning, and is studied in semi-supervised context as the Conditional VAE (CVAE) . The work mostly related to ours is as they consider the dependency tree as the latent variable, but their work takes a second approximation to the variational lower bound by an extra step to sample from the latent dependency tree, without identifying a tractable inference. We show that with the given structure, exact inference on the lower bound is achievable without approximation by sampling, which tightens the lower bound. A dependency graph of a sentence can be regarded as a directed tree spanning all the words of the sentence, including a special "word"-the ROOT-to originate out. Assuming a sentence length of l, a dependency tree can be denoted as T = (< h 1, m 1 >, . . ., < h l−1, m l−1 >), where h t is the index in the sequence of the head word of the dependency connecting the tth word m t as a modifier. Our graph-based parser is constructed by following the standard structured prediction paradigm . In inference, based on the parameterized scoring function S Λ with parameter Λ, the parsing problem is formulated as finding the most probable directed spanning tree for a given sentence x: where T * is the highest scoring parse tree and T is the set of all valid trees for the sentence x. It is common to factorize the score of the entire graph into the summation of its substructures: the individual arc scores : whereT represents the candidate parse tree, and s Λ is a function scoring each individual arc. s Λ (h, m) describes the likelihood of forming an arc from the head h to its modifier m in the tree. Through out this paper, the scoring is based on individual arcs, as we focus on first order parsing. We used the same neural architecture as that in. In this formulation, we first use two parameters to extract two different representations that carry two different types of information: a head seeking for its modifier (h-arc); as well as a modifier seeking for its head (m-arc). Then a nonlinear function maps them to an arc score. For a single sentence, we can form a scoring matrix as shown in Figure 4, by filling each entry in the matrix using the score we obtained. Therefore, the scoring matrix is used to represent the head-modifier arc score of all the possible arcs connecting words in a sentence . Using the scoring arc matrix, we build graph-based parsers. Since exploring neural architectures for scoring is not our focus, we did not explore other architectures, however performance shall be further improved using advanced neural architectures. Variational Autoencoder (VAE). The typical VAE is a directed graphical model with Gaussian latent variables, denoted by z. A generative process first generates a set of z from the prior distribution π(z) and the data x is generated as P θ (x|z) parameterized by θ given input x, In our scenario, x is an input sequence and z is a sequence of latent variables corresponding to it. The VAE framework seeks to maximize the complete log-likelihood log P (x) by marginalizing out the latent variable z. Since direct parameter estimation of log P (x) is usually intractable, a common solution is to maximize its Evidence Lower Bound (ELBO) by introducing an auxiliary posterior Q(x|z) distribution that encodes the input into the latent space. Tree Conditional Random Field. Linear chain CRF models an input sequence x = (x 1 . . . x l) of length l with labels y = (y 1 . . . y l) with globally normalized probability where Y is the set of all the possible label sequences, and S(x, y) the scoring function, usually decomposed as emission (for first order models. Tree CRF models generalize linear chain CRF to trees. For dependency trees, if POS tags are given, the tree CRF model tries to resolve which node pairs should be connected with direction, such that the arcs form a tree. The potentials in the dependency tree take an exponential form, thus the conditional probability of a parse tree T, given the sequence, can be denoted as: where is the partition function that sums over all possible valid dependency trees in the set T(x) of the given sentence x. We extend the original VAE model for sequence labeling to dependency parsing by building a latent representation position-wise to form a sequential latent representation. It has been shown that under the VAE framework the latent representation can reflect the desired properties of the raw input . This inspired us to use the continuous latent variable as neural representations for the dependency parsing task. Typically, each token in the sentence is represented by its latent variable z t, which is a high-dimensional Gaussian variable. This configuration on the one hand ensures the continuous latent variable retains the contextual information from lower-level neural models to assist finding its head or its modifier; on the other hand, it forces tokens of similar properties closer in the euclidean space. We adjust the original VAE setup in our semi-supervised task by considering examples with labels, similar to recent conditional variational formulations (; ;). We propose a full probabilistic model for any certain sentence x, with the unified objective to maximize for supervised and unsupervised parsing as follows: This objective can be interpreted as follows: if the training example has a golden tree T with it, then the objective is the log joint probability P θ,ω (T, x); if the golden tree is missing, then the objective is the log marginal probability P θ (x). The probability of a certain tree is modeled by a tree-CRF in Eq. 1 with parameters ω as P ω (T |x). Given the assumed generative process P (x|z), directly optimizing this objective is intractable, we instead optimize its ELBO (We show the details in the appendix, proving J lap is the ELBO of J in Lemma A.1): [log P ω (T |z)]. Instead of autoencoding the input locally at the sequence level, we could alternatively directly regard the dependency tree as the structured latent variable to reconstruct the input sentence, by building a model containing both a discriminative component and a generative component. The discriminative component builds a neural CRF model for dependency tree construction, and the generative model reconstructs the sentence from the factor graph as a Bayesian network, by assuming a generative process in which each head generates its modifier. Concretely, the latent variable in this model is the dependency tree structure. We model the discriminative component in our model as P Φ (T |x) parameterized by Φ, taking the same form as in Eq. 1. Typically in our model, Φ are the parameters of the underlying neural networks, whose architecture is described in Sec. 3.1. We use a set of conditional categorical distributions to construct our Bayesian network decoder. More specifically, using the head h and modifier m notation, each head reconstructs its modifier with the probability P (m t |h t) for the tth word in the sentence (0th word is always the special "ROOT" word), which is parameterized by the set of parameters Θ. Given Θ as a matrix of |V| by |V|, where |V| is the vocabulary size, θ mh is the item on row m column h denoting the probability that the head word h would generate m. In addition, we have a simplex constraint m∈V θ mh = 1. The probability of reconstructing the input x as modifiers m in the generative process is where l is the sentence length and P (m t |h t) represents the probability a head generating its modifier. With the design of the discriminative component and the generative component of the proposed model, we have a unified learning framework for sentences with or without golden parse tree. The complete data likelihood of a given sentence, if the golden tree is given, is where s Φ,Θ (h, m) = s Φ (h, m) + log θ mh, with m, x and T all observable. For unlabeled sentences, the complete data likelihood can be obtained by marginalizing over all the possible parse trees in the set T(x): where We adapted a variant of's algorithm to marginalize over all possible trees to compute both Z and U, as U has the same structure as Z, assuming a projective tree. We use log-likelihood as our objective function. The objective for a sentence with golden tree is: for sentence x l i with golden parse tree T l i in the labeled data set {x, T} l do Stochastically update the parameter Λ in the encoder using Adam while fixing the decoder. Compute the posterior Q(T) in an arc factored manner for x u i tractably. 10: Compute the expectation of all possible (h(head) → m(modif ier)) occurrence in the sentence x based on Q(T). Update buffer B using the expectation to the power for Obtain Θ globally and analytically based on the buffer B and renew the decoder. 14: end for If the input sentence does not have an annotated golden tree, then the objective is: Thus, during training, the objective function with shared parameters is chosen based on whether the sentence in the corpus has golden parse tree or not. Directly optimizing the loss in Eq.2 is difficult for the unlabeled data, and may lead to undesirable shallow local optima without any constraints. Instead, we derive the evidence lower bound (ELBO) of log P Θ,Φ (m|x) as follows, by denoting Q(T) = P Θ,Φ (T |m, x) as the posterior: Instead of maximizing the log-likelihood directly, we alternatively maximize the ELBO, so our new objective function for unlabeled data becomes max In addition, to account for the unambiguity in the posterior, we incorporate entropy regularization when applying our algorithm, by adding an entropy term − T Q(T) log Q(T) with a non-negative factor σ when the input sentence does not have a golden tree. Adding this regularization term is equivalent as raising the expectation of Q(T) to the power of 1 1−σ. We annealed σ from 1 to 0.3 from the beginning of training to the end, as in the beginning, the generative model is well initialized by sentences with golden trees that resolve disambiguity. In practice, we found the model benefits more by fixing the parameter Φ when the data is unlabeled and optimizing the ELBO w.r.t. the parameter Θ. We attribute this to the strict convexity of the ELBO w.r.t. Θ, by sketching the proof in the appendix. The details of training are shown in Alg. 1. The common approach to approximate the expectation of the latent variables from the posterior distribution Q(T) is via sampling in VAE-type models . In a significant contrast to that, we argue in this model the expectation of the latent variable (which is the dependency tree structure) is analytically tractable by designing a variant of the inside-outside algorithm in an arc decomposed manner. We leave the detailed derivation in the appendix. A high-level explanation is that assuming the dependency tree is projective, specialized belief propagation algorithm exists to compute not only the marginalization but also the expectation analytically, making inference tractable. Data sets First we compared our models' performance with strong baselines on the WSJ data set, which is the Stanford Dependency conversion of the Penn Treebank using the standard section split: 2-21 for training, 22 for development and 23 for testing. Second we evaluated our models on multiple languages, using data sets from UD (Universal Dependency) 2.3 . Since semi-supervised learning is particularly useful for low-resource languages, we believe those languages in UD can benefit from our approach. The statistics of the data used in our experiments are described in Table 3 in appendix. To simulate the low-resource language environment, we used 10% of the whole training set as the annotated, and the rest 90% as the unlabeled. Input Representation and Architecture Since we use the same neural architecture in all of our models, we specify the details of the architecture once, as follows: The internal word embeddings have dimension 100 and the POS embeddings have dimension 25. The hidden layer of the bi-LSTM layer is of dimension 125. The nonlinear layers used to form the head and the modifier representation both have 100 dimension. For LAP, we use separate bi-LSTMs for words and POSs. In GAP, using "POS to POS" decoder only yield the satisfactory performance. This echos the finding that complicated decoders may cause "posterior collapse" (van den ;). Training In the training phase, we use Adam to update all the parameters in both LAP and GAP, except the parameters in the decoder in GAP, which are updated by using their global optima in each epoch. We did not take efforts to tune models' hyper-parameters and they remained the same across all the experiments. We first evaluate our models on the WSJ data set and compared the model performance with other semi-supervised parsing models, including CRFAE , which is originally designed for dependency grammar induction but can be modified for semi-supervised parsing, and "differentiable Perturb-and-Parse" parser (DPPP) . To contextualize the , we also experiment with the supervised neural margin-based parser (NMP) , neural tree-CRF parser (NTP) and the supervised version of LAP and GAP, with only the labeled data. To ensure a fair comparison, our experimental set up on the WSJ is identical as that in DPPP and we use the same 100 dimension skip-gram word embeddings employed in an earlier transition-based system . We show our experimental in Table 1. As shown in this table, both of our LAP and GAP model are able to utilize the unlabeled data to increase the overall performance comparing with only using labeled data. Our LAP model performs slightly worse than the NMP model, which we attribute to the increased model complexity by incorporating extra encoder and decoders to deal with the latent variable. However, our LAP model achieved comparable on semi-supervised parsing as the DPPP model, while our LAP model is simple and straightforward without additional inference procedure. Instead, the DPPP model has to sample from the posterior of the structure by using a "GUMBEL-MAX trick" to approximate the categorical distribution at each step, which is intensively computationally expensive. Further, our GAP model achieved the best among all these methods, by successfully leveraging the the unlabeled data in an appropriate manner. We owe this success to such a fact: GAP is able to calculate the exact expectation of the arc-decomposed latent variable, the dependency tree structure, in the ELBO for the complete data likelihood when the data is unlabeled, rather than using sampling Model UAS DPPP (L) 88.79 DPPP (L+U) 89.50 CRFAE (L+U) 82.34 NMP Table 2: In this table we compare different models on multiple languages from UD. Models were trained in a fully supervised fashion with labeled data only (noted as "L") or semi-supervised (notes as "L+U"). "ST" stands for self-training. to approximate the true expectation. Self-training using NMP with both labeled and unlabeled data is also included as a base-line, where the performance is deteriorated without appropriately using the unlabeled data. We also evaluated our models on multiple languages from the UD data and compared the model performance with the semi-supervised version of CRFAE and the fully supervised NMP and NTP. To fully simulate the low-resource scenario, no external word embeddings were used. We summarize the in Table 2. First, when using labeled data only, LAP and GAP have similar performance as NMP and NTP. Second, we note that our LAP and GAP models do benefit from the unlabeled data, compared to using labeled data only. Both our LAP and GAP model are able to exploit the hidden information in the unlabeled data to improve the performance. Comparing between LAP and GAP, we notice GAP in general has better performance than LAP, and can better leverage the information in the unlabeled data to boost the performance. These validate that GAP is especially useful for low-resource languages with few annotations. We also experimented using self-training on the labeled and unlabeled data with the NMP model. As show, selftraining deteriorate the performance especially when the size of the training data is small. In this paper, we present two semi-supervised parsers, which are locally autoencoding parser (LAP) and globally autoencoding parser (GAP). Both of them are end-to-end learning systems enhanced with neural architecture, capable of utilizing the latent information within the unlabeled data together with labeled data to improve the parsing performance, without using external resources. More importantly, our GAP model outperforms the previous published semisupervised parsing system on the WSJ data set. We attribute this success to two reasons: First, our GAP model consists both a discriminative component and a generative component. These two components are constraining and supplementing each other such that final parsing choices are made in a checked-and-balanced manner to avoid over-fitting. Second, instead of sampling from posterior of the latent variable (the dependency tree) , our model analytically computes the expectation and marginalization of the latent variable, such that the global optima can be found for the decoder, which leads to an improved performance. A APPENDIX Lemma A.1. J lap is the ELBO (evidence lower bound) of the original objective J, with an input sequence x. Denote the encoder Q is a distribution used to approximate the true posterior distribution P φ (z|x), parameterized by φ such that Q encoding the input into the latent space z. Proof. Combining U and L leads to the fact: In practice, similar as VAE-style models, E z∼Q φ (z|x) [log P θ (x|z)] is approximated by, where z j is the jthe sample of N samples sampled from Q φ (z|x). At prediction stage, we simply use µ z rather than sampling z. Here we used a mean field approximation together with the conditional independence assumption by assuming P θ (z|x) ≈ l t=1 Q φ (z t |x t). The generative model P θ (x|z) acting as decoder parameterized by θ tries to regenerate the specific input x t at time step t from the latent space z t, as we assume conditional independence in the generative process among P θ (x t |z t). The encoder and the decoder are trained jointly in the classical variational autoencoder framework, by minimizing the KL divergence between the approximated posterior and the true posterior. We describe the encoder and decoder formulation. We parameterize the encoder Q φ (z t |x t) in such a way: First a bi-LSTM is used to obtain a non-linear transformation h t of the original x t; then two separate MLPs are used to compute the mean µ zt and the variance σ 2 zt. The generative story P θ (x t |z t) follows such parameterization: we used a MLP of two hidden layers in-between to take z t as the input, and then predict the word (or POS tag) over the vocabulary, such that the reconstruction probability can be measured. Following traditional VAE training paradigms, we also apply the "re-parameterization" trick to circumvent the non-differentiable sampling procedure to sample z t from the Q φ (z t |x t). Instead of directly sample from N (µ zt, σ 2 zt), we form z t = µ zt + σ 2 zt by sampling ∼ N (0, I). In addition, to avoid hindering learning during the initial training phases, following previous works , we anneal the temperature on the KL divergence term from a small value to 1. From an empirical Bayesian perspective, rather than fixing the prior using some certain distributions, it is beneficial to estimate the prior distribution directly from the data by treating prior's parameters part of the model parameters. Similar to the approach used in the previous study , LAP also learns the priors from the data by updating them iteratively. We initialize the priors from a standard Gaussian distribution N (0, I), where I is an identity matrix. During the training, the current priors are updated using the last optimized posterior, following the rule: where P (x) represents the empirical data distribution, and k the iteration step. Empirical Bayesian is also named as "maximum marginal likelihood", such that our approach here is to marginalize over the missing observation as a random variable. In previous studies exploring parsing using neural architectures, POS tags and external embeddings have been shown to contain important information characterizing the dependency relationship between a head and a child. Therefore, in addition to the variational autoencoding framework taking as input the randomly initialized word embeddings, optionally we can build the same structure for POS to reconstruct tags and for external embeddings to reconstruct words as well, whose variational objectives are U p and U e respectively. Hence, the final variational objective can be a combination of three: U = U w (The original U in Lemma A.1) + U p + U e (or just U = U w + U p if external embeddings are not used). Assuming the sentence is of length l, and we have obtained a arc decomposed scoring matrix S of size l ×l, and an entry S[i, j] i =j,j =0 stands for the arc score where ith word is the head and jth word the modifier. We first describe the inside algorithm to compute the marginalization of all possible projective trees in Algo.2. We then describe the outside algorithm to compute the outside tables in Algo. 3. In this algorithm, stands for the logaddexp operation. Finally, with the inside table α, outside table β and the marginalization Z of all possible latent trees, we can compute the expectation of latent tree in an arc-decomposed manner. Algo. 4 describes the procedure. It the matrix P containing the expectation of all individual arcs by marginalize over all other arcs except itself. Light modification is needed in our study to calculate the expectation w.r.t. the posterior distribution Q(T) = P Θ,Φ (T |m, x), as we have In this section we derive the strict convexity of ELBO w.r.t. Θ. Since we only care about the term containing Θ, the KL divergence term degenerates to a constant. For sentence i, Q(T i) has been derived in the previous section as matrix P and 1 is the indication function. Q(1(h → m)) log θ mh Q(1(h → m)) is a Bernoulli distribution, indicating whether the arc (h → m) exists. s.t. DATA SET STATISTICS We show the details of the statistics of the WSJ data set, which is the Stanford Dependency conversion of the Penn Treebank and the statistics of the languaes we used in UD (Universal Dependency) 2.3 here.

Represent the TLDR for finding the corresponding scientific article

We describe two end-to-end autoencoding parsers for semi-supervised graph-based dependency parsing.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose Bayesian quantized networks, for which we learn a posterior distribution over their quantized parameters.