Question: Represent the scientific article to find a short TLDR

Question: Graph embedding techniques have been increasingly deployed in a multitude of different applications that involve learning on non-Euclidean data. However, existing graph embedding models either fail to incorporate node attribute information during training or suffer from node attribute noise, which compromises the accuracy. Moreover, very few of them scale to large graphs due to their high computational complexity and memory usage. In this paper we propose GraphZoom, a multi-level framework for improving both accuracy and scalability of unsupervised graph embedding algorithms. GraphZoom first performs graph fusion to generate a new graph that effectively encodes the topology of the original graph and the node attribute information. This fused graph is then repeatedly coarsened into a much smaller graph by merging nodes with high spectral similarities. GraphZoom allows any existing embedding methods to be applied to the coarsened graph, before it progressively refine the embeddings obtained at the coarsest level to increasingly finer graphs. We have evaluated our approach on a number of popular graph datasets for both transductive and inductive tasks. Our experiments show that GraphZoom increases the classification accuracy and significantly reduces the run time compared to state-of-the-art unsupervised embedding methods. Recent years have seen a surge of interest in graph embedding, which aims to encode nodes, edges, or (sub)graphs into low dimensional vectors that maximally preserve graph structural information. Graph embedding techniques have shown promising for various applications such as vertex classification, link prediction, and community detection ; ; . However, current graph embedding methods have several drawbacks. On the one hand, random-walk based embedding algorithms, such as DeepWalk and node2vec , attempt to embed a graph based on its topology without incorporating node attribute information, which limits their embedding power. Later, graph convolutional networks (GCN) are developed with the basic notion that node embeddings should be smooth over the graph . While GCN leverages both topology and node attribute information for simplified graph convolution in each layer, it may suffer from high frequency noise in the initial node features, which compromises the embedding quality . On the other hand, few embedding algorithms can scale well to large graphs with millions of nodes due to their high computation and storage cost (a). For example, graph neural networks (GNNs) such as GraphSAGE collectively aggregate feature information from the neighborhood. When stacking multiple GNN layers, the final embedding vector of a node involves the computation of a large number of intermediate embeddings from its neighbors. This will not only drastically increase the number of computations among nodes but also lead to high memory usage for storing the intermediate . In literature, increasing the accuracy and improving the scalability of graph embedding methods are largely viewed as two orthogonal problems. Hence most research efforts are devoted to addressing only one of the problems. For instance, and proposed multi-level methods to obtain high-quality embeddings by training unsupervised models at every level; but their techniques do not improve scalability due to the additional training overhead. developed a heuristic algorithm to coarsen the graph by merging nodes with similar local structures. They use GCN to refine the embedding on the coarsened graphs, which not only is timeconsuming to train but may also degrade accuracy when multiple GCN layers are stacked together. More recently, proposed a similar strategy to coarsen the graph, where certain properties of the graph structure are preserved. However, this work lacks proper refinement methods to improve the embedding quality. In this paper we propose GraphZoom, a multi-level spectral approach to enhancing the quality and scalability of unsupervised graph embedding methods. Specifically, GraphZoom consists of four kernels: graph fusion, spectral graph coarsening, graph embedding, and embedding refinement. More concretely, graph fusion first converts the node feature matrix into a feature graph and then fuses it with the original topology graph. The fused graph provides richer information to the ensuing graph embedding step to achieve a higher accuracy. Spectral graph coarsening produces a series of successively coarsened graphs by merging nodes based on their spectral similarities. We show that our coarsening algorithm can efficiently and effectively retain the first few eigenvectors of the graph Laplacian matrix, which is critical for preserving the key graph structures. During the graph embedding step, any of the existing unsupervised graph embedding techniques can be applied to obtain node embeddings for the graph at the coarsest level. 1 Embedding refinement is then employed to refine the embeddings back to the original graph by applying a proper graph filter to ensure embeddings are smoothed over the graph. We validate the proposed GraphZoom framework on three transductive benchmarks: Cora, Citeseer and Pubmed citation networks as well as two inductive dataset: PPI and Reddit for vertex classification task. We further test on friendster dataset which contains 8 million nodes and 400 million edges to show the scalability of GraphZoom. Our experiments show that GraphZoom can improve the classification accuracy over all baseline embedding methods for both transductive and inductive tasks. Our main technical contributions are summarized as follows: • GraphZoom generates high-quality embeddings. We propose novel algorithms to encode graph structures and node attribute information in a fused graph and exploit graph filtering during refinement to remove high frequency noise. This in an increase of the embedding accuracy over the prior arts by up to 19.4%. • GraphZoom improves scalability. Our approach can significantly reduce the embedding run time by effectively coarsening the graph without losing the key spectral properties. Experiments show that GraphZoom can accelerate the entire embedding process by up to 40.8x while producing a similar or better accuracy than state-of-the-art techniques. • GraphZoom is highly composable. Our framework is agnostic to underlying graph embedding techniques. Any of the existing unsupervised embedding methods, either transductive or inductive, can be incorporated by GraphZoom in a plug-and-play manner. GraphZoom draws inspiration from multi-level graph embedding and graph filtering to boost the performance and speed of unsupervised embedding methods. Multi-level graph embedding. Multi-level graph embedding attempts to coarsen the graph in a series of levels where graph embedding techniques can be applied on those coarsened graphs with decreasing size. coarsen the graph into several levels and then perform embedding on the hierarchy of graphs from the coarsest to the original one. adopt a similar idea by hierarchical sampling original graph into multi-level graphs whose embedding vectors are concatenated to obtain the final node embeddings of the original graph. Both of these works only focus on improving embedding quality without improving the scalability. Later, Zhang et al. (2018b); attempt to improve graph embedding scalability by only embedding on the coarsest graph. However, their approaches lack proper refinement methods to generate high-quality embeddings of the original graph. propose MILE, which only trains the coarsest graph to obtain coarse embeddings, and leverages GCN as embeddings refinement method to improve embedding quality. Nevertheless, MILE requires to train a GCN model which is very time consuming for large graphs and cannot support inductive embedding models due to the transductive property of GCN. In contrast to the prior multi-level graph embedding techniques, GraphZoom is a simple yet theoretically motivated spectral approach to improving embedding quality as well as scalability of unsupervised graph embedding models. Graph filtering. Graph filters are direct analogs of classical filters in signal processing field, but intended for signals defined on graphs. defined graph filters in both vertex and spectral domains, and applies graph filter in image denoising and reconstruction tasks. showed the fundamental link between graph embedding and filtering by proving that GCN model implicitly exploits graph filter to remove high frequency noise from the node feature matrix; a filter neural network (gfNN) is then proposed to derive a stronger graph filter to improve the embedding . further derived two generalized graph filters and apply them on graph embedding models to improve their embedding quality for various classification tasks. 3 GRAPHZOOM FRAMEWORK Figure 1 shows the proposed GraphZoom framework which consists of four key phases: Phase is graph fusion, which constructs a weighted graph that fuses the information of both the graph topology and node attributes; In Phase, a spectral graph coarsening process is applied to form a hierarchy of coarsened fused graphs with decreasing size; In Phase, any of the prior graph embedding methods can be applied to the fused graph at the coarsest level; In Phase, the embedding vectors obtained at the coarsest level are mapped onto a finer graph using the mapping operators determined during the coarsening phase. This is followed by a refinement (smoothing) procedure; by iteratively applying Phase to increasingly finer graphs, the embedding vectors for the original graph can be eventually obtained. In the rest of this section, we describe each of these four phases in more detail. Graph w/ Node Embeddings Figure 1: Overview of the GraphZoom framework. Graph fusion aims to construct a weighted graph that has the same number of nodes as the original graph but potentially different set of edges (weights) that encapsulate the original graph topology as well as node attribute information. Specifically, given an undirected graph G = (V, E) with N nodes, its adjacency matrix A topo ∈ R N ×N and its node attribute (feature) matrix X ∈ R N ×K, where K corresponds to the dimension of node attribute vector, graph fusion can be interpreted as a function f (·) that outputs a weighted graph G f usion = (V, E f usion) represented by its adjacency matrix A f usion ∈ R N ×N, namely, A f usion = f (A topo, X). Graph fusion first converts the initial attribute matrix X into a weighted node attribute graph G f eat = (V, E f eat) by generating a k-nearest-neighbor (kNN) graph based on the l 2 -norm distance between the attribute vectors of each node pair. Note that a straightforward implementation requires comparing all possible node pairs and then selecting top-k nearest neighbors. However, such a naïve approach has a worst-case time complexity of O(N 2), which certainly does not scale to large graphs. To allow constructing the attribute graph in linear time, we leverage our O(|E|) complexity spectral graph coarsening scheme described with details in Section 3.2. More specifically, our approach starts with coarsening the original graph G to obtain a substantially reduced graph that has much fewer nodes. Note that such a procedure is very similar to spectral graph clustering, which aims to group nodes into clusters of high conductance . Once such node clusters are formed through spectral coarsening, selecting the top-k nearest neighbors within each cluster can be accomplished in O(M 2), where M is the averaged node count within the same cluster. Since we have roughly N/M clusters, the total run time for constructing the approximate kNN graph becomes O(M N). When a proper coarsening ratio (M N) is chosen, say M = 50, the overall run time complexity will become almost linear. For each edge in the attribute graph, we assign its weight w i,j according to the cosine similarity of two nodes' attribute vectors: w i,j = (X i,: · X j,:)/(X i,: X j,:), where X i,: and X j,: are the attribute vectors of node i and j. Finally, we can construct the fused graph by combining the topological graph and the attribute graph: A f usion = A topo + βA f eat, where β allows us to balance the graph topological and node attribute information in the fusion process. The fused graph will enable the underlying graph embedding model to utilize both graph topological and node attribute information, and thus can be fed into any downstream graph embedding procedures to further improve embedding quality. Graph coarsening via global spectral embedding. To reduce the size of the original graph while preserving important spectral properties (e.g., the first few eigenvalues and eigenvectors of the graph Laplacian matrix 2), a straightforward way is to first embed the graph into a k-dimensional space using the first k eigenvectors of the graph Laplacian matrix, which is also known as the spectral graph embedding technique . Next, the graph nodes that are close to each other in the low-dimensional embedding space can be aggregated to form the coarse-level nodes and subsequently the reduced graph. However, it will be very costly to calculate the eigenvectors of the original graph Laplacian, especially for very large graphs. Graph coarsening via local spectral embedding. In this work, we leverage an efficient yet effective local spectral embedding scheme to identify node clusters based on emerging graph signal processing techniques . There are obvious analogies between the traditional signal processing (Fourier analysis) and graph signal processing: The signals at different time points in classical Fourier analysis correspond to the signals at different nodes in an undirected graph; The more slowly oscillating functions in time domain correspond to the graph Laplacian eigenvectors associated with lower eigenvalues or the more slowly varying (smoother) components across the graph. Instead of directly using the first few eigenvectors of the original graph Laplacian, we apply the simple smoothing (low-pass graph filtering) function to k random vectors to obtain smoothed vectors for k-dimensional graph embedding, which can be achieved in linear time. Consider a random vector (graph signal) x that can be expressed with a linear combination of eigenvectors u of the graph Laplacian. Low-pass graph filters can be adopted to quickly filter out the "high-frequency" components of the random graph signal or the eigenvectors corresponding to high eigenvalues of the graph Laplacian. By applying the smoothing function on x, a smoothed vectorx can be obtained, which can be considered as a linear combination of the first few eigenvectors: More specifically, we apply a few (e.g. five to ten) Gauss-Seidel iterations for solving the linear system of equations L G x (i) = 0 to a set of t initial random vectors T = (x,..., x (t) ) that are orthogonal to the all-one vector 1 satisfying 1 x (i) = 0, and L G is the Laplacian matrix of graph G or G f usion. Based on the smoothed vectors in T, each node is embedded into a t-dimensional space such that nodes p and q are considered spectrally similar if their low-dimensional embedding vectors x p ∈ R t and x q ∈ R t are highly correlated. Here the node distance is measured by the spectral node affinity a p,q for neighboring nodes p and q : Once the node aggregation schemes are determined, the graph mapping operators on each level (can be obtained and leveraged for constructing a series of spectrally-reduced We emphasize that the aggregation scheme based on the above spectral node affinity calculations will have a (linear) complexity of O(|E f usion |) and thus allow preserving the spectral (global or structural) properties of the original graph in a highly efficient and effective way. As suggested in , a spectral sparsification procedure can be applied to effectively control densities of coarse level graphs. In this work, a similarity-aware spectral sparsification tool "GRASS" has been adopted for achieving a desired graph sparsity at the coarsest level. Embedding the Coarsest Graph. Once the coarsest graph G m is constructed, node embeddings E m on G m can be obtained by E m = l(G m), where l(·) can be any unsupervised embedding methods. Once the base node embedding are available, we can easily project the node embeddings from graph G i+1 to the fine-grained graph G i with the corresponding projection operator H i i+1: Due to the property of the projection operator, embedding of the node in coarse-grained graph will be directly copied to the nodes of the same aggregation set in the fine-grained graph. In this case, spectrally-similar nodes in the fine-grained graph will have the same embedding if they are aggregated into a single node during the coarsening phase. To further improve the quality of the mapped embeddings, we apply a local refinement process motivated by Tikhonov regularization to smooth the node embeddings over the graph by minimizing the following objective: min where L i and E i are the normalized Laplacian matrix and mapped embedding matrix of the graph at the i-th coarsening level, respectively. The refined embedding matrixẼ i is obtained by solving Eq., whose first term enforces the refined embeddings to agree with mapped embeddings while the second term employs Laplacian smoothing to smoothẼ i over the graph. By taking the derivative of the objective function in Eq. and setting it to zero, we have: where I is the identity matrix. However, obtaining refined embeddings in this way is very time consuming since it involves matrix inversion whose time complexity is O(N 3). Instead, we exploit a more efficient graph filter to smooth the embeddings. Let the term (I + L) −1 denoted by h(L), then its corresponding graph filter in spectral domain is h(λ) = (1 + λ) −1. To avoid the inversion term, we approximate h(λ) by its first-order Taylor expansion, namely,h(λ) = 1 − λ. We then generalizẽ k, where k controls the power of graph filter. After transformingh where A is the adjacency matrix and D is the degree matrix. It can be proved that adding a proper self-loop for every node in the graph can enableh k (L) to more effectively filter out high-frequency noise components (more details are available in Appendix G). Thus, we modify the adjacency matrix asÃ = A + σI, where σ is a small value to ensure every node has its own self-loop. Finally, the low-pass graph filter can be utilized to smooth the mapped embedding matrix, as shown in. We iteratively apply Eq. to obtain the embeddings of the original graph (i.e., E 0). Note that our refinement stage does not involve training and can be simply considered as several (sparse) matrix multiplications, which can be computed efficiently. We have performed comparative evaluation of GraphZoom framework against several existing stateof-the-art unsupervised graph embedding techniques and multi-level embedding frameworks on five standard graph-based dataset (transductive as well as inductive). In addition, we evaluate the scalability of GraphZoom on Friendster dataset that contains 8 million nodes and 400 million edges. Finally, we further analyze GraphZoom kernels separately to show their effectiveness. Datasets. The statistics of datasets used in our experiments are demonstrated in Table 1. We use Cora, Citeseer, Pubmed, Friendster for transductive task and PPI, Reddit for inductive task. We choose the same training and testing size used in;. Transductive baseline models. Many existing graph embedding techniques are essentially transductive learning methods, which require all nodes in the graph be present during training, and their embedding models have to be retrained whenever a new node is added. We compare GraphZoom with transductive models DeepWalk, node2vec, and Deep Graph Infomax (DGI) (Velikovi et al., and MILE , which have shown improvement upon DeepWalk and node2vec in either embedding quality or scalability. Inductive baseline models. Inductive graph embedding models can be trained without seeing the whole graph structure and their trained models can be applied on new nodes added to graph. To show GraphZoom can also enhance inductive learning, we compare it against GraphSAGE using four different aggregation functions. More details of datasets and baselines are available in Appendix A and B. We optimize hyperparameters of DeepWalk, node2vec, DGI and GraphSAGE on original datasets as embedding baseline, and then we choose the same hyper-parameters to embed coarsened graph in HARP, MILE and our GraphZoom framework. We run all the experiments on a machine running Linux with an Intel Xeon Gold 6242 CPU (32 cores, 2.40GHz) and 384 GB of RAM. Tables 2 and 3, respectively. We report the mean classification accuracy for transductive task and micro-averaged F1 score for inductive task as well as CPU time after 10 runs for all the baselines and GraphZoom. We measure the CPU time for graph embedding as the total run time of DeepWalk, node2vec, DGI, and GraphSAGE. We use the sum of CPU time for graph coarsening, graph embedding, and embedding refinement as total run time of HARP and MILE. Similarly, we sum up the CPU time for graph fusion, graph coarsening, graph embedding, and embedding refinement as total run time of GraphZoom. We also perform fine-tuning on the hyper-parameters. For both DeepWalk and node2vec, we use 10 walks with a walk length of 80, a window size of 10, and an embedding dimension of 128; we further set p = 1 and q = 0.5 in node2vec. For DGI, we choose early stopping strategy with a learning rate of 0.001, an embedding dimension of 512. For GraphSAGE, we train a two-layer model for one epoch, with a learning rate of 0.00001, an embedding dimension of 128, and a batch size of 256. Comparing GraphZoom with baseline embedding methods. We show the of GraphZoom with coarsening level varying 1 to 3 for transductive learning and 1 to 2 for inductive learning. Results with larger coarsening level are available in Figure 3 (blue curve) and the Appendix I. Our demonstrate that GraphZoom is agnostic to underlying embedding methods and capable of boosting the accuracy and speed of state-of-the-art unsupervised embedding methods on various datasets. More specifically, for transductive learning task, GraphZoom improves classification accuracy upon both DeepWalk and node2vec by a margin of 8.3%, 19.4%, and 10.4% on Cora, Citeseer, and Pubmed, respectively, while achieving up to 40.8x run-time reduction. In regards to comparing with DGI, GraphZoom achieves comparable or better accuracy with speedup up to 11.2×. Similarly, GraphZoom outperforms all the baselines by a margin of 3.4% and 3.3% on PPI and Reddit for inductive learning task, respectively, with speedup up to 7.6x. Our indicate that reducing graph size while properly retaining the key spectral properties of graph Laplacian and smoothing embeddings will not only boost the embedding speed but also lead to high embedding quality. Comparing GraphZoom with multi-level frameworks. As shown in Table 2, HARP only slightly improves and sometimes even worsens the classification accuracy while significantly increasing the CPU time. Although MILE improves both accuracy and CPU time compared to baseline embedding methods in some cases, the performance of MILE becomes worse with increasing coarsening levels (e.g., the classification accuracy of MILE drops from 0.708 to 0.618 on Pubmed dataset with node2vec as the embedding kernel). GraphZoom achieves a better accuracy and speedup compared to MILE with the same coarsening level across all datasets. Moreover, when increasing coarsening levels, namely, decreasing number of nodes on the coarsened graph, GraphZoom still produces comparable or even a better embedding accuracy with much shorter CPU times. This further confirms GraphZoom can retain the key graph structure information to be utilized by underlying embedding models to generate high-quality node embeddings. More of GraphZoom on non-attributed graph for both node classification and link prediction tasks are available in Appendix J. GraphZoom for large graph embedding. To show GraphZoom can significantly improve performance and scalability of underlying embedding model on large graph, we test GraphZoom and MILE on Friendster dataset, which contains 8 million nodes and 400 million edges, using DeepWalk as the embedding kernel. As shown in Figure 2, GraphZoom drastically boosts the Micro-F1 score up to 47.6% compared to MILE and 49.9% compared to DeepWalk with speedup up to 119.8×. When increasing coarsening level, GraphZoom achieves a higher speedup while the embedding accuracy decreases gracefully, which shows the key strength of GraphZoom: it can effectively coarsen large graph by merging many redundant nodes that are spectrally similar, which preserves the most important graph spectral (structural) properties key to underlying embedding model. When applying basic embedding model on coarsest graph, it can learn more global information from spectral domain, leading to high-quality node embeddings. On the contrary, heuristic graph coarsening algorithm used in MILE fails to preserve a meaningful coarsest graph, especially when coarsening graph by a large reduction ratio. Comparisons of different kernel combinations in GraphZoom and MILE in classification accuracy on Cora, Citeseer, and Pubmed datasets -We choose DeepWalk (DW) as the embedding kernel. GZoom F, GZoom C, GZoom R denote the fusion, coarsening, and refinement kernels proposed in GraphZoom, respectively; MILE C and MILE R denote the coarsening and refinement kernels in MILE, respectively; The blue curve is basically GraphZoom and the yellow one is MILE. To study the effectiveness of our proposed GraphZoom kernels separately, we compare each of them against the corresponding kernel in MILE with other kernels fixed. As shown in Figure 3, when fixing coarsening kernel and comparing refinement kernel of GraphZoom with that of MILE (shown in purple curve and yellow curve), GraphZoom refinement kernel can improve embedding upon MILE refinement kernel, especially when the coarsening level is large, which indicates that our proposed graph filter in refinement kernel can successfully filter out high frequency noise from graph to improve embedding quality. Similarly, when comparing coarsening kernels in GraphZoom and MILE with refinement kernel fixed (shown in light blue curve and yellow curve), GraphZoom coarsening kernel can also improve embedding quality upon MILE coarsening kernel, which shows that our spectral graph coarsening algorithm can indeed retain key graph structure for underlying graph embedding models to exploit. When combining GraphZoom coarsening kernel and refinement kernel (green curve), we can achieve better classification accuracy compared with the ones using any kernel in MILE (i.e., light blue curve, purple curve and yellow curve), which means that GraphZoom coarsening kernel and refinement kernel play different roles to boost embedding performance and their combination can further improve embedding . Moreover, when adding graph fusion kernel with the combination of GraphZoom coarsening and refinement kernels (blue curve, which is our GraphZoom framework), it improves classification accuracy by a large margin, which betokens that graph fusion can properly incorporate both graph topology and node attribute information and lifts the embedding quality of downstream embedding models. Results of each kernel CPU time and speedup comparison are available in Appendix F and Appendix H. In this work we propose GraphZoom, a multi-level framework to improve embedding quality and scalability of underlying unsupervised graph embedding techniques. GraphZoom encodes graph structure and node attribute in a single graph and exploiting spectral coarsening and refinement methods to remove high frequency noise from the graph. Experiments show that GraphZoom improves both classification accuracy and embedding speed on a number of popular datasets. An interesting direction for future work is to derive a proper way to propagate node labels to the coarsest graph, which would allow GraphZoom to support supervised graph embedding models. Transductive task. We follow the experiments setup in for three standard citation network benchmark datasets: Cora, Citeseer, and Pubmed. In all these three citation networks, nodes represent documents and edges correspond to citations. Each node has a sparse bag-of-word feature vector and a class label. We allow only 20 labels per class for training and 1, 000 labeled nodes for testing. In addition, we further evaluate on Friendster dataset , which contains 8 million nodes and 400 million edges, with 2.5% of the nodes used for training and 0.3% nodes for testing. In Friendster, nodes represent users and a pair of nodes are linked if they are friends; each node has a class label but is not associated with a feature vector. Inductive task. We follow for setting up experiments on both proteinprotein interaction (PPI) and Reddit dataset. PPI dataset consists of graphs corresponding to human tissues, where nodes are proteins and edges represent interaction effects between proteins. Reddit dataset contains nodes corresponding to users' posts: two nodes are connected through an edge if the same users comment on both posts. We use 60% nodes for training, 40% for testing on PPI and 65% for training and 35% for testing on Reddit. DeepWalk first generates random walks based on graph structure. Then, walks are treated as sentences in a language model and Skip-Gram model is exploited to obtain node embeddings. node2vec is different from DeepWalk in terms of generating random walks by introducing the return parameter p and the in-out parameter q, which can combine DFS-like and BFS-like neighborhood exploration. Deep Graph Infomax (DGI) is an unsupervised approach that generates node embeddings by maximizing mutual information between patch representations (local information) and corresponding high-level summaries (global information) of graphs. GraphSAGE embeds nodes in an inductive way by learning an aggregation function that aggregates node features to obtain embeddings. GraphSAGE supports four different aggregation functions: GraphSAGE-GCN, GraphSAGE-mean, GraphSAGE-LSTM and GraphSAGE-pool. HARP coarsens the original graph into several levels and apply underlying embedding model to train the coarsened graph at each level sequentially to obtain the final embeddings on original graph. Since the coarsening level is fixed in their implementation, we run HARP in our experiments without changing the coarsening level. MILE is the state-of-the-art multi-level unsupervised graph embedding framework and similar to our GraphZoom framework since it also contains graph coarsening and embedding refinement kernels. More specifically, MILE first uses its heuristic-based coarsening kernel to reduce the graph size and trains underlying unsupervised graph embedding model on coarsest graph. Then, its refinement kernel employs Graph Convolutional Network (GCN) to refine embeddings on the original graph. We compare GraphZoom with MILE on various datasets, including Friendster that contains 8 million nodes and 400 million edges (shown in Table 2 and Figure 2). Moreover, we further compare each kernel in GraphZoom and MILE in Figure 3. The details of graph size at different coarsening level on all six datasets are shown in Table 4. APPENDIX E SPECTRAL COARSENING Note that the mapping operator H i+1 i ∈ {0, 1} |Vi+1|×|Vi| is a matrix containing only 0s and 1s. It has following properties: • The row (column) index of H i+1 i corresponds to the node index in graph G i+1 (G i). • It is a surjective mapping of the node set, where (H i+1 i) p,q = 1 if node q in graph G i is aggregated to super-node p in graph G i+1, and (H i+1 i) p,q = 0 for all nodes p ∈ {v ∈ V i+1 : v = p}. • It is a locality-preserving operator, where the coarsened version of G i induced by the nonzero entries of (H i+1 i) p,: is connected for each p ∈ V i+1. Algorithm 2: spectral coarsening algorithm Input: Adjacency matrix A i ∈ R |Vi|×|Vi| Output: Adjacency matrix A i+1 ∈ R |Vi+1|×|Vi+1| of the reduced graph G i+1, mapping operator H i+1 i ∈ R |Vi+1|×|Vi| 11 n = |V i |, n c = n; 12 [graph reduction ratio] γ max = 1.8, δ = 0.9; 13 for each edge (p, q) ∈ E i do 14 [spectral node affinity set] C ← a p,q defined in Eq. 2; 15 end 16 for each node p ∈ V i do 17 Figure 4 (note that the y axis is in logarithmic scale), the GraphZoom embedding kernel dominates the total CPU time, which can be more effectively reduced with a greater coarsening level L. All other kernels in GraphZoom are very efficient, which enable the GraphZoom framework to drastically reduce the total graph embedding time. Figure 5a shows the original distribution of graph Laplacian eigenvalues which also can be interpreted as frequencies in graph spectral domain (smaller eigenvalue means lower frequency). The proposed graph filter for embedding refinement (as shown in Figure 5e) can be considered as a bandstop filter that passes all frequencies with the exception of those within the middle stop band that is greatly attenuated. Therefore, the band-stop filter may not be very effective for removing highfrequency noises from the graph signals. Fortunately, it has been shown that by adding self-loops to each node in the graph as followsÃ = A+σI (shown in Figure 5b, 5c, 5d, where σ = 0.5, 1.0, 2.0), the distribution of Laplacian eigenvalues can be squeezed to the left (towards zero) . By properly choosing σ such that large eigenvalues will mostly lie in the stop band (e.g., σ = 1.0, 2.0 shown in Figure 5c and 5d), the graph filter will be able to effectively filtered out high-frequency components (corresponding to high eigenvalues) while retaining low-frequency components, which is similar to a low-pass graph filter as shown in Figure 5f. It is worth noting that if σ is too large, then most eigenvalues will be very close to zero, which makes the graph filter less effective for removing noises. In this work, we choose σ = 2.0 for all our experiments. As shown in Figure 6, the combination of GraphZoom coarsening and refinement kernels can always achieve the greatest speedups (green curves); adding GraphZoom fusion kernel (blue curves) will lower the speedups by a small margin but further boost the embedding quality, showing a clear tradeoff between embedding quality and runtime efficiency: to achieve the highest graph embedding quality, the graph fusion kernel should be included. To further show that GraphZoom can work on non-attributed datasets, we evaluate it on PPI(Homo Sapiens) and Wiki datasets, following the same dataset configuration as used in;. As shown in Table 5, GraphZoom (without fusion kernel) improves

A multi-level spectral approach to improving the quality and scalability of unsupervised graph embedding.

Represent the short summary for finding the corresponding scientific article (long).

We propose an efficient, provable and data independent method for network compression via neural pruning using coresets of neurons -- a novel construction proposed in this paper.


Question: Represent the article for finding a single-sentence TLDR

Question: Deep generative models have been enjoying success in modeling continuous data. However it remains challenging to capture the representations for discrete structures with formal grammars and semantics, e.g., computer programs and molecular structures. How to generate both syntactically and semantically correct data still remains largely an open problem. Inspired by the theory of compiler where syntax and semantics check is done via syntax-directed translation (SDT), we propose a novel syntax-directed variational autoencoder (SD-VAE) by introducing stochastic lazy attributes. This approach converts the offline SDT check into on-the-fly generated guidance for constraining the decoder. Comparing to the state-of-the-art methods, our approach enforces constraints on the output space so that the output will be not only syntactically valid, but also semantically reasonable. We evaluate the proposed model with applications in programming language and molecules, including reconstruction and program/molecule optimization. The demonstrate the effectiveness in incorporating syntactic and semantic constraints in discrete generative models, which is significantly better than current state-of-the-art approaches. Recent advances in deep representation learning have ed in powerful probabilistic generative models which have demonstrated their ability on modeling continuous data, e.g., time series signals BID20 BID6 and images BID22 BID15. Despite the success in these domains, it is still challenging to correctly generate discrete structured data, such as graphs, molecules and computer programs. Since many of the structures have syntax and semantic formalisms, the generative models without explicit constraints often produces invalid ones. Conceptually an approach in generative model for structured data can be divided in two parts, one being the formalization of the structure generation and the other one being a (usually deep) generative model producing parameters for stochastic process in that formalization. Often the hope is that with the help of training samples and capacity of deep models, the loss function will prefer the valid patterns and encourage the mass of the distribution of the generative model towards the desired region automatically. Arguably the simplest structured data are sequences, whose generation with deep model has been well studied under the seq2seq BID25 framework that models the generation of sequence as a series of token choices parameterized by recurrent neural networks (RNNs). Its widespread success has encourage several pioneer works that consider the conversion of more complex structure data into sequences and apply sequence models to the represented sequences. BID11 (CVAE) is a representative work of such paradigm for the chemical molecule generation, using the SMILES line notation BID26 for representing molecules. Figure 1: Illustration on left shows the hierarchy of the structured data decoding space w.r.t different works and theoretical classification of corresponding strings from formal language theory. SD-VAE, our proposed model with attribute grammar reshapes the output space tighter to the meaningful target space than existing works. On the right we show a case where CFG is unable to capture the semantic constraints, since it successfully parses an invalid program. However, because of the lack of formalization of syntax and semantics serving as the restriction of the particular structured data, underfitted general-purpose string generative models will often lead to invalid outputs. Therefore, to obtain a reasonable model via such training procedure, we need to prepare large amount of valid combinations of the structures, which is time consuming or even not practical in domains like drug discovery. To tackle such a challenge, one approach is to incorporate the structure restrictions explicitly into the generative model. For the considerations of computational cost and model generality, contextfree grammars (CFG) have been taken into account in the decoder parametrization. For instance, in molecule generation tasks, BID18 proposes a grammar variational autoencoder (GVAE) in which the CFG of SMILES notation is incorporated into the decoder. The model generates the parse trees directly in a top-down direction, by repeatedly expanding any nonterminal with its production rules. Although the CFG provides a mechanism for generating syntactic valid objects, it is still incapable to regularize the model for generating semantic valid objects BID18. For example, in molecule generation, the semantic of the SMILES languages requires that the rings generated must be closed; in program generation, the referenced variable should be defined in advance and each variable can only be defined exactly once in each local context (illustrated in Fig 1b).All the examples require cross-serial like dependencies which are not enforceable by CFG, implying that more constraints beyond CFG are needed to achieve semantic valid production in VAE.In the theory of compiler, attribute grammars, or syntax-directed definition has been proposed for attaching semantics to a parse tree generated by context-free grammar. Thus one straightforward but not practical application of attribute grammars is, after generating a syntactic valid molecule candidate, to conduct offline semantic checking. This process needs to be repeated until a semantically valid one is discovered, which is at best computationally inefficient and at worst infeasible, due to extremely low rate of passing checking. As a remedy, we propose the syntax-direct variational autoencoder (SD-VAE), in which a semantic restriction component is advanced to the stage of syntax tree generator. This allows the generator with both syntactic and semantic validation. The proposed syntax-direct generative mechanism in the decoder further constraints the output space to ensure the semantic correctness in the tree generation process. The relationships between our proposed model and previous models can be characterized in Figure 1a.Our method brings theory of formal language into stochastic generative model. The contribution of our paper can be summarized as follows:• Syntax and semantics enforcement: We propose a new formalization of semantics that systematically converts the offline semantic check into online guidance for stochastic generation using the proposed stochastic lazy attribute. This allows us effectively address both syntax and semantic constraints.• Efficient learning and inference: Our approach has computational cost O(n) where n is the length of structured data. This is the same as existing methods like CVAE and GVAE which do not enforce semantics in generation. During inference, the SD-VAE runs with semantic guiding onthe-fly, while the existing alternatives generate many candidates for semantic checking.• Strong empirical performance: We demonstrate the effectiveness of the SD-VAE through applications in two domains, namely the subset of Python programs and molecules. Our approach consistently and significantly improves the in evaluations including generation, reconstruction and optimization. Before introducing our model and the learning algorithm, we first provide some knowledge which is important for understanding the proposed method. The variational autoencoder BID16 BID23 provides a framework for learning the probabilistic generative model as well as its posterior, respectively known as decoder and encoder. We denote the observation as x, which is the structured data in our case, and the latent variable as z. The decoder is modeling the probabilistic generative processes of x given the continuous representation z through the likelihood p θ (x|z) and the prior over the latent variables p(z), where θ denotes the parameters. The encoder approximates the posterior p θ (z|x) ∝ p θ (x|z)p(z) with a model q ψ (z|x) parametrized by ψ. The decoder and encoder are learned simultaneously by maximizing the evidence lower bound (ELBO) of the marginal likelihood, i.e., DISPLAYFORM0 where X denotes the training datasets containing the observations. Context free grammar A context free grammar (CFG) is defined as G = V, Σ, R, s, where symbols are divided into V, the set of non-terminal symbols, Σ, the set of terminal symbols and s ∈ V, the start symbol. Here R is the set of production rules. Each production rule r ∈ R is denoted as r = α → β for α ∈ V is a nonterminal symbol, and β = u 1 u 2... u |β| ∈ (V Σ) * is a sequence of terminal and/or nonterminal symbols. Attribute grammar To enrich the CFG with "semantic meaning", BID17 formalizes attribute grammar that introduces attributes and rules to CFG. An attribute is an attachment to the corresponding nonterminal symbol in CFG, written in the format v.a for v ∈ V. There can be two types of attributes assigned to non-terminals in G: the inherited attributes and the synthesized attributes. An inherited attribute depends on the attributes from its parent and siblings, while a synthesized attribute is computed based on the attributes of its children. Formally, for a production u 0 → u 1 u 2... u |β|, we denote I(u i) and S(u i) be the sets of inherited and synthesized attributes of u i for i ∈ {0, . . ., |β|}, respectively. We here exemplify how the above defined attribute grammar enriches CFG with non-context-free semantics. We use the following toy grammar, a subset of SMILES that generates either a chain or a cycle with three carbons: DISPLAYFORM0 where we show the production rules in CFG with → on the left, and the calculation of attributes in attribute grammar with ← on the left. Here we leverage the attribute grammar to check (with attribute matched) whether the ringbonds come in pairs: a ringbond generated at atom 1 should match the bond type and bond index that generated at atom 2, also the semantic constraint expressed by s.ok requires that there is no difference between the set attribute of atom 1 and atom 2. Such constraint in SMILES is known as cross-serial dependencies (CSD) BID4 which is non-context-free BID24. See Appendix A.3 for more explanations. FIG2 illustrates the process of performing syntax and semantics check in compilers. Here all the attributes are synthetic, i.e., calculated in a bottom-up direction. So generally, in the semantic correctness checking procedure, one need to perform bottom-up procedures for calculating the attributes after the parse tree is generated. However, in the top-down structure generating process, the parse tree is not ready for semantic checking, since the synthesized attributes of each node require information from its children nodes, which are not generated yet. Due to such dilemma, it is nontrivial to use the attribute grammar to guide the top-down generation of the tree-structured data. One straightforward way is using acceptance-rejection sampling scheme, i.e., using the decoder of CVAE or GVAE as a proposal and the semantic checking as the threshold. It is obvious that since the decoder does not include semantic guidance, the proposal distribution may raise semantically invalid candidate frequently, therefore, wasting the computational cost in vain.3 SYNTAX-DIRECTED VARIATIONAL AUTOENCODER As described in Section 2.2.1, directly using attribute grammar in an offline fashion (i.e., after the generation process finishes) is not efficient to address both syntax and semantics constraints. In this section we describe how to bring forward the attribute grammar online and incorporate it into VAE, such that our VAE addresses both syntactic and semantic constraints. We name our proposed method Syntax-Directed Variational Autoencoder (SD-VAE). By scrutinizing the tree generation, the major difficulty in incorporating the attributes grammar into the processes is the appearance of the synthesized attributes. For instance, when expanding the start symbol s, none of its children is generated yet. Thus their attributes are also absent at this time, making the s.matched unable to be computed. To enable the on-the-fly computation of the synthesized attributes for semantic validation during tree generation, besides the two types of attributes, we introduce the stochastic lazy attributes to enlarge the existing attribute grammar. Such stochasticity transforms the corresponding synthesized attribute into inherited constraints in generative procedure; and lazy linking mechanism sets the actual value of the attribute, once all the other dependent attributes are ready. We demonstrate how the decoder with stochastic lazy attributes will generate semantic valid output through the same pedagogical example as in Section 2.2.1. FIG3 visually demonstrates this process. The tree generation procedure is indeed sampling from the decoder p θ (x|z), which can be decomposed into several steps that elaborated below: Sample production rule r = (α → β) ∈ R ∼ p θ (r|ctx, node, T). The conditioned variables encodes the semantic constraints in tree generation. DISPLAYFORM0 DISPLAYFORM1 ctx ← RNN(ctx, r) update context vector 6: DISPLAYFORM0 ) node creation with parent and siblings' attributes 8:GenTree(v i, T) recursive generation of children nodes 9:Update synthetic and stochastic attributes of node with v i Lazy linking 10:end for 11: end procedure i) stochastic predetermination: in FIG3 (a), we start from the node s with the synthesized attributes s.matched determining the index and bond type of the ringbond that will be matched at node s. Since we know nothing about the children nodes right now, the only thing we can do is to'guess' a value. That is to say, we associate a stochastic attribute s.sa ∈ {0, 1} Ca ∼ Ca i=1 B(sa i |z) as a predetermination for the sake of the absence of synthesized attribute s.matched, where B(·) is the Bernoulli distribution. Here C a is the maximum cardinality possible 1 for the corresponding attribute a. In above example, the 0 indicates no ringbond and 1 indicates one ringbond at both atom 1 and atom 2, respectively.ii) constraints as inherited attributes: we pass the s.sa as inherited constraints to the children of node s, i.e., atom 1 and atom 2 to ensure the semantic validation in the tree generation. For example, FIG3 (b)'sa=1' is passed down to atom 1.iii) sampling under constraints: without loss of generality, we assume atom 1 is generated before atom 2. We then sample the rules from p θ (r| atom 1, s, z) for expanding atom 1, and so on and so forth to generate the subtree recursively. Since we carefully designed sampling distribution that is conditioning on the stochastic property, the inherited constraints will be eventually satisfied. In the example, due to the s.sa ='1', when expanding atom 1, the sampling distribution p θ (r| atom 1, s, z) only has positive mass on rule atom →'C' bond digit. iv) lazy linking: once we complete the generation of the subtree rooted at atom 1, the synthesized attribute atom 1.set is now available. According to the semantic rule for s.matched, we can instantiate s.matched = atom 1.set = {'-1'}. This linking is shown in FIG3 (d)(e). When expanding atom 2, the s.matched will be passed down as inherited attribute to regulate the generation of atom 2, as is demonstrated in FIG3 (f)(g).In summary, the general syntax tree T ∈ L(G) can be constructed step by step, within the languages L(G) covered by grammar G. In the beginning, T = root, where root. symbol = s which contains only the start symbol s. At step t, we will choose an nonterminal node in the frontier 2 of partially generated tree T (t) to expand. The generative process in each step t = 0, 1,... can be described as:1. Pick node v (t) ∈ F r(T (t) ) where its attributes needed are either satisfied, or are stochastic attributes that should be sampled first according to Bernoulli distribution B(·|v (t), T (t) ); DISPLAYFORM1, and DISPLAYFORM2, i.e., expand the nonterminal with production rules defined in CFG. DISPLAYFORM3, grow the tree by attaching β (t) to v (t). Now the node v (t) has children represented by symbols in β (t).The above process continues until all the nodes in the frontier of T (T) are all terminals after T steps. Then, we obtain the algorithm 1 for sampling both syntactic and semantic valid structures. In fact, in the model training phase, we need to compute the likelihood p θ (x|z) given x and z. The probability computation procedure is similar to the sampling procedure in the sense that both of them requires tree generation. The only difference is that in the likelihood computation procedure, the tree structure, i.e., the computing path, is fixed since x is given; While in the sampling procedure, it is sampled following the learned model. Specifically, the generative likelihood can be written as: DISPLAYFORM4 where ctx = z and ctx (t) = RNN(r t, ctx (t−1) ). Here RNN can be commonly used LSTM, etc.. As we introduced in section 2, the encoder, q ψ (z|x) approximates the posterior of the latent variable through the model with some parametrized function with parameters ψ. Since the structure in the observation x plays an important role, the encoder parametrization should take care of such information. The recently developed deep learning models BID9 BID19 provide powerful candidates as encoder. However, to demonstrate the benefits of the proposed syntax-directed decoder in incorporating the attribute grammar for semantic restrictions, we will exploit the same encoder in BID18 for a fair comparison later. We provide a brief introduction to the particular encoder model used in Kusner et al. FORMULA0 for a self-contained purpose. Given a program or a SMILES sequence, we obtain the corresponding parse tree using CFG and decompose it into a sequence of productions through a pre-order traversal on the tree. Then, we convert these productions into one-hot indicator vectors, in which each dimension corresponds to one production in the grammar. We will use a deep convolutional neural networks which maps this sequence of one-hot vectors to a continuous vector as the encoder. Our learning goal is to maximize the evidence lower bound in Eq 1. Given the encoder, we can then map the structure input into latent space z. The variational posterior q(z|x) is parameterized with Gaussian distribution, where the mean and variance are the output of corresponding neural networks. The prior of latent variable p(z) = N (0, I). Since both the prior and posterior are Gaussian, we use the closed form of KL-divergence that was proposed in BID16. In the decoding stage, our goal is to maximize p θ (x|z). Using the Equation FORMULA2, we can compute the corresponding conditional likelihood. During training, the syntax and semantics constraints required in Algorithm 1 can be precomputed. In practice, we observe no significant time penalty measured in wall clock time compared to previous works. Generative models with discrete structured data have raised increasing interests among researchers in different domains. The classical sequence to sequence model BID25 and its variations have also been applied to molecules BID11. Since the model is quite flexible, it is hard to generate valid structures with limited data, though Dave Janz FORMULA0 shows that an extra validator model could be helpful to some degree. Techniques including data augmentation BID2, active learning BID14 and reinforcement learning also been proposed to tackle this issue. However, according to the empirical evaluations from BID1, the validity is still not satisfactory. Even when the validity is enforced, the models tend to overfit to simple structures while neglect the diversity. Since the structured data often comes with formal grammars, it is very helpful to generate its parse tree derived from CFG, instead of generating sequence of tokens directly. The Grammar VAE BID18 introduced the CFG constrained decoder for simple math expression and SMILES string generation. The rules are used to mask out invalid syntax such that the generated sequence is always from the language defined by its CFG. BID21 uses a RecursiveReverse-Recursive Neural Network (R3NN) to capture global context information while expanding with CFG production rules. Although these works follow the syntax via CFG, the context sensitive information can only be captured using variants of sequence/tree RNNs (; BID8 BID27, which may not be time and sample efficient. In our work, we capture the semantics with proposed stochastic lazy attributes when generating structured outputs. By addressing the most common semantics to harness the deep networks, it can greatly reshape the output domain of decoder BID13 . As a , we can also get a better generative model for discrete structures. Code is available at https://github.com/Hanjun-Dai/sdvae.We show the effectiveness of our proposed SD-VAE with applications in two domains, namely programs and molecules. We compare our method with CVAE BID11 and GVAE BID18 . CVAE only takes character sequence information, while GVAE utilizes the context-free grammar. To make a fair comparison, we closely follow the experimental protocols that were set up in BID18 . The training details are included in Appendix B.Our method gets significantly better than previous works. It yields better reconstruction accuracy and prior validity by large margins, while also having comparative diversity of generated structures. More importantly, the SD-VAE finds better solution in program and molecule regression and optimization tasks. This demonstrates that the continuous latent space obtained by SD-VAE is also smoother and more discriminative. Here we first describe our datasets in detail. The programs are represented as a list of statements. Each statement is an atomic arithmetic operation on variables (labeled as v0, v1, · · ·, v9) and/or immediate numbers (1, 2, . . ., 9). Some examples are listed below: v9=v3-v8;v5=v0 * v9;return:v5 v2=exp(v0);v7=v2 * v0;v9=cos(v7);v8=cos(v9);return: v8 Here v0 is always the input, and the variable specified by return (respectively v5 and v8 in the examples) is the output, therefore it actually represent univariate functions f: R → R. Note that a correct program should, besides the context-free grammar specified in Appendix A.1, also respect the semantic constraints. For example, a variable should be defined before being referenced. We randomly generate 130, 000 programs, where each consisting of 1 to 5 valid statements. Here the maximum number of decoding steps T = 80. We hold 2000 programs out for testing and the rest for training and validation. DISPLAYFORM0 For molecule experiments, we use the same dataset as in BID18. It contains 250, 000 SMILES strings, which are extracted from the ZINC database BID11. We use the same split as BID18, where 5000 SMILES strings are held out for testing. Regarding the syntax constraints, we use the grammar specified in Appendix A.2, which is also the same as BID18. Here the maximum number of decoding steps T = 278.For our SD-VAE, we address some of the most common semantics:Program semantics We address the following: a) variables should be defined before use, b) program must return a variable, c) number of statements should be less than 10.Molecule semantics The SMILES semantics we addressed includes: a) ringbonds should satisfy cross-serial dependencies, b) explicit valence of atoms should not go beyond permitted. For more details about the semantics of SMILES language, please refer to Appendix A.3. We use the held-out dataset to measure the reconstruction accuracy of VAEs. For prior validity, we first sample the latent representations from prior distribution, and then evaluate how often the model can decode into a valid structure. Since both encoding and decoding are stochastic in VAEs, we follow the Monte Carlo method used in BID18 to do estimation: a) reconstruction: for each of the structured data in the held-out dataset, we encode it 10 times and decoded (for each encoded latent space representation) 25 times, and report the portion of decoded structures that are the same as the input ones; b) validity of prior: we sample 1000 latent representations z ∼ N (O, I). For each of them we decode 100 times, and calculate the portion of 100,000 decoded that corresponds to valid Program or SMILES sequences. Program We show in the left part of Table 1 that our model has near perfect reconstruction rate, and most importantly, a perfect valid decoding program from prior. This huge improvement is due to our model that utilizes the full semantics that previous work ignores, thus in theory guarantees perfect valid prior and in practice enables high reconstruction success rate. For a fair comparison, we run and tune the baselines in 10% of training data and report the best . In the same place we also report the reconstruction successful rate grouped by number of statements. It is shown that our model keeps high rate even with the size of program growing. SMILES Since the settings are exactly the same, we include CVAE and GVAE directly from BID18. We show in the right part of Table 1 that our model produces a much higher rate of successful reconstruction and ratio of valid prior. FIG6 in Appendix C.2 also demonstrates some decoded molecules from our method. Note that the we reported have not included the semantics specific to aromaticity into account. If we use an alternative kekulized form of SMILES to train the model, then the valid portion of prior can go up to 97.3%. Figure 4: On the left are best programs found by each method using Bayesian Optimization. On the right are top 3 closest programs found by each method along with the distance to ground truth (lower distance is better). Both our SD-VAE and CVAE can find similar curves, but our method aligns better with the ground truth. In contrast the GVAE fails this task by reporting trivial programs representing linear functions. Finding program In this application the models are asked to find the program which is most similar to the ground truth program. Here the distance is measured by log(1 + MSE), where the MSE (Mean Square Error) calculates the discrepancy of program outputs, given the 1000 different inputs v0 sampled evenly in [−5, 5]. In Figure 4 we show that our method finds the best program to the ground truth one compared to CVAE and GVAE. Molecules Here we optimize the drug properties of molecules. In this problem, we ask the model to optimize for octanol-water partition coefficients (a.k.a log P), an important measurement of druglikeness of a given molecule. As Gómez-Bombarelli et al. FORMULA0 suggests, for drug-likeness assessment log P is penalized by other properties including synthetic accessibility score BID10 ). In Figure 5 we show the the top-3 best molecules found by each method, where our method found molecules with better scores than previous works. Also one can see the molecule structures found by SD-VAE are richer than baselines, where the latter ones mostly consist of chain structure. Figure 5: Best top-3 molecules and the corresponding scores found by each method using Bayesian Optimization. Method LL RMSE LL RMSE CVAE -4.943 ± 0.058 3.757 ± 0.026 -1.812 ± 0.004 1.504 ± 0.006 GVAE -4.140 ± 0.038 3.378 ± 0.020 -1.739 ± 0.004 1.404 ± 0.006 SD-VAE -3.754 ± 0.045 3.185 ± 0.025 -1.697 ± 0.015 1.366 ± 0.023 Table 2: Predictive performance using encoded mean latent vector. Test LL and RMSE are reported. The VAEs also provide a way to do unsupervised feature representation learning BID11. In this section, we seek to to know how well our latent space predicts the properties of programs and molecules. After the training of VAEs, we dump the latent vectors of each structured data, and train the sparse Gaussian Process with the target value (namely the error for programs and the drug-likeness for molecules) for regression. We test the performance in the held-out test dataset. In Table 2, we report the in Log Likelihood (LL) and Regression Mean Square Error (RMSE), which show that our SD-VAE always produces latent space that are more discriminative than both CVAE and GVAE baselines. This also shows that, with a properly designed decoder, the quality of encoder will also be improved via end-to-end training. Similarity Metric MorganFp MACCS PairFp TopologicalFp GVAE 0.92 ± 0.10 0.83 ± 0.15 0.94 ± 0.10 0.71 ± 0.14 SD-VAE 0.92 ± 0.09 0.83 ± 0.13 0.95 ± 0.08 0.75 ± 0.14 Table 3: Diversity as statistics from pair-wise distances measured as 1 − s, where s is one of the similarity metrics. So higher values indicate better diversity. We show mean ± stddev of 100 2 pairs among 100 molecules. Note that we report from GVAE and our SD-VAE, because CVAE has very low valid priors, thus completely only failing this evaluation protocol. Inspired by BID1, here we measure the diversity of generated molecules as an assessment of the methods. The intuition is that a good generative model should be able to generate diverse data and avoid mode collapse in the learned space. We conduct this experiment in the SMILES dataset. We first sample 100 points from the prior distribution. For each point, we associate it with a molecule, which is the most frequent occurring valid SMILES decoded (we use 50 decoding attempts since the decoding is stochastic). We then, with one of the several molecular similarity metrics, compute the pairwise similarity and report the mean and standard deviation in Table 3. We see both methods do not have the mode collapse problem, while producing similar diversity scores. It indicates that although our method has more restricted decoding space than baselines, the diversity is not sacrificed. This is because we never rule-out the valid molecules. And a more compact decoding space leads to much higher probability in obtaining valid molecules. We seek to visualize the latent space as an assessment of how well our generative model is able to produces a coherent and smooth space of program and molecules. Program Following Bowman et al. FORMULA0, we visualize the latent space of program by interpolation between two programs. More specifically, given two programs which are encoded to p a and p b respectively in the latent space, we pick 9 evenly interpolated points between them. For each point, we pick the corresponding most decoded structure. In TAB3 we compare our with previous works. Our SD-VAE can pass though points in the latent space that can be decoded into valid programs without error and with visually more smooth interpolation than previous works. Meanwhile, CVAE makes both syntactic and semantic errors, and GVAE produces only semantic errors (reference of undefined variables), but still in a considerable amount. CVAE GVAE SD-VAE v6=cos;v8=exp;v2=v8*v0;v9=v2/v6;return:v9 v6=cos;v8=exp;v2=v8*v0;v9=v2/v6;return:v9 v6=cos;v8=exp;v2=v8*v0;v9=v2/v6;return:v9 v8=cos;v7=exp;v5=v7*v0;v9=v9/v6;return:v9 v3=cos;v6=exp;v6=v8*v0;v9=v2/v6;return:v9 v6=cos;v8=exp;v2=v8*v0;v9=v2/v6;return:v9 v4=cos;v8=exp;v2=v2*v0;v9=v8/v6;return:v9 v3=cos;v6=2/8;v6=v5*v9;v5=v8v5;return:v5 v6=cos;v8=exp;v3=v8*v0;v9=v3/v8;return:v9 v6=cos;v8=sin;v5=v4*1;v5=v3/v4;return:v9 v3=cos;v6=2/9;v6=v5+v5;v5=v1+v6;return:v5 v6=cos;v8=v6/9;v1=7*v0;v7=v6/v1;return:v7 v9=cos;v7=sin;v3=v1*5;v9=v9+v4;return:v9 v5=cos;v1=2/9;v6=v3+v2;v2=v5-v6;return:v2 v6=cos;v8=v6/9;v1=7*v6;v7=v6+v1;return:v7 v6=cos;v3=sin (10;;v9=8*v8;v7=v2/v2;return:v9 v5=sin;v3=v1/9;v6=v3-v3;v2=v7-v6;return:v2 v6=cos;v8=v6/9;v1=7*v8;v7=v6+v8;return:v7 v5=exp(v0;v4=sin(v0);v3=8*v1;v7=v3/v2;return:v9 v1=sin;v5=v5/2;v6=v2-v5;v2=v0-v6;return:v2 v6=exp(v0);v8=v6/2;v9=6*v8;v7=v9+v9;return:v7 v5=exp(v0);v1=sin;v5=2*v3;v7=v3+v8;return:v7 v1=sin;v7=v8/2;v8=v7/v9;v4=v4-v8;return:v4 v6=exp(v0);v8=v6-4;v9=6*v8;v7=v9+v8;return:v7 v4=exp(v0);v1=v7-8;v9=8*v3;v7=v3+v8;return:v7 v8=sin;v2=v8/2;v8=v0/v9;v4=v4-v8;return:v4 v6=exp(v0);v8=v6-4;v9=6*v6;v7=v9+v8;return:v7 v4=exp(v0);v9=v6-8;v6=2*v5;v7=v3+v8;return:v7 v6=exp(v0);v2=v6-4;v8=v0*v1;v7=v4+v8;return:v7 v6=exp(v0);v8=v6-4;v4=4*v6;v7=v4+v8;return:v7 v6=exp(v0);v8=v6-4;v4=4*v8;v7=v4+v8;return:v7 v6=exp(v0);v8=v6-4;v4=4*v8;v7=v4+v8;return:v7 v6=exp(v0);v8=v6-4;v4=4*v8;v7=v4+v8;return:v7 Observe that when a model passes points in its latent space, our proposed SD-VAE enforces both syntactic and semantic constraints while making visually more smooth interpolation. In contrast, CVAE makes both kinds of mistakes, GVAE avoids syntactic errors but still produces semantic errors, and both methods produce subjectively less smooth interpolations. SMILES For molecules, we visualize the latent space in 2 dimensions. We first embed a random molecule from the dataset into latent space. Then we randomly generate 2 orthogonal unit vectors A. To get the latent representation of neighborhood, we interpolate the 2-D grid and project back to latent space with pseudo inverse of A. Finally we show decoded molecules. In FIG5, we present two of such grid visualizations. Subjectively compared with figures in BID18, our visualization is characterized by having smooth differences between neighboring molecules, and more complicated decoded structures. DISPLAYFORM0 In this paper we propose a new method to tackle the challenge of addressing both syntax and semantic constraints in generative model for structured data. The newly proposed stochastic lazy attribute presents a the systematical conversion from offline syntax and semantic check to online guidance for stochastic generation, and empirically shows consistent and significant improvement over previous models, while requiring similar computational cost as previous model. In the future work, we would like to explore the refinement of formalization on a more theoretical ground, and investigate the application of such formalization on a more diverse set of data modality. Since our proposed SD-VAE differentiate itself from previous works (CVAE, GVAE) on the formalization of syntax and semantics, we therefore use the same deep neural network model architecture for a fair comparison. In encoder, we use 3-layer one-dimension convolution neural networks (CNNs) followed by a full connected layer, whose output would be fed into two separate affine layers for producing µ and σ respectively as in reparameterization trick; and in decoder we use 3-layer RNNs followed by a affine layer activated by softmax that gives probability for each production rule. In detail, we use 56 dimensions the latent space and the dimension of layers as the same number as in BID18. As for implementation, we use Kusner et al. FORMULA0's open sourced code for baselines, and implement our model in PyTorch framework 3.In a 10% validation set we tune the following hyper parameters and report the test from setting with best valid loss. For a fair comparison, all tunings are also conducted in the baselines. We use ReconstructLoss + αKLDivergence as the loss function for training. A natural setting is α = 1, but BID18 suggested in their open-sourced implementation 4 that using α = 1/LatentDimension would leads to better . We explore both settings. The Bayesian optimization is used for searching latent vectors with desired target property. For example, in symbolic program regression, we are interested in finding programs that can fit the given input-output pairs; in drug discovery, we are aiming at finding molecules with maximum drug likeness. To get a fair comparison with baseline algorithms, we follow the settings used in BID18.Specifically, we first train the variational autoencoder in an unsupervised way. After obtaining the generative model, we encode all the structures into latent space. Then these vectors and corresponding property values (i.e., estimated errors for program, or drug likeness for molecule) are used to train a sparse Gaussian process with 500 inducing points. This is used later for predicting properties in latent space. Next, 5 iterations of batch Bayesian optimization with the expected improvement (EI) heuristic is used for proposing new latent vectors. In each iteration, 50 latent vectors are proposed. After the proposal, the newly found programs/molecules are then added to the batch for next round of iteration. During the proposal of latent vectors in each iteration, we perform 100 rounds of decoding and pick the most frequent decoded structures. This helps regulates the decoding due to randomness, as well as increasing the chance for baselines algorithms to propose valid ones. We visualize some reconstruction of SMILES in FIG6. It can be observed that, in most cases the decoder successfully recover the exact origin input. Due to the stochasticity of decoder, it may have some small variations.

A new generative model for discrete structured data. The proposed stochastic lazy attribute converts the offline semantic check into online guidance for stochastic decoding, which effectively addresses the constraints in syntax and semantics, and also achieves superior performance

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: Modeling informal inference in natural language is very challenging. With the recent availability of large annotated data, it has become feasible to train complex models such as neural networks to perform natural language inference (NLI), which have achieved state-of-the-art performance. Although there exist relatively large annotated data, can machines learn all knowledge needed to perform NLI from the data? If not, how can NLI models benefit from external knowledge and how to build NLI models to leverage it? In this paper, we aim to answer these questions by enriching the state-of-the-art neural natural language inference models with external knowledge. We demonstrate that the proposed models with external knowledge further improve the state of the art on the Stanford Natural Language Inference (SNLI) dataset. Reasoning and inference are central to both human and artificial intelligence. Natural language inference (NLI) is concerned with determining whether a natural-language hypothesis h can be inferred from a natural-language premise p. Modeling inference in human language is very challenging but is a basic problem towards true natural language understanding -NLI is regarded as a necessary (if not sufficient) condition for true natural language understanding BID20.The most recent years have seen advances in modeling natural language inference. An important contribution is the creation of much larger annotated datasets such as SNLI BID5 and MultiNLI BID37. This makes it feasible to train more complex inference models. Neural network models, which often need relatively large amounts of annotated data to estimate their parameters, have shown to achieve the state of the art on SNLI and MultiNLI BID5 BID25 BID27 BID30 BID26 BID8 BID1. While these neural networks have shown to be very effective in estimating the underlying inference functions by leveraging large training data to achieve the best , they have focused on end-toend training, where all inference knowledge is assumed to be learnable from the provided training data. In this paper, we relax this assumption, by exploring whether external knowledge can further help the best reported models, for which we propose models to leverage external knowledge in major components of NLI. Consider an example from the SNLI dataset:• p: An African person standing in a wheat field.• h: A person standing in a corn field. If the machine cannot learn useful or plenty information to distinguish the relationship between wheat and corn from the large annotated data, it is difficult for a model to predict that the premise contradicts the hypothesis. In this paper, we propose neural network-based NLI models that can benefit from external knowledge. Although in many tasks learning tabula rasa achieved state-of-the-art performance, we believe complicated NLP problems such as NLI would benefit from leveraging knowledge accumulated by humans, at least in a foreseeable future when machines are unable to learn that with limited data. A typical neural-network-based NLI model consists of roughly four components -encoding the input sentences, performing co-attention across premise and hypothesis, collecting and computing local inference, and performing sentence-level inference judgment by aggregating or composing local information information. In this paper, we propose models that are capable of leveraging external knowledge in co-attention, local inference collection, and inference composition components. We demonstrate that utilizing external knowledge in neural network models outperforms the previously reported best models. The advantage of using external knowledge is more significant when the size of training data is restricted, suggesting that if more knowledge can be obtained, it may yielding more benefit. Specifically, this study shows that external semantic knowledge helps mostly in attaining more accurate local inference information, but also benefits co-attention and aggregation of local inference. Early work on natural language inference (also called recognizing textual textual) has been performed on quite small datasets with conventional methods, such as shallow methods BID13, natural logic methods BID20, among others. These work already shows the usefulness of external knowledge, such as WordNet BID22, FrameNet BID1, and so on. More recently, the large-scale dataset SNLI was made available, which made it possible to train more complicated neural networks. These models fall into two kind of approaches: sentence encodingbased models and inter-sentence attention-based models. Sentence encoding-based models use Siamese architecture BID7 -the parameter-tied neural networks are applied to encode both the premise and hypothesis. Then a neural network classifier (i.e., multilayer perceptron) is applied to decide the relationship between the two sentence representations. Different neural networks have been utilized as sentence encoders, such as LSTM BID5, GRU BID33, CNN BID23, BiLSTM and its variants BID17 BID9, and more complicated neural networks BID6 BID24 BID1. The advantage of encoding-based models is that the encoders transform sentences into fixed-length vector representations, which can help a wide range of transfer tasks BID12. However, this architecture ignores the local interaction between two sentences, which is necessary in traditional natural language inference procedure BID20.Therefore, inter-sentence attention-based models were proposed to relieve this problem. In this framework, local inference information is collected by the attention mechanism and then fed into neural networks to compose as fixed-sized vectors before the final classification. Many related works follow this route BID29 BID34 BID11 BID27 BID8. Among them, BID29 were the first to propose neural attention-based models for NLI. BID8 proposed an enhanced sequential inference model (ESIM), which is one of the best models so far and regarded as the baseline in this paper. In general, external knowledge have been shown to be effective in a wide range of NLP tasks, including machine translation BID32, language modeling BID0, and dialogue system. For NLI, to the best of our knowledge, we are the first to utilize external knowledge together with neural networks. In this paper, we first show that a neural network equipped with external knowledge obtains further improvement over the already strong baseline, and achieves an accuracy of 88.6% on the SNLI benchmark. Furthermore, we show that the gain is more significant when using less training samples. External knowledge needs to be converted to a numerical representation for enriching natural language inference model. One of approaches to represent external knowledge is using knowledge graph embeddings, such as TransE BID4, TransH BID35, TransG BID38, and so on. However, these kind of approaches usually need to train a knowledge-graph embedding beforehand. In this paper, we propose to use relation features to describe relationship between the words in any word pair, which can be easily obtained from various knowledge graphs, such as WordNet BID22, and Freebase BID2. Specifically, we use WordNet to measure the semantic relatedness of the word in a pair using various relation types, including synonymy, antonymy, hypernymy, and so on. Each of these features is a real number on the interval. The definition and instances of pair features derived from WordNet are indicated in TAB0. The setting of features refers to , but we add a new feature same hypernym, which improve the significantly in our experiments. Intuitively, the synonymy, hypernymy and hyponymy features help model the entailment of word pairs; the antonymy and same hypernym features help model contradiction in word pairs. We regard the vector r ∈ R Dr as the relation feature derived from external knowledge, where D r is 5 in our experiments. The r will be enriched in the neural inference model to capture external semantic knowledge. TAB1 reports some key statistics of the relation features from WordNet. We present here our natural inference models which are composed of the following major components: input encoding, knowledge enriched co-attention, knowledge enriched local inference collection, and knowledge enriched inference composition. Figure 1 shows a high-level view of the architecture. First, the premise and hypothesis are encoded by the input encoding components as context-dependent representations. Second, co-attention is calculated to obtain word-level softalignment between two sentences. Third, local inference information is collected to prepare for final prediction. Fourth, the inference composition component applies aggregation of the whole sentences and makes final prediction based on the fixed-size vector. Among them, external knowledge is regard as the auxiliary component to improve the ability of calculating co-attention, collecting local inference information and composing inference. Figure 1: A high-level view of our neural inference networks. Given two sentence, i.e., the premise "The child is getting a pedicure", and the hypothesis "The kid is getting a manicure", the model needs to predict the relationship among them: entailment, contradiction, or neutral. Given the word sequences of the premise a = (a 1, . . ., a M) and the hypothesis b = (b 1, . . ., b N), where M and N are the lengths of the sentences, the final objective is to predict a label y that indicates the logic relationship between a and b. The formula is y = arg max DISPLAYFORM0 Specifically, "<BOS>" and "<EOS>" are inserted as the first and last token, respectively. First, a and b are embedded into a D e -dimensional vectors [E(a 1),..., E(a M)] and [E(b 1),..., E(b N)] using an embedding matrix E ∈ R De×V, where V is the vocabulary size and E can be initialized with some pre-trained word embeddings from a universal corpus. To represent the words of the premise and hypothesis in a context-dependent way, the two sentences are fed into the encoders to obtain context-dependent hidden states a s and b s. The formula is DISPLAYFORM1 We employ bidirectional LSTMs (BiLSTMs) BID15 as encoders, which is a common choice for natural language. A BiLSTM runs a forward and a backward LSTM on a sequence starting from the left and the right end, respectively. The hidden states generated by these two LSTMs at each time step are concatenated to represent that time step and its context: DISPLAYFORM2 The hidden states of the unidirectional LSTM (h → t or h ← t) is calculated as follows: DISPLAYFORM3 DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 DISPLAYFORM7 DISPLAYFORM8 where σ is the sigmoid function, is the element-wise multiplication of two vectors. DISPLAYFORM9 are parameters to be learned. D is the dimension of the hidden states in the LSTM. The LSTM utilizes a set of gating functions for each input vector x t, i.e., the input gate i t, forget gate f t, and output gate o t, together with a memory cell c t to generate a hidden state h t. In this component, we acquire soft-alignment of word pairs between the premise and hypothesis based on our knowledge-enriched co-attention mechanism. Given the relation features r ij ∈ R Dr between the premise's i-th word and the hypothesis's j-th word from the external knowledge, the co-attention is calculated as e ij = (a DISPLAYFORM0 The function F can be any non-linear or linear function. Here we use F (r ij) = λ1(r ij), where λ is a hyper-parameter tuned on the development set and 1 is the indication function.1(r ij) = 1 if r ij is not zero vector; 0 if r ij is zero vector.Intuitively, the word pairs with semantic relationship in various features are probably aligned together. Soft-alignment is determined by the co-attention matrix e ∈ R M ×N computed in Equation, which is used to obtain the local relevance between the premise and hypothesis. For the hidden state of a word in a premise, i.e., a s i (already encoding the word itself and its context), the relevant semantics in the hypothesis is identified into a context vector a c i using e ij, more specifically with Equation. DISPLAYFORM1, a DISPLAYFORM2 DISPLAYFORM3 where α ∈ R M ×N and β ∈ R M ×N are the normalized attention weight matrices with respect to the 2-axis and 1-axis. The same calculation is performed for each word in the hypothesis, i.e., b DISPLAYFORM4 DISPLAYFORM5 where a heuristic matching trick with difference and element-wise product is used BID23 BID8. The last term in Equation FORMULA0 aims to obtain the local inference relationship between the original vectors (a In this component, we introduce knowledge-enriched inference composition. To determine the overall inference relationship between a premise and a hypothesis, we need to explore a composition layer to compose the local inference vectors (a m and b m) collected above. The formula is DISPLAYFORM0 Here, we also use BiLSTMs as building blocks for the composition layer. The BiLSTMs read local inference vectors (a m and b m) and learn to judge the type of local inference relationship and distinguish crucial local inference vectors for overall sentence-level inference relationship. The responsibility of BiLSTMs in the inference composition layer is completely different from the BiLSTMs in the input encoding layer. Our inference model converts the output hidden vectors of BiLSTMs to a fixed-length vector with pooling operations and puts it into the final classifier to determine the overall inference class. Particularly, besides using mean pooling and max pooling similarly to ESIM BID8, we propose to use weighted pooling based on external knowledge to obtain a fixed-length vector as in Equation FORMULA0. Intuitively, the final prediction is mostly determined by those word pairs appearing in the external knowledge. BID9 uses a similar idea called gated-attention but they do not use external knowledge. DISPLAYFORM1 In our experiments, we regard the function H as a 1-layer feed-forward neural network with ReLU activation function. We concatenate all pooling vectors, i.e., mean, max, and weighted pooling, into a fixed-length vector and then put the vector into a final multilayer perceptron (MLP) classifier. The MLP has a hidden layer with tanh activation and softmax output layer in our experiments. The entire model is trained end-to-end, through minimizing the cross-entropy loss. The Stanford Natural Language Inference (SNLI) dataset BID5 focuses on three basic relationships between a premise and a potential hypothesis: the premise entails the hypothesis (entailment), they contradict each other (contradiction), or they are not related (neutral). We use the same data split as in previous work, and use classification accuracy as the evaluation metric, as in related work. WordNet 3.0 BID22 is used to extract semantic relation features between words, as described in Section 3.1. The words are lemmatized using Stanford CoreNLP 3.7.0 to match words in WordNet, but the input word sequences for the input encoding layer are only tokenized, without lemmatization. We release our code at [xxx] to make it replicatibility purposes. The models are selected on the development set. Some of our training details are as follows: the dimension of the hidden states of LSTMs and word embeddings are 300. The word embeddings are initialized by 300D GloVe 840B BID28, and out-of-vocabulary words among them are initialized randomly. All word embeddings are updated during training. Adam BID16 ) is used for optimization with an initial learning rate 0.0004. The mini-batch size is set to 32. Dropout with a keep rate of 0.5 and early stopping with patience of 7 are used to avoid overfitting. The gradient is clipped with a maximum L2-norm 10. The trade-off λ for calculating co-attention in Equation FORMULA10 is selected in [0.1, 0.2, 0.5, 1, 2, 5, 10, 20, 50] based on the development set. LSTM BID5 80.6 GRU BID33 81.4 Tree CNN BID23 82.1 SPINN-PI BID6 83.2 NTI BID25 83.4 Intra-Att BiLSTM 84.2 Self-Att BiLSTM BID17 84.2 NSE BID24 84.6 Gated-Att BiLSTM BID9 85.5 DiSAN BID31 85.6 LSTM Att BID29 83.5 mLSTM BID34 86.1 LSTMN BID11 86.3 Decomposable Att BID27 86.8 NTI BID25 87.3 Re-read LSTM BID30 87.5 BiMPM 87.5 btree-LSTM BID26 87.6 DIM BID14 88.0 ESIM BID8 88.0 KIM 88.6 HIM (ESIM+Syntactic TreeLSTM) BID8 88.6 BiMPM (Ensemble) 88.8 DIIN (Ensemble) 88.9 KIM (Ensemble) 89.1 TAB2 shows the of different models on the SNLI dataset. The first group of models use sentence-encoding based approaches. BID5 employs LSTMs as encoders for both the premise and hypothesis into two fixed-size sentence vectors. Then the sentence representation is put into a MLP classifier to predict the final inference relationship. The accuracy on the test set is 80.6%. Many related works follow this framework, using different neural networks as encoders. Their performances are also listed in the first group in TAB2. Among them, gated-Att BiLSTM BID9 achieves an accuracy of 85.5%, which is state of the art for sentenceencoding based approaches. The second group of models uses a cross-sentence attention mechanism, which can obtain softalignment information between cross-sentence word pairs. BID34 proposes a matching-LSTM to compare the inference information of locally-aligned words, and obtains a higher accuracy of 86.1%, even better than the state-of-the-art sentence-encoding models. Other related models are also listed in the second group in TAB2. Among them, ESIM BID8 ) is the previous state-of-the-art system, whose accuracy in test set is 88.0%. The proposed model, namely Knowledge-based Inference Model (KIM), which enriches ESIM with external knowledge, obtains an accuracy of 88.6%. The difference between ESIM and KIM is statistically significant under the one-tailed paired t-test at the 99% significance level. To be best of our knowledge, this is a new state of the art. Our ensemble model, which averages the probability distributions from ten individual single KIMs with different initialization, achieves an even higher accuracy, 89.1%. To compare the importance of external knowledge under different training data scales, we randomly sample different ratio of the whole training set, i.e., 0.8%, 4%, 20% and 100%. "A" indicates adding external knowledge in calculating the co-attention matrix as in Equation FORMULA10, "I" indicates adding external knowledge in collecting local inference information as in Equation FORMULA0, and "C" indicates adding external knowledge in composing inference as in Equation. When we only have restricted training data, i.e., 0.8% training set (about 4,000 samples), our baseline ESIM has a poor accuracy of 62.4%. When we only add external knowledge in calculating co-attention ("A"), the accuracy increases to 66.6% (+ absolute 4.2%). When we only utilize external knowledge in collecting local inference information ("I"), the accuracy has a significant gain, to 70.3% (+ absolute 7.9%). When we only add external knowledge in inference composition ("C"), the accuracy gets a smaller gain to 63.4% (+ absolute 1.0%). The comparison indicates that "I" plays the most important role among the three components in using external knowledge. Moreover, when we compose the three components ("A,I,C"), we obtain the best of 72.6% (+ absolute 10.2%). When we use more training data, i.e., 4%, 20%, 100% of the training set, only utilizing external knowledge in local inference information collected ("I") achieves a significant gain, but "A" or "C" do not bring any significant improvement. The indicate that external semantic knowledge only helps in co-attention and composition when there is limited training data, but always helps in collecting local inference information. Meanwhile, for less training data, λ is usually set to a larger value. For example, the optimal λ tuned on the development set is 20 for 0.8% training set, 2 for the 4% training set, 1 for the 20% training set and 0.2 for the 100% training set. Figure 3 displays the of using different ratio of external knowledge for different training data size. Note that here we only use external knowledge in collecting local inference information, because it always works well for different scale of the training set. Better accuracies are achieved when using more external knowledge. Especially under the condition of restricted training data (0.8%), the model obtains a large gain when using more than half of the external knowledge. Our enriched neural network-based model for natural language inference with external knowledge, namely KIM, achieves a new state-of-the-art accuracy on the SNLI dataset. The model is equipped with external knowledge in the major informal inference components, specifically, in calculating co-attention, collecting local inference, and composing inference. The proposed models of infusing neural networks with external knowledge may also help shed some light on tasks other than NLI, such as question answering and machine translation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

the proposed models with external knowledge further improve the state of the art on the SNLI dataset.

represent the short summary for finding the corresponding scientific article (long)

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.