Question: Represent the scientific article to find a short TLDR

Question: Representation learning is one of the foundations of Deep Learning and allowed important improvements on several Machine Learning tasks, such as Neural Machine Translation, Question Answering and Speech Recognition. Recent works have proposed new methods for learning representations for nodes and edges in graphs. Several of these methods are based on the SkipGram algorithm, and they usually process a large number of multi-hop neighbors in order to produce the context from which node representations are learned. In this paper, we propose an effective and also efficient method for generating node embeddings in graphs that employs a restricted number of permutations over the immediate neighborhood of a node as context to generate its representation, thus ego-centric representations. We present a thorough evaluation showing that our method outperforms state-of-the-art methods in six different datasets related to the problems of link prediction and node classification, being one to three orders of magnitude faster than baselines when generating node embeddings for very large graphs. Many important problems involving graphs require the use of learning algorithms to make predictions about nodes and edges. These predictions and inferences on nodes and edges from a graph are typically done using classifiers with carefully engineered features BID13. These features, besides taking time and manual labor to be developed and acquired, usually do not generalize well to other problems or contexts. The field of Natural Language Processing (NLP) has had many advances due to the use of algorithms that learn word representations, instead of manually extracted features. Originally proposed by BID5 and commonly used with Word2Vec algorithms like CBOW and SkipGram (a), word embeddings are used in many state-of-the-art solutions for neural machine translation (; BID12, question answering BID2 and natural language generation . Recent works have proposed new methods for learning representations for nodes and edges in graphs, based on random walks (; BID13 or auto-encoding adjacency vectors .In this paper, we propose a new general purpose method for generating node embeddings in very large graphs, which we call Neighborhood Based Node Embeddings (or simply NBNE). NBNE is based on the SkipGram algorithm and uses nodes neighborhoods as contexts. NBNE outperforms state-of-the-art DeepWalk and Node2Vec BID13 for the tasks of link prediction and node classification on six collections, being one to three orders of magnitude faster. In this work, we considered DeepWalk and Node2Vec as baselines. The main reason for this improvement on effectiveness and efficiency is that we concentrate learning on the "predictable" parts of the graph. A study by Facebook research BID8 found that each person in the world (at least among the people active on Facebook) is connected to every other person by an average 3.57 other people. In a graph of this magnitude and connectedness, learning node embeddings by maximizing the log-probability of predicting nearby nodes in a random walk (with a window size of 5) can be highly inefficient and make it'harder' for the embeddings to be constructed, even if these random walks are biased like in Node2Vec. We suspect this can also make them more unstable, which would explain why they need more iterations before embedding convergence. The definition of node similarity and finding general purpose node and/or edge representations are non-trivial challenges (Lü &). Many definitions of similarity in graphs use the notion of first and second order proximity. First-order proximity is the concept that connected nodes in a graph should have similar properties, while the second-order proximity indicates that nodes with similar neighborhoods should have common characteristics. Some earlier works on finding these embeddings use various matrix representations of the graph, together with dimensionality reduction techniques, to obtain the nodes' representations . A problem with these approaches is that they usually depend on obtaining the matrix' eigenvectors, which is infeasible for large graphs (O(n 2.376)) with the Coppersmith-Winograd algorithm BID6 ). Recent techniques attempt to solve this problem by dynamically learning representations for nodes in a graph using non-linear techniques based either on first and second order proximities or random walks (; BID13 .Other recent works focus on finding representations for specific types of graphs. TriDNR uses a graph structure together with node content and labels to learn node representations in two citation networks. Their work can be directly applied to any graph where nodes have labels and/or text contents. TEKE and KR-EAR find representations for entities in knowledge graphs and metapath2vec BID7 finds node representations in heterogeneous networks. The method LINE finds a d dimensional representation for each node based on first and second-order graph proximities, not being feasible for large graphs, because its cost function depends on the whole adjacency matrix (O(|V | 2)).Another method, Structural Deep Network Embedding (SDNE) , is also based on first and second order proximities. It uses autoencoders to learn a compact representation for nodes based on their adjacency matrix (second-order proximity), while forcing representations of connected nodes to be similar (first-order proximity) by using an hybrid cost function. SDNE is also not feasible for large graphs, since the autoenconders are trained on the complete adjacency vectors. Each vector has size O(|V |) and is created at least once, creating a lower bound on time complexity DISPLAYFORM0 The method DeepWalk generates k random walks starting on each vertex in the graph to create sentences where each "word" is a node. These sentences are then trained using the SkipGram algorithm to generate node embeddings. This method has a time complexity bounded by O(|V | log |V |).Node2Vec BID13 ) also uses random walks with SkipGram and can be seen as a generalization of DeepWalk. The difference between the two methods is that Node2Vec's walks are random, but biased by two pre-assigned parameters p and q. During the creation of the walks, these parameters are used to increase the chance of the walk returning to a parent node or going farther from it. This method uses a semi-supervised approach which requires several models to be generated and a small sample of labeled nodes to be used so that the best parameters p and q can be chosen. Node2Vec is not efficient for densely connected graphs, since its time and memory dependencies on the graph's branching factor b are O(b 2).In this work, we considered DeepWalk and Node2Vec BID13 as baselines, since they are scalable, having a time complexity (O(|V | log |V |)). The main differences between NBNE and the two baselines are: (i) we use a different sentence sampling strategy which is based in a node's neighborhood instead of random walks, (ii) NBNE is more effective than both Node2Vec and DeepWalk, as supported by our experiments in six different datasets, and (iii) NBNE is efficient for both dense and sparse graphs and scalable for very large applications, having a faster training time than both Node2Vec and DeepWalk. The context of a word is not a straightforward concept, but it is usually approximated by the words surrounding it. In graphs, a node's context is an even more complex concept. As explained above, DeepWalk and Node2Vec use random walks as sentences and consequently as contexts in which nodes appear. In this work, the contexts are based solely on the neighborhoods of nodes, defined here as the nodes directly connected to it, focusing mainly on the second-order proximities. Consequently, nodes' representations will be mainly defined by their neighborhoods and nodes with similar neighborhoods (contexts) will be associated with similar representations. In our Neighborhood Based Node Embedding's (NBNE) method, as the name implies, sentences are created based on the neighborhoods of nodes. There are two main challenges in forming sentences from neighborhoods, as follows:• A sentence containing all the neighbors from a specific highly connected root node might be of little use. Most neighbors would be distant from each other in the sentence, not influencing each other's representations, and not directly influencing the root node.• There is no explicit order in the nodes in a neighborhood. So there is no clear way to choose the order in which they would appear in a sentence. In this work, the solution is to form small sentences, with only k neighbors in each, using random permutations of these neighborhoods. Algorithm 1 presents the code for generating sentences. As a trade-off between training time and increasing the training dataset the user can select the number of permutations n. Selecting a higher value for n creates a more uniform distribution on possible neighborhood sentences, but also increases training time. Algorithm 1 Sentence Sampling 1: procedure GETSENTENCES(graph, n) 2: DISPLAYFORM0 for j in 0: n do for node in graph.nodes do neighbors ← random permutation(node.neighbors) 6:for i in 0: len(neighbors)/k do 7: DISPLAYFORM0 sentences.append(sentence) As described in Section 3.1, Algorithm 1 forms a set of sentences S, where each word is actually a node from the graph. We train the vector representations of nodes by maximizing the log probability of predicting a node given another node in a sentence and given a set of representations r. We use a window of size k which is equal to the size of the generated sentences, so that each node in a sentence predicts all the others. The log probability maximized by NBNE is given by: DISPLAYFORM0 where p (s|r) is the probability of each sentence, which is given by: DISPLAYFORM1 where v i is a vertex in the graph and v j are the other vertices in the same sentence. The probabilities in this model are learned using the feature vectors r vi, which are then used as the vertex representations. The probability p (v j |v i, r) is given by: DISPLAYFORM2 where r T vj is the transposed output feature vector of vertex j, used to make predictions. The representations r v and r v are learned simultaneously by optimizing Equation. This is done using stochastic gradient ascent with negative sampling (b).By optimizing this log probability, the algorithm maximizes the predictability of a neighbor given a node, creating node embeddings where nodes with similar neighborhoods have similar representations. Since there is more than one neighbor in each sentence, this model also makes connected nodes have similar representations, because they will both predict each others neighbors, ing in representations also having some first order similarities. A trade-off between first and second order proximity can be achieved by changing the parameter k, which simultaneously controls both the size of sentences generated and the size of the window used in the SkipGram algorithm. A further discussion on this effect can be seen in Appendix B.3. When using large values of n (i.e., number of permutations) on graphs with few edges per node, some overfitting can be seen on the representations, as shown in details in Section 5.1 and in Appendix B.2. This overfitting can be avoided by sequentially training on increasing values of n and testing the embeddings on a validation set every few iterations, stopping when performance stops improving, as shown in Algorithm 2.Algorithm 2 NBNE without Overfitting DISPLAYFORM0 sentences ← get sentences(graph, max n) DISPLAYFORM1 for j in 0: log 2 (max n) do brief description of each dataset can be found in Appendix A and an analysis of their assortativity properties in Appendix B.1. We present for the link prediction problem in Section 4.1 and for the node classification problem in Section 4.2. For all experiments we used sentences of size k = 5 and embeddings of size d = 128, while the number of permutations was run for n ∈ {1, 5, 10}. The best value of n was chosen according to the precision on the validation set and we used early stopping, as described in Section 3.3.On both these tasks, DeepWalk and Node2Vec were used as baselines, having been trained and tested under the same conditions as NBNE and using the parameters as proposed in BID13. More specifically, we trained them with the same training, validation and test sets as NBNE and used a window size of 10 (k), walk length (l) of 80 and 10 runs per node (r). For Node2Vec, which is a semi-supervised algorithm, we tuned p and q on the validation set, doing a grid search on values p, q ∈ {0.25; 0.5; 1; 2; 4}. We also evaluated NBNE on two synthetic graphs with different sizes and sparseness, which can be seen on Appendix C, and an author name disambiguation task, on Appendix F. A comparison between NBNE and SDNE can be seen on Appendix D. Setup. Link prediction attempts to estimate the likelihood of the existence of a link between two nodes, based on observed links and the nodes' attributes (Lü &). Typical approaches to this task use similarity metrics, such as Common Neighbors or Adamic-Adar BID1. Instead of these hand made similarity metrics, we propose to train a logistic classifier based on the concatenation of the embeddings from both nodes that possibly form an edge and predict the existence or not of the edge. To train NBNE on this task, we first obtained a sub-graph with 90% randomly select edges from each dataset, and obtained the node embeddings by training NBNE on this sub-graph. We, then, separated a small part of these sub-graph edges as a validation set, using the rest to train a logistic regression with the learned embeddings as features. After the training was completed, the unused 10% of the edges were used as a test set to predict new links. 10-fold cross-validation was used on the entire training process to access the statistical significance of the , analyzing statistical difference between the baselines and NBNE. To evaluate the on this task, we used as metrics: AUC (area under the ROC curve) , and training time. 3 The logistic regressions were all trained and tested using all available edges (respectively in the training or test set), and an equal sized sample of negative samples, which, during training, included part of the 10% removed edges. Results. TAB1 presents for this task. Considering AUC scores on the Link Prediction task, NBNE was statistically better 4 than both DeepWalk and Node2Vec on the Astro and PPI datasets, with more than 7% improvement, also showing a 4.67% performance gain in Wikipedia and a small, but statistically significant, gain on Blog. Only losing by a small percentage on Facebook, with a difference that was not statistically significant. In DBLP, NBNE again presents the best AUC score, although this difference was small and its statistical significance could not be verified due to the large training times of the baselines. This dataset contains the largest graph analyzed in this work (317,080 nodes and 1,049,866 edges) and in it, to train a single fold, Node2Vec took 3,285m59s (more than 54 hours) and DeepWalk took 164m34s (approximately 2 hours and 44 minutes), while NBNE took only 14m30s, which represents a 226/11 times improvement over Node2Vec and DeepWalk, respectively. Considering training time for this task, NBNE presents the biggest improvements on sparser networks of medium size, like Astro, PPI and Wikipedia datasets. On these graphs, the best are for n = 1, ing in more than 50x faster training than DeepWalk and more than 1,500 times faster than Node2Vec, achieving a 6,049 times faster training than Node2Vec on Wikipedia. For the Blog and Facebook datasets the best are for n = 5, ing in larger training times, but still more than one order of magnitude faster than DeepWalk and more than 350 times faster than Node2Vec. For the DBLP dataset, the best were achieved with n = 10, still much faster than the baselines. Setup. Given a partially labeled graph, node classification is the task of inferring the classification of the unknown nodes, using the structure of the graph and/or the properties of the nodes. In this task, the node embeddings were trained using NBNE on the complete graph. After obtaining the node embeddings, 80% of the labeled nodes in the graph were used to train a logistic classifier that predicted the class of each node, while 5% of the nodes were used for validation and the remaining 15% nodes were used as a test set. This test was repeated for 10 different random seed initializations to access the statistical relevance of the . Results. Results on the Blog, PPI and Wikipedia datasets are shown in TAB2 and are presented in terms of Macro F1 scores and training times. NBNE produces statistically similar to its baselines, in terms of Macro F1, on both PPI and Wikipedia, while showing a statistically significant 22.45% gain in the Blog dataset, indicating that NBNE's embeddings did not only get a better accuracy on Blog, but also that correct answers were better distributed across the 39 possible classes. Considering training times, NBNE is more than 10 times faster than DeepWalk on these three datasets and is [300 ∼ 600] times faster than Node2Vec. NBNE didn't show statistically worse in any dataset analyzed here 5, while having an order of magnitude faster training time than DeepWalk and more than two orders of magnitude faster training time than Node2Vec. The quality of NBNE's embeddings depends on both the size of the embeddings (d) and the number of permutations (n). For highly connected graphs, larger numbers of permutations should be chosen (n ∈) to better represent distributions, while for sparser graphs, smaller values can be used (n ∈). FIG0 shows AUC scores versus embedding sizes for several values of n on the Facebook link prediction task. Quadratic functions approximating log(auc score) were plotted to allow for a better understanding of the . It can be seen that for larger numbers of permutations (n > 100) improve with embedding size, while for small ones (n = 1) they decrease. The plot also shows that n = 10 gives fairly robust values for all tested embedding sizes. A further analysis can be seen in TAB3, which shows that graphs with more edges per node tend to get better with larger values of n, while graphs with a smaller branching factor have better with only one permutation (n = 1). Other factors also enter into account when choosing n, like graph size. For example, link prediction on the DBLP graph had its best for n = 10, although its branching size was only 3.31. Further experiments on this parameter can be seen in Appendices B.2 and C.1. which is the size of the vocabulary (|V |), the algorithm will take a time bounded by: DISPLAYFORM0 Figure 2 (Top-Left and Top-Right) show training time is indeed linear on both embedding size and number of permutations. It also shows that Node2Vec is considerably slower than DeepWalk, and only has a similar training time to running NBNE with at least n = 1000. NBNE with n < 10 was by far the fastest algorithm. NBNE, Node2Vec and DeepWalk run in a time bounded by O(|V | log |V |), as can be seen in. FIG1 (Bottom-Right) shows that NBNE's time complexity is linear in the branching factor b, while Node2Vec's is quadratic. DeepWalk's running time is constant in this parameter, but for a graph with a larger branching factor, a higher number of walks per node should be used to train this algorithm, which would make it indirectly dependent on this factor. The proposed node embedding method NBNE shows similar or better than the state-of-theart algorithms Node2Vec and DeepWalk on several different datasets. It shows promising in two application scenarios: link prediction and node classification, while being efficient and easy to compute for large graphs, differently from other node embedding algorithms, such as LINE or SDNE . NBNE focuses learning on node's immediate neighbors, creating more ego-centric representations, which we suspect makes them more stable and faster to learn. Empirical show that, although it has a similar time complexity, NBNE can be trained in a fraction of the time taken by DeepWalk (10 to 190 times faster) or Node2Vec (200 to 6,000 times faster), giving fairly robust . Since embeddings are learned using only a node's immediate neighbors, we suspect it should also be easier to implement more stable asynchronous distributed algorithms to train them, and we leave this as future work. We used a total of six graph datasets to evaluate NBNE, with DeepWalk and Node2Vec being used as baselines. Next we briefly describe these datasets:1. Facebook : A snapshot of a subgraph of Facebook, where nodes represent users and edges represent friendships.2. Astro BID15: A network that covers scientific collaborations between authors whose papers were submitted to the Astrophysics category in Arxiv.3. Protein-Protein Interactions (PPI) : We use the same subgraph of the PPI network for Homo Sapiens as in BID13. This subgraph contains nodes with labels from the hallmark gene sets BID16 Assortativity, also referred to as homophily in social network analysis, is a preference of nodes to attach themselves to others which are similar in some sense. In this section, we further investigate assortativity properties related to both the representations generated by our algorithm, as of the graphs themselves. In Section B.1, we do a quantitative analysis on the homophily inherent to the datasets considered in this work. In Section B.2, we make a qualitative analysis of how assortativity varies depending on the number of permutations n. In Section B.3, we make a qualitative analysis on the trade-off of first and second order proximities based on the choice of k. There are several ways to quantitatively capture the homophily present in a graph. Jensen & Neville describe relational auto-correlation, which is Pearson's contingency coefficient on the characteristics of nodes which share edges BID14 ). Park & Barabási define dyadicity and heterophilicity, which respectively measure how a graph's nodes share common/different characteristics in edges, compared to a random model. TAB5 presents both degree and label assortativity properties of the six graphs analysed here, calculated using the definition of. We can see in this table that the datasets analyzed in this work cover a broad specter of assortativity properties. PPI, Wikipedia and Blog graphs present negative degree assortativity, which means nodes in these graphs are more likely to connect with nodes of different connectivity degrees. At the same time, Facebook, Astro and DBLP present positive degree assortativity, which indicates that their nodes tend to connect to others with similar degrees. We also analyze graphs with both positive and negative label assortativity in our label classification task. While PPI and Blog datasets present positive label assortativity, with connected nodes more frequently sharing classes, Wikipedia has a negative assortativity, with its connected nodes being more likely to have different classes. Here, we further analyze how the number of permutations (n) influences both homophily and overfitting in our learned representations. We qualitatively measure homophily by comparing either cosine or euclidean distances between nodes on edges to the distances in non-edges. The cosine distances for the PPI dataset, shown by the box plots in FIG4 (top-left), clearly show for larger values of n how the embeddings overfit to the specific graph structure, with the learned similarity on edges not generalizing to the links which were previously removed. In this graph, for larger numbers of permutation the removed edges have a distribution more similar to the non edges than to the edges used during training, which is a tendency that can be observed in the other graphs, although in a smaller scale. The box plots in FIG4 (top-right) show the cosine distance for Facebook nodes. We can see that for n = 5 there is a larger separation between removed edges and non edges, which justifies the algorithm's choice of this value. For larger values of n we can again see an overlap between the distributions, caused by the embeddings overfitting. On the other hand, the cosine distances for the DBLP in FIG4 (bottom-left) show the largest separation for n = 10.Finally, the box plots in FIG4 (bottom-right) show cosine distances for the Blog dataset. We can see that for n = 1 and n = 5 there is actually a larger cosine distance between nodes in removed edges than in non edges, with this situation only inverting for n ≥ 10. This happens due to this graph's negative degree homophily. This is also observed for similar graphs in the PPI and Wikipedia datasets, though with a smaller intensity in the PPI graph, which has a smaller absolute value of degree assortativity and where only embeddings for n = 1 present this property. The box plots from FIG4 further support our intuition that graphs with larger branching factors should have larger values of n. At the same time, this choice also depends on the graph size and structure, as shown by the algorithms choice of n = 10 for the DBLP dataset, which contains the largest degree assortativity. The best choice of n depends on the analyzed task, but we believe that, at least for link prediction, this choice is both directly proportional to a graph's branching size and degree assortativity. Nonetheless, the difficulty in analyzing these graphs supports our choice for a semi-supervised approach, automatically choosing n on a per graph instance. Considering again the experiment on the PPI dataset with the number of permutations n = 1 in FIG4 (top-left), in FIG5 we present in detail the euclidean distances between nodes that share or not an edge for this number of permutations. We can see that the distribution of removed edges is a lot closer to the edges used for training than to the non edges. Removed Edge Non Edge Connection type The window size and the number of neighbors in a sentence are both adjusted by a single variable, k, and this variable also controls a trade-off between first and second order proximities in the node embeddings. This can be explained intuitively by analyzing both the sentence sampling method in Algorithm 1 and Equations 1, 2 and 3, in Section 3.2.When a smaller k is chosen, each node's embedding r vi will be mainly used to predict its own neighbors. This causes nodes with shared neighbors to have closer representations (second order proximity). When larger values of k are chosen, nodes will appear more often in its neighbors sentences, and will predict not only its own neighbors, but his neighbors' neighbors. This will in connected nodes to have more similar embeddings, increasing first order similarity. We further analyze this by examining the distribution of cosine distances between nodes at different graph distances. For this analysis, we use three different synthetic graphs: Barabási-Albert BID4; Erdõs-Rényi BID9; Watts-Strogatz . We choose these graphs because of their structural differences, believing they cover an ample specter of different graphs' properties. These graphs were created with |V | = 2000 and b = 20, and Watts-Strogatz graphs had a probability β = 0.2 of generating non-lattice edges. To train our representations we used n = 10 and d = 128. FIG6 shows box plots of these cosine distances of nodes' representations versus their graph distance on these different artificial random graphs. In this figure, we can see that, for both Barabàsi-Albert and Erdõs-Rényi graphs, when using a sentence size (k) equal to 1, the cosine similarity is larger for nodes which are two steps away than for nodes which share an edge (second order proximity), while for larger values of k, nodes which share an edge have larger similarity (first order proximity). The box plots in FIG6 also show that the difference in similarity increases with the value of k. The larger the value of k, the larger the difference between similarities of nodes which share an edge and nodes with larger distances, as can be seen in detail in FIG7 for the Barabási-Albert graph. In this section, we analyze how a graph's sparseness (represented here by its branching factor) and size (represented here by its number of vertices) affect the choice of the number of permutations (n) and of the window/sentence size (k). With this purpose we ran several link prediction experiments on two different synthetic graphs: Watts-Stogratz and Barabási-Albert. 6 These graphs were generated for different sizes (|V |) and sparseness (b), and we ran experiments with the same setup as in Section 4.1, with Watts-Stogratz graphs having again β = 0.2.7 Section C.1 presents this analysis for the number of permutations (n) and Section C.2 contains the analysis for different window/sentence sizes (k). We analyze, in this section, how a graph's size and sparseness affect the choice of the number of permutations (n), for both Watts-Stogratz and Barabási-Albert graphs. Analyzing the graphs in Figure 8, we see a correlation between the best choice of n and a graph's number of vertices (|V |) and branching factor (b). In Figure 8a, which contains the experiments in the most sparse graphs, for n = 1 are better for all graph sizes. A random algorithm would return an AUC score of 0.5, so bellow this value clearly expose a problem in the learning algorithm. This is the case for both n = 10 and n = 5 in these graphs, which overfit its representations. In Figure 8b we can see that, when considering a graph with a branching size of b = 4, for smaller graphs a smaller value of n is preferable, while for larger graphs a larger number of permutations gives better (n = 10). In Figure8c we can see that, for a branching size of b = 8, for larger values of n are always better than for n = 1. Notice also that, while for b = 2 and b = 4 were around 0.55 ∼ 0.7, for b = 8 are closer to 0.9, showing that this algorithm is better at learning with more information. Our experiments in link prediction using synthetic Barabási-Albert graphs present slightly more complex . FIG9 shows that for smaller branching factors (b ≤ 8), n = 1 indeed generate better for small graphs, but for larger graphs, a larger number of permutations is necessary. For intermediary branch sizes the best value of n is harder to determine, and only for b = 64 we start to see a tendency of larger number of permutations consistently giving better . We can also see from FIG9 that edges in Barabási-Albert graphs are considerably more difficult to predict, specially for smaller branching sizes. Most of our are around 60% and our best AUC scores in these graphs are around 70%.Again, n's dependency on these graph properties (|V | and b) depends highly on the graph's structure, further supporting our choice of a semi-supervised approach, choosing n on a per graph instance by validating on a small validation set. This can be considered as a form of early stopping when training these node embeddings. In this section, we again use Watts-Stogratz and Barabási-Albert graphs, this time to analyze how a graph's size and sparseness affect for different window and sentence sizes (k) in our model. For these experiments we keep n = 5 fixed. Barabási-Albert graphs' edges are considerably harder for our algorithm to predict, as shown in the previous section, so we only report for larger values of b (the algorithm, with our choice of hyper-parameters, overfits for smaller values). We can see from FIG0 that larger values of k usually produce better for this graph, but are more propense to overfit, specially when being applied to larger sparse graphs (|V | ≥ 800 and b = 16). Further analysis on the representations' properties for different values of k could provide better motivation on its choice, but we leave this to future studies, keeping our choice of k = 5 constant in this work. Studying if geometric operations between representations have comprehensible meanings would also be interesting, such as was done for Word2Vec algorithms, but this is also left as future work. Structural Deep Network Embedding is another algorithm used for learning node embeddings in graphs. As described in Section 2, SDNE is based on first and second order proximities, using autoencoders to learn compact representations based on a node's adjacency matrix (second-order proximity), while forcing representations of connected nodes to be similar (first-order proximity) by using an hybrid cost function. This algorithm has a time complexity of O(|V | 2), but its main computation, which is calculating the gradients of its cost function and updating model parameters, can be highly parallelized by using modern GPUs and Deep Learning frameworks. In this section, we compare NBNE and SDNE in terms of both efficiency and efficacy, analysing both AUC/Macro F1 scores and training time. With this objective, we trained SDNE embeddings using both a dedicated K40 GPU with CUDA 8.0 and a dedicated 16 core linux server. In their original work, SDNE was run in a semi-supervised setting, finding the best value of α, β and ν by tuning them on a small validation set. In this work we fix α = 0.2 and β = 10, since in their work they state that these values commonly give the best , while only choosing ν in a semi-supervised manner. We use SDNE's architecture with [10,300; 1,000; 128] nodes on each layer and test it on both Link Prediction and Node Classification tasks, using the same steps described in Sections 4.1 and 4.2. We train these embeddings using ν ∈ {0.1, 0.01, 0.001} and choose the best value on the same validation sets used to tune n for NBNE and p and q for Node2vec. TAB6 shows using both NBNE and SDNE embeddings on Link Prediction tasks. In this table we can see that both algorithms produce similar in terms of AUC scores, with each having a statistically significant better on two datasets, and NBNE having a non statistically significant, but slightly better on the fifth. It is clear that even when training SDNE using a K40 GPU, NBNE still has more than an order of magnitude faster training time on all datasets, being more than two orders of magnitude faster on most. When comparing to SDNE trained on a CPU, NBNE has more than three orders of magnitude faster training time. On Astro, the dataset with the largest number of nodes analyzed here, NBNE had a 2,009 times faster training time compared to SDNE on a GPU and 44,896 times faster compared to SDNE on CPU. TAB7 shows the of running NBNE and SDNE on the Node Classification task. On this task NBNE gave statistically better on two datasets, with an impressive gain of 29.27% on PPI and 46.94% on Blog, only losing on Wikipedia with an also large gain of −20.20%. We can again see that NBNE has a more than an order of magnitude faster training time than SDNE on a GPU in this dataset, being more than two orders of magnitude faster when SDNE is trained on a CPU. Analyzing both these tables we can also see that the largest gains in training time occur when using NBNE on a large but sparse network, such as Astro. This agrees with our theoretical expectations, since SDNE's time complexity grows quadratically with the number of nodes O(|V 2 |) and NBNE's grows with O(|V | · log(|V |) · b), which is close to linear on the number of nodes for large graphs. In this section, we extend the presented in Section 4, considering now the precision on the training and test sets. We present for the link prediction problem in TAB8 and for the node classification problem in TAB9 NBNE produces statistically similar to its baselines, in terms of Macro F1, on the node classification task using both PPI and Wikipedia datasets, while showing a statistically significant 13% (11.34x, 226 .52x) † average of 10 fold ‡ no statistical tests were run, due to the time necessary to run one fold 22.45% gain in the Blog dataset. Node classification in the PPI dataset had the smallest precision among all datasets, with only approximately 14%, but there are 49 classes in it and a classifier which always guessed the most common class would only get 2.95% precision. NBNE only shows a statistically worse in test precision for node classification on the Wikipedia dataset, losing to DeepWalk, but having an order of magnitude faster training time than DeepWalk and more than two orders of magnitude faster training time than Node2Vec. On all other experiments in either node classification or link prediction it presented either statistically better or similar to its baselines, while showing much faster training times. F AUTHOR NAME DISAMBIGUATION One of the hardest problems faced by current scholarly digital libraries is author name ambiguity BID10. This problem occurs when an author publishes works under distinct names or distinct authors publish works under similar names BID11. Automatic solutions, which are effective, efficient and practical in most situations, are still in need . In this section, we test our algorithm against the case where distinct authors publish works under similar names. Using these co-authorship networks, embeddings were obtained by training on the graphs with 20% of the papers from each ambiguous author removed. After the embeddings had already been learned for each author, the probability of each possible author-coauthors "sentence" was calculated as: s possible author = [v possible author, v coauthor 1, ..., v coauthor j] This probability is given by: DISPLAYFORM0 (log (p (v t+j |v t))) DISPLAYFORM1 where v 1 = author, which comes from the NBNE model itself. As a baseline, we used the typical solution that classifies the closest of the possible ambiguous authors as co-author for each of the test papers. If no path on the graph existed to any of the possible ambiguous authors, or if there was a tie between the distances to two or more of them, a random one was chosen between the possible ones. DeepWalk and Node2Vec were not used as baselines for this task due to the size of the 14 graphs analyzed here, most with more than 100,000 nodes and 500,000 edges, which would in a prohibitive training time. F.2 EXPERIMENTAL TAB0 presents the for the author name disambiguation task for each chosen author. This experiment was run using NBNE as an unsupervised algorithm with a fixed number of permutations n = 10, having no validation set. We also used sentences of size k = 5 and node embeddings of size d = 128.only required computing the probability of each possible author-coauthors "sentence" (p(s)), while the baseline had to dynamically get the distance between the papers' co-authors and the possible authors. It can be seen in TAB0 that for all but two authors the precision was higher when using the NBNE embeddings instead of the graph baseline, while for the other two precision score remained the same.

A faster method for generating node embeddings that employs a number of permutations over a node's immediate neighborhood as context to generate its representation.

represent the tldr for finding the corresponding scientific article!


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Orthogonal recurrent neural networks address the vanishing gradient problem by parameterizing the recurrent connections using an orthogonal matrix. This class of models is particularly effective to solve tasks that require the memorization of long sequences. We propose an alternative solution based on explicit memorization using linear autoencoders for sequences. We show how a recently proposed recurrent architecture, the Linear Memory Network, composed of a nonlinear feedforward layer and a separate linear recurrence, can be used to solve hard memorization tasks. We propose an initialization schema that sets the weights of a recurrent architecture to approximate a linear autoencoder of the input sequences, which can be found with a closed-form solution. The initialization schema can be easily adapted to any recurrent architecture. We argue that this approach is superior to a random orthogonal initialization due to the autoencoder, which allows the memorization of long sequences even before training. The empirical analysis show that our approach achieves competitive against alternative orthogonal models, and the LSTM, on sequential MNIST, permuted MNIST and TIMIT. Several sequential problems require the memorization of long sequences of patterns. As an example, a generative model for music should be able to memorize long sequences of notes and be able to repeat them, as it is typically done in musical pieces. RNNs and LSTMs struggle to solve even simple memorization tasks . Therefore, it is important to study alternative solutions to this problem. Orthogonal RNNs are a class of recurrent architectures that solve the vanishing gradient problem by constraining the recurrent connections to be an orthogonal or unitary matrix . They are particularly effective to solve long memorization tasks. In this paper, we address the problem of memorization with orthogonal RNNs and linear autoencoders. Our objective is to find a solution to the problem of memorization of long sequences. The memorization of long sequences with orthogonal models can require a large number of hidden units, increasing the hidden state size and the cost in time and space. If we assume that the input sequences in the training data lay in a low-dimensional manifold, as is typically believed for real-world data, then we can train an autoencoder with a small number of hidden units, sufficient to encode the entire sequence. If we restrict ourselves to the linear case, we can compute the optimal autoencoder with a closedform solution . This can be exploited to initialize recurrent architectures to approximate the linear autoencoder for the input sequences. In our experiments we use the RNN and the Linear Memory Network (LMN) . The LMN with the autoencoder initialization is a recurrent architecture equivalent to the Elman RNN but able to solve the vanishing gradient problem and memorize input sequences with a minimal hidden state. We test our approach on classic benchmarks for orthogonal RNNs, showing that our proposed approach behaves similarly to orthogonal architectures on pure memorization tasks, and even improving the performance on real-world datasets. Finally, we show that the model can also be used in situations where a strict orthogonal parameterization struggles , like the TIMIT benchmark . Work on orthogonal models typically focuses on the properties of an orthogonal parameterization at the backward step, to address the vanishing gradient. In our work instead, we focus on the forward step, by investigating the effect of an orthogonal parameterization against an autoencoder-based solution. For these reasons, we are not particularly interested in enforcing exact orthogonality constraints, but in the study of the effectiveness of an autoencoder-based memorization mechanism. Our proposed approach requires the memorization of the entire sequence within the hidden state activations. It is necessary to note that this approach can be quite inefficient whenever the underlying task does not require complete memorization of the input sequences. In this situation, the hidden state size necessary to encode the entire sequence could be much larger than the minimal hidden state size necessary to solve the problem. Therefore, it is fundamental to allow the model to diverge from the orthogonality or the autoencoder solution by only imposing soft orthogonality constraints. Nonetheless, we believe that even when complete full memorization is not necessary, the autoencoder initialization can help to speed up training convergence by allowing the model to remember long sequences, which can then be gradually forgotten during training if deemed uninformative. In summary, the main contributions of the paper are: • the proposal of a novel initialization schema designed for the explicit memorization of long sequences; • highlighting the connection between orthogonal models and linear autoencoders for sequences; • an empirical analysis that shows the effectiveness of our proposed initialization schema to solve tasks that require long memorization. The Linear Memory Network (LMN) is a recurrent neural network that computes a hidden state h t and a separate memory state m t. The hidden state h t is computed with a fully connected layer from the input and the previous state followed by an activation function. The memory state m t is computed with a linear transformation of the current hidden state and the previous memory state. The equations for the model update are the following: where W xh, W mh, W hm, W mm are the model's parameters and σ is a non-linear activation function (tanh in our experiments), and m t is the final output of the layer. The linear recurrence can be used to control the memorization properties of the model, as we will show in the subsequent sections. The linearity of the memory update can be exploited to guarantee a constant propagation of the gradient by constraining W mm to be an orthogonal matrix. This property allows the LMN to avoid the vanishing gradient problem. Notice that to guarantee the constant gradient propagation it is also necessary to truncate the gradient between m t−1 and h t by setting ∂h t ∂m t−1 = 0. This approach is similar to the truncated backpropagation proposed originally for the LSTM , which is necessary to guarantee that an LSTM without a forget gate propagates the gradients without vanishing effects. The modification to the training algorithm is trivial to implement with the current automatic differentiation packages, as done in . However, in this work, we focus on the effect of the orthogonality on the forward propagation and therefore we only use the full gradient in our experimental . The linearity of the memory update of the LMN might seem a major limitation of the architecture. However, it is easy to show that an RNN can always be rewritten as an equivalent LMN. Given an RNN with parameters V and U such that h rnn ), we can define an equivalent LMN such that m t = h t rnn ∀t by setting the parameters in the following way: Therefore, the linearity does not limit the expressivity of the LMN. Differences between the two architectures however become important during training. The linear update of the LMN without proper regularization can be affected by the exploding gradient. This problem is less frequent in RNNs due to the non-linearity (which often favors the vanishing gradient). In preliminary experiments, we found that training LMNs without any kind of regularization can lead to instability during the training. Fortunately, the gradient propagation can be controlled in the linear recurrence by tuning the spectral radius ρ of the recurrent matrix W mm, for example, by using an orthogonal parameterization. In our experiments we use soft orthogonality constraints by adding a regularization term λ W W − I 2 as in. Alternatively, it is possible to build a fading memory system by imposing ρ < 1 . Another possible cause of instability is the uncontrolled growth of the norm of the memory state m t. The LSTM solves this problem by using a forget gate to reset the memory cell activations. This is not an issue in RNNs with a bounded nonlinear activation. We overcome this problem by adding to the loss function a regularization term β 2 that penalizes the norm of the hidden activations as in. The linear recurrence of the LMN allows to exploit from the field of linear autoencoder for sequences (LAES) (; 2015;). In this section, we show how to initialize the LMN and RNN models using a linear autoencoder for sequences trained on the input samples. This initialization allows the ing model to encode the input sequences into a minimal hidden state. A linear autoencoder for sequences (LAES) is a linear model composed of a linear autoencoder, which is a linear dynamical system, and a linear decoder. The autoencoder takes as input a sequence x 1,..., x T and computes an internal state m t with the following equations: where A and B are the encoder parameters, C is the decoding matrix andx t andm t−1 are the reconstructions of the current input and the previous memory state. provides a closed-form solution for the optimal linear autoencoder. The corresponding decoder matrix C can be reconstructed from the autoencoder parameters as C = A B. By means of the linear autoencoder for sequences, we can construct a simple linear recurrent model which uses the autoencoder to encode the input sequences within a single vector, the memory state of the autoencoder. To predict the desired target, we can train a linear layer that takes as input the states of the autoencoder to predict the target. The equations for the model are as follows: where A and B are the parameters of the autoencoder trained to reconstruct the input sequences and W o are the parameters of the readout trained to the predict the target. Figure 1a shows a schematic view of the architecture. This initialization approximates the linear RNN. The quality of the approximation depends on the values of the hidden activations m t. With small values close to zero, the tanh activation function is approximately linear, and therefore we obtain a good approximation of the autoencoder. However, for large values the correspondence between the autoencoder activations and the initialized RNN degrades quickly since the tanh activation function enters into the nonlinear part and cannot represent values > 1 or < −1. In practice, we find that the correspondence between the linear autoencoder and the hidden states of the initialized RNN with a tanh activation function tends to degrade quickly due to the accumulation of the error through time. To solve this problem, we propose to adopt a similar initialization scheme to initialize the weights of an LMN using the linear RNN defined in Eq. 1 and 2. The LMN model is initialized as follows: where the parameters of the LMN are initialized to approximate the linear RNN: The initialization is only an approximation of the linear RNN but since the nonlinearity only affects the input transformation Ax t and not the recurrent part By t, the approximation is closer to the original linear RNN. Figures 1b and 1c show a schematic view of the architectures. Experimental in Section 5 provide a quantitative comparison between the two approaches. These show that the accuracy of the initialized LMN is the same as the linear RNN. To give an intuition on the operations enabled by the use of an autoencoder-based memorization mechanism, we discuss how it is possible to use the decoder to build an explicit model that is able to reconstruct the entire sequence, and to process the unrolled sequence with a feedforward layer that combines weights for each specific time delay. Let us assume to have the optimal autoencoder with parameters A and B trained on the input sequence x 1,..., x T. We further assume that the autoencoder has enough hidden units to have exactly zero reconstruction error. The output of the entire model can be computed with a linear layer as follows: where y t is the output vector at time t and W o the parameters of the output layer. Given the hidden state m t at time t of the autoencoder, we can reconstruct the input at time t − k using the decoder In fact, after k applications of the decoder we find: which shows how to reconstruct any input vector x τ, τ = 1,..., t − 1, from the autoencoder memory m t. The output layer can reconstruct the entire input sequence using the decoder. This can be made explicit with a decomposition of W o as: where W i is the matrix applied to the input at delay i. Using this decomposition, the output y t can be expressed as: This ing model requires explicit parameters for each time delay, with a total number of parameters that grows linearly with the length T of the sequence (or the maximum dependency length k < T), and it is expensive to train. However, using the autoencoder, these two operations are combined within a single matrix multiplication. Notice that in general W o will not be decomposed with this factorization. However, any model of this form can be potentialy learned by gradient descent (given a suitable loss function). Therefore it is in principle possible for an autoencoder-based model to reconstruct the entire sequence and process each element separately. The unrolled model shows the expressiveness of the autoencoder. By comparison, an orthogonal parameterization contains the activations of the entire sequence but it is not possible, in principle, to separate them and reconstruct the original input. Furthermore, since the encoding is not explicitly optimized, memorization with an orthogonal parameterization might become inefficient and be prone to require a larger hidden state size. In this section, we evaluate the performance of the orthogonal LMN and the proposed initialization scheme on synthetic tasks and real-world datasets: the copy task , digit classification with sequential and permuted MNIST , and framewise classification with TIMIT . These datasets are standard benchmarks for the assessment of orthogonal recurrent neural networks, and they offer the opportunity to evaluate the proposed approach in different settings. While for permuted MNIST orthogonal models reach state-of-the-art , and showed that orthogonality constraints can reduce the performance of the trained model on TIMIT. We also compare the LAES initialization for the RNN and the LMN, showing that the LMN is a better approximation in practical scenarios. Each model is trained with Adam , with a learning rate in {10 −3, 10 −4, 10 −5} chosen by selecting the best model on a separate validation set. A soft orthogonality constraint λ W W − I 2 is added to the cost function as in , with λ chosen {0, 10 −5, 10 −4, 10 −3, 10 −2}. The copy task is a synthetic benchmark often used to test the ability of a model to capture long-term dependencies. We use the same setup as in. The objective is the memorization of a sequence of S elements, which must be repeated sequentially after T timesteps. The input is a sequence of 2S + T timesteps where the first S elements are randomly generated from a set of K elements, followed by S − 1 blank elements, an output delimiter, and S blank elements. The target sequence contains S + T blank elements followed by the first S elements of the input sequence. To solve this task, the network must be able to memorize S elements into its hidden state and remember them for T timesteps. Each model is a single layer recurrent architectures with 100 hidden units and has been trained with 10 000 batches containing 64 samples each. The copy task can be easily solved with linear orthogonal models. showed a handcrafted solution for the problem. However, adding a nonlinearity makes the task difficult to solve even for orthogonal models . The LMN solves this limitation by separating the nonlinearity used to compute the hidden activations from the linear memory update. As shown in Table 1, the LMN can solve the task even using a saturating nonlinearity (tanh in our experiments). In more general terms, the LMN is able to combine nonlinear activations with the memorization properties of orthogonal linear models. Table 1 shows the on the copy task for T = 100 and T = 500, with S = 10, and K = 8. We compare the to a memoryless baseline that outputs the blank element for S + T timesteps and random elements for the last S elements. The LMN and the linear RNN solve the task perfectly in both cases. The RNN with a tanh nonlinearity fails to beat the baseline even with T = 100. The LSTM beats the baseline in both settings but does not solve the task optimally. These confirm our expectations that we can use the LMN architecture with a saturating nonlinearity even for tasks that require learning long-term dependencies. MNIST are two standard benchmarks to test the ability of recurrent neural networks to learn long-term dependencies. They consist of sequences extracted from the MNIST dataset by scanning each image one pixel at a time, in order (sequential MNIST) or with a random fixed permutation (permuted MNIST). We use these datasets to compare the LAES initialization to a random orthogonal initialization. A validation set is extracted by sampling 10 000 random images from the training set and separating them from the training data to use them to perform the model selection. These datasets provide an ideal setting to test a pure memorization approach like the LAES initialization since the output layer can use the final memory state m t to reconstruct the entire sequence of hidden activations. Table 2 shows the on the two MNNIST benchmarks. We compare the RNN and LMN with the LAES initialization and a random orthogonal initialization with several orthogonal models. We also show the of the linear RNN used to initialize the models with the LAES initialization. All the models have 128 hidden units and a single hidden layer. Each model has been trained for 100 epochs with a batch size of 64. The best are achieved by the AntisymmetricRNN , while the LMN with the LAES initialization obtains slightly lower , but still improves compared to the of the other orthogonal models. 1024 --94.0 93.7 KRU 512 96.6 96.4 94.7 94.5 LSTM 128 98.1 97.8 91.7 91.3 ASRNN 128 -98.0 -95.8 ASRNN(gated) The strength of the LAES initialization is the of the ability of the LAES to efficiently compress the entire sequence and the high performance that can be obtained on MNIST using a linear model 1 and the linear RNN. TIMIT is a speech recognition corpus . In our experiments we follow the setup of for the framewise classification problem . Differently from MNIST, this dataset is more difficult for orthogonal models than for non-orthogonal models.; have shown that orthogonality constraints can be detrimental on this dataset. Therefore, it is ideal to study the performance of the LMN and the LAES initialization in a more challenging setup. All the models have been trained for 50 epochs with a batch size of 1. We compare models with a single layer and a varying number of hidden units to keep the number of parameters approximately constant across the models (about 400k). Table 3 shows the of the experiments. We found that without any regularization the norm of the memory state m t tended to grow indefinitely, which degraded the final performance. We did not find the same problem on the MNIST dataset. A penalization on the memory state norm was sufficient to solve this issue. We selected λ by cross-validation in {0, 1, 10, 100}. It is important to note that, while on this benchmark the best is achieved by the LMN, the best model does not use the LAES initialization. The best configuration for the LMN use an orthogonal initialization but does not use soft orthogonality constraints since the best configuration is obtained when the hyperparameter for the soft orthogonality constraints is λ = 0. This confirms the other in the literature and shows that the architecture is general enough to model problems that do not require explicit memorization. In Section 3 we remarked that LAES initialization can incur in large errors due to the nonlinearity that affects the hidden state of the autoencoder. To verify if these problems happen in practical scenarios, we trained a linear autoencoder with 128 hidden units on sequences from sequential and permuted MNIST and initialized an LMN and an RNN with it. Table 4 shows the . The show that the error of the RNN initialization is large enough to cause a significant drop in the accuracy, from 85.5 of the linear RNN to 11.7 of the initialized model. Conversely, the initialized LMN reaches the same accuracy of the linear RNN. Therefore, it seems that the linear recurrence is the model is necessary to achieve a good approximation of the autoencoder. The on TIMIT show a similar trend. Orthogonal RNNs solve the vanishing gradient problem by parameterizing the recurrent connections with an orthogonal or unitary matrix. Some orthogonal models exploit a specific parameterization or factorization of the matrix (; ;) to guarantee the orthogonality. Other approaches constrain the parameters with soft or hard orthogonality constraints (; ; ; Lezcano-Casado & Martínez-). have shown that hard orthogonality constraints can hinder training speed and final performance. Linear autoencoders for sequences can be trained optimally with a closed-form solution . They have been used to pretrain RNNs . The LMN is a recurrent neural network with a separate linear connection. The memorization properties of untrained models are studied in the field of echo state echo networks , a recurrent model with untrained recurrent parameters. showed that untrained RNNs with a random weight initialization have a Markovian bias, which in the clustering of input with similar suffixes in the hidden state space. Tiňo & and study the short-term memory properties of linear and orthogonal memories. In this work, we studied the problem of building an autoencoder-based memorization mechanism. This system has the ability to encode and decode the input sequences to compute the desired target. Using for the linear autoencoder for sequences, we showed how to initialize a recurrent neural network by approximating a linear autoencoder of the input sequences. The architecture exploits a linear recurrence to obtain a better approximation of the autoencoder. The show that an autoencoder-based initialization can be effective for learning memorization tasks. In the future, we plan to extend this work by studying the effect of the autoencoder during training, possibly by enforcing the encoding of the input sequence even during the learning phase. Another possible avenue of research is the study of better optimization algorithms for the parameters of the linear component, where the linearity could be exploited to speed up the training process through dedicated learning algorithms.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show how to initialize recurrent architectures with the closed-form solution of a linear autoencoder for sequences. We show the advantages of this approach compared to orthogonal RNNs.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: This paper improves upon the line of research that formulates named entity recognition (NER) as a sequence-labeling problem. We use so-called black-box long short-term memory (LSTM) encoders to achieve state-of-the-art while providing insightful understanding of what the auto-regressive model learns with a parallel self-attention mechanism. Specifically, we decouple the sequence-labeling problem of NER into entity chunking, e.g., Barack_B Obama_E was_O elected_O, and entity typing, e.g., Barack_PERSON Obama_PERSON was_NONE elected_NONE, and analyze how the model learns to, or has difficulties in, capturing text patterns for each of the subtasks. The insights we gain then lead us to explore a more sophisticated deep cross-Bi-LSTM encoder, which proves better at capturing global interactions given both empirical and a theoretical justification. Named entity recognition is an important task in information extraction in which we seek to locate entity chunks in text and classify their entity types. Originally a structured prediction task, NER has since been formulated as a task of sequential token labeling, much like text chunking and part-ofspeech tagging. With the ability to compute representations of past and future context respectively for each token, bidirectional LSTM (Bi-LSTM) has proved a robust building block for sequencelabeling NER BID7 BID13 BID0. However, it has been predominantly used as a black box; research directed to understanding how the model learns to tackle the task is minimal. In this work, we decouple sequence-labeling NER into the entity chunking and entity typing subtasks, and seek insight into what patterns LSTM learns to capture or has difficulties capturing. We propose the use of a fast and effective parallel self-attention mechanism alongside Bi-LSTM. Unlike traditional attention mechanisms used for tasks such as machine translation BID12 and sentence classification BID2 BID11, our self-attentive Bi-LSTM uses the hidden state of each token as its own query vector and computes context vectors for all tokens in parallel. For both subtasks, we then find important global patterns that cross past and future context, and in particular discover the way multi-chunk entities are handled. Furthermore, we discover that the theoretical limitations of traditional Bi-LSTMs harms performance on the task, and hence propose using a cross construction of deep Bi-LSTMs. As a , with these cross structures, both selfattentive Bi-LSTM and cross-Bi-LSTM achieve new state-of-the-art on sequence-labeling NER.In Section 3, the normal Bi-LSTM-CNN model is formulated. Section 4 details the computation of the parallel self-attention mechanism. Section 5 presents the empirical and detailed analyses of the models, with a particular focus on patterns captured for {B, I, E} labels. Finally in Section 6, cross-Bi-LSTM-CNN is formulated and evaluated on a theoretical basis. Our contribution is threefold:• We provide insightful understanding of how a sequence-labeling model tackles NER and the difficulties it faces;• We propose using cross-Bi-LSTM-CNN for sequence-labeling NER with theoreticallygrounded improvements. Many have attempted tackling the NER task with LSTM-based sequence encoders BID7 BID13 BID0 BID9. Among these, the most similar to the proposed Bi-LSTM-CNN is the model proposed by BID0. In contrast to previous work, BID0 stack multiple layers of LSTM cells per direction, and also use a CNN to compute character-level word vectors alongside pre-trained word vectors. We largely follow their work in constructing the Bi-LSTM-CNN, including the selection of raw features, the CNN, and the multi-layer Bi-LSTM. The subtle difference is that they send the output of each direction through separate affine-softmax classifiers and then sum their probabilities, effectively forming an ensemble of forward and backward LSTM-CNNs. Another difference is that they focus on proposing a new representation of external lexicon features, which we do not make use of in this work. The modeling of global context for sequential-labeling NER has been accomplished using traditional models with intensive feature engineering and conditional random fields (CRF). BID17 build the Illinois NER tagger with feature-based perceptrons. In their analysis, the usefulness of Viterbi decoding is minimal, as class transition patterns only occur in small chunks and greedy decoding can handle them comparatively well. On the other hand, recent research on LSTM or CNNbased encoders report empirical improvements brought by CRF BID7 BID13 BID9 BID18, as it discourages illegal predictions by explicitly modeling class transition probabilities. In contrast, the cross structures of self-attention and crossBi-LSTM studied in this work provide for the direct capture of global patterns and extraction of better features to improve class observation likelihoods. Various attention mechanisms have been proposed and shown success in natural language tasks. They lighten the LSTM's burden of compressing all relevant information into a single hidden state by consulting past memory. For seq2seq models, attention has been used for current decoder hidden states BID12. For models computing sentence representations, trainable weights are used for self-attention BID2 BID11. In this work, we propose using a token-level parallel self-attention mechanism for sequential token-labeling and show that it enables the model to capture cross interactions between past and future contexts.3 BI-LSTM-CNN FOR SEQUENCE LABELING All models in our experiments use the same set of raw features: word embedding, word capitalization pattern type, character embedding, and character type. For character embedding, 25d vectors are randomly initialized and trained end-to-end with the model. Appended to these are 4d one-hot character-type features indicating whether a character is uppercase, lowercase, digit, or punctuation BID0. In addition, an unknown character vector and a padding character vector are also trained. We unify the word token length to 20 by truncation and padding. The ing 20-by-(25+4) feature map of each token are applied to a character-trigram CNN with 20 kernels per length 1 to 3 and max-over-time pooling to compute a 60d character-based word vector BID8 BID0 BID13.For word embedding, pre-trained 300d GloVe word vectors BID15 are used without further tuning. In addition, 4d one-hot word capitalization features indicate whether a word is uppercase, upper-initial, lowercase, or mixed-caps BID1 BID0.Throughout this paper, we use X to denote the n-by-d x matrix of raw sequence features, with n denoting the number of word tokens in a sentence and d x = 60 + 300 + 4. Given a sequence of input feature vectors x 1, x 2,..., x T ∈ R d1, an LSTM cell computes a sequence of hidden feature vectors h 1, h 2,..., h T ∈ R d2 by DISPLAYFORM0 are trainable weight matrices and biases, tanh denotes hyperbolic tangent, σ denotes sigmoid function, and denotes element-wise multiplication. Bidirectional LSTMs (Bi-LSTMs) are used to capture the future and the past for each time step. Following BID0, 4 distinct LSTM cells -two in each direction -are stacked to capture higher level representations: DISPLAYFORM1 where DISPLAYFORM2 H denote the ing feature matrices of the stacked application, and || denotes row-wise concatenation. In all our experiments, 100d LSTM cells are used, so H ∈ R n×d h and d h = 200. Finally, suppose there are d p token classes, the probability of each of which is given by the composition of affine and softmax transformations: DISPLAYFORM0 where H t is the t th row of H, W p ∈ R d h ×dp, b ∈ R dp are a trainable weight matrix and bias, and s ti and s tj are the i-th and j-th elements of s t.Following BID0, we use the 5 chunk labels O, S, B, I, and E to denote if a word token is {O}utside any entities, the {S}ole token of an entity, the {B}eginning token of a multitoken entity, {I}n the middle of a multi-token entity, or the {E}nding token of a multi-token entity. Hence when there are P types of named entities, the actual number of token classes d p = P × 4 + 1 for sequence labeling NER. We propose using a token-level self-attention mechanism (FIG0) that is computed after the autoregressive Bi-LSTM in Section 3.2. This has two benefits over traditional auto-regressive attention, which wraps stacked LSTM cells to look at past tokens at each time step for each direction of Bi-LSTM. First, it allows each token to look at both past and future sequences simultaneously with one combined hidden state of past and future, thus capturing cross interactions between the two contexts. And secondly, since all time steps run in parallel with matrix computations, it introduces little computation time overhead. Specifically, given the hidden features H of a whole sequence, we project each hidden state to different subspaces, depending on whether it is used as the {q}uery vector to consult other hidden states for each word token, the {k}ey vector to compute its dot-similarities with incoming queries, or the {v}alue vector to be weighted and actually convey information to the querying token. Moreover, as different aspects of a task can call for different attention, multiple "attentions" running in parallel are used, i.e., multi-head attention BID19.Formally, let m be the number of attention heads and d c be the subspace dimension. For each head i ∈ {1..m}, the attention weight matrix and context matrix are computed by DISPLAYFORM0 where W qi, W ki, W vi ∈ R d h ×dc are trainable projection matrices and σ performs softmax along the second dimension. Each row of the ing α 1, α 2,..., α m ∈ R n×n contains the attention weights of a token to its context, and each row of C 1, C 2,..., C m ∈ R n×dc is its context vector. DISPLAYFORM1 H, the computation of α i and C i models the cross interaction between past and future. Finally, for Bi-LSTM-CNN augmented with the attention mechanism, the hidden vector and context vectors of each token are considered together for classification: We conduct experiments on the challenging OntoNotes 5.0 English NER corpus BID6 BID16. OntoNotes is an ambitious project that collects large corpora from diverse sources and provides multi-layer annotations for joint research on constituency parsing, semantic role labeling, coreference resolution, and NER. The data sources include newswires, web, broadcast news, broadcast conversations, magazines, and telephone conversations. Some are transcriptions of talk shows and some are translated from Chinese or Arabic. Such diversity and noisiness requires that models are robust and able to capture a multitude of linguistic patterns. BID16, excluding the New Testament corpus as it contains no entity annotations. Despite this million-token corpus with over 100K annotated entities, previous work has struggled to reach state-of-the-art NER on the dataset. This is due partly to the fact that there are 18 types of entities to be classified. Eleven of these are classes of general names, with NORP including nationalities such as American, FAC including facilities such as The White House, and WORK OF ART including titles of books, songs, and so on. Moreover, various forms of values of the seven numerical classes must also be identified. DISPLAYFORM0 The hyperparameters of our models were given in Sections 3 and 4. When training the models, we minimized per-token cross-entropy loss with the Nadam optimizer BID3. In addition, we randomly dropped 35% hidden features (dropout) and upscaled the same amount during training. Following previous lines of work, we evaluated NER performance with the per-entity F1 score. The tokens for an entity were all to be classified correctly to count as a correct prediction; otherwise it was counted as either a false positive prediction or a false negative non-prediction. We stopped training when the validation F1 had not improved for 20 epochs. All models were initialized and trained 5 times; we report the mean precision, recall, and F1 scores (%) of the experiments. Validation scores are also reported for future research on this task. TAB1 shows the overall of our models against notable previous work. It can be seen that simple LSTM-based sequence encoders already beat the previous best without using external lexicons BID0, document-level context BID18, or constituency parsers BID10. Furthermore, with the proposed parallel self-attention mechanism (ATT), we achieve a new state-of-the-art (88.29 F1) with a clear margin over past systems. More importantly, the attention mechanism allows us to conduct insightful analyses in the following sections, yielding important understanding of how Bi-LSTM learns or has difficulty tackling the different sequence-labeling NER subtasks: entity chunking and entity typing. We decouple the entity chunking task from sequence-labeling NER. Specifically, for a sentence such as {Barack Obama moves out of the White House .}, the task is to correctly label each token as TAB3 shows the performance of different setups on validation data. We take the pre-trained models from TAB1 without re-training for this subtask. {O, S, B, I, E} are the chunk classes. The column of HC all lists the performance of the full Bi-LSTM-CNN+ATT model on each chunk class, where C all stands for C 1,..., C 5. Other columns list the performance of other setups compared to the full model. Columns H to C 5 are when the full model is deprived of all other information by zeroing all other vectors for the affine-softmax classification layer in testing time, except for those specified by the column header. NativeH is the native Bi-LSTM-CNN trained without attention. The figures shown in the table are the per-token recalls for each chunk class, which tells if a part of the model is responsible for signaling the whole model to predict the class. DISPLAYFORM0 Looking at the three columns on the left, the first thing we discover is that Bi-LSTM-CNN+ATT designates the task of predicting {I} to the attention mechanism. The model performance on tokens {I}n the middle of an entity significantly degrades (-28.18) in the absence of global context C all, when token hidden state H is left alone. On the other hand, without the information on the token itself, it is clear that the model strongly favors predicting I (-3.80) given its global context C all.Taking this one step further and zeroing out all other vectors except for each attention head, the roles of context for entity chunking become even clearer. C 2 and C 3 send strong signals (-36.45,-39.19) on entity chunk {E}nding to the model, plus weak signals (-60.56,-50.19) on entity chunk {I}nside, while C 4 sends a strong signal (-12.21) on entity chunk {B}eginning plus weak signals (-57.19) on {I}nside. When all these heads fire simultaneously, the model produces a strong signal to {I}.However, NativeH -Bi-LSTM-CNN trained without attention -underperforms in chunk labels {B} (-0.63), {I} (-0.41), {E} (-0.38) in comparison to HC all, the model trained with ATT. This suggests that entity chunking is indeed a crucial aspect in sequence-labeling NER, and that it is difficult for pure LSTM encoders to compress all necessary information in each hidden state to correctly label all the tokens of a multi-token entity. Aside from knowing that entity chunking is a crucial, challenging aspect in sequence-labeling NER for Bi-LSTM, one remaining question is how exactly the encoder is attempting to properly classify the {B}egin, {I}nside, and {E}nd of a multi-token entity. To shed light on this question, we visualize samples from validation data and discover consistent patterns in the attention weight heat maps across sentences and entities. FIG2 shows one of the samples, where the attention weights α 2, α 3, α 4 of a sentence containing the B White I house E are visualized. The full Bi-LSTM-CNN+ATT (HC all) classifies the tokens correctly, but when in the absence of the context vectors (H), the predictions become the B White S house E. For Bi-LSTM-CNN trained without attention at all (NativeH), the predictions are the O White S house O. Each row of the matrix shows the attention weight distribution for the diagonal token in bold font. We observe that α 2 and especially α 3 have a tendency to focus on the previous tokens: the diagonal shifted left. In contrast, α 4 tends to look at the immediate following tokens: the diagonal shifted right. By looking for previous tokens that belong to the same entity chunk and finding some, an attention head, via its context vector, can signal to the model that the token spoken of might be the {E}nding token or {I}nside token. The same is true for an attention head looking at next tokens, but this time signaling for {B}egin and {I}nside. This also dictates that both signals need to be weaker for {I} but stronger when combined. This behavior can be observed throughout the heat maps of α 2, α 3, α 4. In particular for the White house, C all predicts the B White I house O as Saturday is wrongly focused by α 4 for house. 27.27 -9.09 From TAB3, we already know that NativeH has some difficulties in handling multi-token entities, being more inclined to predict {S}ingle-token entities, and that HC all mitigates this problem by delegating work to C all, especially by relying on the latter to signal for {I}n tokens. The heat maps further tell the story of how the related labels {B, I, E} are handled collectively. In addition, this also suggests that modeling interactions between future and past contexts is crucial for sequencelabeling NER and motivates the use of a deep cross-Bi-LSTM encoder in Section 6. DISPLAYFORM0 When the entity chunking task is decoupled from sequence-labeling NER, the remaining entity typing task requires a model to label {Barack Obama moves out of the White House .} as {Barack PERSON Obama PERSON moves NONE out NONE of NONE the FAC White FAC House FAC . NONE}. TAB4 shows the entity classes for which HC all yields notably different performance (> 2%) from that of NativeH. Of particular interest is C 5's strong signal (27.27) for LAN (language) in comparison to the NativeH's struggles (-9.09) on this class without attention. Qualitatively, we study the two sentences shown in Figure 3, containing Dutch LAN into NONE English LAN and Chinese LAN and NONE English LAN. HC all classifies the tokens correctly, but both H and N ativeH wrongly predict Dutch NORP into NONE English LAN and Chinese NORP and NONE English LAN. Here NORP stands for nationality, meaning that both models without attention wrongly judge that Dutch and Chinese here refer to people from these countries. With attention, in Figure 3, we see that α 1 attends to Dutch and English at the same time for the two tokens and attends to Chinese and English at the same time for the other two. On the other hand, α 5 focuses on all possible LAN tokens, including a small mis-attention to Taiwanese in the second sentence, which is actually a NORP in this case. These attention weights signify that the model learns a pattern of cross interaction between entities: when two ambiguous entities of NORP, LAN occur together in the same context, the model predicts both as LAN. In Section 4.1, we briefly mentioned that the computation of attention weights α i and context features C i models the cross interaction between past and future. Mathematically, since H = − → H || ← − H, the computation of attention scores can be rewritten as DISPLAYFORM0 The un-shifted covariance matrix of the projected (− → H || ← − H) thus computes the interaction between past context and future context for each token, capturing cross-context patterns that the deep Bi-LSTM-CNN specified in Section 3 cannot. The consequence of this inability has been empirically shown in Section 5. Here, we further consider the following four simple phrases that form an XOR: {Key and Peele} WOA; {You and I} WOA; {Key and I}; {You and Peele} where WOA stands for WORK OF ART. The first two phrases are respectively a show title and a song title. The other two are not entities, where the last one actually occurs in an interview with Keegan-Michael Key. Suppose the phrases themselves are the only available context for the classification of and. Then the Bi-LSTM-CNN cannot capture good enough features to classify and correctly simultaneously for the four cases, even if they are the training data, no matter how many LSTM cells are stacked. The key is that given the same half-context of past or future, and is sometimes {WOA : I} but sometimes {NONE : O}. It is only when patterns that cross past and future are captured that the model is able to decide the correct label. Motivated by the limitation of the conventional Bi-LSTM-CNN for sequence labeling, we propose the use of Cross-Bi-LSTM-CNN by changing the deep structure in Section 3.2 to Note that when computing sentence embeddings for tasks such as sentence classification, both directions of a normal Bi-LSTM look at the whole sentence. However, when computing hidden node features for sequence labeling, each direction of a normal Bi-LSTM looks at only half of the sentence. Cross-Bi-LSTM remedies this problem by interleaving the hidden features between LSTM layers. The output of the first layers of both directions are sent to the second layers of both directions, allowing higher layers to capture interactions between past and future contexts for each token. Empirically, we experiment with cross construction 5 times and find it further improves the performance of Bi-LSTM-CNN from 87.56 (±0.07) to 88.09 (±0.16). In this paper, we have decoupled named entity recognition into entity chunking and entity typing and demonstrated how sequence-labeling models can learn to handle each of these two subtasks. By using a fast parallel self-attention mechanism, we have discovered how the beginning and ending of a multi-token entity is determined and how they are jointly correlated to locate the inside tokens. Further, through our quantitative and qualitative analyses for both chunking and typing, we have shown that it is crucial to capture global patterns that cross both sides of a token. We demonstrate the theoretical limitation of the conventional deep Bi-LSTM-CNN used in sequence labeling tasks. In addition to the interpretability of the proposed parallel self-attention, it is shown that it constitutes a way to correlate past and future contexts. We have also provided deep cross-Bi-LSTM-CNN as another way to extract global context features. With their respective cross structures, both selfattentive Bi-LSTM and cross-Bi-LSTM achieve new state-of-the-art on sequence-labeling NER.

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A fully connected architecture is used to produce word embeddings from character representations, outperforms traditional embeddings and provides insight into sparsity and dropout.