Question: Represent the scientific article to find a short TLDR

Question: Up until very recently, inspired by a mass of researches on adversarial examples for computer vision, there has been a growing interest in designing adversarial attacks for Natural Language Processing (NLP) tasks, followed by very few works of adversarial defenses for NLP. To our knowledge, there exists no defense method against the successful synonym substitution based attacks that aim to satisfy all the lexical, grammatical, semantic constraints and thus are hard to perceived by humans. We contribute to fill this gap and propose a novel adversarial defense method called Synonym Encoding Method (SEM), which inserts an encoder before the input layer of the model and then trains the model to eliminate adversarial perturbations. Extensive experiments demonstrate that SEM can efficiently defend current best synonym substitution based adversarial attacks with little decay on the accuracy for benign examples. To better evaluate SEM, we also design a strong attack method called Improved Genetic Algorithm (IGA) that adopts the genetic metaheuristic for synonym substitution based attacks. Compared with existing genetic based adversarial attack, IGA can achieve higher attack success rate while maintaining the transferability of the adversarial examples. Deep Neural Networks (DNNs) have made great success in various machine learning tasks, such as computer vision , and Natural Language Processing (NLP) (; ;). However, recent studies have discovered that DNNs are vulnerable to adversarial examples not only for computer vision tasks but also for NLP tasks , causing a serious threat to their safe applications. For instance, spammers can evade spam filtering system with adversarial examples of spam emails while preserving the intended meaning. In contrast to numerous methods proposed for adversarial attacks (; ;) and defenses (; ;) in computer vision, there are only a few list of works in the area of NLP, inspired by the works for images and emerging very recently in the past two years . This is mainly because existing perturbation-based methods for images cannot be directly applied to texts due to their discrete property in nature. Furthermore, if we want the perturbation to be barely perceptible by humans, it should satisfy the lexical, grammatical and semantic constraints in texts, making it even harder to generate the text adversarial examples. Current attacks in NLP can fall into four categories, namely modifying the characters of a word , adding or removing words , replacing words arbitrarily , and substituting words with synonyms . The first three categories are easy to be detected and defended by spell or syntax check . As synonym substitution aims to satisfy all the lexical, grammatical and semantic constraints, it is hard to be detected by automatic spell or syntax check as well as human investigation. To our knowledge, currently there is no defense method specifically designed against the synonym substitution based attacks. In this work, we postulate that the model generalization leads to the existence of adversarial examples: a generalization that is not strong enough causes the problem that there usually exists some neighbors x of a benign example x in the manifold with a different classification. Based on this hypothesis, we propose a novel defense mechanism called Synonym Encoding Method (SEM) that encodes all the synonyms to a unique code so as to force all the neighbors of x to have the same label of x. Specifically, we first cluster the synonyms according to the Euclidean Distance in the embedding space to construct the encoder. Then we insert the encoder before the input layer of the deep model without modifying its architecture, and train the model again to defend adversarial attacks. In this way, we can defend the synonym substitution based adversarial attacks efficiently in the context of text classification. Extensive experiments on three popular datasets demonstrate that the proposed SEM can effectively defend adversarial attacks, while maintaining the efficiency and achieving roughly the same accuracy on benign data as the original model does. To our knowledge, SEM is the first proposed method that can effectively defend the synonym substitution based adversarial attacks. Besides, to demonstrate the efficacy of SEM, we also propose a genetic based attack method, called Improved Genetic Algorithm (IGA), which is well-designed and more efficient as compared with the first proposed genetic based attack algorithm, GA . Experiments show that IGA can degrade the classification accuracy more significantly with lower word substitution rate than GA. At the same time IGA keeps the transferability of adversarial examples as GA does. Let W denote the word set containing all the legal words. Let x = {w 1, . . ., w i, . . ., w n} denote an input text, C the corpus that contains all the possible input texts, and Y ∈ N K the output space. The classifier f: C → Y takes an input x and predicts its label f (x), and let S m (x, y) denote the confidence value for the y-th category at the softmax layer. Let Syn(w, σ, k) represent the set of first k synonyms of w within distance σ, namely where w − w p is the p-norm distance evaluated on the corresponding embedding vectors. Suppose we have an ideal classifier c: C → Y that could always output the correct label for any input text x. For a subset of (train or test) texts T ⊆ C and a small constant, we could define the natural language adversarial examples as follows: where d(x − x adv) is a distance metric to evaluate the dissimilarity between the benign example x = {w 1, . . ., w i, . . ., w n} and the adversarial example x adv = {w 1, . . ., w i, . . ., w n}. It is usually defined as the p-norm distance: In this subsection, we provide a brief overview of three popular synonym substitution based adversarial attack methods. Greedy Search Algorithm (GSA). propose a greedy search algorithm to substitute words with their synonyms so as to maintain the semantic and syntactic similarity. GSA first constructs a synonym set W s for an input text x = {w 1, . . ., w i, . . ., w n}: Initially, let x adv = x. Then at each stage for x adv = {w 1, . . ., w i, . . ., w n}, GSA finds a wordŵ i ∈ W that satisfies the syntactic constraint and minimizes S m (x, y true) wherex = {w 1, . . ., w i−1,ŵ i, w i+1, . . ., w n}, and updates x adv =x. Such process iterates until x adv becomes an adversarial example or the word replacement rate reaches a threshold. Genetic Algorithm (GA). propose a population-based algorithm to replace words with their synonyms so as to generate semantically and syntactically similar adversarial examples. There are three operators in GA: • M utate(x): Randomly choose a word w i in text x that has not been updated and substitute w i with w i, one of its synonyms Syn(w i, σ, k) that does not violate the syntax constraint by the "Google 1 billion words language model" and minimize S m (x, y true) wherex = {w 1, . . ., w i−1, w i, w i+1, . . ., w n} and S m (x, y true) < S m (x, y true); • Sample(P): Randomly sample a text from population P with a probability proportional to 1 − S m (x i, y true) where x i ∈ P; • Crossover (For a text x, GA first generates an initial population P 0 of size m: Then at each iteration, GA generates the next generation of population through crossover and mutation operators: GA terminates when it finds an adversarial example or reaches the maximum number of iteration limit. Probability Weighted Word Saliency (PWWS). propose a new synonym substitution method called Probability Weighted Word Saliency (PWWS), which considers the word saliency as well as the classification confidence. Given a text x = {w 1, . . ., w i, . . ., w n}, PWWS first calculates the saliency of each word S(x, w i): where "unk" means the word is removed. Then PWWS calculates the maximum possible change in the classification confidence ed from substituting word w i with one of its synonyms: Then, PWWS sequentially checks the words in descending order of φ(S(x, w i), and substitutes the current word w i with its optimal synonym w * i: PWWS terminates when it finds an adversarial example x adv or it has replaced all the words in x. There exist very few works for text adversarial defenses. • In the character-level, propose to place a word recognition model in front of the downstream classifier to defend character-level adversarial attacks by combating adversarial spelling mistakes. • In the word level, for defenses on synonym substitution based attacks, only and incorporate the adversarial training strategy proposed in the image domain with their text attack methods respectively, and demonstrate that adversarial training can promote the model's robustness. However, there is no defense method specifically designed to defend the synonym substitution based adversarial attacks. We first introduce our motivation, then present the proposed text defense method, Synonym Encoding Method (SEM). Let X denote the input space, V (x) denote the -neighborhood of data point x ∈ X, where V (x) = {x ∈ X | x − x <}. As illustrated in Figure 1 (a), we postulate that the generalization of the model leads to the existence of adversarial examples. More generally, given a data point x ∈ X, ∃x ∈ V (x), f (x) = y true where x is an adversarial example of x. Ideally, to defend the adversarial attack, we need to train a classifier f which not only guarantees f (x) = y true, but also assures ∀x ∈ V (x), f (x) = y true. Thus, the most effective way is to add more labeled data to improve the adversarial robustness . Ideally, as illustrated in Figure 1 (b), if we have infinite labeled data, we can train a model f: ∀x ∈ V (x), f (x) = y true with high probability so that the model f is robust enough to adversarial examples. Practically, however, labeling data is very expensive and it is impossible to have infinite labeled data. Because it is impossible to have enough labeled data to train a robust model, as illustrated in Figure 1 (c), propose to construct a convex outer bound and guarantee that all data points in this bound share the same label. The goal is to train a model f: ∀x ∈ V, f (x) = f (x) = y true. Specifically, they propose a linear-programming (LP) based upper bound on the robust loss by adopting a linear relaxation of the ReLU activation and minimize this upper bound during the training. Then they bound the LP optimal value and calculate the elementwise bounds on the activation functions based on a backward pass through the network. Although their method does not need any extra data, it is hard to scale to realistically-sized networks due to its high complexity. In this work, as illustrated in Figure 1 (d), we propose a novel way to find a mapping m: X → X where ∀x ∈ V (x), m(x) = x. In this way, we force the classification to be more smooth and we do not need any extra data to train the model or modify the architecture of the model. All we need to do is to insert the mapping before the input layer and train the model on the original training set. Now the problem turns into how to locate the neighbors of data point x. For image tasks, it is hard to find all images in the neighborhood of x in the input space, and there could be infinite number of neighbors. For NLP tasks, however, utilizing the property that words in sentences are discrete tokens, we can easily find almost all neighbors of an input text. Based on this insight, we propose a new method called Synonym Encoding Method to locate the neighbors of an input x. We assume that the closer the meaning of two sentences is, the closer their distance is in the input space. Thus, we can suppose that the neighbors of x are its synonymous sentences. To find the synonymous sentence, we can substitute words in the sentence with their synonyms. To construct the mapping m, all we need to do is to cluster the synonyms and allocate a unique token for each cluster, which we call the Synonym Encoding Method (SEM). The details are in Algorithm 1. else 8: end if 10: for each word w j in Syn(w i, σ, k) do The current synonym substitution based text adversarial attacks (; ;) have a constraint that they only substitute words at the same position once or replace words with the first k synonyms of the word in the original input x. This constraint can lead to local minimum for adversarial examples, and it is hard to choose a suitable k as different words may have different number of synonyms. To address this issue, we propose an Improved Genetic Algorithm (IGA), which allows to substitute words in the same position more than once based on the current text x. In this way, IGA can traverse all synonyms of a word no matter what value k is. Meanwhile, we can avoid local minimum to some extent as we allow the substitution of the word by the original word in the current position. In order to guarantee that the substituted word is still a synonym of the original word, each word in the same position can be replaced at most λ times. Differs to the first genetic based text attack algorithm of , we change the structure of the algorithm, including the operators for crossover and mutation. For more details of IGA, see Appendix A. We evaluate SEM with four attacks, GSA , GA , PWWS and our IGA, on three popular datasets involving three neural network classification models. The demonstrate that SEM can significantly improve the robustness of neural networks and IGA can achieve better attack performance as compared with existing attacks. And we further provide discussion on the hyper-parameter of SEM in Appendix B. We first provide an overview of the datasets and classification models used in the experiments. Datasets. In order to evaluate the efficacy of SEM, we choose three popular datasets: IMDB, AG's News, and Yahoo! Answers. IMDB is a large dataset for binary sentiment classification, containing 25, 000 highly polarized movie reviews for training and 25, 000 for testing. AG's News consists news article pertaining four classes: World, Sports, Business and Sci/Tech. Each class contains 30, 000 training examples and 1, 900 testing examples. Yahoo! Answers is a topic classification dataset from the "Yahoo! Answers Comprehensive Questions and Answers" version 1.0 dataset with 10 categories, such as Society & Culture, Science & Mathematics, etc. Each class contains 140,000 training samples and 5,000 testing samples. Models. To better evaluate our method, we adopt several state-of-the-art models for text classification, including Convolution Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). The embedding dimension for all models are 300. We replicate the CNN's architecture from , which contains three convolutional layers with filter size of 3, 4, and 5 respectively, a max-pooling layer and a fully-connected layer. LSTM consists of three LSTM layers where each layer has 300 LSTM units and a fully-connected layer . Bi-LSTM contains a bi-directional LSTM layer whose forward and reverse have 300 LSTM units respectively and a fully-connected layer. Baselines. We take the method of adversarial training as our baseline. However, due to the low efficiency of text adversarial attacks, we cannot implement adversarial training as it is in the image domain. In the experiments, we adopt PWWS, which is quicker than GA and IGA, to generate 10% adversarial examples of the training set, and re-train the model incorporating adversarial examples with the training data. To evaluate the efficiency of the SEM method, we randomly sample 200 correctly classified examples on different models from each dataset and use the above attack methods to generate adversarial examples with or without defense. The more effective the defense method is, the smaller the classification accuracy of the model drops. Table 1 shows the efficacy of various attack and defense methods. For each network model, we look at each row to find the best defense under the setting of no attack, or GSA, PWWS, GA, and IGA attacks: • Under the setting of no attack, adversarial training (AT) could improve the classification accuracy of the models on all datasets, as adversarial training (AT) is also the way to augment the training data. Our defense method SEM reaches an accuracy closing to normal training (NT). • Under the four attacks, however, the classification accuracy with normal training (NT) and adversarial training (AT) drops significantly. For normal training (NT), the accuracy degrades more than 75%, 42% and 40% on the three datasets respectively. And adversarial training (AT) cannot defend these attacks effectively, especially for PWWS and IGA on IMDB and Yahoo! Answers, where AT only improves the accuracy a little (smaller than 5%). By contrast, SEM can remarkably improve the robustness of the deep models for all the four attacks. In the image domain, the transferability of adversarial attack refers to its ability to decrease the accuracy of models using adversarial examples generated based on other models . find that the adversarial examples in NLP also exhibite a good transferability. Therefore, a good defense method not only could defend the adversarial attacks but also resists the transferability of adversarial examples. To evaluate the ability of preventing the transferability of adversarial examples, we generate adversarial examples on each model under normal training, and test them on other models with or without defense on Yahoo! Answers. The are shown in Table 2. Almost on all models with adversarial examples generated by other models, SEM could yield the highest classification accuracy. For text attacks, we compare the proposed IGA with GA from various aspects, including attack efficacy, transferability and human evaluation. Attack Efficacy. As shown in Table 1, looking at each column, we see that under normal training (NT) and adversarial training (AT), IGA can always achieve the lowest classification accuracy, which corresponds to the highest attack success rate, on all models and datasets among the four attacks. Under the third column of SEM defense, though IGA may not be the best among all attacks, IGA always outperforms GA. Besides, as depicted in Table 3, IGA can yield lower word substitution rate than GA on most models. Note that for SEM, GA can yield lower word substitution rate, because GA may not replace the word as most words cannot bring any benefit for the first replacement. This indicates that GA stops at local minimum while IGA continues to substitute words and gain a lower classification accuracy, as demonstrated in Table 1. Transferability. As shown in Table 2, the adversarial examples generated by IGA maintain roughly the same transferability as those generated by GA. For instance, if we generate adversarial examples on Word-CNN (column 2, NT), GA can achieve better transferability on LSTM with NT (column 5) while IGA can achieve better transferability on LSTM with AT and SEM (column 6, 7). Human Evaluation. To further verify that the perturbations in the adversarial examples generayed by IGA are hard for humans to perceive, we also perform a human evaluation on IMDB with 35 volunteers. We first randomly choose 100 benign examples that can be classified correctly and generate adversarial examples by GA and IGA on the three models so that we have a total of 700 examples. Then we randomly split them into 7 groups where each group contains 100 examples. We ask every five volunteers to classify one group independently. The accuracy of human evaluation on benign examples is 93.7%. As shown in Figure 2, the classification accuracy of human on adversarial examples generated by IGA is slightly higher than those generated by GA, and is slightly closer to the accuracy of human on benign examples. Summary. IGA can achieve the highest attack success rate when compared with previous synonyms substitution based adversarial attacks and yield lower word replacement rate than GA. Besides, the adversarial examples generated by IGA maintains the same transferability as GA does and are a little bit harder for humans to distinguish. Several generated adversarial examples by GA and IGA are listed in Appendix C. Synonym substitution based adversarial attacks are currently the best text attack methods, as they are hard to be checked by automatic spell or syntax check as well as human investigation. In this work, we propose a novel defense method called Synonym Encoding Method (SEM), which encodes the synonyms of each word to defend adversarial attacks for text classification task. Extensive experiments show that SEM can defend adversarial attacks efficiently and degrade the transferability of adversarial examples, at the same time SEM maintains the classification accuracy on benign data. To our knowledge, this is the first and efficient text defense method in word level for state-of-the-art synonym substitution based attacks. In addition, we propose a new text attack method called Improved Genetic Attack (IGA), which in most cases can achieve much higher attack success rate as compared with existing attacks, at the same time IGA could maintain the transferability of adversarial examples. Here we introduce our Improved Genetic Algorithm (IGA) in details and show how IGA differs from the first proposed generic attack method, GA . Regard a text as a chromosome, there are two operators in IGA: • • M utate(x, w i): For a text x = {w 1, . . ., w i−1, w i, w i+1, . . ., w n} and a position i, replace w i withŵ i whereŵ i ∈ Syn(w, σ, k) to get a new textx = {w 1, . . ., w i−1,ŵ i, w i+1, . . ., w n} that minimizes S m (x, y true). The details of IGA are described in Algorithm 2. Algorithm 2 The Improved Genetic Algorithm Input: x: input text, y true: true label for x, M: maximum number of iterations Output: x adv: adversarial example 1: for each word w i ∈ x do 2: if f (x adv) = y true then Randomly sample parent 1, parent 2 from P g−1: Randomly choose a word w in child 14: Compared with GA, IGA has the following differences: • Initialization: GA initializes the first population randomly, while IGA initializes the first population by replacing each word by its optimal synonym, so our population is more diversified. • M utation: Different from GA, IGA allows to replace the word that has been replaced before so that we can avoid local minimum. • Crossover: To better simulate the reproduction and biological crossover, we randomly cut the text from two parents and concat two fragments into a new text rather than randomly choose a word of each position from the two parents. The selection of the next generation is similar to GA, greedily choose one offspring, and then generate other offsprings by M utate(Crossover(·, ·)) on two randomly chosen parents. But as M utate and Crossover are different, IGA has very different offsprings. To explore how hyper-parameter of SEM influences the efficacy, we try different ranging from 0 to 1.2 for three models on IMDB with or without adversarial attacks. The are illustrated in Figure 3. On benign data, as shown in Figure 3(a), the classification accuracy of the models decreases a little when increases. Because a bigger indicates that we need less words to train the model, which could degrade the efficacy of the models. Nevertheless, the classification accuracy does not decrease much as SEM could maintain the semantic invariance of the original text after encoding. Then we show the defense efficacy of SEM on the three models when changing the value of, as shown in Figure 3 (b)-(d). When = 0 where SEM could not take any impact, we see that the accuracy is the lowest under all attacks. When increases, SEM starts to defend the attacks, the accuracy increases rapidly and reach the peak when = 0.5. Then the accuracy decays slowly if we continue to increase. Thus, we choose = 0.5 to be a good trade-off on the accuracy of benign examples and adversarial examples. C ADVERSARIAL EXAMPLES GENERATED BY GA AND IGA To show the generated adversarial examples, we randomly pick some benign examples from IMDB and generate adversarial examples by GA and IGA respectively on several models. The examples are shown in Table 5 to Table 6. We see that IGA substitutes less words than GA on these models under normal training. I am sorry but this is the worst film I have ever seen in my life. I cannot believe that after making the first one in the series, they were able to get a budget to make another. This is the least scary film I have ever watched and laughed all the way through to the end. This is a unique masterpiece made by the best director ever lived in the ussr. He knows the art of film making and can use it very well. If you find this movie, buy or copy it! 50.6 0 I cared this film which I thought was well written and acted, there was plenty of humour and a igniting storyline, a tepid and enjoyable experience with an emotional ending. GA 92.7 1 I am sorry but this is the harshest film I have ever seen in my life. I cannot believe that after making the first one in the series, they were able to get a budget to make another. This is the least scary film I have ever watched and laughed all the way through to the end. This is a sole masterpiece made by the nicest director permanently lived in the ussr. He knows the art of film making and can use it much well. If you find this movie, buy or copy it! 88.3 0 I enjoyed this film which I think was well written and acted, there was plenty of humour and a causing storyline, a lukewarm and enjoyable experience with an emotional ending. IGA 70.8 1 I am sorry but this is the hardest film I have ever seen in my life. I cannot believe that after making the first one in the series, they were able to get a budget to make another. This is the least scary film I have ever watched and laughed all the way through to the end. This is a sole masterpiece made by the best director permanently lived in the ussr. He knows the art of film making and can use it very well. If you find this movie, buy or copy it! I am sorry but this is the worst film I have ever seen in my life. I cannot believe that after making the first one in the series, they were able to get a budget to make another. This is the least scary film I have ever watched and laughed all the way through to the end. This is a unique masterpiece made by the best director ever lived in the ussr. He knows the art of film making and can use it very well. If you find this movie, buy or copy it! 88.2 0 I enjoyed this film which I thought was well written and proceeded, there was plenty of humorous and a igniting storyline, a tepid and enjoyable experience with an emotional terminate. GA 99.9 1 I am sorry but this is the hardest film I have ever seen in my life. I cannot believe that after making the first one in the series they were able to get a budget to make another. This is the least terrifying film I have ever watched and laughed all the way through to the end. This is a unique masterpiece made by the best superintendent ever lived in the ussr. He knows the art of film making and can use it supremely alright. If you find this movie, buy or copy it! 72.1 0 I enjoyed this film which I thought was well written and acted, there was plenty of humour and a provoking storyline, a lukewarm and agreeable experience with an emotional ending. IGA 99.8 1 I am sorry but this is the hardest film I have ever seen in my life. I cannot believe that after making the first one in the series, they were able to get a budget to make another. This is the least scary film I have ever watched and laughed all the way through to the end. This is a sole masterpiece made by the best director ever lived in the ussr. He knows the art of film making and can use it very well. If you find this movie, buy or copy it! Table 6: The adversarial examples generated by GA and IGA on IMDB using Bi-LSTM model. 99.6 1 I enjoyed this film which I thought was well written and acted, there was plenty of humour and a provoking storyline, a warm and enjoyable experience with an emotional ending. Original 97.0 0 I am sorry but this is the worst film I have ever seen in my life. I cannot believe that after making the first one in the series, they were able to get a budget to make another. This is the least scary film I have ever watched and laughed all the way through to the end. This is a unique masterpiece made by the best director ever lived in the ussr. He knows the art of film making and can use it very well. If you find this movie, buy or copy it! 98.2 0 I enjoyed this film which I thought was well written and proceeded, there was plenty of humorous and a igniting storyline, a tepid and enjoyable experiment with an emotional terminate. GA 78.8 1 I am sorry but this is the hardest film I have ever seen in my life. I cannot believe that after making the first one in the series, they were able to get a budget to make another. This is the least scary film I have ever watched and laughed all the way through to the end. This is a unique masterpiece made by the best superintendent ever lived in the ussr. He knows the art of film making and can use it supremely alright. If you find this movie buy or copy it! 81.2 0 I enjoyed this film which I thought was alright written and acted, there was plenty of humour and a arousing storyline, a lukewarm and enjoyable experiment with an emotional ending. IGA 78.8 1 I am sorry but this is the hardest film I have ever seen in my life. I cannot believe that after making the first one in the series they were able to get a budget to make another. This is the least scary film I have ever watched and laughed all the way through to the end. This is a sole masterpiece made by the best director ever lived in the ussr. He knows the art of film making and can use it very alright. If you find this movie buy or copy it!

The first text adversarial defense method in word level, and the improved generic based attack method against synonyms substitution based attacks.

represent this text to find a much much longer article about it with say hundreds-thousands of words

For example, 'Neural net for graph-based semi-supervised learning; revisits the classics and propagates *labels* rather than feature representations' should be close to 'Graph networks have recently attracted considerable interest, and in particular in the context of semi-supervised learning. These methods typically work by generating node representations that are propagated throughout a given weighted graph. Here we argue that for semi-supervised learning, it is more natural to consider propagating labels in the graph instead. Towards this end, we propose a differentiable neural version of the classic Label Propagation (LP) algorithm. This formulation can be used for learning edge weights, unlike other methods where weights are set heuristically. Starting from a layer implementing a single iteration of LP, we proceed by adding several important non-linear steps that significantly enhance the label-propagating mechanism. Experiments in two distinct settings demonstrate the utility of our approach. We study the problem of graph-based semi-supervised learning (SSL), where the goal is to correctly label all nodes of a graph, of which only a few are labeled. Methods for this problem are often based on assumptions regarding the relation between the graph and the predicted labels. One such assumption is smoothness, which states that adjacent nodes are likely to have similar labels. Smoothness can be encouraged by optimizing an objective where a loss term L over the labeled nodes is augmented with a quadratic penalty over edges: Here, y are the true labels, f are "soft" label predictions, S is the set of labeled nodes, and w are non-negative edge weights. The quadratic term in Eq. is often referred to as Laplacian Regularization since (for directed graphs) it can equivalently be expressed using the graph Laplacian BID5. Many early methods for SSL have adopted the general form of Eq. BID51 BID50 BID4 BID6 BID0 BID42 BID47. Algorithms such as the seminal Label Propagation BID51 are simple, efficient, and theoretically grounded but are limited in two important ways. First, predictions are parameterized either naïvely or not at all. Second, edge weights are assumed to be given as input, and in practice are often set heuristically. Recent deep learning methods address the first point by offering intricate predictive models that are trained discriminatively BID47 BID38 BID48 BID28 BID20 BID21 BID34. Nonetheless, many of them still require w as input, which may be surprising given the large body of work highlighting the importance of good weights BID51 BID24 BID46 BID4 BID25. While some methods consider some form of weight learning BID45 BID35, to some extent they have drifted away from the original quadratic criterion. Other works address the second point by proposing disciplined ways for learning w. However, these either assume specific simple parameterizations BID49 BID25, or altogether consider weights disjointly from predictions BID46 BID32.Our goal in this paper is to simultaneously addresses both issues. We propose a framework that, given a graph, jointly learns both a parametric predictive model and the edge weights. To do this, we begin by revisiting the Label Propagation (LP), and casting it as a differentiable neural network. Each layer in the network corresponds to a single iterative update, making a forward pass equivalent to a full run of the algorithm. Since the network is differentiable, we can then optimize the weights of the LP solution using gradient descent. As we show, this can be done efficiently with a suitable loss function. The key modeling point in our work is that labeled information is used as input to both the loss and the network. In contrast to most current methods, our network's hidden layers directly propagate labeling information, rather than node or feature representations. Each layer is therefore a self-map over the probability simplex; special care is therefore needed when introducing non-linearities. To this end, we introduce two novel architectural components that are explicitly designed to operate on distributions. The first is an information-gated attention mechanism, where attention is directed based on the informativeness and similarity of neighboring nodes' states. The second is a novel "bifurcation" operator that dynamically controls label convergence, and acts as a balancing factor to the model's depth. Our main guideline in designing our model was to tailor it to the semi-supervised setting. The is a slim model having relatively few parameters and only one model-specific hyper-parameter (depth), making it suitable for tasks where only few labeled nodes are available. The final network provides a powerful generalization of the original propagation algorithm that can be trained efficiently. Experiments on benchmark datasets in two distinct learning settings show that our model compares favorably against strong baselines. Many SSL methods are based on Eq. or on similar quadratic forms. These differ in their assumed input, the optimization objective, and the parametric form of predictions. Classic methods such as LP BID51 assume no parametric form for predictions, and require edge weights as inputs. When node features are available, weights are often set heuristically based on some similarity measure (e.g., w ij = exp x i − x j 2 2 /σ 2). LP constrains predictions on S to agree with their true labels. Other propagation methods relax this assumption BID4 BID6, add regularization terms BID0, or use other Laplacian forms BID50.Some methods aim to learn edge weights, but do not directly optimize for accuracy. Instead, they either model the relations between the graph and features BID46 BID32 or simply require f as input BID14 BID23 BID17. Methods that focus on accuracy are often constrained to specific parameterizations or assumptions BID25. BID49 optimize the leave-one-out loss (as we do), but require a series of costly matrix inversions. Several recent works in deep learning have been focused on graph inputs in general BID2 and specifically for inductive SSL. The main idea behind these methods is to utilize a weighted graph to create meaningful vector representations of nodes, which are then fed into a classifier. Methods are typically designed for one of two settings: when the input includes only a graph, and when node features are available. When the input includes only a graph (and no features), node representations are generated using embedding techniques. BID38 use a SkipGram model over random walks on the graph, which are used to define context. BID20 further this idea by introducing expressive parameterized random walks, while BID43 focus on optimizing similarities between pairs of node embeddings. Various methods have been proposed to utilize node features, when available. Spectral methods, stemming from a CNN formulation for graphs BID11, include different approximations of spectral graph convolutions BID16 BID28 adaptive convolution filters BID34, or attention mechanisms BID45 Embedding approaches have been suggesting for handling bag-of-words representations BID48 and general node attributes such as text or continuous features BID18. Many of the above methods can be thought of as propagating features over the graph in various forms. Our method, in contrast, propagates labels. The main advantage label propagation is that labeled information is used not only to penalize predictions (in the loss), but also to generate predictions. We begin by describing the learning setup and introducing notation. The input includes a (possibly directed) graph G = (V, E), for which a subset of nodes S ⊂ V are labeled by y S = {y i} i∈S with y i ∈ {1, . . ., C}. We refer to S as the "seed" set, and denote the unlabeled nodes by U = V \ S, and the set of i's (incoming) neighbors by N i = {j : (j, i) ∈ E}. We use n = |V |, m = |E|, = |S|, and u = |U | so that n = + u. In a typical task, we expect to be much smaller than n. We focus on the transductive setting where the goal is to predict the labels of all i ∈ U. Most methods (as well as ours) output "soft" labels f i ∈ ∆ C, where ∆ C is the C-dimensional probability simplex. For convenience we treat "hard" labels y i as one-hot vectors in ∆ C. All predictions are encoded as a matrix f with entries f ic = P[y i = c]. For any matrix M, we will use M A to denote the sub-matrix with rows corresponding to A. Under this notation, given G, S, y S, and possibly x, our goal is to predict soft labels f U that match y U.In some cases, the input may also include features for all nodes x = {x i} i∈V. Importantly, however, we do not assume the input includes edge weights w = {w e} e∈E, nor do we construct these from x. We denote by W the weighted adjacency matrix of w, and useW andw for the respective (row)-normalized weights. Many semi-supervised methods are based on the notion that predictions should be smooth across edges. A popular way to encourage such smoothness is to optimize a (weighted) quadratic objective. Intuitively, the objective encourages the predictions of all adjacent nodes to be similar. There are many variations on this idea; here we adopt the formulation of BID51 where predictions are set to minimize a quadratic term subject to an agreement constraint on the labeled nodes: DISPLAYFORM0 In typical applications, w is assumed to be given as input. In contrast, our goal here is to learn them in a discriminative manner. A naïve approach would be to directly minimize the empirical loss. For a loss function L, regularization term R, and regularization constant λ, the objective would be: DISPLAYFORM1 While appealing, this approach introduces two main difficulties. First, f * is in itself the solution to an optimization problem (Eq.), and so optimizing Eq. FORMULA1 is not straightforward. Second, the constraints in Eq. ensure that fIn what follows, we describe how to overcome these issues. We begin by showing that a simple algorithm for approximating f * can be cast as a deep neural network. Under this view, the weights (as well as the algorithm itself) can be parametrized and optimized using gradient descent. We then propose a loss function suited to SSL, and show how the above network can be trained efficiently with it. Recall that we would like to learn f * (w; S). When w is symmetric, the objective in Eq. FORMULA10 is convex and has a closed form solution. This solution, however, requires the inversion of a large matrix, which can be costly, does not preserve sparsity, and is non-trivial to optimize. The LP algorithm BID51 circumvents this issue by approximating f * using simple iterative averaging updates. Let f (t) be the set of soft labels at iteration t andw ij = w ij / k w ik, then for the following recursive relation: DISPLAYFORM0 it holds that lim t→∞ f (t) = f * for any initial f BID51. In practice, the iterative algorithm is run up to some iteration T, and predictions are given using f (T). This dynamic process can be thought of as labels propagating over the graph from labeled to unlabeled nodes over time. Motivated by the above, the idea behind our method is to directly learn weights for f (T), rather than for f *. In other words, instead of optimizing the quadratic solution, our goal is to learn weights under which LP preforms well. This is achieved by first designing a neural architecture whose layers correspond to an "unrolling" of the iterative updates in Eq., which we describe next. The main building block of our model is the basic label-propagation layer, which takes in two main inputs: a set of (predicted) soft labels h = {h i} n i=1 for all nodes, and the set of true labels y A for some A ⊆ S. For clarity we use A = S throughout this section. As output, the layer produces a new set of soft labels h = {h i} n i=1 for all nodes. Note that both h i and h i are in ∆ C. The layer's functional form borrows from the LP update rule in Eq. where unlabeled nodes are assigned the weighted-average values of their neighbors, and labeled nodes are fixed to their true labels. For a given w, the output is: DISPLAYFORM0 whereW is the row-normalized matrix of w. A basic network is obtained by composing T identical layers: DISPLAYFORM1 where the model's parameters w are shared across layers, and the depth T is the model's only hyper-parameter. The input layer h is initialized to y i for each i ∈ S and to some prior ρ i (e.g., uniform) for each i ∈ U. Since each layer h (t) acts as a single iterative update, a forward pass unrolls the full algorithm, and hence H can be thought of as a parametrized and differentiable form of the LP algorithm. In practice, rather than directly parameterizing H by w, it may be useful to use more sophisticated forms of parameterization. We will denote such general networks by H(θ), where θ are learned parameters. As a first step, given edge features {φ e} e∈E, we can further parametrize w using linear scores s ij and normalizing with softmax: DISPLAYFORM2 where θ φ ∈ R d are learned parameters. We propose using three types of features (detailed in Appendix B): Node-feature similarities: when available, node features can be used to define edge features by incorporating various similarity measures such as cosine similarity (φ ij = x i x j / x i x j) and Gaussian similarity DISPLAYFORM3 DISPLAYFORM4 where each similarity measure induces a distinct feature. Graph measures: these include graph properties such as node attributes (e.g., source-node degree), edge centrality measures (e.g., edge betweenness), path-ensemble features (e.g., Katz distance), and graph-partitions (e.g., k-cores). These allow generalization across nodes based on local and global edge properties. Seed relations: relations between the edge (i, j) and nodes in S, such the minimal (unweighted) distance to i from some s ∈ S. Since closer nodes are more likely to be labeled correctly, these features are used to quantify the reliability of nodes as sources of information, and can be class-specific. The label propagation layers in H pass distributions rather than node feature representations. It is important to take this into account when adding non-linearities. We therefore introduce two novel components that are explicitly designed to handle distributions, and can be used to generalize the basic layer in Eq. The general layer (illustrated in FIG1) replaces weights and inputs with functions of the previous layer's output: DISPLAYFORM0 whereÃ(·) is a normalized weight matrix (replacingW), µ(·) is a soft-label matrix (replacing h (t) ), and θ α and θ τ are corresponding learned parameters. The edge-weight functionÃ offers an information-gated attention mechanism that dynamically allocates weights according to the "states" of a node and its neighbors. The labeling function µ is a time-dependent bifurcation mechanism which controls the rate of label convergence. We next describe our choice ofÃ and µ in detail. The LP update (Eq.) uses fixed weights w. The importance of a neighbor j is hence predetermined, and is the same regardless of, for instance, whether h j is close to some y, or close to uniform. Here we propose to relax this constraint and allow weights to change over time. Thinking of h (t) i as the state of i at time t, we replace w ij with dynamic weights a (t) ij that depend on the states of i and j through an attention mechanism α: DISPLAYFORM0 where θ α are the attention parameters. Ã in Eq. FORMULA8 is the corresponding row-normalized weight matrix. determined by e and d (boxed bars), which are computed using h (t) and θ. Here θ directs attention at the informative and similar neighbor (thick arrow), and the update amplifies the value of the correct label. (Right) The bifurcation mechanism for C = 3 and various τ. Arrows map each h ∈ ∆ C to bif(h) ∈ ∆ C. DISPLAYFORM1 When designing α, one should take into account the nature of its inputs. Since both h i and h j are label distributions, we have found it useful to let α depend on information theoretic measures and relations. We use negative entropy e to quantify the certainty of a label, and negative KL-divergence d to measure cross-label similarity. Both are parameterized by respective class-dependent weights θ e c and θ d c, which are learned: DISPLAYFORM2 where: DISPLAYFORM3 In a typical setting, unlabeled nodes start out with uniform labels, making the overall entropy high. As distributions pass through the layers, labeled information propagates, and both entropy and divergence change. The attention of node i is then directed according to the informativeness (entropy) and similarity (divergence) of the states of its neighbors. As we show in the experiments (Sec. 5), this is especially useful when the data does not include node features (from which weights are typically derived). FIG2 (left) exemplifies this. Although the updates in Eq. converge for any w, this can be slow. Even with many updates, predictions are often close to uniform and thus sensitive to noise BID39. One effective solution is to dynamically bootstrap confident predictions as hard labels BID29 BID19. This process speeds up the rate of convergence by decreasing the entropy of low-entropy labels. Here we generalize this idea, and propose a flexible bifurcation mechanism. This mechanism allows for dynamically increasing or decreasing the entropy of labels. For node i and some τ ∈ R, h ic is replaced with: DISPLAYFORM0 Note that since h i ∈ ∆ C, this definition ensures that, for any τ, we have that µ(h i) ∈ ∆ C as well. When τ > 1 and as τ increases, entropy decreases, and confident labels are amplified. In contrast, when 0 < τ < 1 and as approaches 0, entropy decreases, and labels become uniform. For τ < 0 the effects are reversed, and setting τ = 1 gives µ(h i) = h i, showing that Eq. FORMULA8 DISPLAYFORM1 Thus, when θ τ = 0, Eq. FORMULA1 Recall that our goal is to learn the parameters θ of the network H(θ; S). Note that by Eq., for all i ∈ S it holds that H i (θ; S) = y i. In other words, as in LP, predictions for all labeled nodes are constrained to their true value. Due to this, the standard empirical loss becomes degenerate, as it penalizes H i (θ; S) only according to y i, and the loss becomes zero for all i ∈ S and for any choice of θ. As an alternative, we propose to follow BID49 and minimize the leave-one-out loss: DISPLAYFORM0 where S −i = S \ {i}, L is a loss function, R is a regularization term with coefficient λ, and θ contains all model parameters (such as θ φ, θ α, and θ τ). Here, each true label y i is compared to the model's prediction given all labeled points except i. Thus, the model is encouraged to propagate the labels of all nodes but one in a way which is consistent with the held-out node. In practice we have found it useful to weight examples in the loss by the inverse class ratio (estimated on S).The leave-one-out loss is a well-studied un-biased estimator of the expected loss with strong generalization guarantees BID26 BID8 BID22. In general settings, training the model on all sets {S −i} i∈S introduces a significant computational overhead. However, in SSL, when is sufficiently small, this becomes feasible BID49. For larger values of, a possible solution is to instead minimize the leave-k-out loss, using any number of sets with k randomly removed examples. When λ is small, θ is unconstrained, and the model can easily overfit. Intuitively, this can happen when only a small subset of edges is sufficient for correctly propagating labels within S. This should in noisy labels for all nodes in U. In the other extreme, when λ is large, w approaches 0, and by Eq. w is uniform. The current graph-SSL literature includes two distinct evaluation settings: one where the input includes a graph and node features, and one where a graph is available but features are not. We evaluate our method in both settings, which we refer to as the "features setting" and "no-features setting", respectively. We use benchmark datasets that include real networked data (citation networks, social networks, product networks, etc.). Our evaluation scheme follows the standard SSL setup 1 BID51 BID50 BID12 BID49 BID38 BID20 BID43 BID34. First, we sample k labeled nodes uniformly at random, and ensure at least one per class. Each method then uses the input (graph, labeled set, and features when available) to generate soft labels for all remaining nodes. Hard labels are set using argmax. We repeat this for 10 random splits using k = 1% labeled nodes. For further details please see Appendices A and C.For each experimental setting we use different Label Propagation Network (LPN) variant that differ in how edge weights are determined. Both variants use bifurcation with linear time-dependency (Sec. 3.2), and include a-symmetric bi-directional edge weights. In all tasks, LPN was initialized to simulate vanilla LP with uniform weights by setting θ = 0. Hence, we expect the learned model deviate from LP (by utilizing edge features, attention, or bifurcation) only if this in more accurate predictions. We choose T ∈ {10, 20, . . ., 100} by running cross-validation on LP rather than LPN. This process does not require learning and so is extremely fast, and due to bifurcation, quite robust (see FIG5). For training we use a class-balanced cross-entropy loss with 2 regularization, and set λ by 5-fold cross-validation. We optimize with Adam using a learning rate of 0.01. We use all relevant datasets from the LINQS collection BID41. These include three citation graphs, where nodes are papers, edges link citing papers, and features are bag-of-words. As described in Sec. 2.2, for this setting we use a model (LPN φ) where w is parameterized using a linear function of roughly 30 edge features (φ). These are based on the given "raw" node features, the graph, and the labeled set. See Appendix B for more details. Baselines include LP BID51 with uniform (LP U) and RBF (LP RBF) weights, the LP variant ADSORPTION BID0, ICA BID33, Graph Convolutional Networks , and Graph Attention Networks (GAT, BID45). 2 We also add a features-only baseline (RIDGEREG) and a graph-only baseline (NODE2VEC). We use the FLIP collection BID40, which includes several types of real networks. As no features are available, for generating meaningful weights we equip our model (LPN α) with the attention mechanism (Sec. 3.1), letting weights vary according to node states. Baselines include LP with uniform weights (LP U), the spectral embedding LEM BID3, and the deep embedding DEEPWALK BID38, LINE BID43, and NODE2VEC BID20.Results: TAB4 includes accuracies for both features and no-features settings, each averaged over 10 random splits. As can be seen, LPN outperforms other baselines on most datasetes, and consistently ranks high. Since LPN generalizes LP, the comparison to LP U and LP RBF quantifies the gain achieved by learning weights (as opposed to setting them heuristically). When weights are parameterized using features, accuracy improves by 13.4% on average. When attention is used, accuracy improves by a similar 13.7%.While some deep methods perform well on some datasets, they fail on others, and their overall performance is volatile. This is true for both learning settings. A possible explanation is that, due to their large number of parameters and hyper-parameters, they require more labeled data. Deep methods tend to perform well when more labeled nodes are available, and when tuning is done on a large validation set, or even on an entire dataset (see, e.g., BID48 BID28 ; BID34 BID45). In contrast, LPN requires relatively few parameters (θ) and only a singe model-specific hyper-parameter (T). Analysis: FIG4 gives some insight as to why LPN learns good weights. It is well known that the Laplacian's eigenvalues (and specifically λ 2, the second smallest one) play an important role in the generalization of spectral methods BID4. The figure shows how λ 2 and accuracy change over the training process. As can be seen, learning leads to weights with increasing λ 2, followed by an increase in accuracy. FIG5 demonstrates the effect of bifurcation for different depths T. As can be seen, a model with bifurcation (LPN bif) clearly outperforms the same model without it (LPN nobif). While adding depth generally improves LPN bif, it is quite robust across T. This is mediated by larger values of τ that increase label convergence rate for smaller T. Interestingly, LPN nobif degrades with large T, and even τ slightly above 1 makes a difference. In this work we presented a deep network for graph-based SSL. Our design process revolved around two main ideas: that edge weights should be learned, and that labeled data should be propagated. We began by revisiting the classic LP algorithm, whose simple structure allowed us to encode it as a differentiable neural network. We then proposed two novel ad-hoc components: information-gated attention and bifurcation, and kept our design slim and lightly parameterized. The ing model is a powerful generalization of the original algorithm, that can be trained efficiently using the leave-one-out loss using few labeled nodes. We point out two avenues for future work. First, despite its non-linearities, the current network still employs the same simple averaging updates that LP does. An interesting challenge is to design general parametric update schemes, that can perhaps be learned. Second, since the Laplacian's eigenvalues play an important role in both theory and in practice, an interesting question is whether these can be used as the basis for an explicit form of regularization. We leave this for future work. Dataset statistics are summarized in TAB1. As described in Sec. 5, there are two collections of data, LINQS 3 BID41 and FLIP 4 BID40 Although possible, parameterizing H directly by w will likely lead to overfitting. Instead, we set edge weights to be a function of edge features φ ij ∈ R d and parameters θ φ ∈ R d, and normalize using softmax over scores: DISPLAYFORM0 Our main guideline in choosing features is to keep in line with the typical SSL settings where there are only few labeled nodes. To this end, we use only a handful of features, thus keeping the number of model parameters to a minimum. We propose three types of features suited for different settings. Most works consider only "raw" node features (e.g., bag-of-words for papers in a citation network). The model, however, requires edge features for parameterizing edge weights. Edge features are therefore implicitly constructed from node features, typically by considering node-pair similarities in features space. This has three limitations. First, node feature spaces tend to be large and can thus lead to over-parameterization and eventually overfitting. Second, edge features are inherently local, as they are based only on the features of corresponding nodes, and global graph-dependent properties of edges are not taken into account. Third, parameterization is completely independent of the labeled set, meaning that edges are treated similarly regardless of whether they are in an informative region of the graph (e.g., close to labeled nodes) or not (e.g., far from any labeled node).In accordance, we propose three types of features that overcome these issues by leveraging raw features, the graph, and the labeled "seed" set. Raw features (φ x): When the data includes node features x i, a simple idea is to use a small set of uncorrelated (unparameterized) similarity measures. Examples include feature similarity measures such as cosine (x i x j / x i x j) or Gaussian (exp{− x i − x j 2 2 /σ 2}) and the top components of dimensionality reduction methods. Graph features (φ G): When the graph is real (e.g., a social network), local attributes and global roles of nodes are likely to be informative features. These can include node attributes (e.g., degree), centrality measures (e.g., edge betweenness), path-ensembles (e.g., Katz distance), and graph-partitions (e.g., k-cores). These have been successfully used in other predictive tasks on networked data (e.g., BID13).Seed features (φ S): Since labels propagate over the graph, nodes that are close to the labeled set typically have predictions that are both accurate and confident. One way of utilizing this is to associate an incoming edge with the lengths of paths that originate in a labeled node and include it. This acts as a proxy for the reliability of a neighbor as a source of label information. In general, features should be used if they lead to good generalization; in our case, this depends on the available data (such as node features), the type of graph (e.g., real network vs. k-NN graph), and by the layout of the labeled set (e.g., randomly sampled vs. crawled). TAB4 provides a list of some useful features of each of the above types. These were used in our experiments. BID30 |Γ(u) ∩ Γ(v)|/|Γ(u) ∪ Γ(v)| Graph Link Prediction Edge Adamic Adar Index BID30 w∈Γ FORMULA10 x, y denote feature vectors of two different nodes. Γ(u) is the set of neighbors of u. σ(s, t) is the number of shortest paths from s to t and σ(s, t|e) is those that pass through e.• DEEPWALK: used source code provided by the authors.• NODE2VEC: used source code provided by the authors.• LINE: used source code provided by the authors.'

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the article for finding a single-sentence TLDR

Question: A major drawback of backpropagation through time (BPTT) is the difficulty of learning long-term dependencies, coming from having to propagate credit information backwards through every single step of the forward computation. This makes BPTT both computationally impractical and biologically implausible. For this reason, full backpropagation through time is rarely used on long sequences, and truncated backpropagation through time is used as a heuristic. However, this usually leads to biased estimates of the gradient in which longer term dependencies are ignored. Addressing this issue, we propose an alternative algorithm, Sparse Attentive Backtracking, which might also be related to principles used by brains to learn long-term dependencies. Sparse Attentive Backtracking learns an attention mechanism over the hidden states of the past and selectively backpropagates through paths with high attention weights. This allows the model to learn long term dependencies while only backtracking for a small number of time steps, not just from the recent past but also from attended relevant past states. Recurrent Neural Networks (RNNs) are state-of-the-art for many machine learning sequence processing tasks. Examples where models based on RNNs shine include speech recognition BID21 BID3, image captioning BID32 BID29 BID17, machine translation BID1 BID26 BID18, and speech synthesis BID20. It is common practice to train these models using backpropagation through time (BPTT), wherein the network states are unrolled in time and gradients are backpropagated through the unrolled graph. Since the parameters of an RNN are shared across the different time steps, BPTT is more prone to vanishing and exploding gradients (; BID2 BID11 than equivalent deep feedforward networks with as many stages. This makes credit assignment particularly difficult for events that have occurred many time steps in the past, and thus makes it challenging in practice to capture long-term dependencies in the data (; BID2 . Having to wait for the end of the sequence in order to compute gradients is neither practical for machines nor animals when the dependencies extend over very long timescales. Training is slowed down considerably by long waiting times, as the rate of convergence crucially depends on how often parameters can be updated. In practice, proper long-term credit assignment in RNNs is very inconvenient, and it is common practice to employ truncated versions of BPTT for long sequences BID23 BID24 . In truncated BPTT (TBPTT), gradients are backpropagated only for a fixed and limited number of time steps and parameters are updated after each such subsequence. Truncation is often motivated by computational concerns: memory, computation time and the advantage of faster learning obtained when making more frequent updates of the parameters rather than having to wait for the end of the sequence. However, it makes capturing correlations across distant states even harder. Regular RNNs are parametric: their hidden state vector has a fixed size. We believe that this is a critical element in the classical analysis of the difficulty of learning long-term dependencies BID2. Indeed, the fixed state dimension becomes a bottleneck through which information has to flow, both forward and backward. We thus propose a semi-parametric RNN, where the next state is potentially conditioned on all the previous states of the RNN, making it possible-thanks to attention-to jump through any distance through time. We distinguish three types of states in our proposed semi-parametric RNN:• The fixed-size hidden state h (t), the conventional state of an RNN model at time t;• The monotonically-growing macrostate M = {m,..., m (s) }, the array of all past microstates, which plays the role of a random-access memory;• And the fixed-size microstate m (i), which is the ith hidden state (one of the h (t) ) that was chosen for inclusion within the macrostate M. There are as many hidden states as there are timesteps in the sequence being analyzed by the RNN. A subset of them will become microstates, and this subset is called the macrostate. The computation of the next hidden state h (t+1) is based on the whole macrostate M, in addition to the external input x (t). The macrostate being variable-length, we must devise a special mechanism to read from this ever-growing array. As a key component of our model, we propose to use an attention mechanism over the microstate elements of the macrostate. The attention mechanism in the above setting may be regarded as providing adaptive, dynamic skip connections: any past microstate can be linked, via a dynamic decision, to the current hidden state. Skip connections allow information to propagate over very long sequences. Such architectures should naturally make it easier to learn long-term dependencies. We name our algorithm sparse attentive backtracking (SAB). SAB is especially well-suited to sequences in which two parts of a task are closely related yet occur very far apart in time. Inference in SAB involves examining the macrostate and selecting some of its microstates. Ideally, SAB will not select all microstates, instead attending only to the most salient or relevant ones (e.g., emotionally loaded, in animals). The attention mechanism will select a number of relevant microstates to be incorporated into the hidden state. During training, local backpropagation of gradients happens in a short window of time around the selected microstates only. This allows for the updates to be asynchronous with respect to the time steps we attend to, and credit assignment takes place more globally in the proposed algorithm. With the proposed framework for SAB, we present the following contributions:• A principled way of doing sparse credit assignment, based on a semi-parametric RNN.• A novel way of mitigating exploding and vanishing gradients, based on reducing the number of steps that need to be backtracked through temporal skip connections.• Competitive compared to full backpropagation through time (BPTT), and much better as compared to Truncated Backpropagation through time, with significantly shorter truncation windows in our model. Mechanisms such as SAB may also be biologically plausible. Imagine having taken a wrong turn on a roadtrip and finding out about it several miles later. Our mental focus would most likely shift directly to the location in time and space where we had made the wrong decision, without replaying in reverse the detailed sequence of experienced traffic and landscape impressions. Neurophysiological findings support the existence of such attention mechanisms and their involvement in credit assignment and learning in biological systems. In particular, hippocampal recordings in rats indicate that brief sequences of prior experience are replayed both in the awake resting state and during sleep, both of which conditions are linked to memory consolidation and learning BID7 BID6 BID8. Moreover, it has been observed that these replay events are modulated by the reward an animal does or does not receive at the end of a task in the sense that they are more pronounced in the presence of a reward signal and less pronounced or absent in the absence of a reward signal BID0. Thus, the mental look back into the past seems to occur exactly when credit assignment is to be performed.2 RELATED WORK 2.1 TRUNCATED BACKPROPAGATION THROUGH TIME When training on very long sequences, full backpropagation through time becomes computationally expensive and considerably slows down training by forcing the learner to wait for the end of each (possibly very long sequence) before making a parameter update. A common heuristic is to backpropagate the loss of a particular time step through only a limited number of time steps, and hence truncate the backpropagation computation graph BID30. While truncated backpropagation through time is heavily used in practice, its inability to perform credit assignment over longer sequences is a limiting factor for this algorithm, ing in failure cases even in simple tasks, such as the Copying Memory and Adding task in BID12. The Decoupled Neural Interfaces method BID14 replaces full backpropagation through time with synthetic gradients, which are essentially small networks, mapping the hidden unit values of each layer to an estimator of the gradient of the final loss with respect to that layer. While training the synthetic gradient module requires backpropagation, each layer can make approximate gradient updates for its own parameters in an asynchronous way by using its synthetic gradient module. Thus, the network learns how to do credit assignment for a particular layer from a few examples of the gradients from backpropagation, reducing the total number of times that backpropagation needs to be performed. Online credit assignment in RNNs without backtracking remains an open research problem. One approach BID22 attempts to solve this problem by estimating gradients using an approximation to forward mode automatic differentiation instead of backpropagation. Forward mode automatic differentiation allows for computing unbiased gradient estimates in an online fashion, however it normally requires storage of the gradient of the current hidden state values with respect to the parameters, which is O(N 3) where N is the number of hidden units. The Unbiased Online Recurrent Optimization (UORO) ) method gets around this by updating a rank-1 approximation to this gradient tensor, which is shown to keep the estimate of the gradient unbiased, but potentially at the risk of increasing the variance of the gradient estimator. Neural architectures such as Residual Networks BID9 and Dense Networks BID13 ) allow information to skip over convolutional processing blocks of an underlying convolutional network architecture. In the case of Residual Networks identity connections are used to skip over convolutional processing blocks and this information is recombined using addition. This construction provably mitigates the vanishing gradient problem by allowing the gradient at any given layer to be bounded. Densely-connected convolutional networks alleviate the vanishing gradient problem by allowing a direct path from any point in the network to the output. In contrast, here we propose and explore what one might regard as a form of dynamic skip connection, modulated by an attention mechanism. We now introduce the idea of Sparse Attentive Backtracking (SAB). Classical RNN models such as those based on LSTMs or GRUs only use the previous hidden state in the computation of the next one, and therefore struggle with extremely long-range dependencies. SAB sidesteps this limitation by additionally allowing the model to select and use (a subset of) any of the past microstates in the computation of the next hidden state. In doing so the model may potentially reference microstates computed arbitrarily long ago in time. Since the classic RNN models do not support such operations on their past, we make a few architectural additions. On the forward pass of a training step, a mechanism is introduced that selects microstates from the macrostate, summarizes them, then incorporates this summary into the next hidden state. The hidden state may or may not become a microstate. On the backward pass, the gradient is allowed to flow not only through the (truncated) master chain linking consecutive hidden states, but also to the microstates which are selected in the forward pass. In the forward pass, the microstate selection process can be denser or sparser, and the summarization and incorporation can be more or less sophisticated. In the backward pass, the gating of gradient flow from a hidden state to its ancestor microstates can also be denser or sparser, although it can be no denser than the forward pass was. t{−8,−7,−6,−5,−4,−3,−2,−1} are evaluated, first by broadcasting the current provisional hidden stateĥ (t) against the macrostate (which, in the presented case of k att = 1, consists of all past hidden states), concatenating, then passing the to an MLP. The attention weights are then run through the sparsifier which selects the k top = 3 attention weights, while the others are zeroed out. Black arrows show the microstates corresponding to the non-zero sparse attention weights {ã DISPLAYFORM0}, being weighted, summed, then incorporated intoĥ (t) to compute the current final hidden state h (t).For instance, it is possible for the forward pass to be dense, incorporating a summary of all microstates, but for the backward pass to be sparse, only allowing gradient flow to some of the microstate contributors to the hidden state (Dense Forward, Sparse Backward). Another possibility is for the forward pass to be sparse, making only a few, hard, microstate selections for the summary. In this case, the backward pass will necessarily also be sparse, since few microstates will have contributed to the hidden state, and therefore to the loss (Sparse Forward, Sparse Backward).Noteworthy is that not all hidden states need be eligible to become microstates. In practice, we have found that restricting the pool of eligible hidden states to only every k att'th one still works well, while reducing both memory and computation expense. Such an increase in the granularity of microstate selection can also improve performance, by preventing the model from attending exclusively to the most recent hidden states and temporally spreading microstates out from each other. The SAB algorithm is widely applicable, and is compatible with numerous RNN architectures, including vanilla, GRU and LSTM models. However, since it necessarily requires altering the hiddento-hidden transition function substantially, it's currently incompatible with the accelerated RNN kernels offered by e.g. NVIDIA on its GPU devices through cuDNN library BID5.For vanilla and GRU-inspired RNN architectures, SAB's selection and incorporation mechanisms operate over the (hidden) state. For the LSTM architecture, which we adopt for our experiments, they operate over the hidden state but not the cell state. The microstate selection mechanism determines which microstate subset of the macrostate will be selected for summarization on the forward pass of the RNN, and which subset of that subset will receive gradient on the backward pass during training. This makes it the core of the attention mechanism of a SAB implementation. While the selection mechanism may use hard-coded attention heuristics, there is no reason why the microstate selection mechanism could not itself be a (deep) neural network trained alongside the RNN model over which it operates. In the models we use here, the selection mechanism is chosen to be a 1-hidden-layer Linear-TanhLinear MLP that computes a scalar attention weight a i for each eligible microstate vector m (i), and a sparsifier that masks out all but the k top greatest attention weights, producing the sparse attention weightsã i. We empirically demonstrate that even this simple mechanism learns to focus on past time steps relevant to the current one, thus successfully performing credit assignment. The use of a higher complexity model here would be an interesting avenue for future research. The selected microstates must be somehow combined into a fixed-size summary for incorporation into the next hidden state. While many options exist for doing so, we choose to simply perform a summation of the microstates, weighted by their sparsified attention weightã i. Lastly, the summary must be incorporated into the hidden state. Again, multiple options exist, such as addition (as done in ResNets) or concatenation (as done in DenseNets).For our purposes we choose to simply sum the summary into the provisional hidden state output h (t) computed by the LSTM cell to produce the final hidden state h (t) that will be conditioned upon in the next timestep. We now give the equations for the specific SAB-augmented LSTM model we use in our experiments. At time t, the underlying LSTM receives a vector of hidden states h (t−1), a vector of cell states c (t−1) and an input x (t), and computes a provisional hidden state vectorĥ (t) that also serves as a provisional output. We next use an attention mechanism that is similar to BID1, but modified to produce sparse discrete attention decisions. First, the provisional hidden state vectorĥ (t) is concatenated to each microstate vector m (i). Then, an MLP maps each such concatenated vector to an attention weight a (t)i representing the salience of the microstate i at the current time t. This can be expressed as: a DISPLAYFORM0 DISPLAYFORM1 where the weights matrices w 1, w 2 and w 3 are learned parameters. Following this, we apply a piece-wise linear function that sparsifies the attention while making discrete decisions. (This is different from typical attention mechanisms that normalize attention weights using a Softmax function BID1, whose output is never sparse). Let a (t) ktop be the k top th greatest-valued attention weight at time t; then the sparsified attention weights are computed asã DISPLAYFORM2 This has the effect of zeroing all attention weights less than a DISPLAYFORM3 ktop, thus masking out all but the k top most salient microstates in M. The few selected microstates receive gradient information, while no gradient flows to the rest. A summary vector s (t) is then obtained using a weighted sum over the macrostate, employing the sparse attention weights: DISPLAYFORM4 Given that this sum is very sparse, the summary operation is very fast. To incorporate the summary into the final hidden state at timestep t, we simply sum the summary and the provisional hidden state: DISPLAYFORM5 Lastly, to compute the output at the time step t, we concatenate h (t) and the sparse attention weights a (t), then apply an affine output transform to compute the output. This can be equivalently expressed as: DISPLAYFORM6 where the weights matrices V 1 and V 2 and bias vector b are learned parameters. In summary, for a given time step t, a hidden state h (i) selected by the hard-attention mechanism has two paths contributing to the hidden states h (t) in the forward pass. One path is the regular sequential forward path in an RNN; the other path is through the dynamic skip connections in the attention mechanism. When we perform backpropagation through the skip connections, gradient only flows from h (t) to microstates m (i) selected by the attention mechanism (those for which DISPLAYFORM7 In the preparation of this work, it was discovered that the attention mechanism absolutely must include a non-linearity in the computation of the raw attention weights a (t) i. Our failure to do so in an early iteration of the work ed in a catastrophic cancellation in the subsequent sparsification of the weights toã (t) i. This is because in, a rectified difference between a (t) i is computed to zero all but the k top greatest attention weights. Subtraction is linear; And since our earlier attention mechanism was linear as well, it could be separated into two parts, a first half to which only the microstate m (i) contributed and a second half to which only the hidden state h (t) contributed. This second half of the contribution is catastrophically cancelled in the difference a DISPLAYFORM0 ktop, because it was computed from the same h (t) for both, and therefore equal. We now report and discuss the of an empirical study that analyses the performance of SAB using five different tasks. We first study synthetic tasks-the copying and adding problems designed to measure models' abilities to learn long-term dependenciesmeant to confirm that SAB can successfully perform credit assignment for events that have occurred many time steps in the past. We then study more realistic tasks and larger datasets. Baselines We compare the quantitative performance of our model against two LSTM baselines BID12. The first is trained with backpropagation through time (BPTT) and the second is trained using truncated backpropagation through time (TBTPP). Both methods are trained using teacher forcing BID31. We also used gradient clipping (that is, we clip the gradients to 1 to avoid exploding gradients). Hyperparameters that are task-specific are discussed in the tasks' respective subsections, other hyperparameters that are also used by SAB and that we set to the same value are discussed below. Compared to standard RNNs, our model has two additional hyperparameters:• k top, the number of most-salient microstates to select at each time step for passing gradients in the backward pass• k att, the granularity of attention. Every k att th hidden state is chosen to be a microstate. The special case k att = 1 corresponds to choosing all hidden states to be microstates as well. In addition, we also study the impact of the TBPTT truncation length, which we denote as k trunc. This determines how many timesteps backwards to propagate gradients through in the backward pass. This effect of this hyperparameter will also be studies for the LSTM with TBTPP baseline. For all experiments we used a learning rate of 0.001 with the Adam BID15 optimizer unless otherwise stated. For SAB, we attend to every second hidden states, i.e. k att =2, unless otherwise stated. Our main findings are:1. SAB performs almost optimally and significantly outperforms both full backpropagtion through time (BPTT), and truncated backpropagation through time (TBPTT) on the synthetic copying task.2. For the synthetic adding, two language modelling task (using PennTree Bank and Text8), and permuted sequential MNIST classification tasks, SAB reaches the performance of BPTT and outperforms TBPTT. In addition, for the adding task, SAB outperforms TBPTT using much shorter truncation lengths. The copying memory task tests the model's ability to memorize salient information for long time periods. We follow the setup of the copying memory problem from BID12. In details, the network is given a sequence of T + 20 inputs consisting of: a) 10 (randomly generated) digits (digits 1 to 8) followed by; b) T blank inputs followed by; c) a special end-ofsequence character followed by; d) 10 additional blank inputs. After the end-of-sequence character the network must output a copy of the initial 10 digits. TAB1, and 3 report both accuracy and cross-entropy (CE) of the models' predictions on unseen sequences. We note that SAB is able to learn this copy task almost perfectly for all sequence-lengths T. Further, SAB outperforms all baselines. This is particularly noticeable for longer sequences, for example, when T is 300 the best baseline achieves 35.9% accuracy versus SAB's 98.9%.To better understand the learning process of SAB, we visualized the attention weights while learning the copying task (T = 200, k trunc = 10, k top = 10). FIG5 (appendix) shows the attention weights (averaged over a single mini-batch) at three different learning stages of training, all within the first epoch. We note that the attention quickly (and correctly) focuses on the first ten timesteps which contain the input digits. Furthermore, we experimented with LSTM with self-attention trained using full BPTT. The setup is very similar to unidirectional LSTM with self-attention BID16. Due to GPU memory constraints, we are only able to run this experiment up to small sequence lengths. For T=200 and T = 300, we could see that SAB performs comparably to LSTM with full self-attention trained with full BPTT. The adding task requires the model to sum two specific entries in a sequence of T (input) entries BID12. In the spirit of the copying task, larger values of T will require the model to keep track of longer-term dependencies. The exact setup is as follows. Each example in the task consists of 2 input vectors of length T. The first, is a vector of uniformly generated values between 0 and 1. The second vector encodes binary a mask that indicates which 2 entries in the Table 1: Test accuracy and cross-entropy loss performance on the copying task with sequence lengths of T = 100. Models that use TBPTT cannot solve this task while SAB and BPTT can both achieve optimal performance. Table 3: Test accuracy and cross-entropy loss performance on copying task with sequence lengths of T = 300. On these long sequences SAB's performance can still be very close to optimal. first input to sum (it consists of T − 2 zeros and 2 ones). The mask is randomly generated with the constraint that masked-in entries must be from different halves of the first input vector. TAB2 report the cross-entropy (CE) of the model's predictions on unseen sequences. When T = 200, SAB's performance is similar to the best performance of both baselines. With even longer sequences (T = 400), SAB outperforms the TBPTT but is outperformed by BPTT. We evaluate our model on language modelling task using the Penn TreeBank dataset BID19. Our LSTM baselines use 1000 hidden units and a learning rate of 0.002. We used nonoverlapping sequences of 100 in the batches of 32. We trained SAB for 100 epochs. We evaluate the performance of our model using the bits-per-character (BPC) metric. As shown in Table 6, we perform slightly worse than BPTT, but better than TBPTT. Table 4: Performance on unseen sequences of the T = 200 adding task. We note that all methods have configurations that allow them to perform near optimally. Table 6: BPC evaluation on the validation set of the character-level PTB (lower is better). This dataset is derived from the text of Wikipedia and consists of a sequence of a total of 100M characters (non-alphabetical and non-space characters were removed). We follow the setup of; use the first 90M characters for training, the next 5M for validation and the final 5M characters for testing. We train on non-overlapping sequences of length 180. Due to computational constraints, all baselines use 1000 hidden units. We trained all models using a batch size of 64. We trained SAB for a maximum of 30 epochs. We have not done any hyperparameter search for our model as it's computationally expensive. TAB3: Bit-per-character (BPC) Results on the validation and test set for Text8 (lower is better). Our last task is a sequential version of the MNIST classification dataset. The task involves predicting the label of the image after being given the image pixel by pixel (pixels are processed in a fixed random order.). All models use an LSTM with 128 hidden units. The prediction is produced by passing the final hidden state of the network into a softmax. We used a learning rate of 0.001. We trained our model for about 100 epochs, and did early stopping based on the validation set. An interesting direction for future development of the Sparse Attentive Backtracking method from the machine learning standpoint would be improving the computational efficiency when the sequences in question are very long. Since the Sparse Attentive Backtracking method uses selfattention on every step, the memory requirement grows linearly in the length of the sequence and computing the attention mechanism requires computing a scalar between the current hidden states and all previous hidden states (to determine where to attend). It might be possible to reduce the memory requirement by using a hierarchical model as done by BID4, and then recomputing the states for the lower levels of the hierarchy only when our attention mechanism looks at the corresponding higher level of the hierarchy. It might also be possible to reduce the computational cost of the attention mechanism by considering a maximum inner product search algorithm BID25, instead of naively computing the inner product with all hidden states values in the past. Improving the modeling of long-term dependencies is a central challenge in sequence modeling, and the exact gradient computation by BPTT is not biologically plausible as well as inconvenient computationally for realistic applications. Because of this, the most widely used algorithm for training recurrent neural networks on long sequences is truncated backpropagation through time, which is known to produced biased estimates of the gradient, focusing on short-term dependencies. We have proposed Sparse Attentive Backtracking, a new biologically motivated algorithm which aims to combine the strengths of full backpropagation through time and truncated backpropagation through time. It does so by only backpropagating gradients through paths selected by its attention mechanism. This allows the RNN to learn long-term dependencies, as with full backpropagation through time, while still allowing it to only backtrack for a few steps, as with truncated backpropagation through time, thus making it possible to update weights as frequently as needed rather than having to wait for the end of very long sequences. The time complexity of the forward pass of both training and inference in SAB is O(tn 2), with t the number of timesteps and n the size of the hidden state, although our current implementation scales as O(t 2 n 2). The space complexity of the forward pass of training is unchanged at O(tn), but the space complexity of inference in SAB is now O(tn) rather than O(n). However, the time cost of the backward pass of training cost is very difficult to formulate. Hidden states depend on a sparse subset of past microstates, but each of those past microstates may itself depend on several other, even earlier microstates. The web of active connections is, therefore, akin to a directed acyclic graph, and it is quite possible in the worst case for a backpropagation starting at the last hidden state to touch all past microstates several times. However, if the number of microstates truly relevant to a task is low, the attention mechanism will repeatedly focus on them to the exclusion of all others, and pathological runtimes will not be encountered. Our method approximates the true gradient but in a sense it's no different than the kind of approximation made with truncated gradient, except that instead of truncating to the last k trunc time steps, we truncate to one skip-step in the past, which can be arbitrarily far in the past. This provides a way of combating exploding and vanishing gradient problems by learning long-term dependencies. To verify the fact, we ran our model on all the datasets (Text8, Pixel-By-Pixel MNIST, char level PTB) with and without gradient clipping. We empirically found, that we need to use gradient clipping only for text8 dataset, for all the other datasets we observed little or no difference with gradient clipping. We visualize how the attention weights changes during training for the Copying Memory Task in section 4.1. The attention weights are averaged over the batch. The salient information in a copying task are in the first 10 steps. The figure shows how the attention learns to move towards and concentrate on the beginning of the sequence as training procedes. Note these all happened with the first epoch of training, such that the model learns in a reasonable amount of time.

Represent the TLDR for finding the corresponding scientific article

Towards Efficient Credit Assignment in Recurrent Networks without Backpropagation Through Time

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A theoretical analysis of a new class of RNNs, trained on neuroscience tasks, allows us to identify the role of dynamical dimensionality and cell classes in neural computations.