Question: Represent the scientific article to find a short TLDR.

Question: We describe Kernel RNN Learning (KeRNL), a reduced-rank, temporal eligibility trace-based approximation to backpropagation through time (BPTT) for training recurrent neural networks (RNNs) that gives competitive performance to BPTT on long time-dependence tasks. The approximation replaces a rank-4 gradient learning tensor, which describes how past hidden unit activations affect the current state, by a simple reduced-rank product of a sensitivity weight and a temporal eligibility trace. In this structured approximation motivated by node perturbation, the sensitivity weights and eligibility kernel time scales are themselves learned by applying perturbations. The rule represents another step toward biologically plausible or neurally inspired ML, with lower complexity in terms of relaxed architectural requirements (no symmetric return weights), a smaller memory demand (no unfolding and storage of states over time), and a shorter feedback time. Animals and humans excel at learning tasks that involve long-term temporal dependencies. A key challenge of learning such tasks is the problem of spatiotemporal credit assignment: the learner must find which of many past neural states is causally connected to the currently observed error, then allocate credit across neurons in the brain. When the time-dependencies between network states and errors are long, learning becomes difficult. In machine learning, the current standard for training recurrent architectures is Backpropagation Through Time (, BID18). BPTT assigns temporal credit or blame by unfolding a recurrent neural network in time up to a horizon length T, processing the input in a forward pass, and then backpropagating the error back in time in a backward pass (see FIG0).From a biological perspective, BPTT -like backpropagation in feedforward neural networks -is implausible for many reasons. For each weight update, BPTT requires using the transpose of the recurrent weights to transmit errors backwards in time and assign credit for how past activity affected present performance. Running the network with transposed weights requires that the network either has two-way synapses, or uses a symmetric copy of the feedforward weights to backpropagate error. In either case, the network must alternatingly gate its dynamical process to run forward or backward, and switch from nonlinear to linear dynamics, depending on whether activity or errors are being sent through the network. From both biological and engineering perspectives, there is a heavy memory demand: the complete network states, going T timesteps back in time, must be stored. The time-complexity of computation of the gradient in BPTT scales like T, making each iteration slow when training tasks with long time scale dependencies. Although T should match the length of the task or the maximum temporal lag between network states and errors for unbiased gradient learning, in practice T is often truncated to mitigate these computational costs, introducing a bias. The present work is another step in the direction of providing heuristics and relaxed approximations to backpropagation-based gradient learning for recurrent networks. KeRNL confronts the problems of efficiency and biological plausibility. It replaces the lengthy, linearized backward-flowing backpropagation phase with a product of a forward-flowing temporal eligibility trace and a spatial sensitivity weight. Instead of storing all details of past states in memory, synapses integrate their past activity during the forward pass (see FIG0). The network does not have to recompute the entire gradient at each update, as the time-scales of the eligibility trace and the sensitivity weight are learned over time. In recent years, much work has been devoted to implementing backpropagation algorithms in a more biologically plausible way, partly in the hope that more plausible implementations might also be simpler. The symmetry requirement between the forwards and backwards weights can be alleviated by using random return weights BID9 and BID10 ), however, learning still requires a separate backward pass through a network with linearized dynamics. Neurons may be able to extract error information in the time derivative of their firing rates using an STDP-like learning rule BID0 ), with error backpropagation computed as a relaxation to equilibrium BID14 ), at least for learning fixed points. Other work has focused on replacing batch learning with online learning. Typically, BPTT is implemented in a setting where data is prepared into batches of fixed sequence length T and used to perform learning in a T -step unrolled graph; however, online learning, with a constant stream of data error signals, is a more natural description of how the world supplies a learning system with data. BPTT without truncation struggles with online learning, as it must repeatedly backpropagate the error all the way through a continuously expanding graph. Since computation of the unbiased gradient scales with the length of the graph, gradient computation increases linearly with time. For a task with T timesteps, the total computation of the gradients scales like T 2.Real Time Recurrent Learning and Unbiased Online Gradient Optimization (UORO, BID15, BID11) deal with this issue by keeping track of how the synaptic weights affect the hidden state in a feedforward way. Decoupled Neural Interfaces (DNI BID6) estimates the truncated part of the gradient by continually predicting the future loss with respect to the hidden state. KeRNL offers this same advantage, in addition to other benefits. RTRL requires that the network keep track of an unwieldy rank-3 tensor, which could not be stored by any known biological entities. UORO factorizes this into rank-2 objects but still requires non-local computations like vector norm operations. Finally, DNI requires an entire separate network to keep track of the synthetic gradient. KeRNL is distinguished by its simplicity, requiring only rank-2 tensors. All computations are local, and synapses need to integrate over only a few relevant quantities. Consider a single-layer RNN in discrete time (indexed by t) with readout, input, and hidden layer activations given by y t, x t, and h t, respectively (boldface represents vectors, with vector entries denoting the activity of individual units). The dynamics of the recurrently connected hidden units are given by: DISPLAYFORM0 where W rec, W in are the recurrent and input weights, b are the hidden biases, σ is a general pointwise non-linearity, and g t represents the summed inputs (pre-nonlinearity) to the neurons at time t. The readout is given by DISPLAYFORM1 whereŷ T is the target output, in the case where error feedback is received at the end of an episode of length T, and C = T t=0 C t when errors DISPLAYFORM2 DISPLAYFORM3 DISPLAYFORM4 is the gradient of the cost with respect to the current hidden state, β ij is a set of learned sensitivity weights, and DISPLAYFORM5 is a local eligibility trace consisting of a temporally filtered version of the product of presynaptic activation and a postsynaptic activity factor. The temporal filter or kernel, K, in the eligibility has learnable time-scales; in this manuscript we use the simplest version of a lowpass temporal filter, a decaying exponential with a single time-constant γ j per neuron: K(τ, γ j) = exp(−γ j τ), though one can imagine many other function choices with multiple timescales. The role of the eligibility is to specify how strongly a synapse W jk should be held responsible for any errors in neuron j at the present time, on the basis on how far in the past the presynaptic neuron k was active. Here s DISPLAYFORM6 } stands in for the activation of the neuron presynaptic to the synapse being updated.1. Since the eligibility trace can be computed during the forward pass, KeRNL does not require backpropagating the error through time. Furthermore, KeRNL only uses at most rank-2 tensors, so neurons and synapses could plausibly do all of the required computation. The contrast between BPTT and KeRNL is depicted in FIG0,b. KeRNL emerges from the following Ansatz: DISPLAYFORM7 We call ∂h DISPLAYFORM8, a key term in the computation of the gradient, the sensitivity tensor in an extension of the usage in BID1 ). This sensitivity describes how the activity of neuron j at a previous time t − τ affects the activity of neuron i at the current time t. While the true sensitivity is a 4-index tensor summarizing many interactions based on the many paths through which activity propagates forward in a recurrent network, we approximate it with a product of a (learnable) rank-2 sensitivity weight matrix β and a temporal kernel K with (learnable) inversetime coefficients γ. The sensitivity weights β ij describe how strongly neuron j affects neuron i on average, while the temporal kernel describes how far into the future the activity of a neuron affects the other neurons for learning. We describe how to learn these parameters (β, γ) in the next section. We arrive at KeRNL by using our Ansatz for the sensitivity in the computation of a gradientbased weight update, instead of using the true sensitivity. First, we write down the full gradient rule for a recurrent network. If the parameters W ij are treated as functions that can vary over time during a trial, then the derivative can be written as a functional derivative: DISPLAYFORM9 This is simply mathematical notation for the "unfolding-in-time" trick, in which the network and weights are assumed to be replicated for each time-step of the dynamics of a recurrent network, and separate gradients are computed for each time-replica of the weights; the actual weight updates are simply the average of the separate weight variations for each time-replica. We next apply the sensitivity lemma BID1 to express gradients with respect to weights as gradients with respect to input activations, times the presynaptic activity: DISPLAYFORM10 DISPLAYFORM11 By replacing the sensitivity δh i (t)/δh j (t − τ) with our Ansatz, we arrive at our learning rule, KeRNL.The time-dependent part of the computation-a leaky integral of the product of the presynaptic activity multiplied by the instantaneous change in the postsynaptic activity-can be computed during the forwards pass, without any backpropagation of activity or error signals. For our Ansatz to align as well as possible with the gradient, we allow the sensitivity weights β and inverse-timescales γ to be learned. We learn these parameters by tracking the effect of small i.i.d. hidden perturbations ξ during the forward pass. In order to do so our hidden neurons must store two values, the true hidden state h, and a perturbed hidden stateh, which is generated by applying noise to the neurons during the forward pass: DISPLAYFORM0 The effect of previous noise on the current hidden state can be computed using the sensitivitỹ DISPLAYFORM1 We train γ, β to predict the network's response to these noisy perturbations. We take gradients with respect to the objective function. DISPLAYFORM2 which we have generated by substituting our Ansatz into. Taking gradients with respect to this objective function gives us the following update rule for the sensitivity weights and inverse-timescales. DISPLAYFORM3 represents the error in reconstructing the effect of the perturbation via the sensitivity weights and DISPLAYFORM4 are integrals that neuron h j performs over the applied perturbation ξ. In our implementation, we update these parameters immediately before we compute the gradient using. The full update rule is described in the pseudocode table.2 If we don't care about the size of the gradients and only the direction, we can use the cost function DISPLAYFORM5 where DISPLAYFORM6. This cost function trains the parameters to predict the correct direction of the perturbed hidden state minus the hidden state and works for algorithms where the gradient is divided by a running average of its magnitude (RMSProp, Adam). DISPLAYFORM7 We test KeRNL on several benchmark tasks that require memory and computation over time, showing that it is competitive with BPTT across these tasks. We implemented batch learning with KeRNL and BPTT on two tasks: the adding problem BID4; BID5 ) and pixel-by-pixel MNIST BID8 ). We implemented an online version of KeRNL with an LSTM network on the A n, B n task BID3 ) to compare with from the UORO algorithm BID15 ). The tuned hyperparameters for BPTT and KeRNL were the learning rate, η, and the gradient clipping parameter, gc BID12 ). For KeRNL, we additionally permitted a shared learning rate parameter for the sensitivity weights and kernels, η m. In practice, the same hyperparameter settings η, gc tended to work well for both BPTT and KeRNL. The additional hyperparameter for KeRNL, η m, did not need to be find tuned, and often worked well across a broad range (across several orders of magnitude, so long as it not too small but smaller than η).We implemented both the RMSprop BID16 ) and Adam (Kingma & Ba FORMULA0) optimizers and reported the best . In the adding problem, the network receives two input streams, one a sequence of random numbers in, and the second a mask vector of zeros, with two entries set randomly to one in each trial. The network's task is to sum the input from the first stream whenever there is a non-zero entry in the second. This task requires remembering sparse pieces of information over long time scales and ignoring long sequences of noise, which is difficult for RNNs when the sequences are long. We tested the performance of two networks on a variety of sequence lengths, up to 400, using both BPTT and KeRNL, Table 2. The networks were an IRNN, which is an RNN with a ReLU non-linearity where the recurrent weight matrix is initialized to identity, and a RNN with tanh nonlinearity. The implementation details are described in Appendix A.Untruncated BPTT applied to an IRNN performed very well on this task, but less so on the RNN with tanh nonlinearity. KeRNL was somewhat unstable on the IRNN, but it outperformed BPTT with the tanh nonlinearity FIG3.We believe that KeRNL outperforms BPTT on the tanh nonlinearity because our Ansatz allows the sensitivity tanh nonlinearity. By applying gradients generated by our Ansatz (instead of the true gradients) we push our network toward a solution with longer time scales via a feedback alignment-like mechanism BID9 BID10 ), as schematized in FIG0.To investigate the importance of learning the kernel timescales, we implemented KeRNL without training the sensitivity weights (β) or the inverse timescales (γ). When these parameters are not learned, KeRNL is still able to perform the task for the shorter 200-length sequence (Table 2) implying that a feedback-alignment-like mechanism BID9 BID10 ) may be enabling learning even when the error signals are not delivered along the instantaneous gradients. For longer sequences, however, learning the sensitivity and timescale parameters is important. Surprisingly, learning the inverse timescales is even more important than learning the sensitivity weights. We hypothesize that as long as the timescales over which error is correlated with outcome are appropriate, sensitivity weights are relatively less important because of feedback-alignment-like mechanisms. We show an example of how the timescales may change in FIG5. Our second task is pixel-by-pixel MNIST BID8. Here the RNN is given a stream of pixels left-to-right, top-to-bottom for a given handwritten digit from the MNIST data set. At the end of the sequence, the network is tasked with identifying the digit it was shown. This problem is difficult, as the RNN must remember an long sequence of 784 singly-presented pixels. We tuned over the same hyperparameters as in the adding problem, looking at performance after 100, 000 minibatches. Neither KeRNL nor BPTT worked well with a tanh nonlinearity, but both performed relatively well on an IRNN, FIG5. KeRNL preferred a slightly lower learning rate η than BPTT. While the KeRNL algorithm is able to learn almost as quickly on pixel-by-pixel MNIST, it does not reach as high an asymptotic performance. Still, it performs reasonably well relative to BPTT on the task. Table 2: Learning of KeRNL parameters. Left: Histogram of inverse time coefficients before training (blue) and after 7 × 10 4 minibatches (orange) on the adding problem FORMULA3: the network learns the relative importance of certain time-scales. Right: Examining the relative importance of learnable parameters in KeRNL: Performance on BPTT and various versions of KeRNL using a tanh RNN after 7 × 10 4 minibatches: fixing the sensitivities, β, while learning the inverse timescales, γ, is better than doing the reverse. While KeRNL is comparable in speed to BPTT for batch learning, we expect it to be significantly faster for online learning when the time-dependencies are of length T. Untruncated BPTT requires information sent back T steps in time for each weight update, thus the wallclock speed of computation of the gradients at each weight update in online learning scales as T, and the total scaling is thus of order T 2. If BPTT updates are truncated S < T steps back in time, the scaling is ST. KeRNL requires no backward unrolling in time, thus online KeRNL requires only O time per weight update, for a total scaling of T. As a , optimized-speed online-KeRNL should run faster than truncated online BPTT by a factor T when the trunctation time is similar to the total time-dependencies in the problem. We tested the performance of online KeRNL against UORO, another online learning algorithm, and online BPTT on the A n, B n task, where the network must predict the next character in a stream of letters. Each stream consists first of a sequence of n As followed by a sequence of n Bs. The length, n, of the sequences is randomly generated in some range. The network cannot solve this task perfectly, as it can not predict the number of As before it has seen the sequence, but can do well by matching the number of Bs to the number of As. We generated n ∈ {1, 32}. The minimum achievable average bit-loss for this task is 0.14.To compare with in the literature, we implemented KeRNL in an LSTM layer, with h representing a concatenation of the hidden and cell states (Details in Appendix B). Instead of optimizing common hyperparameters, we simply used the values from BID15, which included decaying the learning rate in time as η t = η/(1 + α √ t). However, we varied the learning rate η m, with η DISPLAYFORM0 Results other than those for KeRNL are from BID15 Table 5: Average cross-entropy bit-loss (over 10 4 minibatches) on the online A n, B n task after 10 6 minibatches entropy. Although 17-step BPTT and UORO outperformed KeRNL, we expect speed-optimized versions of KeRNL to be much faster (wall clock speed) in direct comparisons. To test how computation time for truncated-BPTT and KeRNL compare in the online setting, we implemented a dummy RNN, where the required tensor operations were performed using a random vector for both the input data and the error signal TAB5, both algorithms were implemented in Python for uniformity; Details in Appendix A). KeRNL is faster than truncated BPTT beyond very short truncation lengths. Step BPTT 3Step BPTT 10Step BPTT 20Step BPTT CPU Time 14.1 4.23 7.22 17.8 30.9 In this paper we show that KeRNL, a reduced-rank and forward-running approximation to backpropagation in RNNs, is able to perform roughly comparably to BPTT on a range of hard RNN tasks with long time-dependencies. One may view KeRNL as imposing a strong prior on the way in which neural activity from the past should be assigned credit for current performance, through the choice of the temporal kernels K in the eligibility trace, and the choice of the sensitivity weights β. This product of two rank-2 tensors in KeRNL (replacing the rank-4 sensitivity tensor for backpropagation in RNNs), assumes that the strength of influence of a neuron on another at fixed time-delay can be summarized by a simple sensitivity weight matrix, β ki, 4, and a decay due to the time difference given by K. This strong simplifying assumption is augmented or mitigated by the ability to (meta)learn the parameters of the sensitivity weights and kernels in the eligibility trace, giving the rule simultaneous simplicity and flexibility. The form of the KeRNL ansatz or prior, if wellsuited to learning problems in recurrent networks, serves as a regularizer on the types of solutions the network can find, and could even, for good choices of kernel K, provide better solutions than BPTT. We present limited evidence that KeRNL may combat the vanishing gradient problem with tanh units by imposing a prior of long time-dependencies through the eligibility. Finally, we show that KeRNL can be implemented online, where it has a shorter computation cycle than BPTT.KeRNL is a step toward biologically plausible learning. It eschews the segmented two phase backpropagation algorithm for a computation that is largely feedforward. It does not require the segmentation and storage of all past states, instead using an integrated activity or eligibility trace, and it gives rise to a naturally asymmetric structure that is more similar to the brain. While we show empirically that KeRNL performs hill-climbing, there is no guarantee that the gradients computed by KeRNL are unbiased. In the future, we hope to show empirically that KeRNL is able to perform well on more realistic tasks, and obtain some analytical guarantees on the performance of KeRNL. We hope the present contribution inspires more work on training RNNs with shorter, more plausible feedback paths. More generally, we hope that the present work shows how, with the use of reduced-rank tensor products and eligibility traces, to construct entire nested families of relaxed approximations to gradient learning in RNNs. For the adding problem and pixel-by-pixel MNIST, we tested performance by varying η and gc over several orders of magnitude: η = {1e − 03, 1e − 04, 1e − 05, 1e − 06, 1e − 07}, gc = {1, 10, 100}, using both Adam BID7 ) and RMSProp BID16 ). We then varied ηm = {1e − 03, 1e − 04, 1e − 05, 1e − 06, 1e − 07} on KeRNL. We found that KeRNL was relatively robust across ηm. For all sequence lengths, we used the hyperparameters that performed best on the task with sequence length 400. Besides the recurrent weights of the IRNN, all other weight matrices were initialized using Xavier initialization BID2 ). We initialized β with Xavier initialization for the tanh RNN, and to the identity for the IRNN. This choice was motivated by the initial sensitivity of the IRNN (we used the alternative cost function described in footnote 2. We trained on both of these tasks using the Python numpy (Walt et al. FORMULA0) package. For the dummy RNN, we used the Python numpy package BID17 ) to perform matrix algebra on a RNN with 100 hidden nodes, 100 input nodes and a tanh nonlinearity. We called "matmul" for matrix multiplication and "einsum" for other tensor operations. We used the "tanh" and "cosh" functions to compute the nonlinearity and its derivatives. In this section we describe how to implement KeRNL on an in more detail. The dynamics of the LSTM (without peepholes) are as follows DISPLAYFORM0 where net t j represents the presynaptic input to f t j. The other gradients can be calculated in an analogous manner. In the interest of full disclosure, we note that KeRNL did not perform well on next word prediction on the PennTreebank dataset. We tested an LSTM network across a wide variety of learning rates and gradient clippings and were not able to achieve near state of the art performance using KeRNL.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A biologically plausible learning rule for training recurrent neural networks

represent this text to find a much much longer article about it with say hundreds-thousands of words
Fewshot example: "GLMP: Global memory encoder (context RNN, global pointer) and local memory decoder (sketch RNN, local pointer) that share external knowledge (MemNN) are proposed to strengthen response generation in task-oriented dialogue." == "End-to-end task-oriented dialogue is challenging since knowledge bases are usually large, dynamic and hard to incorporate into a learning framework. We propose the global-to-local memory pointer (GLMP) networks to address this issue. In our model, a global memory encoder and a local memory decoder are proposed to share external knowledge. The encoder encodes dialogue history, modifies global contextual representation, and generates a global memory pointer. The decoder first generates a sketch response with unfilled slots. Next, it passes the global memory pointer to filter the external knowledge for relevant information, then instantiates the slots via the local memory pointers. We empirically show that our model can improve copy accuracy and mitigate the common out-of-vocabulary problem. As a , GLMP is able to improve over the previous state-of-the-art models in both simulated bAbI Dialogue dataset and human-human Stanford Multi-domain Dialogue dataset on automatic and human evaluation. Task-oriented dialogue systems aim to achieve specific user goals such as restaurant reservation or navigation inquiry within a limited dialogue turns via natural language. Traditional pipeline solutions are composed of natural language understanding, dialogue management and natural language generation BID32 BID28, where each module is designed separately and expensively. In order to reduce human effort and scale up between domains, end-to-end dialogue systems, which input plain text and directly output system responses, have shown promising based on recurrent neural networks BID10 BID14 and memory networks BID24. These approaches have the advantages that the dialogue states are latent without hand-crafted labels and eliminate the needs to model the dependencies between modules and interpret knowledge bases (KB) manually. However, despite the improvement by modeling KB with memory network BID0, end-to-end systems usually suffer from effectively incorporating external KB into the system response generation. The main reason is that a large, dynamic KB is equal to a noisy input and hard to encode and decode, which makes the generation unstable. Different from chit-chat scenario, this problem is especially harmful in task-oriented one, since the information in KB is usually the expected entities in the response. For example, in TAB0 the driver will expect to get the correct address to the gas station other than a random place such as a hospital. Therefore, pointer networks BID26 or copy mechanism BID8 ) is crucial to successfully generate system responses because directly copying essential words from the input source to the output not only reduces the generation difficulty, but it is also more like a human behavior. For example, in TAB0, when human want to reply others the Valero's address, they will need to "copy" the information from the table to their response as well. Therefore, in the paper, we propose the global-to-local memory pointer (GLMP) networks, which is composed of a global memory encoder, a local memory decoder, and a shared external knowledge. Unlike existing approaches with copy ability BID9 BID8, which the only information passed to decoder is the encoder hidden states, our model shares the external knowledge and leverages the encoder and the external knowledge to learn a global memory pointer and global contextual representation. Global memory pointer modifies the external knowledge by softly filtering words that are not necessary for copying. Afterward, instead of generating system responses directly, the local memory decoder first uses a sketch RNN to obtain sketch responses without slot values but sketch tags, which can be considered as learning a latent dialogue management to generate dialogue action template. Then the decoder generates local memory pointers to copy words from external knowledge and instantiate sketch tags. We empirically show that GLMP can achieve superior performance using the combination of global and local memory pointers. In simulated out-of-vocabulary (OOV) tasks in the bAbI dialogue dataset BID0, GLMP achieves 92.0% per-response accuracy and surpasses existing end-to-end approaches by 7.5% in full dialogue. In the human-human dialogue dataset, GLMP is able to surpass the previous state of the art on both automatic and human evaluation, which further confirms the effectiveness of our double pointers usage. Our model 1 is composed of three parts: global memory encoder, external knowledge, and local memory decoder, as shown in FIG0 (a). The dialogue history X = (x 1, . . ., x n) and the KB information B = (b 1, . . ., b l) are the input, and the system response Y = (y 1, . . ., y m) is the expected output, where n, l, m are the corresponding lengths. First, the global memory encoder uses a context RNN to encode dialogue history and writes its hidden states into the external knowledge. Then the last hidden state is used to read the external knowledge and generate the global memory pointer at the same time. On the other hand, during the decoding stage, the local memory decoder first generates sketch responses by a sketch RNN. Then the global memory pointer and the sketch RNN hidden state are passed to the external knowledge as a filter and a query. The local memory pointer returns from the external knowledge can copy text from the external knowledge to replace the sketch tags and obtain the final system response. Our external knowledge contains the global contextual representation that is shared with the encoder and the decoder. To incorporate external knowledge into a learning framework, end-to-end memory networks (MN) are used to store word-level information for both structural KB (KB memory) and temporal-dependent dialogue history (dialogue memory), as shown in FIG0. In addition, the MN is well-known for its multiple hop reasoning ability BID24, which is appealing to strengthen copy mechanism. Global contextual representation. In the KB memory module, each element b i ∈ B is represented in the triplet format as (Subject, Relation, Object) structure, which is a common format used to represent KB nodes BID18. For example, the KB in the TAB0 will be denoted as {(Tom's house, distance, 3 miles),..., (Starbucks, address, 792 Bedoin St)}. On the other hand, the dialogue context X is stored in the dialogue memory module, where the speaker and temporal encoding are included as in BID0 like a triplet format. For instance, the first utterance from the driver in the TAB0 will be denoted as {($user, turn1, I), ($user, turn1, need), ($user, turn1, gas)}. For the two memory modules, a bag-of-word representation is used as the memory embeddings. During the inference time, we copy the object word once a memory position is pointed to, for example, 3 miles will be copied if the triplet (Toms house, distance, 3 miles) is selected. We denote Object function as getting the object word from a triplet. Knowledge read and write. Our external knowledge is composed of a set of trainable embedding matrices C = (C 1, . . ., C K+1), where C k ∈ R |V |×d emb, K is the maximum memory hop in the MN, |V | is the vocabulary size and d emb is the embedding dimension. We denote memory in the external knowledge as M = [B; X] = (m 1, . . ., m n+l), where m i is one of the triplet components mentioned. To read the memory, the external knowledge needs a initial query vector q 1. Moreover, it can loop over K hops and computes the attention weights at each hop k using DISPLAYFORM0 where DISPLAYFORM1 is the embedding in i th memory position using the embedding matrix C k, q k is the query vector for hop k, and B is the bag-of-word function. Note that p k ∈ R n+l is a soft memory attention that decides the memory relevance with respect to the query vector. Then, the model reads out the memory o k by the weighted sum over c k+1 and update the query vector q k+1. Formally, DISPLAYFORM2 In FIG1 (a), a context RNN is used to model the sequential dependency and encode the context X. Then the hidden states are written into the external knowledge as shown in FIG0 (b). Afterward, the last encoder hidden state serves as the query to read the external knowledge and get two outputs, the global memory pointer and the memory readout. Intuitively, since it is hard for MN architectures to model the dependencies between memories, which is a serious drawback especially in conversational related tasks, writing the hidden states to the external knowledge can provide sequential and contextualized information. With meaningful representation, our pointers can correctly copy out words from external knowledge, and the common OOV challenge can be mitigated. In addition, using the encoded dialogue context as a query can encourage our external knowledge to read out memory information related to the hidden dialogue states or user intention. Moreover, the global memory pointer that learns a global memory distribution is passed to the decoder along with the encoded dialogue history and KB information. Context RNN. A bi-directional gated recurrent unit (GRU) BID2 ) is used to encode dialogue history into the hidden states H = (h e 1, . . ., h e 1), and the last hidden state h e n is used to query the external knowledge as the encoded dialogue history. In addition, the hidden states H are written into the dialogue memory module in the external knowledge by summing up the original memory representation with the corresponding hidden states. In formula, c DISPLAYFORM0 Global memory pointer. Global memory pointer G = (g 1, . . ., g n+l) is a vector containing real values between 0 and 1. Unlike conventional attention mechanism that all the weights sum to one, each element in G is an independent probability. We first query the external knowledge using h e n until the last hop, and instead of applying the Softmax function as in FORMULA0, we perform an inner product followed by the Sigmoid function. The memory distribution we obtained is the global memory pointer G, which is passed to the decoder. To further strengthen the global pointing ability, we add an auxiliary loss to train the global memory pointer as a multi-label classification task. We show in the ablation study that adding this additional supervision does improve the performance. Lastly, the memory readout q K+1 is used as the encoded KB information. In the auxiliary task, we define the label DISPLAYFORM1 by checking whether the object words in the memory exists in the expected system response Y. Then the global memory pointer is trained using binary cross-entropy loss Loss g between G and G label. In formula, DISPLAYFORM2 Given the encoded dialogue history h e n, the encoded KB information q K+1, and the global memory pointer G, our local memory decoder first initializes its sketch RNN using the concatenation of h e n and q K+1, and generates a sketch response that excludes slot values but includes the sketch tags. For example, sketch RNN will generate "@poi is @distance away", instead of "Starbucks is 1 mile away." At each decoding time step, the hidden state of the sketch RNN is used for two purposes: 1) predict the next token in vocabulary, which is the same as standard sequence-to-sequence (S2S) learning; 2) serve as the vector to query the external knowledge. If a sketch tag is generated, the global memory pointer is passed to the external knowledge, and the expected output word will be picked up from the local memory pointer. Otherwise, the output word is the word that generated by the sketch RNN. For example in FIG1 (b), a @poi tag is generated at the first time step, therefore, the word Starbucks is picked up from the local memory pointer as the system output word. DISPLAYFORM0 We use the standard cross-entropy loss to train the sketch RNN, we define Loss v as. DISPLAYFORM1 We replace the slot values in Y into sketch tags based on the provided entity table. The sketch tags ST are all the possible slot types that start with a special token, for example, @address stands for all the addresses and @distance stands for all the distance information. Local memory pointer. Local memory pointer L = (L 1, . . ., L m) contains a sequence of pointers. At each time step t, the global memory pointer G first modify the global contextual representation using its attention weights, DISPLAYFORM2 and then the sketch RNN hidden state h d t queries the external knowledge. The memory attention in the last hop is the corresponding local memory pointer L t, which is represented as the memory distribution at time step t. To train the local memory pointer, a supervision on top of the last hop memory attention in the external knowledge is added. We first define the position label of local memory pointer L label at the decoding time step t as DISPLAYFORM3 The position n+l+1 is a null token in the memory that allows us to calculate loss function even if y t does not exist in the external knowledge. Then, the loss between L and L label is defined as DISPLAYFORM4 Furthermore, a record R ∈ R n+l is utilized to prevent from copying same entities multiple times. All the elements in R are initialized as 1 in the beginning. During the decoding stage, if a memory position has been pointed to, its corresponding position in R will be masked out. During the inference time,ŷ t is defined aŝ DISPLAYFORM5 where is the element-wise multiplication. Lastly, all the parameters are jointly trained by minimizing the weighted-sum of three losses (α, β, γ are hyper-parameters): DISPLAYFORM6 3 EXPERIMENTS We use two public multi-turn task-oriented dialogue datasets to evaluate our model: the bAbI dialogue BID0 and Stanford multi-domain dialogue (SMD). The bAbI dialogue includes five simulated tasks in the restaurant domain. Task 1 to 4 are about calling API calls, modifying API calls, recommending options, and providing additional information, respectively. Task 5 is the union of tasks 1-4. There are two test sets for each task: one follows the same distribution as the training set and the other has OOV entity values. On the other hand, SMD is a human-human, multi-domain dialogue dataset. It has three distinct domains: calendar scheduling, weather information retrieval, and point-of-interest navigation. The key difference between these two datasets is, the former has longer dialogue turns but the regular user and system behaviors, the latter has few conversational turns but variant responses, and the KB information is much more complicated. The model is trained end-to-end using Adam optimizer BID12, and learning rate annealing starts from 1e −3 to 1e −4. The number of hop K is set to 1,3,6 to compare the performance difference. The weights α, β, γ summing up the three losses are set to 1. All the embeddings are initialized randomly, and a simple greedy strategy is used without beam-search during the decoding stage. The hyper-parameters such as hidden size and dropout rate are tuned with grid-search over Table 2: Per-response accuracy and completion rate (in the parentheses) on bAbI dialogues. GLMP achieves the least out-of-vocabulary performance drop. Baselines are reported from Query Reduction Network BID20, End-to-end Memory Network BID0, Gated Memory Network BID15, Point to Unknown Word BID9, and Memory-to-Sequence. DISPLAYFORM0 Ptr-Unk Mem2Seq GLMP K1 GLMP K3 GLMP K6 T1 99.4 (-) 99.9 (99.6) 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 T2 99.5 (-) 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 the development set (per-response accuracy for bAbI Dialogue and BLEU score for the SMD). In addition, to increase model generalization and simulate OOV setting, we randomly mask a small number of input source tokens into an unknown token. The model is implemented in PyTorch and the hyper-parameters used for each task and the dataset statistics are reported in the Appendix. bAbI Dialogue. In Table 2, we follow BID0 to compare the performance based on per-response accuracy and task-completion rate. Note that for utterance retrieval methods, such as QRN, MN, and GMN, cannot correctly recommend options (T3) and provide additional information (T4), and a poor generalization ability is observed in OOV setting, which has around 30% performance difference in Task 5. Although previous generation-based approaches (Ptr-Unk, Mem2Seq) have mitigated the gap by incorporating copy mechanism, the simplest cases such as generating and modifying API calls (T1, T2) still face a 6-17% OOV performance drop. On the other hand, GLMP achieves a highest 92.0% task-completion rate in full dialogue task and surpasses other baselines by a big margin especially in the OOV setting. No per-response accuracy loss for T1, T2, T4 using only the single hop, and only decreases 7-9% in task 5.Stanford Multi-domain Dialogue. For human-human dialogue scenario, we follow previous dialogue works BID10 to evaluate our system on two automatic evaluation metrics, BLEU and entity F1 score 2. As shown in TAB3, GLMP achieves a highest 14.79 BLEU and 59.97% entity F1 score, which is a slight improvement in BLEU but a huge gain in entity F1. In fact, for unsupervised evaluation metrics in task-oriented dialogues, we argue that the entity F1 might be a more comprehensive evaluation metric than per-response accuracy or BLEU, as shown in that humans are able to choose the right entities but have very diversified responses. Note that the of rule-based and KVR are not directly comparable because they simplified the task by mapping the expression of entities to a canonical form using named entity recognition and linking 3. Moreover, human evaluation of the generated responses is reported. We compare our work with previous state-of-the-art model Mem2Seq 4 and the original dataset responses as well. We randomly select 200 different dialogue scenarios from the test set to evaluate three different responses. Amazon Mechanical Turk is used to evaluate system appropriateness and human-likeness on a scale from 1 to 5. As the shown in TAB3, we see that GLMP outperforms Mem2Seq in both measures, which is coherent to previous observation. We also see that human performance on this assessment sets the upper bound on scores, as expected. More details about the human evaluation are reported in the Appendix. Ablation Study. The contributions of the global memory pointer G and the memory writing of dialogue history H are shown in TAB4. We compare the using GLMP with K = 1 in bAbI OOV setting and SMD. GLMP without H means that the context RNN in the global memory encoder does not write the hidden states into the external knowledge. As one can observe, our model without H has 5.3% more loss in the full dialogue task. On the other hand, GLMP without G means that we do not use the global memory pointer to modify the external knowledge, and an 11.47% entity F1 drop can be observed in SMD dataset. Note that a 0.4% increase can be observed in task 5, it suggests that the use of global memory pointer may impose too strong prior entity probability. Even if we only report one experiment in the table, this OOV generalization problem can be mitigated by increasing the dropout ratio during training. Visualization and Qualitative Evaluation. Analyzing the attention weights has been frequently used to interpret deep learning models. In Figure 3, we show the attention vector in the last hop for each generation time step. Y-axis is the external knowledge that we can copy, including the KB information and the dialogue history. Based on the question "what is the address?" asked by the driver in the last turn, the gold answer and our generated response are on the top, and the global memory pointer G is shown in the left column. One can observe that in the right column, the final memory pointer successfully copy the entity chevron in step 0 and its address 783 Arcadia Pl in step 3 to fill in the sketch utterance. On the other hand, the memory attention without global weighting is reported in the middle column. One can find that even if the attention weights focus on several point of interests and addresses in step 0 and step 3, the global memory pointer can mitigate the issue as expected. More dialogue visualization and generated including several negative examples and error analysis are reported in the Appendix. Task-oriented dialogue systems. Machine learning based dialogue systems are mainly explored by following two different approaches: modularized and end-to-end. For the modularized systems BID29 BID28, a set of modules for natural language understanding BID32 BID1, dialogue state tracking BID13 BID35, dialogue management BID23, and natural language generation BID22 are used. These approaches achieve good stability via combining domain-specific knowledge and slot-filling techniques, but additional human labels are needed. On the other hand, end-to-end approaches have shown promising recently. Some works view the task as a next utterance retrieval problem, for examples, recurrent entity networks share parameters between RNN BID30, query reduction networks modify query between layers BID20, and memory networks BID0 BID15 perform multi-hop design to strengthen reasoning ability. In addition, some approaches treat the task as a sequence generation problem. BID14 Delexicalized Generation: @poi is at @address Final Generation: chevron is at 783_arcadia_pl Gold: 783_arcadia_pl is the address for chevron gas_station Figure 3: Memory attention visualization in the SMD navigation domain. Left column is the global memory pointer G, middle column is the memory pointer without global weighting, and the right column is the final memory pointer. these approaches can encourage more flexible and diverse system responses by generating utterances token-by-token. Pointer network. BID26 uses attention as a pointer to select a member of the input source as the output. Such copy mechanisms have also been used in other natural language processing tasks, such as question answering BID3 BID10, neural machine translation BID9 BID8, language modeling BID17, and text summarization BID19. In task-oriented dialogue tasks, first demonstrated the potential of the copy-augmented Seq2Seq model, which shows that generationbased methods with simple copy strategy can surpass retrieval-based ones. Later, augmented the vocabulary distribution by concatenating KB attention, which at the same time increases the output dimension. Recently, combines end-to-end memory network into sequence generation, which shows that the multi-hop mechanism in MN can be utilized to improve copy attention. These models outperform utterance retrieval methods by copying relevant entities from the KBs. Others. BID10 proposes entity indexing and introduces recorded delexicalization to simplify the problem by record entity tables manually. In addition, our approach utilized recurrent structures to query external memory can be viewed as the memory controller in Memory augmented neural networks (MANN) BID6. Similarly, memory encoders have been used in neural machine translation BID27 and meta-learning applications. However, different from other models that use a single matrix representation for reading and writing, GLMP leverages end-to-end memory networks to perform multiple hop attention, which is similar to the stacking self-attention strategy in the Transformer BID25. In the work, we present an end-to-end trainable model called global-to-local memory pointer networks for task-oriented dialogues. The global memory encoder and the local memory decoder are designed to incorporate the shared external knowledge into the learning framework. We empirically show that the global and the local memory pointer are able to effectively produce system responses even in the out-of-vocabulary scenario, and visualize how global memory pointer helps as well. As a , our model achieves state-of-the-art in both the simulated and the human-human dialogue datasets, and holds potential for extending to other tasks such as question answering and text summarization. A.1 TRAINING PARAMETERS Table 5: Selected hyper-parameters in each dataset for different hops. The values is the embedding dimension and the GRU hidden size, and the values between parenthesis is the dropout rate. For all the models we used learning rate equal to 0.001, with a decay rate of 0.5. T2 T3 T4 T5 SMD GLMP K1 64 FORMULA0 2) GLMP K3 64 FORMULA0 2) GLMP K6 64 FORMULA0 A.2 DATASET STATISTICS For bAbI dialogues, the mistakes are mainly from task 3, which is recommending restaurants based on their rating from high to low. We found that sometimes the system will keep sending those restaurants with the higher score even if the user rejected them in the previous turns. On the other hand, SMD is more challenging for response generation. First, we found that the model makes mistakes when the KB has several options corresponding to the user intention. For example, once the user has more than one doctor appointment in the table, the model can barely recognize. In addition, since we do not include the domain specific and user intention supervision, wrong delexicalized responses may be generated, which in an incorrect entity copy. Lastly, we found that the copied entities may not be matched to the generated sketch tags. For example, an address tag may in a distance entity copy. We leave the space of improvement to future works. One of the reviewers suggested us to compare our work to some existing dialogue framework such as PyDial 5. To the best of our knowledge, in the PyDial framework, it requires to have the dialogue acts labels for the NLU module and the belief states labels for the belief tracker module. The biggest challenge is we do not have such labels in the SMD and bAbI datasets. Moreover, the semi tracker in PyDial is rule-based, which need to re-write rules whenever it encounters a new domain or new datasets. Even its dialogue management module could be a learning solution like policy networks, the input of the policy network is still the hand-crafted state features and labels. Therefore, without the rules and labels predefined in the NLU and belief tracker modules, PyDial could not learn a good policy network. Truly speaking, based on the data we have (not very big size) and the current state-of-the-art machine learning algorithms and models, we believe that a well and carefully constructed task-oriented dialogue system using PyDial in a known domain using human rules (in NLU and Belief Tracker) with policy networks may outperform the end-to-end systems (more robust). However, in this paper, without additional human labels and human rules, we want to explore the potential and the advantage of end-to-end systems. Besides easy to train, for multi-domain cases, or even zero-shot domain cases, we believe end-to-end approaches will have better adaptability compared to any rule-based systems. Delexicalized Generation: the nearest @poi_type is @poi, @distance away at @address Final Generation: the nearest grocery_store is willows_market, 3_miles away at 409_bollard_st Gold: we are 3_miles away from willows_market but there is a car_collision_nearby Figure 5: Memory attention visualization from the SMD navigation domain. Delexicalized Generation: the nearest @poi_type is @poi, @distance away at @address Final Generation: the nearest grocery_store is sigona_farmers_market, 4_miles away at 819_alma_st Gold: there are whole_foods 2_miles away and sigona_farmers_market 4_miles away where do we go? Delexicalized Generation: okay, setting a reminder for your @event with @party @date at @time Final Generation: okay, setting a reminder for your dinner with marie 7pm at the_6th Gold: i will set a reminder for dinner at 7pm, for the_6th of this month with marie Figure 9: Memory attention visualization from the SMD schedule domain. Delexicalized Generation: your @event is on @date at @time Final Generation: your tennis_activity is on the_4th at 5pm Gold: your tennis_activity is on the_4th at 5pm and your sister will be attending Delexicalized Generation: your next @event is on @date at @time Final Generation: your next tennis_activity is on the_13th at 6pm Gold: your next tennis_activity is on the_4th at 5pm with your mother Delexicalized Generation: your @event is on @date at @time with @party Final Generation: your doctor is on the_5th at 6pm with alex Gold: are you talking about the doctor_appointment on wednesday or the one on the_5th? Delexicalized Generation: it will not be @weather_attribute in @location @weekly_time Final Generation: it will not be drizzle in redwood_city weekend Gold: there will be no drizzle in redwood_city this weekend Delexicalized Generation: there will be @weather_attribute in @location on @date Final Generation: there will be clear_skies in danville on thursday Gold: dew is predicted in danville on thursday FIG0: Memory attention visualization from the SMD weather domain."

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the scientific article to find a short TLDR

Question: We present Deep Graph Infomax (DGI), a general approach for learning node representations within graph-structured data in an unsupervised manner. DGI relies on maximizing mutual information between patch representations and corresponding high-level summaries of graphs---both derived using established graph convolutional network architectures. The learnt patch representations summarize subgraphs centered around nodes of interest, and can thus be reused for downstream node-wise learning tasks. In contrast to most prior approaches to unsupervised learning with GCNs, DGI does not rely on random walk objectives, and is readily applicable to both transductive and inductive learning setups. We demonstrate competitive performance on a variety of node classification benchmarks, which at times even exceeds the performance of supervised learning. encoder models based on graph convolutions BID10, it is unclear whether randomwalk objectives actually provide any useful signal, as these encoders already enforce an inductive bias that neighboring nodes have similar representations. In this work, we propose an alternative objective for unsupervised graph learning that is based upon mutual information, rather than random walks. Recently, scalable estimation of mutual information was made both possible and practical through Mutual Information Neural Estimation , which relies on training a statistics network as a classifier of samples coming from the joint distribution of two random variables and their product of marginals. Following on MINE, introduced Deep InfoMax (DIM) for learning representations of highdimensional data. DIM trains an encoder model to maximize the mutual information between a high-level "global" representation and "local" parts of the input (such as patches of an image). This encourages the encoder to carry the type of information that is present in all locations (and thus are globally relevant), such as would be the case of a class label. DIM relies heavily on convolutional neural network structure in the context of image data, and to our knowledge, no work has applied mutual information maximization to graph-structured inputs. Here, we adapt ideas from DIM to the graph domain, which can be thought of as having a more general type of structure than the ones captured by convolutional neural networks. In the following sections, we introduce our method called Deep Graph Infomax (DGI). We demonstrate that the representation learned by DGI is consistently competitive on both transductive and inductive classification tasks, often outperforming both supervised and unsupervised strong baselines in our experiments. Contrastive methods. An important approach for unsupervised learning of representations is to train an encoder to be contrastive between representations that capture statistical dependencies of interest and those that do not. For example, a contrastive approach may employ a scoring function, training the encoder to increase the score on "real" input (a.k.a, positive examples) and decrease the score on "fake" input (a.k.a., negative samples). Contrastive methods are central to many popular word-embedding methods BID6 BID27 BID26, but they are found in many unsupervised algorithms for learning representations of graphstructured input as well. There are many ways to score a representation, but in the graph literature the most common techniques use classification BID32 BID13 BID24 BID16, though other scoring functions are used BID9 BID2. DGI is also contrastive in this respect, as our objective is based on classifying local-global pairs and negative-sampled counterparts. Sampling strategies. A key implementation detail to contrastive methods is how to draw positive and negative samples. The prior work above on unsupervised graph representation learning relies on a local contrastive loss (enforcing proximal nodes to have similar embeddings). Positive samples typically correspond to pairs of nodes that appear together within short random walks in the graph-from a language modelling perspective, effectively treating nodes as words and random walks as sentences. Recent work by BID2 uses node-anchored sampling as an alternative. The negative sampling for these methods is primarily based on sampling of random pairs, with recent work adapting this approach to use a curriculum-based negative sampling scheme (with progressively "closer" negative examples; BID45 or introducing an adversary to select the negative examples BID3 .Predictive coding. Contrastive predictive coding is another method for learning deep representations based on mutual information maximization. Like the models above, CPC is also contrastive, in this case using an estimate of the conditional density (in the form of noise contrastive estimation, BID14 as the scoring function. However, unlike our approach, CPC and the graph methods above are all predictive: the contrastive objective effectively trains a predictor between structurally-specified parts of the input (e.g., between neighboring node pairs or between a node and its neighborhood). Our approach differs in that we contrast global / local parts of a graph simultaneously, where the global variable is computed from all local variables. To the best of our knowledge, the sole prior works that instead focuses on contrasting "global" and "local" representations on graphs do so via (auto-)encoding objectives on the adjacency matrix BID41 ) and incorporation of community-level constraints into node embeddings BID42. Both methods rely on matrix factorization-style losses and are thus not scalable to larger graphs. In this section, we will present the Deep Graph Infomax method in a top-down fashion: starting with an abstract overview of our specific unsupervised learning setup, followed by an exposition of the objective function optimized by our method, and concluding by enumerating all the steps of our procedure in a single-graph setting. We assume a generic graph-based unsupervised machine learning setup: we are provided with a set of node features, X = {x 1, x 2, . . ., x N}, where N is the number of nodes in the graph and x i ∈ R F represents the features of node i. We are also provided with relational information between these nodes in the form of an adjacency matrix, A ∈ R N ×N. While A may consist of arbitrary real numbers (or even arbitrary edge features), in all our experiments we will assume the graphs to be unweighted, i.e. A ij = 1 if there exists an edge i → j in the graph and A ij = 0 otherwise. Our objective is to learn an encoder, E: DISPLAYFORM0 for each node i. These representations may then be retrieved and used for downstream tasks, such as node classification. Here we will focus on graph convolutional encoders-a flexible class of node embedding architectures, which generate node representations by repeated aggregation over local node neighborhoods BID10. A key consequence is that the produced node embeddings, h i, summarize a patch of the graph centered around node i rather than just the node itself. In what follows, we will often refer to h i as patch representations to emphasize this point. Our approach to learning the encoder relies on maximizing local mutual information-that is, we seek to obtain node (i.e., local) representations that capture the global information content of the entire graph, represented by a summary vector, s. In order to obtain the graph-level summary vectors, s, we leverage a readout function, R: DISPLAYFORM0, and use it to summarize the obtained patch representations into a graph-level representation; i.e., s = R(E(X, A)).As a proxy for maximizing the local mutual information, we employ a discriminator, D: DISPLAYFORM1 represents the probability scores assigned to this patch-summary pair (should be higher for patches contained within the summary).Negative samples for D are provided by pairing the summary s from (X, A) with patch representations h j of an alternative graph, (X, A). In a multi-graph setting, such graphs may be obtained as other elements of a training set. However, for a single graph, an explicit (stochastic) corruption function, C: DISPLAYFORM2 is required to obtain a negative example from the original graph, i.e. (X, A) = C(X, A). The choice of the negative sampling procedure will govern the specific kinds of structural information that is desirable to be captured as a byproduct of this maximization. For the objective, we follow the intuitions from Deep InfoMax and use a noise-contrastive type objective with a standard binary cross-entropy (BCE) loss between the samples from the joint (positive examples) and the product of marginals (negative examples). Following their work, we use the following objective DISPLAYFORM3 This approach effectively maximizes mutual information between h i and s, based on the JensenShannon divergence 2 between the joint and the product of marginals. As all of the derived patch representations are driven to preserve mutual information with the global graph summary, this allows for discovering and preserving similarities on the patch-level-for example, distant nodes with similar structural roles (which are known to be a strong predictor for many node classification tasks; BID8 . Note that this is a "reversed" version of the argument given by : for node classification, our aim is for the patches to establish links to similar patches across the graph, rather than enforcing the summary to contain all of these similarities (however, both of these effects should in principle occur simultaneously). We now provide some intuition that connects the classification error of our discriminator to mutual information maximization on graph representations. DISPLAYFORM0 be a set of node representations drawn from an empirical probability distribution of graphs, p(X), with finite number of elements, |X|, such that p(DISPLAYFORM1 be a deterministic readout function on graphs and s (k) = R(X (k) ) be the summary vector of the k-th graph, with marginal distribution p(s). The optimal classifier between the joint distribution p(X, s) and the product of marginals p(X)p(s), assuming class balance, has an error rate upper bounded by DISPLAYFORM2 the set of all graphs in the input set that are mapped to s (k) by R, i.e. DISPLAYFORM3 As R(·) is deterministic, samples from the joint, (X (k), s (k) ) are drawn from the product of marginals with probability p(s (k) )p(X (k) ), which decomposes into: DISPLAYFORM4 This probability ratio is maximized at 1 when DISPLAYFORM5. The probability of drawing any sample of the joint from the product of marginals is then bounded above by DISPLAYFORM6. As the probability of drawing (DISPLAYFORM7, we know that classifying these samples as coming from the joint has a lower error than classifying them as coming from the product of marginals. The error rate of such a classifier is then the probability of drawing a sample from the joint as a sample from product of marginals under the mixture probability, which we can bound by Err ≤ DISPLAYFORM8, with the upper bound achieved, as above, when R(·) is injective for all elements of {X (k) }.It may be useful to note that DISPLAYFORM9 The first is obtained via a trivial application of Jensen's inequality, while the other extreme is reached only in the edge case of a constant readout function (when every example from the joint is also an example from the product of marginals, so no classifier performs better than chance). Corollary 1. From now on, assume that the readout function used, R, is injective. Assume the number of allowable states in the space of s, | s|, is greater than or equal to |X|. Then, for s, the optimal summary under the classification error of an optimal classifier between the joint and the product of marginals, it holds that | s | = |X|.Proof. By injectivity of R, we know that s = argmin s Err *. As the upper error bound, Err *, is a simple geometric sum, we know that this is minimized when p(s (k) ) is uniform. As R(·) is deterministic, this implies that each potential summary state would need to be used at least once. Combined with the condition | s| ≥ |X|, we conclude that the optimum has | s | = |X|. Theorem 1. s = argmax s MI(X; s), where MI is mutual information. Proof. This follows from the fact that the mutual information is invariant under invertible transforms. As | s | = |X| and R is injective, it has an inverse function, R −1. It follows then that, for any s, MI(X; s) ≤ H(X) = MI(X; X) = MI(X; R(X)) = MI(X; s), where H is entropy. Theorem 1 shows that for finite input sets and suitable deterministic functions, minimizing the classification error in the discriminator can be used to maximize the mutual information between the input and output. However, as was shown in, this objective alone is not enough to learn useful representations. As in their work, we discriminate between the global summary vector and local high-level representations. DISPLAYFORM10 be the neighborhood of the node i in the k-th graph that collectively maps to its high-level features, DISPLAYFORM11 where n is the neighborhood function that returns the set of neighborhood indices of node i for graph X (k), and E is a deterministic encoder function. Let us assume that DISPLAYFORM12 Proof. Given our assumption of |X i | = | s|, there exists an inverse X i = R −1 (s), and therefore DISPLAYFORM13 ) mapping s to h i. The optimal classifier between the joint p(h i, s) and the product of marginals p(h i)p(s) then has (by Lemma 1) an error rate upper bound of DISPLAYFORM14. Therefore (as in Corollary 1), for the optimal DISPLAYFORM15 which by the same arguments as in Theorem 1 maximizes the mutual information between the neighborhood and high-level features, MI(X DISPLAYFORM16 This motivates our use of a classifier between samples from the joint and the product of marginals, and using the binary cross-entropy (BCE) loss to optimize this classifier is well-understood in the context of neural network optimization. Assuming the single-graph setup (i.e., (X, A) provided as input), we will now summarize the steps of the Deep Graph Infomax procedure:1. Sample a negative example by using the corruption function: (X, A) ∼ C(X, A).2. Obtain patch representations, h i for the input graph by passing it through the encoder: DISPLAYFORM0 3. Obtain patch representations, h j for the negative example by passing it through the encoder: DISPLAYFORM1. Summarize the input graph by passing its patch representations through the readout function: s = R(H). This algorithm is fully summarized by Figure 1. Figure 1: A high-level overview of Deep Graph Infomax. Refer to Section 3.4 for more details. DISPLAYFORM0 We have assessed the benefits of the representation learnt by the DGI encoder on a variety of node classification tasks (transductive as well as inductive), obtaining competitive . In each case, DGI was used to learn patch representations in a fully unsupervised manner, followed by evaluating the node-level classification utility of these representations. This was performed by directly using these representations to train and test a simple linear (logistic regression) classifier. We follow the experimental setup described in BID23 and BID15 on the following benchmark tasks: classifying research papers into topics on the Cora, Citeseer and Pubmed citation networks BID35; predicting the community structure of a social network modeled with Reddit posts; and classifying protein roles within protein-protein interaction (PPI) networks BID49, requiring generalisation to unseen networks. Further information on the datasets may be found in TAB0 and Appendix A. For each of three experimental settings (transductive learning, inductive learning on large graphs, and multiple graphs), we employed distinct encoders and corruption functions appropriate to that setting (described below).Transductive learning. For the transductive learning tasks (Cora, Citeseer and Pubmed), our encoder is a one-layer Graph Convolutional Network (GCN) model BID23, with the following propagation rule: DISPLAYFORM0 whereÂ = A + I N is the adjacency matrix with inserted self-loops andD is its corresponding degree matrix; i.e. D ii = jÂ ij. For the nonlinearity, σ, we have applied the parametric ReLU (PReLU) function BID17, and Θ ∈ R F ×F is a learnable linear transformation applied to every node, with F = 512 features being computed (specially, F = 256 on Pubmed due to memory limitations).The corruption function used in this setting is designed to encourage the representations to properly encode structural similarities of different nodes in the graph; for this purpose, C preserves the original adjacency matrix (A = A), whereas the corrupted features, X, are obtained by row-wise shuffling of X. That is, the corrupted graph consists of exactly the same nodes as the original graph, but they are located in different places in the graph, and will therefore receive different patch representations. We demonstrate DGI is stable to other choices of corruption functions in Appendix C, but we find those that preserve the graph structure in the strongest features. Inductive learning on large graphs. For inductive learning, we may no longer use the GCN update rule in our encoder (as the learned filters rely on a fixed and known adjacency matrix); instead, we apply the mean-pooling propagation rule, as used by GraphSAGE-GCN BID15: DISPLAYFORM1 with parameters defined as in Equation 3. Note that multiplying byD DISPLAYFORM2 actually performs a normalized sum (hence the mean-pooling). While Equation 4 explicitly specifies the adjacency and degree matrices, they are not needed: identical inductive behaviour may be observed by a constant attention mechanism across the node's neighbors, as used by the Const-GAT model BID39.For Reddit, our encoder is a three-layer mean-pooling model with skip connections BID18: DISPLAYFORM3 where is featurewise concatenation (i.e. the central node and its neighborhood are handled separately). We compute F = 512 features in each MP layer, with the PReLU activation for σ. Given the large scale of the dataset, it will not fit into GPU memory entirely. Therefore, we use the subsampling approach of BID15, where a minibatch of nodes is first selected, and then a subgraph centered around each of them is obtained by sampling node neighborhoods with replacement. Specifically, we sample 10, 10 and 25 neighbors at the first, second and third level, respectively-thus, each subsampled patch has 1 + 10 + 100 + 2500 = 2611 nodes. Only the computations necessary for deriving the central node i's patch representation, h i, are performed. These representations are then used to derive the summary vector, s, for the minibatch FIG0 ). We used minibatches of 256 nodes throughout training. To define our corruption function in this setting, we use a similar approach as in the transductive tasks, but treat each subsampled patch as a separate graph to be corrupted (i.e., we row-wise shuffle the feature matrices within a subsampled patch). Note that this may very likely cause the central node's features to be swapped out for a sampled neighbor's features, further encouraging diversity in the negative samples. The patch representation obtained in the central node is then submitted to the discriminator. Inductive learning on multiple graphs. For the PPI dataset, inspired by previous successful supervised architectures BID39, our encoder is a three-layer mean-pooling model with dense skip connections BID18 BID20: DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 where W skip is a learnable projection matrix, and MP is as defined in Equation 4. We compute F = 512 features in each MP layer, using the PReLU activation for σ. In this multiple-graph setting, we opted to use randomly sampled training graphs as negative examples (i.e., our corruption function simply samples a different graph from the training set). We found this method to be the most stable, considering that over 40% of the nodes have all-zero features in this dataset. To further expand the pool of negative examples, we also apply dropout BID36 to the input features of the sampled graph. We found it beneficial to standardize the learnt embeddings across the training set prior to providing them to the logistic regression model. Readout, discriminator, and additional training details. Across all three experimental settings, we employed identical readout functions and discriminator architectures. For the readout function, we use a simple averaging of all the nodes' features: DISPLAYFORM7 where σ is the logistic sigmoid nonlinearity. While we have found this readout to perform the best across all our experiments, we assume that its power will diminish with the increase in graph size, and in those cases, more sophisticated readout architectures such as set2vec BID40 or DiffPool BID46 ) are likely to be more appropriate. The discriminator scores summary-patch representation pairs by applying a simple bilinear scoring function (similar to the scoring used by): DISPLAYFORM8 Here, W is a learnable scoring matrix and σ is the logistic sigmoid nonlinearity, used to convert scores into probabilities of (h i, s) being a positive example. All models are initialized using Glorot initialization BID11 ) and trained to maximize the mutual information provided in Equation 1 on the available nodes (all nodes for the transductive, and training nodes only in the inductive setup) using the Adam SGD optimizer BID22 with an initial learning rate of 0.001 (specially, 10 DISPLAYFORM9 on Reddit). On the transductive datasets, we use an early stopping strategy on the observed training loss, with a patience of 20 epochs 3. On the inductive datasets we train for a fixed number of epochs (150 on Reddit, 20 on PPI). The of our comparative evaluation experiments are summarized in TAB1 For the transductive tasks, we report the mean classification accuracy (with standard deviation) on the test nodes of our method after 50 runs of training (followed by logistic regression), and reuse the metrics already reported in BID23 for the performance of DeepWalk and GCN, as well as Label Propagation (LP) BID48 and Planetoid BID44 )-a representative supervised random walk method. Specially, we provide for training the logistic regression on raw input features, as well as DeepWalk with the input features concatenated. Avg. pooling 0.958 ± 0.001 0.969 ± 0.002For the inductive tasks, we report the micro-averaged F 1 score on the (unseen) test nodes, averaged after 50 runs of training, and reuse the metrics already reported in BID15 for the other techniques. Specifically, as our setup is unsupervised, we compare against the unsupervised GraphSAGE approaches. We also provide supervised for two related architecturesFastGCN BID5 and Avg. pooling ).Our demonstrate strong performance being achieved across all five datasets. We particularly note that the DGI approach is competitive with the reported for the GCN model with the supervised loss, even exceeding its performance on the Cora and Citeseer datasets. We assume that these benefits stem from the fact that, indirectly, the DGI approach allows for every node to have access to structural properties of the entire graph, whereas the supervised GCN is limited to only two-layer neighborhoods (by the extreme sparsity of the training signal and the corresponding threat of overfitting). It should be noted that, while we are capable of outperforming equivalent supervised encoder architectures, our performance still does not surpass the current supervised transductive state of the art (which is held by methods such as GraphSGAN BID7). We further observe that the DGI method successfully outperformed all the competing unsupervised GraphSAGE approaches on the Reddit and PPI datasets-thus verifying the potential of methods based on local mutual information maximization in the inductive node classification domain. Our Reddit are competitive with the supervised state of the art, whereas on PPI the gap is still large-we believe this can be attributed to the extreme sparsity of available node features (over 40% of the nodes having all-zero features), that our encoder heavily relies on. We note that a randomly initialized graph convolutional network may already extract highly useful features and represents a strong baseline-a well-known fact, considering its links to the Weisfeiler- Lehman graph isomorphism test BID43, that have already been highlighted and analyzed by BID23 and BID15. As such, we also provide, as Random-Init, the logistic regression performance on embeddings obtained from a randomly initialized encoder. Besides demonstrating that DGI is able to further improve on this strong baseline, it particularly reveals that, on the inductive datasets, previous random walk-based negative sampling methods may have been ineffective for learning appropriate features for the classification task. Lastly, it should be noted that deeper encoders correspond to more pronounced mixing between recovered patch representations, reducing the effective variability of our positive/negative examples' pool. We believe that this is the reason why shallower architectures performed better on some of the datasets. While we cannot say that these trends will hold in general, with the DGI loss function we generally found benefits from employing wider, rather than deeper models. We performed a diverse set of analyses on the embeddings learnt by the DGI algorithm in order to better understand the properties of DGI. We focus our analysis exclusively on the Cora dataset (as it has the smallest number of nodes, significantly aiding clarity).A standard set of "evolving" t-SNE plots BID25 of the embeddings is given in FIG2. As expected given the quantitative , the learnt embeddings' 2D projections exhibit discernible clustering in the 2D projected space (especially compared to the raw features and Random-Init), which respects the seven topic classes of Cora. The projection obtains a Silhouette score of 0.234, which compares favorably with the previous reported score of 0.158 for Embedding Propagation BID9.We ran further analyses, revealing insights into DGI's mechanism of learning, isolating biased embedding dimensions for pushing the negative example scores down and using the remainder to encode useful information about positive examples. We leverage these insights to retain competitive performance to the supervised GCN even after half the dimensions are removed from the patch representations provided by the encoder. These-and several other-qualitative and ablation studies can be found in Appendix B. We have presented Deep Graph Infomax (DGI), a new approach for learning unsupervised representations on graph-structured data. By leveraging local mutual information maximization across the graph's patch representations, obtained by powerful graph convolutional architectures, we are able to obtain node embeddings that are mindful of the global structural properties of the graph. This enables competitive performance across a variety of both transductive and inductive classification tasks, at times even outperforming relevant supervised architectures. Transductive learning. We utilize three standard citation network benchmark datasets-Cora, Citeseer and Pubmed BID35 -and closely follow the transductive experimental setup of BID44. In all of these datasets, nodes correspond to documents and edges to (undirected) citations. Node features correspond to elements of a bag-of-words representation of a document. Each node has a class label. We allow for only 20 nodes per class to be used for training-however, honouring the transductive setup, the unsupervised learning algorithm has access to all of the nodes' feature vectors. The predictive power of the learned representations is evaluated on 1000 test nodes. Inductive learning on large graphs. We use a large graph dataset (231,443 nodes and 11,606,919 edges) of Reddit posts created during September 2014 (derived and preprocessed as in BID15). The objective is to predict the posts' community ("subreddit"), based on the GloVe embeddings of their content and comments BID31, as well as metrics such as score or number of comments. Posts are linked together in the graph if the same user has commented on both. Reusing the inductive setup of BID15, posts made in the first 20 days of the month are used for training, while the remaining posts are used for validation or testing and are invisible to the training algorithm. Inductive learning on multiple graphs. We make use of a protein-protein interaction (PPI) dataset that consists of graphs corresponding to different human tissues BID49. The dataset contains 20 graphs for training, 2 for validation and 2 for testing. Critically, testing graphs remain completely unobserved during training. To construct the graphs, we used the preprocessed data provided by BID15. Each node has 50 features that are composed of positional gene sets, motif gene sets and immunological signatures. There are 121 labels for each node set from gene ontology, collected from the Molecular Signatures Database BID37, and a node can possess several labels simultaneously. Visualizing discriminator scores. After obtaining the t-SNE visualizations, we turned our attention to the discriminator-and visualized the scores it attached to various nodes, for both the positive and a (randomly sampled) negative example FIG3. From here we can make an interesting observation-within the "clusters" of the learnt embeddings on the positive Cora graph, only a handful of "hot" nodes are selected to receive high discriminator scores. This suggests that there may be a clear distinction between embedding dimensions used for discrimination and classification, which we more thoroughly investigate in the next paragraph. In addition, we may observe that, as expected, the model is unable to find any strong structure within a negative example. Lastly, a few negative examples achieve high discriminator scores-a phenomenon caused by the existence of DISPLAYFORM0 Figure 6: Classification performance (in terms of test accuracy of logistic regression; left) and discriminator performance (in terms of number of poorly discriminated positive/negative examples; right) on the learnt DGI embeddings, after removing a certain number of dimensions from the embedding-either starting with most distinguishing (p ↑) or least distinguishing (p ↓).low-degree nodes in Cora (making the probability of a node ending up in an identical context it had in the positive graph non-negligible).Impact and role of embedding dimensions. Guided by the previous , we have visualized the embeddings for the top-scoring positive and negative examples (FIG4). The analysis revealed existence of distinct dimensions in which both the positive and negative examples are strongly biased. We hypothesize that, given the random shuffling, the average expected activation of a negative example is zero, and therefore strong biases are required to "push" the example down in the discriminator. The positive examples may then use the remaining dimensions to both counteract this bias and encode patch similarity. To substantiate this claim, we order the 512 dimensions based on how distinguishable the positive and negative examples are in them (using p-values obtained from a t-test as a proxy). We then remove these dimensions from the embedding, respecting this order-either starting from the most distinguishable (p ↑) or least distinguishable dimensions (p ↓)-monitoring how this affects both classification and discriminator performance (Figure 6). The observed trends largely support our hypothesis: if we start by removing the biased dimensions first (p ↓), the classification performance holds up for much longer (allowing us to remove over half of the embedding dimensions while remaining competitive to the supervised GCN), and the positive examples mostly remain correctly discriminated until well over half the dimensions are removed. Here, we consider alternatives to our corruption function, C, used to produce negative graphs. We generally find that, for the node classification task, DGI is stable and robust to different strategies. However, for learning graph features towards other kinds of tasks, the design of appropriate corruption strategies remains an area of open research. Our corruption function described in Section 4.2 preserves the original adjacency matrix (A = A) but corrupts the features, X, via row-wise shuffling of X. In this case, the negative graph is constrained to be isomorphic to the positive graph, which should not have to be mandatory. We can instead produce a negative graph by directly corrupting the adjacency matrix. Therefore, we first consider an alternative corruption function C which preserves the features (X = X) but instead adds or removes edges from the adjacency matrix (A = A). This is done by sampling, i.i.d., a switch parameter Σ ij, which determines whether to corrupt the adjacency matrix at position (i, j). Assuming a given corruption rate, ρ, we may define C as performing the following operations: DISPLAYFORM0 DISPLAYFORM1 where ⊕ is the XOR (exclusive OR) operation. This alternative strategy produces a negative graph with the same features, but different connectivity. Here, the corruption rate of ρ = 0 corresponds to an unchanged adjacency matrix (i.e. the positive and negative graphs are identical in this case). In this regime, learning is impossible for the discriminator, and the performance of DGI is in line with a randomly initialized DGI model. At higher rates of noise, however, DGI produces competitive embeddings. We also consider simultaneous feature shuffling (X = X) and adjacency matrix perturbation (A = A), both as described before. We find that DGI still learns useful features under this compound corruption strategy-as expected, given that feature shuffling is already equivalent to an (isomorphic) adjacency matrix perturbation. From both studies, we may observe that a certain lower bound on the positive graph perturbation rate is required to obtain competitive node embeddings for the classification task on Cora. Furthermore, the features learned for downstream node classification tasks are most powerful when the negative graph has similar levels of connectivity to the positive graph. The classification performance peaks when the graph is perturbed to a reasonably high level, but remains sparse; i.e. the mixing between the separate 1-step patches is not substantial, and therefore the pool of negative examples is still diverse enough. Classification performance is impacted only marginally at higher rates of corruption-corresponding to dense negative graphs, and thus a less rich negative example pool-but still considerably outperforming the unsupervised baselines we have considered. This could be seen as further motivation for relying solely on feature shuffling, without adjacency perturbations-given that feature shuffling is a trivial way to guarantee a diverse set of negative examples, without incurring significant computational costs per epoch. Classification accuracy for (X, A) corruption X = X GCN Figure 8: DGI is stable and robust under a corruption function that modifies both the feature matrix (X = X) and the adjacency matrix (A = A) on the Cora dataset. Corruption functions that preserve sparsity (ρ ≈ 1 N) perform the best. However, DGI still performs well even with large disruptions (where edges are added or removed with probabilities approaching 1). N.B. log scale used for ρ.

Represent the TLDR for finding the corresponding scientific article

A new method for unsupervised representation learning on graphs, relying on maximizing mutual information between local and global representations in a graph. State-of-the-art results, competitive with supervised learning.

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.