Question: Represent the scientific article to find a short TLDR

Question: The impressive lifelong learning in animal brains is primarily enabled by plastic changes in synaptic connectivity. Importantly, these changes are not passive, but are actively controlled by neuromodulation, which is itself under the control of the brain. The ing self-modifying abilities of the brain play an important role in learning and adaptation, and are a major basis for biological reinforcement learning. Here we show for the first time that artificial neural networks with such neuromodulated plasticity can be trained with gradient descent. Extending previous work on differentiable Hebbian plasticity, we propose a differentiable formulation for the neuromodulation of plasticity. We show that neuromodulated plasticity improves the performance of neural networks on both reinforcement learning and supervised learning tasks. In one task, neuromodulated plastic LSTMs with millions of parameters outperform standard LSTMs on a benchmark language modeling task (controlling for the number of parameters). We conclude that differentiable neuromodulation of plasticity offers a powerful new framework for training neural networks. Neural networks that deal with temporally extended tasks must be able to store traces of past events. Often this memory of past events is maintained by neural activity reverberating through recurrent connections; other methods for handling temporal information exist, including memory networks BID36 or temporal convolutions BID21. However, in nature, the primary basis for long-term learning and memory in the brain is synaptic plasticity -the automatic modification of synaptic weights as a function of ongoing activity BID16 BID14. Plasticity is what enables the brain to store information over the long-term about its environment that would be impossible or impractical for evolution to imprint directly into innate connectivity (e.g. things that are different within each life, such as the language one speaks).Importantly, these modifications are not a passive process, but are actively modulated on a momentto-moment basis by dedicated systems and mechanisms: the brain can "decide" where and when to modify its own connectivity, as a function of its inputs and computations. This neuromodulation of plasticity, which involves several chemicals (particularly dopamine; BID1 He et al. 2015; BID13 BID43, plays an important role in learning and adaptation BID22 BID32). By allowing the brain to control its own modification as a function of ongoing states and events, the neuromodulation of plasticity can filter out irrelevant events while selectively incorporating important information, combat catastrophic forgetting of previously acquired knowledge, and implement a self-contained reinforcement learning algorithm by altering its own connectivity in a reward-dependent manner BID31 BID24 BID7; BID19 BID4 BID38.The complex organization of neuromodulated plasticity is not accidental: it from a long process of evolutionary optimization. Evolution has not only designed the general connection pattern of the brain, but has also sculpted the machinery that controls neuromodulation, endowing the brain with carefully tuned self-modifying abilities and enabling efficient lifelong learning. In effect, this coupling of evolution and plasticity is a meta-learning process (the original and by far most powerful example of meta-learning), whereby a simple but powerful optimization process (evolution guided by natural selection) discovered how to arrange elementary building blocks to produce remarkably efficient learning agents. Taking inspiration from nature, several authors have shown that evolutionary algorithms can design small neural networks (on the order of hundreds of connections) with neuromodulated plasticity (see the "Related Work" section below). However, many of the spectacular recent advances in machine learning make use of gradient-based methods (which can directly translate error signals into weight gradients) rather than evolution (which has to discover the gradients through random weight-space exploration). If we could make plastic, neuromodulated networks amenable to gradient descent, we could leverage gradient-based methods for optimizing and studying neuromodulated plastic networks, expanding the abilities of current deep learning architectures to include these important biologically inspired self-modifying abilities. Here we build on the differentiable plasticity framework BID19 BID20 to implement differentiable neuromodulated plasticity. As a , for the first time to our knowledge, we are able to train neuromodulated plastic networks with gradient descent. We call our framework backpropamine in reference to its ability to emulate the effects of natural neuromodulators (like dopamine) in artificial neural networks trained by backpropagation. Our experimental establish that neuromodulated plastic networks outperform both non-plastic and non-modulated plastic networks, both on simple reinforcement learning tasks and on a complex language modeling task involving a multi-million parameter network. By showing that neuromodulated plasticity can be optimized through gradient descent, the backpropamine framework potentially provides more powerful types of neural networks, both recurrent and feedforward, for use in all the myriad domains in which neural networks have had tremendous impact. Neuromodulated plasticity has long been studied in evolutionary computation. Evolved networks with neuromodulated plasticity were shown to outperform both non-neuromodulated and non-plastic networks in various tasks (e.g. BID34 BID26 see Soltoggio et al. 2017 for a review). A key focus of neuromodulation in evolved networks is the mitigation of catastrophic forgetting, that is, allowing neural networks to learn new skills without overwriting previously learned skills. By activating plasticity only in the subset of neural weights relevant for the task currently being performed, knowledge stored in other weights about different tasks is left untouched, alleviating catastrophic forgetting BID4 BID38. However, evolved networks were historically relatively small and operated on low-dimensional problem spaces. The differentiable plasticity framework BID18 BID20 allows the plasticity of individual synaptic connections to be optimized by gradient descent, in the same way that standard synaptic weights are. However, while it could improve performance in some tasks over recurrence without plasticity, this method only facilitated passive, non-modulated plasticity, in which weight changes occur automatically as a function of pre-and post-synaptic activity. Here we extend this framework to implement differentiable neuromodulated plasticity, in which the plasticity of connections can be modulated moment-to-moment through a signal computed by the network. This extension allows the network itself to decide over its lifetime where and when to be plastic, endowing the network with true self-modifying abilities. There are other conceivable though more complex approaches for training self-modifying networks. For example, the weight modifications can themselves be computed by a neural network BID30 BID28 BID23 BID42. However, none so far have taken the simple approach of directly optimizing the neuromodulation of plasticity itself within a single network, through gradient descent instead of evolution, as investigated here. The present work builds upon the existing differentiable plasticity framework BID18 BID20, which allows gradient descent to optimize not just the weights, but also the plasticity of each connection. In this framework, each connection in the network is augmented with a Hebbian plastic component that grows and decays automatically as a of ongoing activity. In effect, each connection contains a fixed and a plastic component: DISPLAYFORM0 DISPLAYFORM1 where x i (t) is the output of neuron i at time t, σ is a nonlinearity (we use tanh in all experiments), w i,j is the baseline (non-plastic) weight of the connection between neurons i and j, and α i,j is the plasticity coefficient that scales the magnitude of the plastic component of the connection. The plastic content is represented by the Hebbian trace Hebb i,j, which accumulates the product of pre-and post-synaptic activity at connection i, j, as shown in Eq. 2.Hebb i,j is initialized to zero at the beginning of each episode/lifetime, and is updated automatically according to Eq. 2: it is a purely episodic/intra-life quantity. By contrast, w i,j, α i,j and η are the structural components of the network, which are optimized by gradient descent between episodes/lifetimes to minimize the expected loss over an episode. The function Clip(x) in Eq. 2 is any function or procedure that constrains Hebb i,j to the [−1, 1] range, to negate the inherent instability of Hebbian learning. In previous work BID20, this function was either a simple decay term, or a normalization implementing Oja's rule BID25. In the present paper it is simply a hard clip (x ← 1 if x > 1; x ← −1 if x < −1). Compared to previously used operations, this simple operation turned out to produce equal or superior performance on the tasks in this paper. Note the distinction between the η and α i,j parameters: η is the intra-life "learning rate" of plastic connections, which determines how fast new information is incorporated into the plastic component, while α i,j is a scale parameter, which determines the maximum magnitude of the plastic component (since Hebb i,j is constrained to the [-1,1] range).Importantly, in contrast to other approaches using uniform plasticity BID29, including "fast weights" BID0, the amount of plasticity in each connection (represented by α i,j) is trainable, allowing the meta-optimizer to design complex learning strategies (see BID20 for a discussion of this point, and experimental comparisons that demonstrate and explain superior performance of differentiable plasticity over uniform-plastic networks).An important aspect of differentiable plasticity is extreme ease of implementation: implementing a plastic recurrent network only requires less than four additional lines of code over a standard recurrent network implementation BID20 ). The Backpropamine framework described below inherits this simplicity; in particular, the "simple neuromodulation" approach does not require any additional code over differentiable plasticity, but merely a modification of it. Two methods are proposed to introduce neuromodulated plasticity within the differentiable plasticity framework. In both cases, plasticity is modulated on a moment-to-moment basis by a networkcontrolled neuromodulatory signal M (t). The computation of M (t) could be done in various ways; at present, it is simply a single scalar output of the network, which is used either directly (for the simple RL tasks) or passed through a meta-learned vector of weights (one for each connection, for the language modeling task). We now explain how the equations of differentiable plasticity are modified to make use of this neuromodulatory signal. The simplest way to introduce neuromodulation of plasticity in this framework is to make the (global) η parameter depend on the output of one or more neurons in the network. Because η essentially determines the rate of plastic change, placing it under network control allows the network to determine how plastic connections should be at any given time. Thus, the only modification to the equations above in this simple neuromodulation variant is to replace η in Eq. 2 with the network-computed, time-varying neuromodulatory signal M (t). That is, Eq. 2 is replaced with DISPLAYFORM0 More complex schemes are possible. In particular, we introduce an alternative neuromodulation scheme that takes inspiration from the short-term retroactive effects of neuromodulatory dopamine on Hebbian plasticity in animal brains. In several experiments, dopamine was shown to retroactively gate the plasticity induced by past activity, within a short time window of about 1s BID43; BID6 BID2. Thus, Hebbian plasticity does not directly modify the synaptic weights, but creates a fast-decaying "potential" weight change, which is only incorporated into the actual weights if the synapse receives dopamine within a short time window. As a , biological Hebbian traces essentially implement a so-called eligibility trace BID37, keeping memory of which synapses contributed to recent activity, while the dopamine signal modulates the transformation of these eligibility traces into actual plastic changes. Such mechanisms have been modelled in computational neuroscience studies, e.g. (; ; BID5 BID33 BID19) (see BID11 for a recent review of this concept).Our framework easily accommodates this more refined model of dopamine effects on plasticity. We simply replace Eq. 2 above with the two equations, DISPLAYFORM0 Here E i,j (t) (the eligibility trace at connection i, j) is a simple exponential average of the Hebbian product of pre-and post-synaptic activity, with trainable decay factor η. Hebb i,j (t), the actual plastic component of the connection (see Eq. 1), simply accumulates this trace, but gated by the current value of the dopamine signal M (t). Note that M (t) can be positive or negative, approximating the effects of both rises and dips in the baseline dopamine levels BID31. Our first test task is a simple meta-learning problem that emulates an animal behavioral learning task, as described in FIG2.1 (Left). In each episode, one of four input cues is arbitrarily chosen as the Target cue. Repeatedly, the agent is shown two cues in succession, randomly chosen from the possible four, then a Response cue during which the agent must respond 1 if the Target cue was part of the pair, or 0 otherwise. A correct response produces a reward of 1.0, while an incorrect response returns reward -1.0 (this is a two-alternative forced choice task: a response of either 1 or 0 is always produced). This process iterates for the duration of the episode, which is 200 time steps. The cues are binary vectors of 20 bits, randomly generated at the beginning of each episode. To prevent simple time-locked scheduling strategies, a variable number of zero-input time steps are randomly inserted, including at least one after each presentation of the Go cue; as a , the length of each trial varies, and the number of trials per episode is somewhat variable (the mean number of trials per episode is 15).The architecture is a simple recurrent network with 200 neurons in the hidden recurrent layer. Only the recurrent layer is plastic: input and output weights are non-plastic, having only w i,j coefficients. There are 24 inputs: 20 binary inputs for the current cue and one input providing the time elapsed since the start of the episode, as well as two binary inputs for the one-hot encoded response at the previous time step and one real-valued channel for the reward received at the previous time step, in accordance with common meta-learning practice BID39 BID3. There are four outputs: two binary outputs for the one-hot encoded response, plus an output neuron that predicts the sum of future discounted rewards V (t) over the remainder of the episode (as mandated by the A2C algorithm that we use for meta-training, following Wang et al. FORMULA0), and the neuromodulatory signal M (t). The two response outputs undergo a softmax operation to produce probabilities over the response, while the M (t) signal is passed through a tanh nonlinearity and the V (t) output is a pure linear output. All gradients are clipped at norm 7.0, which greatly improved stability. Training curves are shown in FIG2.1 (Right; each curve shows the median and inter-quartile range over 10 runs). Neuromodulatory approaches succeed in learning the task, while non-neuromodulatory networks BID18 BID20 ) and non-plastic, simple recurrent networks fail to learn it. We hypothesize that this dramatic difference is related to the relatively high dimensionality of the input cues: just as non-modulated plastic networks seemed to outperform non-plastic networks specifically when required to memorize arbitrary high-dimensional stimuli BID20, neuromodulation seems to specifically help memorizing reward associations with such arbitrary high-dimensional stimuli (see Appendix).To illustrate the behavior of neuromodulation, we plotted the output of the neuromodulator neuron for several trials. These graphs reveal that neuromodulation reacts to reward in a complex, time-dependent manner (see Appendix). For a more challenging problem, we also tested the approach on the grid maze exploration task introduced by BID20. Here, the maze is composed of 9 × 9 squares, surrounded by walls, in which every other square (in either direction) is occupied by a wall. Thus the maze contains 16 wall squares, arranged in a regular grid except for the center square (Figure 2, left). The shape of the maze is fixed and unchanging over the whole task. At each episode, one non-wall square is randomly chosen as the reward location. When the agent hits this location, it receives a reward and is immediately transported to a random location in the maze. Each episode lasts 200 time steps, during which the agent must accumulate as much reward as possible. The reward location is fixed within an episode and randomized across episodes. Note that the reward is invisible to the agent, and thus the agent only knows it has hit the reward location by the activation of the reward input at the next step (and possibly by the teleportation, if it can detect it).The architecture is the same as for the previous task, but with only 100 recurrent neurons. The outputs consist of 4 action channels (i.e. one for each of the possible actions: left, right, up or down) passed through a softmax, as well as the pure linear V (t) output and the M (t) neuromodulatory signal passed through a tanh nonlinearity. Inputs to the agent consist of a binary vector describing the 3 × 3 Figure 2: Maze navigation task. Left: layout of the maze, including an example agent location (yellow) and reward location (green, for illustration only: the reward is not visible to the agent). Right: Training curves for the maze exploration task: median and inter-quartile range of reward over 9 runs for each episode. Cyan stars (bottom) indicate statistically significant difference between simple neuromodulation and non-modulated plasticity at p < 0.05 (Wilcoxon rank-sum test).neighborhood centered on the agent (each element being set to 1 or 0 if the corresponding square is or is not a wall), plus four additional inputs for the one-hot encoded action taken at the previous time step, and one input for the reward received at the previous time step, following common practice BID39. Again, only recurrent weights are plastic: input-to-recurrent and recurrent-to-output weights are non-plastic. Results in Figure 2 show that modulatory approaches again outperform non-modulated plasticity. Word-level language modeling is a supervised learning sequence problem, where the goal is to predict the next word in a large language corpus. Language modeling requires storing long term context, and therefore LSTM models generally perform well on this task BID44. The goal of this experiment is to study the benefits of adding plasticity and neuromodulation to LSTMs. The Penn Tree Bank corpus (PTB), a well known benchmark for language modeling BID15, is used here for comparing different models. The dataset consists of 929k training words, 73k validation words, and 82k, test words, with a vocabulary of 10k words. For this task, we implemented neuromodulated plasticity in two different models: a basic model with 4.8 million parameters inspired from BID44, and a much larger and more complex model with 24.2 million parameters, forked from. The smaller model allowed for more experimentation, while the larger model showcases the of neuromodulation on a complex model with (at the time of writing) state-of-the-art performance. Detailed experimental descriptions are provided in the Appendix, summarized here: For the basic model, each network consists of an embedding layer, followed by two LSTM layers (approximately of size 200). The size of the LSTM layers is adjusted to ensure that the total number of trainable parameters remains constant across all experiments; note that this includes all plasticity-related additional parameters, i.e. α i,j as well as the additional parameters related to neuromodulation (see Appendix). The final layer is a softmax layer of size 10k. The network is unrolled for 20 time steps during backpropagation through time BID41. The norm of the gradient is clipped at 5. This setup is similar to the non-regularized model described by BID44. One difference is that an extra L2 penalty is added to the weights of the network here (adding this penalty consistently improves for all the models). FORMULA0 is that we do not implement recurrent dropout (feedforward dropout is preserved) and reduce batch size to 7 due to computational limitations. Other hyperparameters are taken "as is" without any tuning. See Appendix for details. Four versions of the smaller model are evaluated here TAB0. FORMULA0 The Baseline LSTM model (described in the previous paragraph) 1. LSTM with Differentiable Plasticity: there are four recurrent connections in each LSTM node and here, plasticity is added to one of them (see A.1 for details) as per equations 1 and 2. Because the number of plastic connections is large, each plastic connection has its own individual η so that their values can be individually tuned by backpropagation. LSTM with Simple Neuromodulation: here simple neuromodulation is introduced following equation 3. The η parameters are replaced by the output of a neuron M (t). M (t) itself receives as input a weighted combination of the hidden layer's activations, where the weights are learned in the usual way. There is one M (t) associated with each LSTM layer. LSTM with Retroactive Neuromodulation: this model is the same as the LSTM with Simple Neuromodulation, except it uses the equations that enable eligibility traces (equations 4 and 5). Additional details for the plastic and neuromodulated plastic LSTMs are described in the Appendix. For each of the four models, we separately searched for the best hyperparameters with equallypowered grid-search. Each model was then run 16 times with its best hyperparameter settings. The mean test perplexity of these 16 runs along with the 95% confidence interval is presented in TAB0. Results show that adding differentiable plasticity to LSTM provides slightly, but statistically significantly better than the Baseline LSTM (Wilcoxon rank-sum test, p = 0.0044). Adding neuromodulation further (and statistically significantly) lowers the perplexity over and above the LSTM with differential plasticity (p = 1e − 6). Overall, retroactive neuromodulation provides about 1.7 perplexity improvement vs. the Baseline LSTM (statistically significant, p = 1e − 7). Retroactive neuromodulation (i.e. with eligibility traces) does outperform simple neuromodulation, but the improvement is just barely not statistically significant at the traditional p < 0.05 cutoff (p = 0.066). Note that while these figures are far from state-of-the-art (which use considerably larger, more complex architectures), they all still outperform published work using similar architectures BID44.For the larger model, we compare a version in which the core LSTM module has been reimplemented to have neuromodulated plastic connections (simple neuromdulation only; no retroactive modulation was implemented), and a baseline model that uses the same LSTM reimplementation but without the plasticity and modulation, in order to make the comparison as equal as possible. Note that in this model, plasticity coefficients are attributed "per neuron": there is only one α i for each neuron i (as opposed to one per connection), which is applied to all the Hebbian traces of the connections incoming to this neuron. This helps limit the total number of parameter. See the Appendix for a more complete description. The modulated plastic model shows a small improvement over the non-plastic version TAB0, confirming the obtained with the smaller model. This paper introduces a biologically-inspired method for training networks to self-modify their weights. Building upon the differentiable plasticity framework, which already improved performance (sometimes dramatically) over non-plastic architectures on various supervised and RL tasks BID18 BID20, here we introduce neuromodulated plasticity to let the network control its own weight changes. As a , for the first time, neuromodulated plastic networks can be trained with gradient descent, opening up a new research direction into optimizing large-scale self-modifying neural networks. As a complement to the benefits in the simple RL domains investigated, our finding that plastic and neuromodulated LSTMs outperform standard LSTMs on a benchmark language modeling task (importantly, a central domain of application of LSTMs) is potentially of great importance. LSTMs are used in real-world applications with massive academic and economic impact. Therefore, if plasticity and neuromodulation consistently improve LSTM performance (for a fixed search space size), the potential benefits could be considerable. We intend to pursue this line of investigation and test plastic LSTMs (both neuromodulated and non) on other problems for which LSTMs are commonly used, such as forecasting. Conceptually, an important comparison point is the "Learning to Reinforcement Learn" (L2RL) framework introduced by BID39. In this meta-learning framework, the weights do not change during episodes: all within-episode learning occurs through updates to the activity state of the network. This framework is explicitly described BID40 as a model of the slow sculpting of prefrontal cortex by the reward-based dopamine system, an analogy facilitated by the features of the A2C algorithm used for meta-training (such as the use of a value signal and modulation of weight changes by a reward prediction error). As described in the RL experiments above, our approach adds more flexibility to this model by allowing the system to store state information with weight changes, in addition to hidden state changes. However, because our framework allows the network to update its own connectivity, we might potentially extend the L2RL model one level higher: rather than using A2C as a hand-designed reward-based weight-modification scheme, the system could now determine its own arbitrary weight-modification scheme, which might make use of any signal it can compute (reward predictions, surprise, saliency, etc.) This emergent weight-modifying algorithm (designed over many episodes/lifetimes by the "outer loop" meta-training algorithm) might in turn sculpt network connectivity to implement the meta-learning process described by BID40. Importantly, this additional level of learning (or "meta-meta-learning") is not just a pure flight of fancy: it has undoubtedly taken place in evolution. Because humans (and other animals) can perform meta-learning ("learning-to-learn") during their lifetime (; BID40, and because humans are themselves the of an optimization process (evolution), then meta-meta-learning has not only occurred, but may be the key to some of the most advanced human mental functions. Our framework opens the tantalizing possibility of studying this process, while allowing us to replace evolution with any gradient-based method in the outermost optimization loop. To investigate the full potential of our approach, the framework described above requires several improvements. These include: implementing multiple neuromodulatory signals (each with their own inputs and outputs), as seems to be the case in the brain BID12; BID27; introducing more complex tasks that could make full use of the flexibility of the framework, including the eligibility traces afforded by retroactive modulation and the several levels of learning mentioned above; and addressing the pitfalls in the implementation of reinforcement learning with reward-modulated Hebbian plasticity (e.g. the inherent interference between the unsupervised component of Hebbian learning and reward-based modifications; BID9 BID8, so as to facilitate the automatic design of efficient, selfcontained reinforcement learning systems. Finally, it might be necessary to allow the meta-training algorithm to design the overall architecture of the system, rather than simply the parameters of a fixed, hand-designed architecture. With such a rich potential for extension, our framework for neuromodulated plastic networks opens many avenues of exciting research. DISPLAYFORM0 j t, f t and o t are used for controlling the data-flow through the LSTM and i t is the actual data. Therefore, plasticity is introduced in the path that goes through i t (adding plasticity to the control paths of LSTM is for future-work). The corresponding pre-synaptic and post-synaptic activations (denoted by x i (t − 1) and x j (t) respectively in equations 1 and 2) are h t−1 and i t. A layer of size 200 has 40k (200×200) plastic connections. Each plastic connection has its own individual η (used in equation 2) that is learned through backpropagation. The plasticity coefficients (α i,j) are used as shown in equation 1. As shown in equation 3, for simple neuromodulation, the η is replaced by the output of a network computed neuron M (t). For neuromodulated LSTMs, individual η for each plastic connection is replaced by the output of a neuron (M (t)) that has a fan-out equal to the number of plastic connections. The input to this neuron is the activations h t−1 of the layer from the previous time-step. Each LSTM layer has its dedicated neuromodulatory neuron. Other variations of this setting include having one dedicated neuromodulatory neuron per node or having one neuromodulatory neuron for the whole network. Preliminary experiments showed that these variations performed worse and therefore they were not further evaluated. All the four models presented in TAB0 are trained using SGD. Initial learning rate was set 1.0. Each model is trained for 13 epochs. The hidden states of LSTM are initialized to zero; the final hidden states of the current minibatch are used as the initial hidden states of the subsequent minibatch. In addition to the previous model, we also applied the Backpropamine framework to the much larger, state-of-the-art model described by. This model consists of three stacked LSTMs with 115, 1150 and 400 cells each, with an input embedding of size 400 and an output softmax layer that shares weights with the input embedding. The model makes use of numerous optimization and regularization techniques. Connections between successive LSTMs implement "variational" dropout, in which a common dropout mask is used for the entire forward and backward pass BID10. Backpropagation through time uses a variable horizon centered on 70 words. After 45 epochs, the optimizer switches from SGD (without momentum) to Averaged-SGD, which consists in computing standard SGD steps but taking the average of the ing successive updated weight vectors. This is all in accordance with. The only differences are that we do not implement weight-dropout in recurrent connections, force the switch to ASGD at 45 epochs for all runs of all models, and limit batch size to 7 due to computational restrictions. Plasticity coefficients are attributed "per neuron": rather than having and independent α i,j for each connection, each neuron i has a plasticity coefficient α i that is applied to all its incoming connection (note that Hebbian traces Hebb i,j are still individually maintained for each connection). This reduces the number of trainable parameters, since α is now a vector of length N rather than a matrix of size N × N (where N is the number of recurrent neurons).We implement simple neuromodulation as described in Equation 3. A single neuromodulator neuron with tanh nonlinearity receives input from all recurrent neurons. This neuromodulator input is then passed through a vector of weights, one per neuron, to produce a different η i for each neuron. In other words, different neurons i have different η i, but these are all fixed multiples of a common value. This is an intermediate solution between having a single η(t) for the whole network, and independently computing a separate η i for each neuron, each with its own input weights (which would require N × N weights, rather than 2 × N for the current solution). Neuromodulation is computed separately to each of the three LSTMs in the model. For the non-plastic network, the total number of trainable parameters is 24 221 600. For the neuromodulated plastic version, we reduce the number of hidden cells in LSTMs from 1150 to 1149, which suffices to bring the total number of parameters down to 24 198 893 trainable parameters (rather than 24 229 703 for 1150-cell LSTMs).All other hyperparameters are taken from , using the instructions provided on the code repository for their model, available at https://github.com/salesforce/ awd-lstm-lm. We did not perform any hyperparameter tuning due to computational constraints. To illustrate the behavior of neuromodulation, we plot the output of the neuromodulator neuron for random trials from several runs of Task 1 (Figure 3). All runs are from well-trained, highly successful network, as seen by the low proportion of negative rewards. For each run, we plot both the value of the neuromodulator output at each time step, and the reward being currently perceived by the network (i.e. the one produced by the response at the previous time step).The plots reveal rich, complex dynamics that vary greatly between runs. The modulator neuron clearly reacts to reward; however, this reaction is complex, time-dependent and varies from run to run. The topmost run for retroactive modulation tends to produce negative neuromodulation in response to positive reward, and vice-versa; while the second-to-last run for simple neuromodulation tends to to the opposite. A common pattern is to produce negative neuromdulation on the time step just following reward perception (especially for simple neuromodulation). Two of the runs for retroactive modulation exhibit a pattern where reward perception is followed by highly positive, then highly negative neuromodulation. Understanding the mechanism by which these complex dynamics perform efficient within-episode learning is an important direction for future work. In the cue-reward association learning task described above, neuromodulated plasticity was able to learn a task that non-modulated plasticity simply could not. What might be the source of this difference? In a previous experiment, we implemented the same task, but using only four fixed 4-bit binary cues for the entire task, namely,'1000','0100','0010' and'0001'. In this simplified version of the task, there is no need to memorize the cues for each episode, and the only thing to be learned for each episode is which of the four known cues is associated with reward. This is in contrast with the version used in the paper above, in which the cues are arbitrary 20-bits vectors randomly generated for each episode. With the fixed, four-bit cues, non-modulated plasticity was able to learn the task, though somewhat more slowly than neuromodulated plasticity (see FIG2).This suggests neuromodulated plasticity could have a stronger advantage over non-modulated plasticity specifically in situations where the association to be learned involves arbitrary high-dimensional cues, which must be memorized jointly with the association itself. This echoes the of BID20, who suggest that plastic networks outperform non-plastic ones specifically on tasks requiring the fast memorization of high-dimensional inputs (e.g. image memorization and reconstruction task in BID20).Clearly, more work is needed to investigate which problems benefit most from neuromodulated plasticity, over non-modulated or non-plastic approaches. We intend to pursue this line of research in future work. Figure 3: Dynamics of neuromodulation. For both simple and retroactive modulation, we show one trial from each of 4 runs. At each time step, gray squares indicate which cue was shown; orange curves indicates reward at the previous time step (i.e. the one currently fed to the network), which is always -1, 0 or 1; and blue curves indicates the value of the modulator output at the current time step. Notice the diversity of dynamics. Training curves for the cue-reward association task with fixed, binary four-bit cues (medians and inter-quartile ranges of rewards per episode over 10 runs). " Soft clip" refers to a different clipping operation used in Equation 2; "Hard clip" is the same as used in the present paper, i.e. the simple clipping described in Methods. Note that non-modulated plastic network succeed in solving this task.

represent the tldr for finding the corresponding scientific article.
------
For example, A new form of attention that works well for the distant supervision setting, and a multitask learning approach to add sentence-level annotations.  should be similar to We propose an effective multitask learning setup for reducing distant supervision noise by leveraging sentence-level supervision. We show how sentence-level supervision can be used to improve the encoding of individual sentences, and to learn which input sentences are more likely to express the relationship between a pair of entities. We also introduce a novel neural architecture for collecting signals from multiple input sentences, which combines the benefits of attention and maxpooling. The proposed method increases AUC by 10% (from 0.261 to 0.284), and outperforms recently published on the FB-NYT dataset. Early work in relation extraction from text used fully supervised methods, e.g., BID2, which motivated the development of relatively small datasets with sentence-level annotations such as ACE 2004 BID2, BioInfer and SemEval 2010. Recognizing the difficulty of annotating text with relations, especially when the number of relation types of interest is large, BID16 pioneered the distant supervision approach to relation extraction, where a knowledge base (KB) and a text corpus are used to automatically generate a large dataset of labeled sentences which is then used to train a relation classifier. Distant supervision provides a practical alternative to manual annotations, but introduces many noisy examples. Although many methods have been proposed to reduce the noise in distantly supervised models for relation extraction (e.g., BID8 BID23 BID22 BID5 BID27 BID11, a rather obvious approach has been understudied: using sentence-level supervision to augment distant supervision. Intuitively, supervision at the sentence-level can help reduce the noise in distantly supervised models by identifying which of the input sentences for a given pair of entities are likely to express a relation. We experiment with a variety of model architectures to combine sentence-and bag-level supervision and find it most effective to use the sentence-level annotations to directly supervise the sentence encoder component of the model in a multi-task learning framework. We also introduce a novel maxpooling attention architecture for combining the evidence provided by different sentences where the entity pair is mentioned, and use the sentence-level annotations to supervise attention weights. The contributions of this paper are as follows:• We propose an effective multitask learning setup for reducing distant supervision noise by leveraging existing datasets of relations annotated at the sentence level.• We propose maxpooled attention, a neural architecture which combines the benefits of maxpooling and soft attention, and show that it helps the model combine information about a pair of entities from multiple sentences.• We release our library for relation extraction as open source. 1 The following section defines the notation we use, describes the problem and provides an overview of our approach. Our goal is to predict which relation types are expressed between a pair of entities (e 1, e 2), given 1 We attach the anonymized code as supplemental material in this submission instead of providing a github link in order to maintain author anonymity. Figure 1: An overview of our approach for augmenting distant supervision with sentence-level annotations. The left side shows one sentence in the labeled data and how it is used to provide direct supervision for the sentence encoder. The right side shows snippets of the text corpus and the knowledge base, which are then combined to construct one training instance for the model, with a bag of three input sentences and two active relations:'founder of' and'ceo of'.all sentences in which both entities are mentioned in a large collection of unlabeled documents. Following previous work on distant supervision, we use known tuples (e 1, r, e 2) in a knowledge base K to automatically annotate sentences where both entities are mentioned. In particular, we group all sentences s with one or more mentions of an entity pair e 1 and e 2 into a bag of sentences B e 1,e 2, then automatically annotate this bag with the set of relation types L distant = {r ∈ R : (e 1, r, e 2) ∈ K}, where R is the set of relations we are interested in. We use'positive instances' to refer to cases where |L| > 0, and'negative instances' when |L| = 0.In this paper, we leverage existing datasets with sentence-level relation annotations in a similar domain, where each example consists of a token sequence s, token indexes for e 1 and e 2 in the sequence, and one relation type (or 'no relation'). Since the relation types annotated at the sentence level may not correspond one-to-one to those in the KB, we replace the relation label associated with each sentence with a binary indicator. (1 indicates that the sentence s expresses one of the relationships of interest.) We do not require the entities to match those in the KB either. Fig. 1 illustrates how we modify neural architectures commonly used in distant supervision, e.g., BID13 to effectively incorporate sentence-level supervision. The model consists of two components: 1) A sentence encoder (displayed in blue) reads a sequence of tokens and their relative distances from e 1 and e 2, and outputs a vector s representing the sentence encoding, as well as P (e 1 ∼ e 2 | s) representing the probability that the two entities are related given this sentence. 2) The bag encoder (displayed in green) reads the encoding of each sentence in the bag for the pair (e 1, e 2) and predicts P (r = 1 | e 1, e 2), ∀r ∈ R. We combine both bag-level (i.e., distant) and sentence-level (i.e., direct) supervision in a multi-task learning framework by minimizing the weighted sum of the cross entropy losses for P (e 1 ∼ e 2 | s) and P (r = 1 | e 1, e 2). By sharing the parameters of sentence encoders used to compute either loss, the sentence encoders become less susceptible to the noisy bag labels. The bag encoder also benefits from sentence-level supervision by using the supervised distribution P (e 1 ∼ e 2 | s) to decide the weight of each sentence in the bag, using a novel architecture which we call maxpooled attention. The model predicts a set of relation types L pred ⊂ R given a pair of entities e 1, e 2 and a bag of sentences B e 1,e 2. In this section, we first describe the sentence encoder part of the model FIG2, then describe the bag encoder FIG2, then we explain how the two types of supervision are jointly used for training the model end-to-end. Given a sequence of words w 1,..., w |s| in a sentence s, a sentence encoder translates this sequence into a fixed length vector s. Input Representation. The input representation is illustrated graphically with a table at the bottom of FIG2. We map word token i in the sentence w i to a pretrained word embedding vector w i. 2 Another crucial input signal is the position of entity mentions in each sentence s ∈ B e 1,e 2. Following BID28, we map the distance between each word in the sentence and the entity mentions 3 to a small vector of learned parameters, namely d i. Instead of randomly initializing position embeddings with mean = 0, we obtain notable performance improvements by randomly initializing all dimensions of the position embedding for distance d around the mean value d. Intuitively, this makes it easier to learn useful parameters since the embedding of similar distances (e.g., d = 10 and d = 11) should be similar, without adding hard constraints on how they should be related. We find that adding a dropout layer with a small probability (p = 0.1) before the sentence encoder reduces overfitting and improves the . To summarize, the input layer for a sentence s is a sequence of vectors: DISPLAYFORM0 Word Composition. Word composition is illustrated with the block CNN in the bottom part of FIG2, which represents a convolutional neural network (CNN) with multiple filter sizes. The outputs of the maxpool operations for different filter sizes are concatenated then projected into a smaller vector using one feed forward linear layer. This is in contrast to previous work BID19 which used Piecewise CNN (PCNN). In PCNN, we convolve three segments of the sentence separately: windows before the left entity, windows inbetween the two entities and windows after the right entity. Every split is maxpooled independently, then the three vectors are concatenated. The intuition is that this helps the model put more emphasis on the middle segment which: Blue box is the sentence encoder, it maps a sentence to a fixed length vector (CNN output) and the probability it expresses a relation between e 1 and e 2 (sigmoid output). Green box is the bag encoder, it takes encoded sentences and their weights and produces a fixed length vector (maxpool output), concatenates it with entity embeddings (pointwise mult. output) then outputs a probability for each relation type r. White boxes contain parameters that the model learns while gray boxes do not have learnable parameters. Sentence-level annotations supervise P (e 1 ∼ e 2 | s). Bag-level annotations supervise P (r = 1 | e 1, e 2).connects the two entities. As discussed later in Section 4.2, we compare CNN and PCNN and find the simpler CNN architecture works better. Sentence encoding s is computed as follows: DISPLAYFORM1 where CNN x is a standard convolutional neural network with filter size x, W 1 and b 1 are model parameters and s is the sentence encoding. We feed the sentence encoding s into a ReLU layer followed by a sigmoid layer with output size 1, representing P (e 1 ∼ e 2 | s), as illustrated in DISPLAYFORM2 where σ is the sigmoid function and W 2, b 2, W 3, b 3 are model parameters. Given a bag B e 1,e 2 of n ≥ 1 sentences, we compute their encodings s 1,..., s n as described earlier and feed them into the bag encoder, which is responsible for combining the information in all sentence encodings and predict the probability P (r = 1 | e 1, e 2), ∀r ∈ R. The bag encoder also makes use of p = P (e 1 ∼ e 2 | s) from Eq. 1 as an estimate of the degree to which sentence s expresses the relation between e 1 and e 2.Maxpooled Attention. To aggregate the sentence encodings s 1,..., s n into a fixed length vector that captures the important features in the bag, BID11 used maxpooling, while BID13 used soft attention. In this work, we propose maxpooled attention, a new form of attention which combines some of the characteristics of maxpooling and soft attention. Given the encoding s j and an unnormalized weight u j for each sentence s j ∈ B e 1,e 2, the bag encoding g is a vector with the same dimensionality as s j with the k-th element computed as: DISPLAYFORM0 Maxpooled attention has the same intuition of soft attention; learning weights for sentences that enable the model to focus on the important sentences. However, maxpooled attention differs from soft attention in two aspects. The first is that every sentence s j is given a probability that indicates how useful the sentence is, independently of the other sentences. Notice how this is different from soft attention where sentences compete for probability mass, i.e., probabilities must sum to 1. This is implemented in maxpooled attention by normalizing the weight of each sentence with a sigmoid function rather than a softmax. This is a better fit for the task at hand because the sentences are not competing. It also makes the weights useful even when |B e 1,e 2 | = 1, while soft attention will always normalize such weights to 1.The second difference between maxpooled attention and soft attention is the use of weighted maxpooling instead of weighted average. Maxpooling is more effective for this task because it can pick the useful features from different sentences. As shown in FIG2, we do not directly use the p from Eq. 1 as weight in maxpooled attention. Instead, we found it useful to feed it into more non-linearities. The unnormalized maxpooled attention weight for s j is computed as: DISPLAYFORM1 Entity Embeddings. , we use entity embeddings to improve our model of relations in the distant supervision setting, although our formulation is closer to that of BID25 who used point-wise multiplication of entity embeddings: m = e 1 e 2, where is point-wise multiplication, and e 1 and e 2 are the embeddings of e 1 and e 2, respectively. In order to improve the coverage of entity embeddings, we use pretrained GloVe vectors BID19 (same embeddings used in the input layer).For entities with multiple words, like "Steve Jobs", the vector for the entity is the average of the GloVe vectors of its individual words. If the entity is expressed differently across sentences, we average the vectors of the different mentions. As discussed in Section 4.2, this leads to big improvement in the , and we believe there's still big room for improvement from having better representation for entities. We feed the output m as additional input to the last block of our model. Output Layer. The final step is to use the bag encoding g and the entity pair encoding m to predict a set of relations L pred which is a standard multilabel classification problem. We concatenate g and m and feed them into a feedforward layer with ReLU non-linearity, followed by a sigmoid layer with an output size of |R|: DISPLAYFORM2 where r is a vector of Bernoulli variables each of which corresponds to one of the relations in R. This is the final output of the model. To train the model on the bag-level labels obtained with distant supervision, we use binary crossentropy loss between the model predictions and the labels obtained with distant supervision, i.e., bag loss = DISPLAYFORM0 where r distant [k] = 1 indicates that the tuple (e 1, r k, e 2) is in the knowledge base. In addition to the bag-level supervision commonly used in distant supervision, we also use sentence-level annotations. One approach is to create a bag of size 1 for each sentence-level annotation, and add the bags to those obtained with distant supervision. However, this approach requires mapping relations in the sentence-level annotations map to those in the KB.Instead, we found that the best use of the supervised data is to improve the model's ability to predict the the potential usefulness of a sentence by using sentence-level annotations to help supervise the sentence encoder module. According to our analysis of baseline models, distinguishing between positive and negative examples is the real bottleneck. This supervision serves two purposes: it improves our encoding of each sentence, and improves the weights used by the maxpooled attention to decide which sentences should contribute more to the bag encoding. We maximize log loss of gold labels in the sentence-level data D according to the model described in Eq. 1: DISPLAYFORM1 where D consists of all the sentence-level annotations in addition to all distantly-supervised negative examples. 4 We jointly train the model on both types of supervision. The model loss is a weighted sum of the sentence-level and the bag-level losses: DISPLAYFORM2 where λ is a parameter that controls the contribution of each loss, tuned on a validation set. This section discusses datasets, metrics, experiment configurations and the models we are comparing with. Metrics. Prior works used precision-recall (PR) curves to show their on the FB-NYT dataset. In this multilabel classification setting, all model predictions for all relation types are sorted by confidence from highest to lowest. Then applying different thresholds gives the points on the PR curve. We use the area under the PR curve (AUC) for early stopping and hyperparameter tuning. Because we are interested in the high-precision extractions, we focus on the high-precision lowrecall part of the PR curve. That is, in our experiments, we only keep points on the PR curve with recall below 0.4 which means that the largest possible value for AUC is 0.4.Configurations. The FB-NYT dataset does not have a validation set for hyper-parameter tuning and early stopping. For these, use the test set, and BID13 use 3-fold cross validation. We use 90% of the training set for training and keep the other 10% for validation. The pretrained word embeddings we use are 300-dimensional GloVe vectors, trained on 42B tokens. Since we do not update word embeddings 6 while training the model, our vocabulary may include any word which appears in the training, validation or test sets with frequency greater than two. When a word with a hyphen (e.g., 'five-star') is not in the GloVe vocabulary, we average the embeddings of its subcomponents. Otherwise, all OOV words are assigned the same random vector (normal with mean 0 and standard deviation 0.05).Our model is implemented using PyTorch and AllenNLP BID6 and trained on machines with P100 GPUs. Each run takes five hours on average. We train for a large number of epochs and use early stopping with patience = 3. The batches of the two datasets are randomly shuffled before every epoch. The optimizer we use is Adam with its default PyTorch parameters. We run every configuration three times with three different random seeds then report the PR curve for the run with the best validation (thresholded) AUC. In the controlled experiments, we report the mean and standard deviation of the AUC metric. Compared Models. Our baseline for comparison is a model that is similar to what is described in Section 3 with the following configurations. It uses our approach for position embedding initialization, encodes sentences using CNN, uses entity embeddings, aggregate sentences using maxpooling and does not use the sentence-level annotation. Our best configuration adds the maxpooled attention and the sentence-level annotations. We also compare with existing models in the literature. The model by BID13 uses an attention mechanism that assigns weights to each sentence followed by a weighted average of sentence encodings. The model by extends the model by BID13 by using soft labels during training. Main Result. FIG3 summarizes the main of our experiments. 8 The AUC of our baseline (green) is comparable to that of (blue), which verifies that we are building on a strong baseline. Adding maxpooled attention and sentence-level supervision (i.e., the full model, in red) substantially improves over the baseline (green). The figure also illustrates that our full model outperforms the strong baseline 8 Results of BID13 and are copied from their papers. 0.271 ± 0.007 + additional bags 0.269 ± 0.001 + sentence loss 0.284 ± 0.007 Table 1: The + and − signs indicate independent changes to the baseline configuration.of in orange. 9 We emphasize that the improved reported here conflate both additional supervision and model improvements. Next, we report the of controlled experiments to quantify the contribution of each modification in Table 1. The first line in the table is the baseline model and configuration described in the previous section and in FIG3, and the + and − signs indicate (independent) additions to and removals from that configuration, respectively. Position Embedding Initialization. The second line in Table 1 shows that removing the distance-based initialization of position embeddings in a large drop in AUC. We hypothesize that the position-based initialization reduces the burden of finding optimal values for position embeddings, without explicit constraints that guarantee similar distances to have similar embeddings. Sentence Encoder. In the next line of Table 1, we replace the simpler CNN in our baseline with the more complex PCNN BID27. Both encoders use filters of sizes 2, 3, 4 and 5. Table 1 shows that using CNN works markedly better than PCNN which is in contrast to the findings of BID27. This could be due to the use of multiple filter sizes and to the improved representation of entity positions in our model, which may obviate the need to have a separate encoding of each segment in the sentence. Entity Embeddings. The next line in Table 1 shows that entity embeddings (which are included in the baseline model) provide valuable information and help predict relations. This information may encode entity type, entity compatibility with each others and entity bias to participate in a relation. Given that our entity embeddings are simple GloVe vectors, we believe there is still a large room for improvement. We compare different ways of aggregating sentences into a single vector including maxpooling (baseline, originally proposed in BID11), attention BID13 and our proposed maxpooled attention. 10 Maxpooling works better than soft attention because it is better at picking out useful features from multiple sentences, while attention can only weight the whole representation of the sentence. We hypothesize that our proposed maxpooled attention works better than both because it combines the soft attention's ability to learn and use different weights for different sentences, and the maxpool's ability to pick out useful features from multiple sentence. Another advantage of maxpooled attention over attention is that it helps in cases where bag size equals 1 because the softmax typically used in attention in a weight of 1 for the sentence rendering that weight useless. The last three lines in Table 1 compare different ways for using sentence-level annotations. The line "baseline + maxpooled att." is copied from the pre-10 Our reimplementation of BID13 attention differs from what was described in the paper. The unnormalized attention weights of BID13 are oj = sj × A × q where sj is the sentence encoding, A is a diagonal matrix and q is the query vector. We tried this but found that implementing it as a feedforward layer with output size = 1 works better. The in Table 1 are for the feedforward implementation. vious line and is the basis for the following two lines. In "additional bags," we add the sentencelevel annotations as additional bags along with the distantly supervised data. In "sentence loss," we use the method described in Section 3 for integrating sentence-level supervision. The show that simply adding the sentence-level supervised data to the distantly supervised data as additional bags has little effect on the performance. This is probably because they change the distribution of the training to differ from the test set. However, adding the sentence-level supervision following our proposed multitask learning improves the considerably because it allows the model to better filter noisy sentences. Selecting Lambda. Although we did not spend much time tuning hyperparameters, we made sure to carefully tune λ (Equation 3) which balances the contribution of the two losses. Early experiments showed that sentence-level loss is typically smaller than bag-level loss, so we experimented with λ ∈ {0, 0.5, 1, 2, 4, 8, 16, 32, 64}. FIG4 shows thresholded AUC for different values of λ, where each point is the average of three runs. It is clear that picking the right value for λ has a big impact on the final . Qualitative Analysis. An example of a positive bag is shown in TAB3. Our model, which incorporates sentence-level supervision, assigns the most weight to the first sentence while the attention model assigns the most weight to the last sentence (which is less informative for the relation between the two entities). Furthermore, the attention model does not use the other two sentences because their weights are dominated by the weight of the last sentence. We also found that the weights from our model usually range between 0 and 0.08, suggesting the relative values of the weights are informative to the model, even when the absolute values are small. Distant Supervision. The term'distant supervision' was coined by BID16 who used relation instances in a KB to identify any sentence in a text corpus where two related entities are mentioned, then developed a classifier to predict the relation. Researchers have since extended this approach for relation extraction (e.g., BID24 BID15 BID21 BID12 . A key source of noise in distant supervision is that sentences may mention two related entities without expressing the relation between them. BID8 used multi-instance learning to address this problem by developing a graphical model for each entity pair which includes a latent variable for each sentence to explicitly indicate the relation expressed by that sentence, if any. Our model can be viewed as an extension of BID8 where the sentence-bound latent variables can also be directly supervised in some of the training examples. Neural Models for Distant Supervision. More recently, neural models have been effectively used to model textual relations (e.g., BID7 BID28 BID17 . Focusing on distantly supervised models, BID27 proposed a neural implementation of multi-instance learning to leverage multiple sentences which mention an entity pair in distantly supervised relation extraction. However, their model picks only one sentence to represent an entity pair, which wastes the information in the neglected sentences. BID11 addresses this limitation by maxpooling the vector encodings of all input sentences for a given entity pair. BID13 independently proposed to use attention to address the same limitation. Results in Section 4.2 suggest that maxpooling is more effective than attention for multi-instance learning. BID26 proposed a method for leveraging dependencies between different relations in a pairwise ranking framework. Sentence-Level Supervision. Despite the substantial amount of work on both fully supervised and distantly supervised relation extraction, the question of how to combine both signals has been mostly ignored in the literature, with a few exceptions. first manually defined a mapping between relation types in YAGO to compatible relation types in ACE 2004 BID4, then trained two separate SVM models using the training portion of ACE 2004 and the distantly supervised sentences. Model predictions are then linearly combined to make the final prediction. In contrast, we use a neural model which combines both sources of supervision in a multi-task learning framework BID3 . We also do not require a strict mapping between the relation types of the KB and those annotated at the sentence level. Another important distinction is the unit of prediction (at the sentence level vs. at the entity pair level), each of which has important practical applications. Also related is BID0 who used active learning to improve the multi-instance multi-label model of BID23. We propose two complementary methods to improve performance and reduce noise in distantly supervised relation extraction. The first is incorporating sentence-level supervision and the second is maxpooled attention, a novel form of attention. The sentence-level supervision improves sentence encoding and provides supervision for attention weights, while maxpooled attention effectively combines sentence encodings and their weights into a bag encoding. Our experiments show a 10% improvement in AUC (from 0.261 to 0.284) outperforming recently published on the FB-NYT dataset.

Neural networks can be trained to modify their own connectivity, improving their online learning performance on challenging tasks.

Represent the short summary for finding the corresponding scientific article (long)

We propose a continual version of Equilibrium Propagation, where neuron and synapse dynamics occur simultaneously throughout the second phase, with theoretical guarantees and numerical simulations.


Question: represent the scientific article to find a short tldr

Question: Deep learning has made remarkable achievement in many fields. However, learning the parameters of neural networks usually demands a large amount of labeled data. The algorithms of deep learning, therefore, encounter difficulties when applied to supervised learning where only little data are available. This specific task is called few-shot learning. To address it, we propose a novel algorithm for fewshot learning using discrete geometry, in the sense that the samples in a class are modeled as a reduced simplex. The volume of the simplex is used for the measurement of class scatter. During testing, combined with the test sample and the points in the class, a new simplex is formed. Then the similarity between the test sample and the class can be quantized with the ratio of volumes of the new simplex to the original class simplex. Moreover, we present an approach to constructing simplices using local regions of feature maps yielded by convolutional neural networks. Experiments on Omniglot and miniImageNet verify the effectiveness of our simplex algorithm on few-shot learning. Deep learning has exhibited outstanding ability in various disciplines including computer vision, natural language processing and speech recognition BID10. For instance, AlexNet has made a breakthrough on recognizing millions of imagery objects by means of deep Convolutional Neural Network (CNN) BID8. In the past five years, the algorithmic capability of comprehending visual concepts has been significantly improved by elaborately well-designed deep learning architectures BID4 BID22. However, training deep neural networks such as the widely employed CNNs of AlexNet BID8, Inception BID23, VGG BID20, and ResNet BID5, needs the supervision of many class labels which are handcrafted. For example, the number of samples of each class in the ImageNet of object recognition benchmark BID17 is more than one thousand. In fact, the number of labelled samples used for learning parameters of CNNs is far more than that because data augmentation is usually applied. This kind of learning obviously deviates from the manner of human cognition. A child can recognize a new object that she/he has never seen only by several examples, from simple shapes like rectangles to highly semantic animals like tigers. However, deep learning algorithms encounter difficulty in such scenarios where only very sparse data are available for learning to recognize a new category, thus raising the research topic of one-shot learning or few-shot learning BID1 BID25.The seminal work BID2 models few-shot learning with the Bayesian framework. Empirical knowledge of available categories is learned and parameterized as a probability density function. The unseen class with a handful of examples is modeled as the posterior by updating the prior. Bayesian theory provides a simple and elegant idea for solving learning problems with little data. If decomposed into parts or programs, an object can be described by the joint distribution of Bayesian criterion. In this manner, human-level performance on one-shot learning has been derived for discovering simple visual concepts such as ancient handwritten characters BID9.With the prevalence of deep learning, the recent work for few-shot learning focuses on the application of deep neural networks that have more capacity to accommodate the complexity of object representations. Siamese neural network facilitates the performance of few-shot recognition by means of twin networks of sharing parameters, optimizing the distances of representative features in intraclasses BID7. The counterpart of learning data structures by distance is also formulated by triplet loss in BID11. Researchers in BID11 assert that the distance metrics can learn the intrinsic manifold structures of training data such that the network is more general and robust when employed for untrained objects. A very recent work pertaining to distance-based optimization, named Prototypical Networks BID21, significantly improves the capability of few-shot recognition. Prototypical Networks attempt to minimize the distance of the test sample to the center of each class and are learned in the end-to-end manner. Memory-augmented architectures are also proposed to help assimilate new classes with more accurate inference BID18. Matching network embeds metric learning in neural network in the light of attention mechanism which is embodied by softmax BID26. In a very recent work, the large-scale memory without the need of resetting during training is formulated as an embedded module for arbitrary neural networks to remember the information of rare events BID6. In order to obtain rapid learning with limited samples, meta learning is exploited both in memory network and matching network. This "learning to learn" technique is extended to deal with few-shot learning from the point of view of optimization BID15. To be specific, a LSTM-based meta learner learns to mimic the exact optimization algorithm and then harnesses the acquired capability to train the learner applied for the few-shot cases. The latest meta learning algorithms also deal with few-shot learning from different angles, e.g. the fast adaptation of neural networks BID3, and temporal convolution BID13.In addition to the application of memory module or attention model in LSTM, there is another type of algorithms digging the effective way of transferring the discriminative power of pre-trained models to few-shot circumstances. Resorting to the correlation between the activations in the last feature layers and the associated parameters for softmax, a transformation is learned to derive the parameters for predicting new classes from corresponding activations BID14.The algorithms based on deep learning can learn more expressive representations for objects, essentially boosting the quality of feature extraction. However, the softmax classifier discriminates all categories by class boundaries, bypassing the steps that carefully characterize the structure of each class. Thus the algorithmic performance will deteriorate grossly if the distribution of new class cannot be accurately modeled by trained networks. Besides softmax, another commonly applied method, k nearest neighbors (KNN), is a point-to-point measurement and is incapable of conveying global structural information. To address this issue, we propose a geometric method for few-shot learning. Our perspective is that accurate geometric characterization for each class is essential when only a handful of samples are available, because such sparse data are usually insufficient to fit well-converged parameterized classifier. To this end, we harness convex polytope to fit a class, in the sense that we construct a convex polytope by selecting the samples in the class as the vertices of the polytope. The volume of the polytope is taken as the measurement of class scatter. Thus the polytopal volume may be improved after including the query sample in the test set during the testing trial. The normalized volume with respect to the original counterpart is applied to compute the distance from the test sample to the test set. To highlight the structural details of object parts, we present the construction of polytope based on convolutional feature maps as well. To the best of our understanding, however, there is no exact formula to calculating the volume of general convex polytope. To make our algorithm feasible, therefore, we use the simplest convex polytope -simplex instead. The volume of a simplex can be expressed by the Cayley-Menger determinant BID0, thus casting the problem of few-shot recognition as a simple calculation of linear algebra. Experiments on Omniglot and miniImageNet datasets verify the effectiveness of our simple algorithm. It is known that by proper embedding, the feature representations of images or documents in the same class spatially cluster together. Each feature vector with a length of d, in its nature, corresponds to a point in d-dimensional space. We model each class as a polytope, with feature representation of each data point in this class as its vertex. Our approach is based on the idea that feature vectors of the test sample will be close to the polytope of its own class, formed by feature vectors of the corresponding examples, and distant from the others. Hence, we can perform the classification by finding the'nearest' class polytope to which the test point belongs, using carefully designed distance metrics associated with the content of polytope. As we point out in the introduction, there is no exact solution to computing the volume of a polytope. Therefore, we resort to the simplex to accomplish our idea. The simplex is the simplest counterpart of convex polytope and its volume admits a closed form expression. So we focus our attention on simplex to develop our algorithm. A simplex is the conceptual extension of a triangle in high-dimensional spaces. To be formal, let Y = {y 0, y 1, . . ., y n} denote a set of points in R d. A simplex is the convex polytope with the condition of n = d, implying that there needs exact d + 1 points to constitute a simplex in the d-dimensional space. For convenience, we call such a simplex the d-simplex. For instance, a line is a 1-simplex, a triangle is a 2-simplex, and a tetrahedron is a 3-simplex. Moreover, a line has the length, a triangle has the area, and a tetrahedron has the volume. In convention, we use the contents to represent the length, the area, and the volume BID27.A particularly intriguing property of the simplex is that its content can be written in a closed form by virtue of the Cayley-Menger determinant BID0 ). To show this, let A = [y 1 −y 0, . . ., y n −y 0] and the Cayley-Menger matrixP DISPLAYFORM0 where e denotes the all-one column vector of length n + 1, T presents the transpose of a matrix or a vector, and the entry P ij of the distance matrix P is of form P ij = y i − y j 2. The content of simplexŶ has two expressions that coincide with each other, showing that DISPLAYFORM1 where det(A T A) is the Gram determinant and det(P) is the Cayley-Menger determinant. Our analysis is based on the application of formula. Let Y = {y 1, . . ., y n} be the feature set of an arbitrary class. These features can be derived from outputs of deep neural networks, e.g., CNN. It is clear thatŶ = {Y ∪y 0}. Let t denote a test sample. It is clear the content C(Y) of the corresponding simplex 1 will be large if data points in Y are sparse and small if compact. Therefore, C(Y) is a plausible measurement for the class scatter. An exact d-simplex will be formed during testing process if the test sample is merged into Y. Then the associated content will be improved from C(Y) to C(Y ∪ t). The incremental content will be marginal if the feature point of the test sample is close to the class simplex, meaning the high correlation of the spatial proximity. Then the dissimilarity measurement of one test sample to one class can be written as DISPLAYFORM0 Here the numerator C(Y) serves to eliminating the quantitative influence of the class scatter. The normalization is indispensable because for a large C(Y), the incremental content C(Y ∪ t) − C(Y) will be prone to be relatively large even if the test sample is close to the simplex. To make it clear, we explicitly write (t, Y). Let the Cayley-Menger matrix pertaining to simplex Y beQ With the same embedding function, the test sample point t is also mapped to this space. Combined with one class, a new simplex (tetrahedron in the 3-dimensional space) is obtained, located in the space of one dimension higher than the original one. It is not surprising that the test sample point t is close to its own class while distant from the others, representing a certain degree of intra-class similarities and inter-class diversities. DISPLAYFORM1 where Q ij = y i − y j 2 and i, j = 1,..., n. Then the content of the simplex formed by Y can be written as DISPLAYFORM2 Substituting FORMULA1 and FORMULA4 into FORMULA2, we derive DISPLAYFORM3 where y 0 = t is implicitly assumed forP. It is straightforward to know that the smaller (t, Y) is, the closer the test point to the class simplex. To help intuitively understand our algorithm, a visual schematic example is shown in FIG0, where the complete procedure is figuratively demonstrated. It is obvious that Y cannot form a d-simplex due to the number |Y| of data points in Y satisfies |Y| = d, violating the definition of the d-simplex. However, our model for the few-shot learning can proceed without any modification. To make this clear, we need to introduce the isometric embedding of the simplex. The points on a geometric entity R de (manifolds or polytopes) can be expressed with coordinates when the entity is placed in an ambient space R da. Usually, the intrinsic dimension d e is much less than the ambient dimension d a, especially when d a is large. Formally, there exists a function For the question that we are interested in, R de is a simplex. Both R de and R da are Euclidean. The isometric embedding means τ i − τ j = f (τ i) − f (τ j) for an arbitrary pair of vertices. For a simplex S in R de, therefore, the contents of C(S) and C(f (S)) satisfy the identity C(S) = C(f (S)) under isometric embedding. That is to say, we can correlate the dimension of the simplex with the number of vertices under isometric embedding in the case where the number of vertices is insufficient to construct a d a -simplex. This reduced simplex in the R da is obvious a low-dimensional one. DISPLAYFORM0 Two visual instances are illustrated in FIG1, where a segment line of fixed length is a 1-simplex. It can be embedded in the 1-dimensional, 2-dimensional and 3-dimensional spaces, respectively. Certainly, we can apply the coordinates of different dimensions for these two endpoints. However, its length is constant in different ambient spaces due to the isometric embedding. When a simplex is mapped into a higher dimensional space by an isometric embedding, the coordinates of its vertices vary, whereas its geometric properties pertaining to distance remain unchanged. Therefore we can employ formula to perform our algorithm. For few-shot classification tasks, such reduced simplices are ubiquitous because feature representations of data points embedded by variously parameterized functions are always of high dimension while the number of available examples for a class is quite few. For example, images classification tasks generally require deep convolutional networks (such as VGGs or Inception) as the embedding function. The VGG16 network produces a long vector of 4096 dimensions after the final fully-connected layer. For a five-shot learning, however, there are only five examples in each class. Hence, the reduced simplices we actually exploit are of dimension four for the computation of C(Y) and dimension five for that of C(Y ∪ t).The above geometric analysis is reminiscent of the theoretic assumption in manifold learning BID24 BID16, especially the Isomap algorithm BID24. Interested readers may refer to these two seminal papers in manifold learning for further understanding. It is known that matching network is the extension of metric learning via attention model. Here we analyze the metric characteristic of our algorithm. This angle of analysis may be useful for digging deep insights of few-shot learning. To this end, we need to reveal the details of formula, which is described in Theorem 1. Theorem 1. The geometric dissimilarity measurement (t, Y) from point t to Y can be expanded as the following form DISPLAYFORM0 where p i = y i − t 2 and DISPLAYFORM1 Proof. For matrices S 11, S 12, S 21, and S 22, the Schur's determinant identity is the form DISPLAYFORM2 Applying the Schur's determinant identity twice, we can obtain the expanded form of det(P) DISPLAYFORM3 By the same way, we can also obtain the analogous form of det(Q) DISPLAYFORM4 Substituting equations and into equation FORMULA5, we achieve the expansion of the measurement (t, Y) in FORMULA7. This concludes the proof of Theorem 1.It is readily to know that both Q and P are not positive definite matrices. Hence Q −1 and P −1 cannot be viewed as metric matrices in because there are negative eigenvalues. However, there still exists the explicit algebraic meaning. The first factor w 1 = e T P −1 e/e T Q −1 e is an incremental ratio with respect to the class and the second factor w 2 = p T Q −1 p is equivalent to the summation of a kind of scale-normalized distance transformation. Motivated by the success of applying the decomposed parts in , we present an approach to constructing a simplex using the spatial feature maps of CNNs. Moreover, the convolutional feature maps have been effectively employed for visual relational reasoning in BID19.While applying deep convolutional networks in classification tasks, the tensor (feature maps) will be flatten to be a 1-D feature vector in the last convolution layer and then it is fed into the fully connected layers. This process makes the feature maps lose adjacency information of spatial structures in the 2-D image. Although flattening seems inevitable in conventional classification pipeline, for few-shot tasks, it is necessary to collect information more effectively from multiple local regions for making an accurate classification decision. In order to play down the negative effect of flattening operation, besides the flattened feature vectors, we take fully advantage of feature maps by applying various scales of attention windows for the last convolution layer. Within each local region, a 3 × 3 region on 5 × 5 feature maps for instance, we perform the same flattening operation, generating a feature vector containing local information. Using this strategy, we are able to generate more feature vectors for simplex construction with few samples. Figure 3 clearly displays the operation of sampling feature vectors on feature maps with local regions. To further carry out this idea of preserving the adjacency information, we tag the l 2 d-dimensional cells with their corresponding coordinates indicating their relative spatial locations. With the extra 2 dimensions, we are able to restrain these cells on an underlying manifold, retaining their relative positions as in the original image. With such a simple manipulation, our model shows an appreciable improvement in performance. In regard to the experiment setup, the number of examples in each class should be constrained so as to fit the"few-shot" scenario. A typical experiment setting is the N -way k-shot task BID26: for each of the N new categories, k examples are provided. Given a set of unlabeled test samples, the model needs to classify them into these N categories. Since the number of available examples is limited (e.g. 1 or 5), training deep convolutional networks either from the scratch or with fine-tuning on new class data will generally lead to over-fitting. We performed our model on two different datasets: miniImageNet BID26 and Omniglot .The algorithms to be compared fall into two categories: conventional methods and deep learning methods. For deep learning methods, we compare our simplex algorithm with three state-of-the-art ones: matching networks BID26, Meta-Learner LSTM BID15, and prototypical networks BID21. Essentially, our algorithm is to measure the distance between a data point and a data set. For conventional methods, therefore, we take the Mahalanobis distance BID12 and Minimum Incremental Coding Length (MICL) BID28 for comparison. The MICL algorithm can be used to measure the structural similarity by coding theory of multivariate Gaussian data. A simple four-block CNN is employed to learn the representations of data both for miniImageNet and Omniglot. The architecture follows the learner network proposed by BID15, which contains four 3 × 3 convolutional layers with 64 filters. Each is followed by batch normalization, ReLU activation and 2 × 2 max-pooling. Following all the above layers is one fully connected layer, and lastly a softmax layer with the number equal to the number of classes being trained upon. The output is optimized with a cross-entropy loss function by the Adam optimizer with a learning rate of 0.001. The architecture is shown in Figure 4. The traditional algorithms compared are all performed on features extracted by this four-layer CNN. Size of local regions 5-way 1-shot Acc.5-way 5-shot Acc. DISPLAYFORM0 42.66% ± 0.48% 58.21% ± 0.47% 3 × 3 42.34% ± 0.42% 58.98% ± 0.40% 4 × 4 40.76% ± 0.43% 58.02% ± 0.44% 5 × 5 32.85% ± 0.70% 58.63% ± 0.68% Table 2: Few-shot accuracy on miniImageNet on 95% confidence interval. *Reported by BID15 Model Fine Tune 5-way 1-shot Acc. 5-way 5-shot Acc. Baseline KNN N 32.69% ± 0.75% 40.32% ± 0.68% Mahalanobis Distance N 32.39% ± 0.70% 60.09% ± 0.39% MICL BID28 N 42.59% ± 0.46% 61.06% ± 0.36%Matching networks* N 43.40% ± 0.78% 51.09% ± 0.71% Matching networks FCE* N 43.56% ± 0.84% 55.31% ± 0.73% Meta-Learner LSTM* N 43.44% ± 0.77% 60.60% ± 0.71% Prototypical networks BID21 N 49.42% ± 0.78% 68.20% ± 0.66% Simplex (ours) N 46.93% ± 0.43% 62.00% ± 0.26% ImageNet is a large-scale image database designed for multiple computer vision tasks. Since it would be extremely time-consuming to test the few-shot performance on the full ImageNet, we turn to miniImageNet instead BID26, the subset of ImageNet with 100 categories selected. For each category, 600 images with size 84 × 84 are provided. Following the same split as BID15, miniImageNet is divided into a 64-class training set, a 16-class validation set and a 20-class test set. We train the embedding network on the 64-class training set and validate the based on the 16-class validation set. The 20-class test set is only for the few-shot experiments. For each image fed into the 4-block CNN, the last feature map (64@5 × 5) is retrieved for simplex modeling. In order to choose a proper local region for conducting the few-shot experiments, we first test the N -way k-shot accuracy of different sizes of regions on the validation set. The size of the local regions varies from 2 × 2 to 5 × 5. 2 The is shown in TAB0. On the validation set, the models with 2 × 2 and 3 × 3 regions perform the best on the 1-shot and 5-shot tasks, respectively. We take them to compare with other models on the test set. Following the same splits proposed by BID15, we compare the performance of our model on miniImageNet directly with other models. We also conduct the K-nearest neighbor on the feature vectors generated from the 4-block model as a comparison (Baseline KNN). The are shown in Table 2. Using the same pre-trained 4-block CNN, our model performs much better than the baseline KNN, meanwhile outperforms the matching networks and Meta-learner LSTM. However, the prototypical networks are better than our simplex algorithm. Omniglot dataset for one-shot learning BID9 contains characters from 50 alphabets ranging from Korean to ancient Greek. Each character is provided with 20 examples handwritten by BID28 95.4% 99.1% 87.3% 96.9%Siamese networks BID7 97.3% 98.4% 88.2% 97.0% Matching networks BID26 98.1% 98.9% 93.8% 98.5% Prototypical networks BID21 Following the training setting in BID26, we split Omniglot into two parts: 1200 characters for training and the rest for validation and few-shot testing. The embedding 4-layer CNN is almost the same as used for training miniImageNet, except that the output feature map is changed to 64@1 × 1 due to the decrease in image size. We compare the N -way k-shot performance of our model with others. The are shown in TAB1. Our models are overall comparable to the other state-of-the-art works. Besides, we conduct more experiments on MICL and our model for a further comparison on robustness. MICL has a distortion parameter 2 in the coding length, i.e. BID28 ) DISPLAYFORM0 where Σ(X) is the corresponding covariance matrix and µ is the center of X. Through adjusting the parameter, the optimal performance of the model can be obtained on the validation set. We follow the same way as MICL and set up the free parameter in our simplex model. To be specific, let λ 1,..., λ n denote the eigenvalues ofQ. It is easy to write det(Q) = n i=1 λ i. To include the free parameter, we employ the following expression instead DISPLAYFORM1 The same computation is also performed for det(P).The experiment is conducted on the different values of distortion 2. According to the in FIG5, it is clear that our model is far more robust than MICL. Although for some value of 2, the accuracy of MICL is close to our model, the overall performance of MICL is instable with respect to 2. On the contrary, the performance of our model almost keeps invariant in a wide spectrum of 2. It is worth noting that our simplex metric can also be combined with very recently published works including prototypical networks BID21 and meta-learning methods, such as BID3 and BID13. For example, the distance measure in prototypical networks can be replaced with our simplex metric and the associated networks can be learned in the end-to-end manner by the supervision of simplex volumes. For few-shot cases, the number of examples in each class is quite limited. So the computational complexity can be well managed. Besides, the meta learning methods are to learn models between tasks instead of data points. Therefore, they are applicable to improve performance based on the simplex metric. We leave these for further explorations. On the other hand, MICL BID28 and our algorithm directly exploit the features of new classes yielded by CNNs that are not retrained, fine-tuned, or performed any relevant refinement of model parameters on new classes. Even so, these two approaches achieve better performance on 5-shot recognition than matching network BID26 and one of meta-learning algorithms BID15 ) that apply more information of new classes and high-level learning techniques. Both MICL and our simplex algorithm harness the geometric characterization of class structures 3. Therefore, our work might inspire the interest of exploring geometry to solve few-shot learning problems, which is paid little attention in the field. In this paper, we designed a novel method to deal with few-shot learning problems. Our idea was from the point of view of high dimensional convex geometry and transformed the learning problem to the study of volumes of simplices. The relation between a test sample and a class was investigated via the volumes of different polytopes. By harnessing the power of simplex, we gave a rigorous mathematical formulation for our approach. We also conduced extensive simulations to validate our method. The on various datasets showed the accuracy and robustness of the geometry-based method, compared to the state-of-the-art in the literature.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A simplex-based geometric method is proposed to cope with few-shot learning problems.

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.