Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Continual learning is the problem of sequentially learning new tasks or knowledge while protecting previously acquired knowledge. However, catastrophic forgetting poses a grand challenge for neural networks performing such learning process. Thus, neural networks that are deployed in the real world often struggle in scenarios where the data distribution is non-stationary (concept drift), imbalanced, or not always fully available, i.e., rare edge cases. We propose a Differentiable Hebbian Consolidation model which is composed of a Differentiable Hebbian Plasticity (DHP) Softmax layer that adds a rapid learning plastic component (compressed episodic memory) to the fixed (slow changing) parameters of the softmax output layer; enabling learned representations to be retained for a longer timescale. We demonstrate the flexibility of our method by integrating well-known task-specific synaptic consolidation methods to penalize changes in the slow weights that are important for each target task. We evaluate our approach on the Permuted MNIST, Split MNIST and Vision Datasets Mixture benchmarks, and introduce an imbalanced variant of Permuted MNIST --- a dataset that combines the challenges of class imbalance and concept drift. Our proposed model requires no additional hyperparameters and outperforms comparable baselines by reducing forgetting. A key aspect of human intelligence is the ability to continually adapt and learn in dynamic environments, a characteristic which is challenging to embed into artificial intelligence. Recent advances in machine learning (ML) have shown tremendous improvements in various problems, by learning to solve one complex task very well, through extensive training on large datasets with millions of training examples or more. However, most of the ML models that are used during deployment in the real-world are exposed to non-stationarity where the distributions of acquired data changes over time. Therefore, after learning is complete, and these models are further trained with new data, responding to distributional changes, performance degrades with respect to the original data. This phenomenon known as catastrophic forgetting or catastrophic interference presents a crucial problem for deep neural networks (DNNs) that are tasked with continual learning , also called lifelong learning . In continual learning, the goal is to adapt and learn consecutive tasks without forgetting how to perform well on previously learned tasks, enabling models that are scalable and efficient over long timescales. In most supervised learning methods, DNN architectures require independent and identically distributed (iid) samples from a stationary training distribution. However, for ML systems in realworld applications that require continual learning, the iid assumption is easily violated when: There is concept drift in the training data distribution. There are imbalanced class distributions and concept drift occuring simultaneously. Data representing all scenarios in which the learner is expected to perform are not initially available. In such situations, learning systems face the "stability-plasticity dilemma" which is a well-known problem for artificial and biological neural networks . This presents a continual learning challenge for an ML system where the model needs to provide a balance between its plasticity (to integrate new knowledge) and stability (to preserve existing knowledge). In biological neural networks, synaptic plasticity has been argued to play an important role in learning and memory (; ;) and two major theories have been proposed to explain a human's ability to perform continual learning. The first theory is inspired by synaptic consolidation in the mammalian neocortex where a subset of synapses are rendered less plastic and therefore preserved for a longer timescale. The general idea for this approach is to consolidate and preserve synaptic parameters that are considered important for the previously learned tasks. This is normally achieved through task-specific updates of synaptic weights in a neural network. The second is the complementary learning system (CLS) theory , which suggests that humans extract highlevel structural information and store it in different brain areas while retaining episodic memories. Recent work on differentiable plasticity has shown that neural networks with "fast weights" that leverage Hebbian learning rules can be trained end-to-end through backpropagation and stochastic gradient descent (SGD) to optimize the standard "slow weights", as well as also the amount of plasticity in each synaptic connection . These works use slow weights to refer to the weights normally used to train vanilla neural networks, which are updated slowly and are often associated with long-term memory. The fast weights represent the weights that are superimposed on the slow weights and change quickly from one time step to the next based on input representations. These fast weights behave as a form of short-term memory that enable "reactivation" of long-term memory traces in the slow weights. showed that simple plastic networks with learned plasticity outperform networks with uniform plasticity on various problems. Moreover, there have been several approaches proposed recently for overcoming the catastrophic forgetting problem in fixed-capacity models by dynamically adjusting the plasticity of each synapse based on its importance for retaining past memories . Here, we extend the work on differentiable plasticity to the task-incremental continual learning setting (van de), where tasks arrive in a batch-like fashion, and have clear boundaries. We develop a Differentiable Hebbian Consolidation 1 model that is capable of adapting quickly to changing environments as well as consolidating previous knowledge by selectively adjusting the plasticity of synapses. We modify the traditional softmax layer and propose to augment the slow weights in the final fully-connected (FC) layer (softmax output layer) with a set of plastic weights implemented using Differentiable Hebbian Plasticity (DHP). Furthermore, we demonstrate the flexibility of our model by combining it with recent task-specific synaptic consolidation based approaches to overcoming catastrophic forgetting such as elastic weight consolidation , synaptic intelligence (b) and memory aware synapses . Our model unifies core concepts from Hebbian plasticity, synaptic consolidation and CLS theory to enable rapid adaptation to new unseen data, while consolidating synapses and leveraging compressed episodic memories in the softmax layer to remember previous knowledge and mitigate catastrophic forgetting. We test our proposed method on established benchmark problems including the Permuted MNIST , Split MNIST (b) and Vision Datasets Mixture benchmarks. We also introduce the Imbalanced Permuted MNIST problem and show that plastic networks with task-specific synaptic consolidation methods outperform networks with uniform plasticity. Neural Networks with Non-Uniform Plasticity: One of the major theories that have been proposed to explain a human's ability to learn continually is Hebbian learning , which suggests that learning and memory are attributed to weight plasticity, that is, the modification of the strength of existing synapses according to variants of Hebb's rule (; ;). It is a form of activity-dependent synaptic plasticity where correlated activation of pre-and post-synaptic neurons leads to the strengthening of the connection between the two neurons. According to the Hebbian learning theory, after learning, the related synaptic strength are enhanced while the degree of plasticity decreases to protect the learned knowledge (a). Recent approaches in the meta-learning literature have shown that we can incorporate fast weights into a neural network to perform one-shot and few-shot learning . proposed a model that augments FC layers preceding the softmax with a matrix of fast weights to bind labels to representations. Here, the fast weights were implemented with non-trainable Hebbian learning-based associative memory. proposed a Hebbian Softmax layer that can improve learning of rare classes by interpolating between Hebbian learning and SGD updates on the output layer using an engineered scheduling scheme. proposed differentiable plasticity, which uses SGD to optimize the plasticity of each synaptic connection, in addition to the standard fixed (slow) weights. Here, each synapse is composed of a slow weight and a plastic (fast) weight that automatically increases or decreases based on the activity over time. Although this approach served to be a powerful new method for training neural networks, it was mainly demonstrated on recurrent neural networks (RNNs) for solving pattern memorization tasks and maze exploration with reinforcement learning. Also, these approaches were only demonstrated on meta-learning problems and not the continual learning challenge of overcoming catastrophic forgetting. Our work also augments the slow weights in the FC layer with a set of plastic (fast) weights, but implements these using DHP. We only update the parameters of the softmax output layer in order to achieve fast learning and preserve knowledge over time. Overcoming Catastrophic Forgetting: This work leverages two strategies to overcome the catastrophic forgetting problem: 1) Task-specific Synaptic Consolidation -Protecting previously learned knowledge by dynamically adjusting the synaptic strengths to consolidate and retain memories. 2) CLS Theory -A dual memory system where, the neocortex (neural network) gradually learns to extract structured representations from the data while, the hippocampus (augmented episodic memory) performs rapid learning and individuated storage to memorize new instances or experiences. There have been several notable works inspired by task-specific synaptic consolidation for overcoming catastrophic forgetting (; b;) and they are often categorized as regularization strategies in the continual learning literature . All of these regularization approaches estimate the importance of each parameter or synapse, Ω k, where least plastic synapses can retain memories for long timescales and more plastic synapses are considered less important. The parameter importance and network parameters θ k are updated in either an online manner or after learning task T n. Therefore, when learning new task T n+1, a regularizer is added to the original loss function L n (θ), so that we dynamically adjust the plasticity w.r.t Ω k and prevent any changes to important parameters of previously learned tasks: where θ n−1 k are the learned network parameters after training on the previous n − 1 tasks and λ is a hyperparameter for the regularizer to control the amount of forgetting (old versus new memories). The main difference in these regularization strategies is on the method used to compute the importance of each parameter, Ω k. In Elastic Weight Consolidation (EWC), used the values given by the diagonal of an approximated Fisher information matrix for Ω k, and this was computed offline after training on a task was completed. An online variant of EWC was proposed by to improve EWC's scalability by ensuring the computational cost of the regularization term does not grow with the number of tasks. Zenke et al. (2017b) proposed an online method called Synaptic Intelligence (SI) for computing the parameter importance where, Ω k is the cumulative change in individual synapses over the entire training trajectory on a particular task. Memory Aware Synapses (MAS) from is an online method that measures Ω k by the sensitivity of the learned function to a perturbation in the parameters, instead of measuring the change in parameters to the loss as seen in SI and EWC. Our work draws inspiration from CLS theory which is a powerful computational framework for representing memories with a dual memory system via the neocortex and hippocampus. There have been numerous approaches based on CLS principles involving pseudo-rehersal (; ;), exact or episodic replay and generative replay . Exact replay methods require storage of the data from previous tasks which are later replayed. Generative replay methods train a separate generative model to generate images to be replayed. iCaRL performs rehearsal and regularization, where an external memory is used to store exemplar patterns from old task data and rehearse the model via distillation. However, in our work, we are primarily interested in neuroplasticity techniques inspired from CLS theory for alleviating catastrophic forgetting. Earlier work from; showed how each synaptic connection can be composed of a fixed weight where slow learning stores long-term knowledge and a fast-changing weight for temporary associative memory. This approach involving slow and fast weights is analogous to properties of CLS theory to overcome catastrophic forgetting during continual learning. Recent research in this vein has included replacing soft attention mechanism with fast weights in RNNs , the Hebbian Softmax layer , augmenting slow weights in the FC layer with a fast weights matrix , differentiable plasticity and neuromodulated differentiable plasticity . We did not evaluate and compare against neuroplasticity-inpired CLS methods as baselines because they were designed for meta-learning problems and would be unfair to evaluate their performance on continual learning benchmark problems given some of their limitations. All of these methods were designed for rapid learning on simple tasks or meta-learning over a distribution of tasks or datasets, where a few number of examples from a class are seen by the network when training on different tasks to perform one-shot and few-shot learning. For instance, the Hebbian Softmax layer modifies its parameters by annealing between Hebbian and SGD updates based on an engineered scheduling scheme which achieves fast binding for rarer classes. However, when a large number of examples are observed frequently from the same class, the annealing function switches completely to SGD updates. Thus, when evaluating this model in continual learning setups, the effect of the fast weights memory storage becomes non-existent as the network learns from a large number of examples per class on each task. With a focus on continual learning, the goal of our work is to metalearn a local learning rule for the fast weights via the fixed (slow) weights and an SGD optimizer. In our model, each synaptic connection in the softmax layer has two weights: 1) The slow weights, θ ∈ R m×d, where m is the number of units in the final hidden layer and d is the number of outputs of the last layer. 2) A Hebbian plastic component of the same cardinality as the slow weights, composed of the plasticity coefficient, α, and the Hebbian trace, Hebb. The α is a scaling parameter for adjusting the magnitude of the Hebb. The Hebbian traces accumulate the mean hidden activations of the final hidden layer h for each target label in the mini-batch {y 1:B} of size B which are denoted byh ∈ R 1×m (refer to Algorithm 1). Given the pre-synaptic activations of neurons i in h, we can formally compute the post-synaptic activations of neurons j using Eq. 2 and obtain the unnormalized log probabilities (softmax pre-activations) z. The softmax function is then applied on z to obtain the desired predicted probabilitiesŷ thus,ŷ = softmax(z). The η parameter in Eq. 3 is a scalar value that dynamically learns how quickly to acquire new experiences into the plastic component, and thus behaves as the learning rate for the plastic connections. The η parameter also acts as a decay term for the Hebb to prevent instability caused by a positive feedback loop in the Hebbian traces. The network parameters α i,j, η and θ i,j are optimized by gradient descent as the model is trained sequentially on different tasks in the continual learning setup. In standard neural networks the weight connection has only fixed (slow) weights, which is equivalent to setting the plasticity coefficients α = 0 in Eq. 2. Algorithm 1 Batch update Hebbian traces. , where w i,j is the change in weight at connection i, j and a k i, a k j denote the activation levels of neurons i and j, respectively, for the k th input. Therefore, in our model, w =h the Hebbian weight update, a i = h the hidden activations of the last hidden layer, a j = y the corresponding target class in y 1:B and N = s the number of inputs for the corresponding class in y 1:B (see Algorithm 1). Across the model's lifetime, we only update the Hebbian traces during training as it learns tasks in a continual manner. Therefore, during test time, we maintain and use the most recent Hebb traces to make predictions. Our model explores an optimization scheme where hidden activations are accumulated directly into the softmax output layer weights when a class has been seen by the network. This in better initial representations and can also retain these learned deep representations for a much longer timescale. This is because memorized activations for one class are not competing for space with activations from other classes. Fast learning, enabled by a highly plastic weight component, improves test accuracy for a given task. Between tasks this plastic component decays to prevent interference, but selective consolidation into a stable component protects old memories, effectively enabling the model to learn to remember by modelling plasticity over a range of timescales to form a learned neural memory (see Section 4.1 ablation study). In comparison to an external memory, the advantage of DHP Softmax is that it is simple to implement, requiring no additional space or computation. This allows it to scale easily with increasing number of tasks. Figure 1: An example of a Hebbian update for the class, c = 6 ∈ y 1:B. Here, we are given the hidden activations of the final hidden layer, h. Multiple hidden activations corresponding to class c = 6 (represented by the pink boxes) are averaged into one vector denoted byh ∈ R 1×m. This Hebbian update visualization reflects Lines 4-6 in Algorithm 1 and is repeated for each unique class in the target vector y 1:B. The plastic component learns rapidly and performs sparse parameter updates to quickly store memory traces for each recent experience without interference from other similar recent experiences. Furthermore, the hidden activations corresponding to the same class, c, are accumulated into one vectorh, thus forming a compressed episodic memory in the Hebbian traces to reflect individual episodic memory traces (similar to the hippocampus in biological neural networks ). As a , this method improves learning of rare classes and speeds up binding of class labels to deep representations of the data without introducing any additional hyperparameters. In Appendix B, we provide a sample implementation of the DHP Softmax using PyTorch. Hebbian Synaptic Consolidation: Following the existing regularization strategies such as EWC , Online EWC , SI (b) and MAS , we regularize the loss L(θ) as in Eq. 1 and update the synaptic importance parameters of the network in an online manner. We rewrite Eq. 1 to obtain the updated quadratic loss for Hebbian Synaptic Consolidation in Eq. 4 and show that the network parameters θ i,j are the weights of the connections between pre-and post-synaptic activities of neurons i and j, We adapt the existing task-specific consolidation approaches to our model and do not compute the synaptic importance parameters on the plastic component of the network, hence we only regularize the slow weights of the network. Furthermore, when training the first task T n=1, the synaptic importance parameter, Ω i,j in Eq. 4, was set to 0 for all of the task-specific consolidation methods that we tested on except for SI. This is because SI is the only method we evaluated that estimates Ω i,j while training, whereas Online EWC and MAS compute Ω i,j after learning a task. The plastic component of the softmax layer in our model can alleviate catastrophic forgetting of consolidated classes by allowing gradient descent to optimize how plastic the connections should be (i.e. less plastic to preserve old information or more plastic to quickly learn new information). In our experiments, we compare our approach to vanilla neural networks with Online EWC, SI and MAS. Since our approach increases the capacity of the DNN due to the addition of plastic weights, we add an extra set of slow weights to the softmax output layer of the standard neural network to match the capacity. We do this to show that it is not the increased model capacity from the plastic weights that is helping mitigate the forgetting when performing sequential task learning, thus ensuring a fair evaluation. We tested our model on the Permuted MNIST, Split MNIST and Vision Datasets Mixture benchmarks, and also introduce the Imbalanced Permuted MNIST problem. For all of the benchmarks, we evaluated the model based on the average classification accuracy on all previously learned tasks as a function of n, the number of tasks trained so far. To determine memory retention and flexibility of the model, we are particularly interested in the test performance on the first task and the most recent one. We also measure forgetting using the backward transfer metric, BWT = , which indicates how much learning new tasks has influenced the performance on previous tasks. R T,i is the test classification accuracy on task i after sequentially finishing learning the T th task. While BWT < 0 directly reports catastrophic forgetting, BWT > 0 indicates that learning new tasks has helped with the preceding tasks. To establish a baseline for comparison of well-known task-specific consolidation methods, we trained neural networks with Online EWC, SI and MAS, respectively, on all tasks in a sequential manner. The hyperparameters of the consolidation methods (i.e. EWC, SI and MAS) remain the same with and without DHP Softmax, and the plastic components are not regularized. Descriptions of the hyperparameters and other details for all benchmarks can be found in Appendix A. In this benchmark, all of the MNIST pixels are permuted differently for each task with a fixed random permutation. Although the output domain is constant, the input distribution changes between tasks and is mostly independent of each other, thus, there exists a concept drift. In the Permuted MNIST and Imbalanced Permuted MNIST benchmarks we use a multi-layered perceptron (MLP) network with two hidden layers consisting of 400 ReLU nonlinearities, and a cross-entropy loss. The η of the plastic component was set to be a value of 0.001 and we emphasize that we spent little to no effort on tuning the initial value of this parameter (see Appendix A.5 for a sensitivity analysis). We first compare the performance between our network with DHP Softmax and a fine-tuned vanilla MLP network we refer to as Finetune in Figure 2a and no task-specific consolidation methods involved. The network with DHP Softmax alone showed improvement in its ability to alleviate catastrophic forgetting across all tasks compared to the baseline network. Then we compared the performance with and without DHP Softmax using the same task-specific consolidation methods. Figure 2a shows the average test accuracy as new tasks are learned for the best hyperparameter combination for each task-specific consolidation method. We find our DHP Softmax with consolidation maintains a higher test accuracy throughout sequential training of tasks than without DHP Softmax. Ablation Study: We further examine the structural parameters of the network and Hebb traces to provide further interpretability into the behaviour of our proposed model. The left plot in Figure 8 shows the behaviour of η during training as 10 tasks in the Permuted MNIST benchmark are learned continually. Initially, in task T 1, η increases very quickly from 0.001 to 0.024 suggesting that the synaptic connections become more plastic to quickly acquire new information. Eventually, η decays after the 3 rd task to reduce the degree of plasticity to prevent interference between the learned representations. We also observe that within each task from T 4 to T 10, η initially increases then decays. The Frobenius Norm of the Hebb trace (middle plot in Figure 8) suggests that Hebb grows without runaway positive feedback every time a new task is learned, maintaining a memory of which synapses contributed to recent activity. The Frobenius Norm of α (right plot in Figure 8) indicates that the plasticity coefficients grow within each task, indicating that the network is leveraging the structure in the plastic component. It is important to note that gradient descent and backpropagation are used as meta-learning to tune the structural parameters in the plastic component. We introduce the Imbalanced Permuted MNIST problem which is identical to the Permuted MNIST benchmark but, now each task is an imbalanced distribution where training samples in each class were artificially removed based on some random probability (see Appendix A.2). This benchmark was motivated by the fact that class imbalance and concept drift can hinder predictive performance, and the problem becomes particularly challenging when they occur simultaneously. Appendix A.6, Figure 5 shows the average test accuracy for the best hyperparameters of each method. We see that DHP Softmax achieves 80.85% after learning 10 tasks with imbalanced class distributions in a sequential manner, thus providing significant 4.41% improvement over the standard neural network baseline of 76.44%. The significance of the compressed episodic memory mechanism in the Hebbian traces is more apparent in this benchmark because the plastic component allows rare classes that are encountered infrequently to be remembered for a longer period of time. We find that DHP Softmax with MAS achieves a 0.04 decrease in BWT, ing in an average test accuracy of 88.80% and a 1.48% improvement over MAS alone; also outperforming all other methods and across all tasks. We split the original MNIST dataset into a sequence of 5 binary classification tasks: T 1 = {0/1}, T 2 = {2/3}, T 3 = {4/5}, T 4 = {6/7} and T 5 = {8/9}. The output spaces are disjoint between tasks, unlike the previous two benchmarks. Similar to the network used by Zenke et al. (2017b), we use an MLP network with two hidden layers of 256 ReLU nonlinearities each, and a cross-entropy loss. The initial η value was set to 0.001 as seen in previous benchmark experiments. We found that different values of η yielded very similar final test performance after learning T 5 tasks (see Appendix A.5). We observed that DHP Softmax alone achieves 98.23% thus, provides a 7.80% improvement on test performance compared to a finetuned MLP network (Figure 2b). Also, combining DHP Softmax with task-specific consolidation consistently decreases BWT, leading to a higher average test accuracy across all tasks, especially the most recent one, T 5. Following previous works , we perform continual learning on a sequence of 5 vision datasets: MNIST, notMNIST 1, FashionMNIST , SVHN and CIFAR-10 (see Appendix A.4 for dataset details). The MNIST, notMNIST and FashionMNIST datasets are zero-padded to be of size 32×32 and are replicated 3 times to create grayscale images with 3 channels, thus matching the resolution of the SVHN and CIFAR-10 images. Here, we use a CNN architecture that is similar to the one used in (more details in Appendix A.4). The initial η parameter value was set to 0.0001. We train the network with mini-batches of size 32 and optimized using plain SGD with a fixed learning rate of 0.01 for 50 epochs per task. We found that DHP Softmax plus MAS decreases BWT by 0.04 ing in a 2.14% improvement in average test accuracy over MAS on its own (see Table 1 and Appendix A.6, Figure 6). Also, SI with DHP Softmax outperforms other competitive methods with an average test performance of 81.75% and BWT of -0.04 after learning all five tasks. In Table 1, we present a summary of the final average test performance after learning all tasks in the respective continual learning problems. Here, we summarize the average test accuracy and BWT across ten trials for each of the benchmarks. We have shown that the problem of catastrophic forgetting in continual learning environments can be alleviated by adding compressed episodic memory in the softmax layer through DHP and performing task-specific updates on synaptic parameters based on their individual importance for solving previously learned tasks. The compressed episodic memory allows new information to be learned in individual traces without overlapping representations, thus avoiding interference when added to the structured knowledge in the slow changing weights and allowing the model to generalize across experiences. The α parameter in the plastic component automatically learns to scale the magnitude of the plastic connections in the Hebbian traces, effectively choosing when to be less plastic (protect old knowledge) or more plastic (acquire new information quickly). The neural network with DHP Softmax showed noticeable improvement across all benchmarks when compared to a neural network with a traditional softmax layer that had an extra set of slow changing weights. The DHP Softmax does not introduce any additional hyperparameters since all of the structural parameters of the plastic part α and η are learned, and setting the initial η value required very little tuning effort. 1 Originally published at http://yaroslavvb.blogspot.com/2011/09/ notmnist-dataset.html and downloaded from https://github.com/davidflanagan/ notMNIST-to-MNIST. We demonstrated the flexibility of our model where, in addition to DHP Softmax, we can perform Hebbian Synaptic Consolidation by regularizing the slow weights using EWC, SI or MAS to improve a model's ability to alleviate catastrophic forgetting after sequentially learning a large number of tasks with limited model capacity. DHP Softmax combined with SI outperforms other consolidation methods on the Split MNIST and 5-Vision Datasets Mixture. The approach where we combine DHP Softmax and MAS consistently leads to overall superior compared to other baseline methods on the Permuted MNIST and Imbalanced Permuted MNIST benchmarks. This is interesting because the local variant of MAS does compute the synaptic importance parameters of the slow weights θ i,j layer by layer based on Hebb's rule, and therefore synaptic connections i, j that are highly correlated would be considered more important for the given task than those connections that have less correlation. Furthermore, our model consistently exhibits lower negative BWT across all benchmarks, leading to higher average test accuracy over methods without DHP. This gives a strong indication that Hebbian plasticity enables neural networks to learn continually and remember distant memories, thus reducing catastrophic forgetting when learning from sequential datasets in dynamic environments. Furthermore, continual synaptic plasticity can play a key role in learning from limited labelled data while being able to adapt and scale at long timescales. We hope that our work will open new investigations into gradient descent optimized Hebbian consolidation for learning and memory in DNNs to enable continual learning. In the continual learning setup, we train a neural network model on a sequence of tasks T 1:nmax, where n max is the maximum number of tasks the model is to learn in the respective benchmarks. Unlike the conventional supervised learning setup, continual learning trains a model on data that is fetched in sequential chunks enumerated by tasks. Therefore, in a continual learning sequence, the model receives a sequence of tasks T 1:nmax that is to be learned, each with its associated training data (X n, Y n), where X n is the input data and the corresponding label data denoted by Y n. Each task T n has its own task-specific loss L n, that will be combined with a regularizer loss term (refer to Eq. 4) to prevent catastrophic forgetting. After training is complete, the model will have learned an approximated mapping f to the the true underlying functionf. The learned f maps a new input X to the target outputs Y 1:n for all T 1:n tasks the network has learned so far. Also, it is to be noted that the set of classes contained in each task can be different from each other, as we have done in the SplitMNIST and Vision Datasets Mixture benchmarks. All experiments were run on either a Nvidia Titan V or a Nvidia RTX 2080 Ti. We train the network on a sequence of tasks T n=1:10 with mini-batches of size 64 and optimized using plain SGD with a learning rate of 0.01. We train for at least 10 epochs and perform earlystopping once the validation error does not improve for 5 epochs. If the validation error increases for more than 5 epochs, then we terminated the training on the task T n, reset the network weights and Hebbian traces to the values that had the lowest test error, and proceeded to the next task. Hyperparameters: For the Permuted MNIST experiments shown in Figure 2a, the regularization hyperparameter λ for each of the task-specific consolidation methods is set to λ = 100 for Online EWC , λ = 0.1 for SI (b) and λ = 0.1 for MAS . We note that for the SI method, λ refers to the parameter c in the original work (b) but we use λ to keep the notation consistent across other task-specific consolidation methods. In SI, the damping parameter, ξ, was set to 0.1. To find the best hyperparameter combination for each of these synaptic consolidation methods, we performed a grid search using a task sequence determined by a single seed. For Online EWC, we tested values of λ ∈ {10, 20, 50,. . ., 400}, SI -λ ∈ {0.01, 0.05,. . ., 0.5, 1.0} and MAS -λ ∈ {0.01, 0.5, . . ., 1.5, 2.0}. For each task in the Imbalanced Permuted MNIST problem, we artificially removed training samples from each class in the original MNIST dataset based on some random probability. For each class and each task, we draw a different removal probability from a standard uniform distribution U, and then remove each sample from that class with that probability. The distribution of classes in each dataset corresponding to tasks T n=1:10 is given in Table 2. Table 2: Distribution of classes in each imbalanced dataset for the respective tasks T n=1:10. Classes Tasks 1 2 3 4 5 6 7 8 9 10 0 4459 3780 1847 3820 5867 122 1013 4608 908 3933 1 1872 3637 1316 6592 1934 1774 5533 2569 831 886 2 2391 4125 2434 4966 5245 4593 4834 4432 3207 3555 3 4433 1907 1682 278 3027 2315 5761 3293 2545 3749 4 186 2728 2002 151 1435 5829 1284 3910 4593 927 5 4292 2472 2924 1369 4094 4858 2265 3289 1134 1413 6 2339 3403 4771 5569 1414 2851 2921 4074 336 3993 7 4717 3090 4800 2574 4086 1065 3520 4705 5400 3650 8 3295 5493 76 4184 2034 4672 682 196 2409 1709 9 2625 3880 4735 1647 2645 3921 901 4546 4649 2045 Total 30609 34515 26587 31120 31781 32000 28714 35622 26012 25860 For the Imbalanced Permuted MNIST experiments shown in Figure 5, the regularization hyperparameter λ for each of the task-specific consolidation methods is λ = 400 for Online EWC , λ = 1.0 for SI (b) and λ = 0.1 for MAS . In SI, the damping parameter, ξ, was set to 0.1. Similar to the Permuted MNIST benchmark, to find the best hyperparameter combination for each of these synaptic consolidation methods, we performed a grid search using a task sequence determined by a single seed. For Online EWC, we tested values of λ ∈ {50, 100,. . .,1×10 3}, SI -λ ∈ {0.1, 0.5,. . ., 2.5, 3.0} and MAS -λ ∈ {0.01, 0.05, . . ., 1.5, 2.0}. Across all experiments, we maintained the the same random probabilities detemined by a single seed to artificially remove training samples from each class. Hyperparameters: For the Split MNIST experiments shown in Figure 2b, the regularization hyperparameter λ for each of the task-specific consolidation methods is λ = 400 for Online EWC , λ = 1.0 for SI (b) and λ = 1.5 for MAS . In SI, the damping parameter, ξ, was set to 0.001. To find the best hyperparameter combination for each of these synaptic consolidation methods, we performed a grid search using the 5 task binary classification sequence (0/1, 2/3, 4/5, 6/7, 8/9). For Online EWC, we tested values of λ ∈ {1, 25, 50, 100, . . .,1×10 3, 2×10 3}, SI -λ ∈ {0.1, 0.5, 1.0, . . ., 5.0} and MAS -λ ∈ {0.01, 0.05, 1.0,. . ., 4.5, 5.0}. We train the network on a sequence of T n=1:5 tasks with mini-batches of size 64 and optimized using plain SGD with a fixed learning rate of 0.01 for 10 epochs. Dataset Details: The Vision Datasets Mixture benchmark consists of a sequence of 5 tasks where each task is a different image classification dataset: MNIST, notMNIST, FashionMNIST, SVHN and CIFAR-10. The notMNIST dataset consists of font glypyhs corresponding to letters'A' to'J'. The original dataset has 500,000 and 19,000 grayscale images of size 28×28 for training and testing, respectively. However, similar to MNIST, we only use 60,000 images for training and 10,000 for testing. FashionMNIST consists of 10 categories of various articles of clothing, and there are 60,000 and 10,000 grayscale images sized 28×28 for training and testing, respectively. SVHN consists of digits'0' to'9' from Google Street View images and there are 73,257 and 26,032 colour images of size 32×32 for training and testing, respectively. CIFAR-10 consists of 50,000 and 10,000 colour images of size 32×32 from 10 different categories for training and testing, respectively. Architecture: The CNN architecture consists of 2 convolutional layers with 20 and 50 channels respectively, and a kernel size of 5. Each convolution layer is followed by LeakyReLU nonlinearities (negative threshold of 0.3) and 2×2 max-pooling operations with stride 2. The two convolutional layers are followed by an FC layer of size 500 before the final softmax output layer (refer to Table 3). Similar to , a multi-headed approach was used because the class definitions are different between datasets. In the other benchmark problems, we use a single η across all connections. In this benchmark, our model has a trainable η value for each connection in the final output layer thus, η ∈ R m×d and we set the initial η value to be 0.0001. We found that using separate η parameters for each connection improved the stability of optimization and convergence to optimal test performance. This allows each plastic connection to modulate its own rate of plasticity when learning new experiences. It was observed that using a single η value across all connections lead to instability of optimization on the SVHN and CIFAR-10 tasks. Hyperparameters: For the 5-Vision Datasets Mixture experiments shown in Figure 6 the regularization hyperparameter λ for each of the task-specific consolidation methods is λ = 100 for Online EWC , λ = 0.1 for SI (b) and λ = 1.0 for MAS . In SI, the damping parameter, ξ, was set to 0.1. To find the best hyperparameter combination for each of these synaptic consolidation methods, we performed a random search using the same task sequence ordering (MNIST, notMNIST, FashionMNIST, . For Online EWC, we tested values of λ ∈ {10, 50, 100,. . ., 500}, SI -λ ∈ {0.01, 0.05, 0.1,. . ., 1.0} and MAS -λ ∈ {0.01, 0.05, 1.0,. . ., 4.5, 5.0}. We provide a summary of the sensitivity analysis performed on the Hebb decay term η and show its effect on the final average test performance after learning a sequence of tasks in the continual learning setup. The plots on the left and center in Figure 4 show the effect of the initial η value on the final test performance after learning tasks T n=1:10 in a sequential manner for the Permuted MNIST and Imbalanced Permuted MNIST benchmarks, respectively. We swept through a range of values η ∈ {0.1, 0.01, 0.001, 0.0005, 0.0001} and found that setting η to low values led to the best performance in terms of being able to alleviate catastrophic forgetting. Similarly, we also performed a sensitivity analysis on the η parameter for the Split MNIST problem (see the rightmost plot in Figure 4). Table 4 presents the average test accuracy across 5 trials for the MNIST-variant benchmarks, which corresponds to the sensitivity analysis plots in Figure 4. e t a r a t e: i n i t i a l l e a r n i n g r a t e v a l u e o f p l a s t i c c o n n e c t i o n s. 14 hebb: t h e u p d a t e d H e b b i a n t r a c e s f o r t h e n e x t i t e r a t i o n. " " " 16 s e l f. i n f e a t u r e s = i n f e a t u r e s 17 s e l f. o u t f e a t u r e s = o u t f e a t u r e s 18 s e l f. e t a r a t e = e t a r a t e 19 20 # I n i t i a l i z e f i x e d (s l o w) w e i g h t s w i t h He i n i t i a l i z a t i o n. 21 s e l f. w e i g h t = P a r a m e t e r (t o r c h . T e n s o r ( s e l f . i n f e a t u r e s, # I n i t i a l i z e t h e l e a r n i n g r a t e o f p l a s t i c c o n n e c t i o n s . 31 s e l f . e t a = P a r a m e t e r ( ( s e l f . e t a r a t e * t o r c h . o n e s ), 56 r e t u r n V a r i a b l e (t o r c h . z e r o s ( s e l f . i n f e a t u r e s, s e l f . o u t f e a t u r e s), r e q u i r e s g r a d = F a l s e ) Listing 1: PyTorch implementation of the DHP Softmax model which adds a compressed episodic memory to the final output layer of a neural network through plastic connections as described in Algorithm 1. We want to emphasize the simplicity of implementation using popular ML frameworks. Zenke et al. (2017b). First, the network was trained on the full CIFAR-10 dataset (Task T n=1) and sequentially on 5 additional tasks each corresponding to 10 consecutive classes from the CIFAR-100 dataset (Tasks T n=2:6). The test accuracies of CIFAR-10 and the CIFAR-100 splits are reported after having learned the final task in this sequence. The DHP Softmax (purple) alone significantly outperforms Finetune (yellow) on each of the tasks in this class-incremental learning setup. On some tasks, DHP Softmax alone performs as well or better than when training from scratch (light green). The test accuracies of Finetune, when training from scratch and SI (turquoise) were taken from von.

Hebbian plastic weights can behave as a compressed episodic memory storage in neural networks and with the combination of task-specific synaptic consolidation can improve the ability to alleviate catastrophic forgetting in continual learning.

A theoretical analysis of a new class of RNNs, trained on neuroscience tasks, allows us to identify the role of dynamical dimensionality and cell classes in neural computations.


Question: Represent the article for finding a single-sentence TLDR

Question: In order to choose a neural network architecture that will be effective for a particular modeling problem, one must understand the limitations imposed by each of the potential options. These limitations are typically described in terms of information theoretic bounds, or by comparing the relative complexity needed to approximate example functions between different architectures. In this paper, we examine the topological constraints that the architecture of a neural network imposes on the level sets of all the functions that it is able to approximate. This approach is novel for both the nature of the limitations and the fact that they are independent of network depth for a broad family of activation functions. Neural networks have become the model of choice in a variety of machine learning applications, due to their flexibility and generality. However, selecting network architectures and other hyperparameters is typically a matter of trial and error. To make the choice of neural network architecture more straightforward, we need to understand the limits of each architecture, both in terms of what kinds of functions any given network architecture can approximate and how those limitations impact its ability to learn functions within those limits. A number of papers (3; 6; 11; 13) have shown that neural networks with a single hidden layer are a universal approximator, i.e. that they can approximate any continuous function on a compact domain to arbitrary accuracy if the hidden layer is allowed to have an arbitrarily high dimension. In practice, however, the neural networks that have proved most effective tend to have a large number of relatively low-dimensional hidden layers. This raises the question of whether neural networks with an arbitrary number of hidden layers of bounded dimension are also a universal approximator. In this paper we demonstrate a fairly general limitation on functions that can be approximated with the L ∞ norm on compact subsets of a Euclidean input space by layered, fully-connected feedforward neural networks of arbitrary depth and activation functions from a broad family including sigmoids and ReLus, but with layer widths bounded by the dimension of the input space. By a layered network, we mean that hidden nodes are grouped into successive layers and each node is only connected to nodes in the previous layer and the next layer. The constraints on the functions are defined in terms of topological properties of the level sets in the input space. This analysis is not meant to suggest that deep networks are worse than shallow networks, but rather to better understand how and why they will perform differently on different data sets. In fact, these limitations may be part of the reason deep nets have proven more effective on datasets whose structures are compatible with these limitations. By a level set, we mean the set of all points in the input space that the model maps to a given value in the output space. For classification models, a level set is just a decision boundary for a particular cutoff. For regression problems, level sets don't have a common interpretation. The main of the paper, Theorem 1, states that the deep, skinny neural network architectures described above cannot approximate any function with a level set that is bounded in the input space. This can be rephrased as saying that for every function that can be approximated, every level set must be unbounded, extending off to infinity. While a number of recent papers have made impressive progress in understanding the limitations of different neural network architectures, this is notable because it is independent of the number of layers in the network, and because the limitations are defined in terms of a very simple topological property. Topological tools have recently been employed to study the properties of data sets within the field known as Topological Data Analysis, but this paper exploits topological ideas to examine the topology of the models themselves. By demonstrating topological constraints on a widely used family of models, we suggest that there is further potential to apply topological ideas to understand the strengths and weaknesses of algorithms and methodologies across machine learning. After discussing the context and related work in Section 2, we introduce the basic definitions and notation in Section 3, then state the main Theorem and outline the proof in Section 4. The detailed proof is presented in Sections 5 and 6. We present experimental that demonstrate the constraints in Section 7, then in Section 8 we present from this work. A number of papers have demonstrated limitations on the functions that can be approximated by neural networks with particular architectures (2; 12; 14; 15; 18; 19; 21; 22; 23; 24; 25; 27; 29; 32). These are typically presented as asymptotic bounds on the size of network needed to approximate any function in a given family to a given ε. Lu et al BID15 gave the first non-approximation that is independent of complexity, showing that there are functions that no ReLu-based deep network of width equal to the dimension of the input space can approximate, no matter how deep. However, they consider convergence in terms of the L 1 norm on the entire space R n rather than L ∞ on a compact subset. This is a much stricter definition than the one used in this paper so even for ReLu networks, Theorem 1 is a stronger . The closest existing to Theorem 1 is a recent paper by Nguyen, Mukkamala and Hein BID24 which shows that for multi-label classification problems defined by an argmax condition on a higherdimensional output function, if all the hidden layers of a neural network have dimension less than or equal to the input dimension then the region defining each class must be connected. The applies to one-to-one activation functions, but could probably be extended to the family of activation functions in this paper by a similar limiting argument. Universality have been proved for a number of variants of the networks described in Theorem 1. Rojas BID26 showed that any two discrete classes of points can be separated by a decision boundary of a function defined by a deep, skinny network in which each layer has a single perceptron that is connected both to the previous layer and to the input layer. Because of the connections back to the input space, such a network is not layered as defined above, so Theorem 1 doesn't contradict this . In fact, to carry out Rojas' construction with a layered feed-forward network, you would need to put all the perceptrons in a single hidden layer. Sutskever and Hinton BID28 showed that deep belief networks whose hidden layers have the same dimension as the input space can approximate any function over binary vectors. This binary input space can be interpreted as a discrete subset of Euclidean space. So while Theorem 1 does not apply to belief networks, it's worth noting that any function on a discrete set can be extended to the full space in such a way that the ing function satisfies the constraints in Theorem 1.This unexpected constraint on skinny deep nets raises the question of whether such networks are so practically effective despite being more restrictive than wide networks, or because of it. Lin, Tegmark and Rolnick BID14 showed that for data sets with information-theoretic properties that are common in physics and elsewhere, deep networks are more efficient than shallow networks. This may be because such networks are restricted to a smaller search space concentrated around functions that model shapes of data that are more likely to appear in practice. Such a would be consistent with a number of papers showing that there are functions defined by deep networks that can only by approximated by shallow networks with asymptotically much larger number of nodes (4; 7; 10; 20; 31).A slightly different phenomenon has been observed for recurrent neural networks, which are universal approximators of dynamic systems BID6. In this setting, Collins, Sohl-Dickstein and Sussillo BID3 showed that many differences that have been reported on the performance of RNNs are due to their training effectiveness, rather than the expressiveness of the networks. In other words, the effectiveness of a given family of models appears to have less to do with whether it includes an accurate model, and more to do with whether a model search algorithm like gradient descent is likely to find an accurate model within the search space of possible models. A model family is a subset M of the space C(R n, R m) of continuous functions from input space R n to output space R m. For parametric models, this subset is typically defined as the image of a map DISPLAYFORM0, where R k is the parameter space. A non-parametric model family is typically the union of a countably infinite collection of parametric model families. We will not distinguish between parametric and non-parametric families in this section. Given a function g: R n → R m, a compact subset A ⊂ R n and a value > 0, we will say that a second function f (, A)-approximates g if for every x ∈ A, we have |f (x) − g(x)| <. Similarly, we will say that a model family M (, A)-approximates g if there is a function f in M that (, A)-approximates g. More generally, we will say that M approximates f if for every compact A ⊂ R n and value > 0 there is a function f in M that (, A)-approximates g. This is equivalent to the statement that there is a sequence of functions f i ∈ M that converges pointwise (though not necessarily uniformly) to g on all of R n. However, we will use the (, A) definition throughout this paper. We'll describe families of layered neural networks with the following notation: Given an activation function ϕ: R → R and a finite sequence of positive integers n 0, n 1,..., n κ, let N ϕ,n0,n1,...,nκ be the family of functions defined by a layered feed-forward neural network with n 0 inputs, n κ outputs and fully connected hidden layers of width n 1,..., n κ−1.With this terminology, Hornik et al's can be restated as saying that the (non-parametric) model family defined as the union of all families N ϕ,n0,n1,1 approximates any continuous function.(Here, κ = 2 and n 2 = 1.)We're interested in deep networks with bounded dimensional layers, so we'll let N * ϕ,n be the union of all the model families N ϕ,n0,n1,...,nκ−1,1 such that n i ≤ n for all i < κ. For the main , we will restrict our attention to a fairly large family of activation functions. We will say that an activation function ϕ is uniformly approximated by one-to-one functions if there is a sequence of continuous, one-to-one functions that converge to ϕ uniformly (not just pointwise).Note that if the activation function is itself one-to-one (such as a sigmoid) then we can let every function in the sequence be ϕ and it will converge uniformly. For the ReLu function, we need to replace the the large horizontal portion with a function such as 1 n arctan(x). Since this function is one-to-one and negative for x < 0, each function in this sequence will be one-to-one. Since it's bounded between − 1 n and 0, the sequence will converge uniformly to the ReLu function. The main of the paper is a topological constraint on the level sets of any function in the family of models N * ϕ,n. To understand this constraint, recall that in topology, a set C is path connected if any two points in C are connected by a continuous path within C. A path component of a set A is a subset C ⊂ A that is connected, but is not a proper subset of a larger connected subset of A. Definition 1. We will say that a function f: R n → R has unbounded level components if for every y ∈ R, every path component of f −1 (y) is unbounded. The main of this paper states that deep, skinny neural networks can only approximate functions with unbounded level components. Note that this definition is stricter than just requiring that every level set be bounded. The stricter definition in terms of path components guarantees that the property is preserved by limits, a fact that we will prove, then use in the proof of Theorem 1. Just having bounded level sets is not preserved under limits. Theorem 1. For any integer n ≥ 2 and uniformly continuous activation function ϕ: R → R that can be approximated by one-to-one functions, the family of layered feed-forward neural networks with input dimension n in which each hidden layer has dimension at most n cannot approximate any function with a level set containing a bounded path component. The proof of Theorem 1 consists of two steps. In the first step, described in Section 5, we examine the family of functions defined by deep, skinny neural networks in which the activation is one-to-one and the transition matrices are all non-singular. We prove two about this smaller family of functions: First, Lemma 2 states that any function that can be approximated by N * ϕ,n can be approximated by functions in this smaller family. This is fairly immediate from the assumptions on ϕ and the fact that singular transition matrices can be approximated by non-singular ones. Second, Lemma 4 states that the level sets of these functions have unbounded level components. The proof of this Lemma is, in many ways, the core argument of the paper and is illustrated in FIG0. The idea is that any function in this smaller family can be written as a composition of a one-to-one function and a linear projection, as in the top row of the Figure. As suggested in the bottom row, this implies that each level set/decision boundary in the full function is defined by the intersection of the image of the one-to-one function (the gray patch in the middle) with a hyperplane that maps to a single point in the second function. Intuitively, this intersection extends out to the edges of the gray blob, so its preimage in the original space must extend out to infinity in Euclidean space, i.e. it must be unbounded. The second part of the proof of Theorem 1, described in Section 5, is Lemma 5 which states that the limit of functions with unbounded level components also has unbounded level components. This is a subtle technical argument, though it should be intuitively unsurprising that unbounded sets cannot converge to bounded sets. The proof of Theorem 1 is the concatenation of these three Lemmas: If a function can be approximated by N * ϕ,n then it can be approximated by the smaller model family (Lemma 2), so it can be approximated by functions with unbounded level components (Lemma 4), so it must also have unbounded level components (Lemma 5). We will say that a function in N * ϕ,n is non-singular if ϕ is continuous and one-to-one, n i = n for all i < k and the matrix defined by the weights between each pair of layers is nonsingular. Note that if ϕ is not one-to-one, then N * ϕ,n will not contain any non-singular functions. If it is one-to-one then N * ϕ,n will contain a mix of singular and non-singular functions. Define the model family of non-singular functionsN n to be the union of all non-singular functions in families N * ϕ,n for all activation functions ϕ and a fixed n. Lemma 2. If g is approximated by N * ϕ,n for some continuous activation function ϕ that can be uniformly approximated by one-to-one functions then it is approximated byN n.To prove this Lemma, we will employ a technical from point-set topology, relying on the fact that a function in N * ϕ,n can be written as a composition of linear functions defined by the weights between successive layers, and non-linear functions defined by the activation function ϕ. DISPLAYFORM0 is a continuous function. Let A ⊂ R n be a compact subset and choose ε > 0. One can prove Lemma 3 by induction on the number of functions in the composition, choosing each A i ⊂ R ni to be a closed ε-neighborhood of the image of A in the composition up to i. For each new function, the δ on the compact set tells you what δ you need to choose for the composition of the preceding functions. We will not include the details here. Proof of Lemma 2. We'll prove this Lemma by showing thatN n approximates any given function in N * ϕ,n. Then, given ε > 0, a compact set A ⊂ R n and a function g that is approximated by N * ϕ,n, we can choose a function f ∈ N * ϕ,n that (ε/2, A)-approximates g and a function inN n that (ε/2, A)-approximates f. So we will reset the notation, let g be a function in N * ϕ,n, let A ⊂ R n be a compact subset and choose ε > 0. As noted above, g is a composition g = ν κ • κ • · · · • ν 0 • 0 where each i is a linear function defined by the weights between consecutive layers and each ν i is a nonlinear function defined by a direct product of the activation function ϕ.If any of the hidden layers in the network defining g have dimension strictly less than n then we can define the same function with a network in which that layer has dimension exactly n, but the weights in and out of the added neurons are all zero. Therefore, we can assume without loss of generality that all the hidden layers in g have dimension exactly n, though the linear functions may be singular. Let {A i} and δ > 0 be as defined by Lemma 3. We want to find functionsν i andˆ i that (δ, A i)-approximate each ν i and i and whose composition is inN n.For the composition to be inN n, we need eachˆ i to be non-singular. If i is already non-singular, then we chooseˆ i = i. Otherwise, we can perturb the weights that define the linear map i by an arbitrarily small amount to make it non-singular. In particular, we can choose this arbitrarily small amount to be small enough that the function values change by less than δ on A i. Similarly, we want eachν i to be a direct product of a continuous, one-to-one activation functions. By assumption, ϕ can be approximated by such functions and we can choose the tolerance for this approximation to be small enough thatν i (δ, A i)-approximates ν i. In fact, we can choose a single activation function for all the nonlinear layers, on each corresponding compact set. Thus we can choose eachˆ i and an activation functionφ that defines all the functions ν i, so that the composition is inN n and, by Lemma 3, the composition (ε, A)-approximates g. Lemma 2 implies that if N * ϕ,n is universal then so isN n. So to prove Theorem 1, we will show that every function inN n has level sets with only unbounded components, then show that this property extends to any function that it approximates. Lemma 4. If f is a function inN n then every level set f −1 (y) is homeomorphic to an open (possibly empty) subset of R n−1. This implies that f has unbounded level components. Proof. Assume f is a non-singular function inN n, where ϕ is continuous and one-to-one. Let f: R n → R n be the function defined by all but the last layer of the network. Letf: R n → R be the function defined by the map from the last hidden layer to the final output layer so that f =f •f.The functionf is a composition of the linear functions defined by the network weights and the nonlinear function at each step defined by applying the activation function to each dimension. Because f is nonsingular, the linear functions are all one-to one. Because ϕ is continuous and one-to-one, so are all the non-linear functions. Thus the compositionf is also one-to-one, and therefore a homeomorphism from R n onto its image If. Since R n is homeomorphic to an open n-dimensional ball, If is an open subset of R n, as indicated in the top row of FIG0.The functionf is the composition of a linear function to R with ϕ, which is one-to-one by assumption. So the preimagef −1 (y) for any y ∈ R is an (n − 1)-dimensional plane in R n. The preimage f −1 (y) is the preimage inf of this (n − 1)-dimensional plane, or rather the preimage of the intersection If ∩f −1 (y), as indicated in the bottom right/center of the Figure. Since If is open as a subset of R n, the intersection is open as a subset off −1 (y).Sincef is one-to-one, its restriction to this preimage (shown on the bottom left of the Figure) is a homeomorphism from f −1 (y) to this open subset of the (n − 1)-dimensional planef −1 (y). Thus f −1 (y) is homeomorphic to an open subset of R n−1.Finally, recall that the preimage in a continuous function of a closed set is closed, so f −1 (y) is closed as a subset of R n. If it were also bounded, then it would be compact. However, the only compact, open subset of R n−1 is the empty set, so f −1 (y) is either unbounded or empty. Since each path component of a subset of R n−1 is by definition non-empty, this proves that any component of f is unbounded. All that remains is to show that this property extends to the functions thatN n approximates. If M is a model family in which every function has unbounded level components then any function approximated by M has unbounded level components. Proof. Let g: R n → R be a function with a level set g −1 (y) containing a bounded path component C. Note that level sets are closed as subsets of R n and bounded, closed sets are compact so C is compact. We can therefore choose a value µ such that any point of g −1 (y) outside of C is distance greater than µ from every point in C.Let η C be the set of all points that are distance strictly less than µ/2 from C. This is an open subset of R n, shown as the shaded region in the center of FIG1, and we will let F be the frontier of η C -the set of all points that are limit points of both η C and limit points of its complement. By construction, every point in F is distance µ/2 from C so F is disjoint from C. Moreover, since every point in g −1 (y) \ C is distance at least µ from C, F is disjoint from the rest of g −1 (y) as well, so y is in the complement of g(F).The frontier is the intersection of two closed sets, so F is closed. It's also bounded, since all points are a bounded distance from C, so F is compact. This implies that g(F) is a compact subset of R, so its complement is open. Since y is in the complement of g(F), this means that there is an open interval U = (y − ε, y + ε) that is disjoint from g(F).LetĈ be the component of g −1 (U) that contains C, as indicated on the right of the Figure. Note that this set intersects η C but is disjoint from its frontier. SoĈ must be contained in η C, and is therefore bounded as well. In particular, each level set that intersectsĈ has a compact component inĈ.Let x be a point in C ⊂Ĉ. SinceĈ is bounded, there is a value r such that every point inĈ is distance at most r from x. Assume for contradiction that g is approximated by a model family M in which each function has unbounded level components. Choose R > r and let B R (x) be a closed ball of radius R, centered at x. Because this is a compact set and g is approximated by M, we can choose a function f ∈ M that (ε/2, B R (x))-approximates g. Then |f (x) − g(x)| < ε/2 so f (x) ∈ [y − ε/2, y + ε/2] ⊂ U and we will define y = f (x).Since f ∈ M, every path component of f −1 (y) is unbounded, so there is a path ⊂ f −1 (y) from x to a point that is distance R from x. If passes outside of B R (x)), we can replace with the component of ∩ B R (x)) containing x to ensure that stays inside of B R (x)), but still reaches a point that is distance R from x. Since every point x ∈ is contained in B R (x), we have |f (x) − g(x)| < ε/2. This implies g(x) ∈ [y − ε, y + ε] = U so the path is contained in g −1 (U), and thus in the path componentĈ of g −1 (U).However, by construction the path ends at a point whose distance from x is R > r, contradicting the assumption that every point inĈ is distance at most r from x. This contradiction proves that g cannot be approximated by a model family M in which each function has unbounded level components. Proof of Theorem 1. Let g be a function that is approximated by N * ϕ,n, where ϕ is a continuous activation function that can be uniformly approximated by one-to-one functions. By Lemma 2, since g is approximated by N (a) The decision boundary learned with six, two-dimensional hidden layers is an unbounded curve that extends outside the region visible in the image.(b) A network with a single threedimensional hidden layer learns a bounded decision boundary relatively easily. To demonstrate the effect of Theorem 1, we used the TensorFlow Neural Network Playground to train two different networks on a standard synthetic dataset with one class centered at the origin of the two-dimensional plane, and the other class forming a ring around it. We trained two neural networks and examined the plot of the ing functions to characterize the level sets/decision boundaries. In these plots, the decision boundary is visible as the white region between the blue and orange regions defining the two labels. The first network has six two-dimensional hidden layers, the maximum number of layers allowed in the webapp. As shown in Figure 3a, the decision boundary is an unbounded curve that extends beyond the region containing all the data points. The ideal decision boundary between the two classes of points would be a (bounded) loop around the blue points in the middle, but Theorem 1 proves that such a network cannot approximate a function with such a level set. A decision boundary such as the one shown in the Figure is as close as it can get. The extra hidden layers allow the decision boundary to curve around and minimize the neck of the blue region, but they do not allow it to pinch off completely. The second network has a single hidden layer of dimension three -one more than that of the input space. As shown in Figure 3b, the decision boundary for the learned function is a loop that approximates the ideal decision boundary closely. It comes from the three lines defined by the hidden nodes, which make a triangle that gets rounded off by the activation function. Increasing the dimension of the hidden layer would make the decision boundary rounder, though in this case the model doesn't need the extra flexibility. Note that this example generalizes to any dimension n, though without the ability to directly graph the . In other words, for any Euclidean input space of dimension n, a sigmoid neural network with one hidden layer of dimension n + 1 can define a function that cannot be approximated by any deep network with an arbitrary number of hidden layers of dimension at most n. In fact, this will be the case for any activation function that is bounded above or below, though we will not include the details of the argument here. In this paper, we describe topological limitations on the types of functions that can be approximated by deep, skinny neural networks, independent of the number of hidden layers. We prove the using standard set theoretic topology, then present examples that visually demonstrate the .

represent this text

E.g. We consider new variants of optimization algorithms for training deep nets. == We consider new variants of optimization algorithms. Our algorithms are based on the observation that mini-batch of stochastic gradients in consecutive iterations do not change drastically and consequently may be predictable. Inspired by the similar setting in online learning literature called Optimistic Online learning, we propose two new optimistic algorithms for AMSGrad and Adam, respectively, by exploiting the predictability of gradients. The new algorithms combine the idea of momentum method, adaptive gradient method, and algorithms in Optimistic Online learning, which leads to speed up in training deep neural nets in practice. Nowadays deep learning has been shown to be very effective in several tasks, from robotics (e.g. BID15), computer vision (e.g. BID12 ; BID9), reinforcement learning (e.g. BID18, to natural language processing (e.g.). Typically, the model parameters of a state-of-the-art deep neural net is very high-dimensional and the required training data is also in huge size. Therefore, fast algorithms are necessary for training a deep neural net. To achieve this, there are number of algorithms proposed in recent years, such as AMSGRAD , ADAM BID13 ), RMSPROP , ADADELTA , and NADAM BID6 ), etc. All the prevalent algorithms for training deep nets mentioned above combines two ideas: the idea of adaptivity in ADAGRAD BID7 BID17 ) and the idea of momentum as NESTEROV'S METHOD BID19 ) or the HEAVY BALL method BID20 ). ADAGRAD is an online learning algorithm that works well compared to the standard online gradient descent when the gradient is sparse. The update of ADAGRAD has a notable feature: the learning rate is different for different dimensions, depending on the magnitude of gradient in each dimension, which might help in exploiting the geometry of data and leading to a better update. On the other hand, NESTEROV'S METHOD or the Momentum Method BID20 ) is an accelerated optimization algorithm whose update not only depends on the current iterate and current gradient but also depends on the past gradients (i.e. momentum). State-of-the-art algorithms like AMSGRAD and ADAM BID13 ) leverages these two ideas to get fast training for neural nets. In this paper, we propose an algorithm that goes further than the hybrid of the adaptivity and momentum approach. Our algorithm is inspired by OPTIMISTIC ONLINE LEARNING BID4;;; BID0 ). OPTIMISTIC ONLINE LEARNING considers that a good guess of the loss function in the current round of online learning is available and plays an action by exploiting the good guess. By exploiting the guess, those algorithms in OPTIMISTIC ONLINE LEARNING have regret in the form of O(T t=1 g t − m t), where g t is the gradient of loss function in round t and m t is the "guess" of g t before seeing the loss function in round t (i.e. before getting g t). This kind of regret can be much smaller than O(√ T) when one has a good guess m t of g t. We combine the OPTIMISTIC ONLINE LEARNING idea with the adaptivity and the momentum ideas to design new algorithms in training deep neural nets, which leads to NEW-OPTIMISTIC-AMSGRAD and NEW-OPTIMISTIC-ADAM. We also provide theoretical analysis of NEW-OPTIMISTIC-AMSGRAD. The proposed OPTIMISTIC-algorithms not only adapt to the informative dimensions and exhibit momentums but also take advantage of a good guess of the next gradient to facilitate acceleration. We evaluate our algorithms with BID13 ), and BID5 ). Experiments show that our OPTIMISTIC-algorithms are faster than the baselines. We should explain that BID5 proposed another version of optimistic algorithm for ADAM, which is referred to as ADAM-DISZ in this paper. We apply the idea of BID5 ) on AMSGRAD, which leads to AMSGRAD-DISZ. Both ADAM-DISZ and AMSGRAD-DISZ are used as baselines. Both AMSGRAD and ADAM BID13 ) are actually ONLINE LEARNING algorithms. They use REGRET ANALYSIS to provide some theoretical guarantees of the algorithms. Since one can convert an online learning algorithm to an offline optimization algorithm by online-to-batch conversion BID3 ), one can design an offline optimization algorithm by designing and analyzing its counterpart in online learning. Therefore, we would like to give a brief review of ONLINE LEARNING and OPTIMISTIC-ONLINE LEARNING. In the typical setting of online learning, there is a LEARNER playing an action and then receiving a loss function in each round t. Specifically, the learner plays an action w t ∈ K in round t, where w t is chosen in a compact and convex set K ⊆ R n, known as the DECISION SPACE. Then, the learner sees the LOSS FUNCTION t (·) and suffers loss t (w t) for the choice. No distributional assumption is made on the loss functions sequence {1 (·), 2 (·),..., T (·)} in ONLINE LEARNING. Namely, the loss functions can be adversarial. The goal of an online learner is minimizing its REGRET, which is DISPLAYFORM0 We can also define AVERAGE REGRET as Regret T:= DISPLAYFORM1, which is REGRET divided by number of rounds T. In ONLINE LEARNING literature, NO-REGRET ALGORITHMS means online learning algorithms satisfying Regret T → 0 as T → ∞. In recent years, there is a branch of works in the paradigm of OPTIMISTIC ONLINE LEARNING (e.g. BID4 ; ; ; BID0). The idea of OPTIMISTIC ONLINE LEARNING is as follows. Suppose that, in each round t, the learner has a good guess m t (·) of the loss function t (·) before playing an action w t. (Recall that the learner receives the loss function after the learner commits an action!) Then, the learner should exploit the guess m t (·) to choose an action w t, as m t (·) is close to the true loss function t (·). DISPLAYFORM0 where R(·) is a 1-strong convex function with respect to a norm (·) on the constraint set K, L t−1:= t−1 s=1 g s is the cumulative sum of gradient vectors of the convex loss functions (i.e. g s := ∇ s (w s) ) up to but not including t, and η is a parameter. The OPTIMISTIC-FTRL of has update DISPLAYFORM1 where m t is the learner's guess of the gradient vector g t:= ∇ t (w t). Under the assumption that loss functions are convex, the regret of OPTIMISTIC-FTRL satisfies Regret T ≤ O(T t=1 g t − m t *), which can be much smaller than O(√ T) of FTRL if m t is close to g t. Consequently, OPTIMISTIC-FTRL will have much smaller regret than FTRL. On the other hand, if m t is far from g t, then the regret of OPTIMISTIC-FTRL would be a constant factor worse than that of FTRL without optimistic update. In the later section, we provide a way to get m t. Here, we just want to emphasize the importance of leveraging a good guess m t for updating w t to get a fast convergence rate (or equivalently, small regret). We also note that the works of;; ) has been shown to accelerate the convergence of some zero-sum games. ADAM BID13 ) is a very popular algorithm for training deep nets. It combines the momentum idea BID20 with the idea of ADAGRAD BID7 ), which has individual learning rate for different dimensions. The learning rate of ADAGRAD in iteration t for a dimension j is proportional to the inverse of Σ t s=1 g s [j] 2, where g s [j] is the j th element of the gradient vector g s in time s. This adaptive learning rate may help for accelerating the convergence when the gradient vector is sparse BID7 ). However, when applying ADAGRAD to train deep nets, it is observed that the learning rate might decay too fast BID13 Get mini-batch stochastic gradient vector g t ∈ R d at w t.5: DISPLAYFORM0 6: DISPLAYFORM1 8: DISPLAYFORM2 (element-wise division) 9: end for ) proposes using a moving average of gradients (element-wise) divided by the root of the second moment of the moving average to update model parameter w (i.e. line 5,6 and line 8 of Algorithm 1). Yet, ADAM BID13 ) fails at some online convex optmization problems. AMSGRAD fixes the issue. The algorithm of AMSGRAD is shown in Algorithm 1. The difference between ADAM and AMSGRAD lies on line 7 of Algorithm 1. ADAM does not have the update of line 7. adds the step to guarantee a nonincreasing learning rate, ηt √v t. which helps for the convergence (i.e. average regret Regret T → 0.) For the parameters of AMSGRAD, it is suggested that β 1 = 0.9, β 2 = 0.99 and η t = η/ √ t for a number η. As mentioned in the introduction, BID5 proposed one version of optimistic algorithm for ADAM, which is referred to as ADAM-DISZ in this paper. BID5 did not propose an optimistic algorithm for AMSGRAD but such an extension is straightforward which is referred to as AMSGRAD-DISZ.In this section, we propose a new algorithm for training deep nets: NEW-OPTIMISTIC-AMSGRAD, shown in Algorithm 2. NEW-OPTIMISTIC-AMSGRAD has an optimistic update, which is line 9 of Algorithm 2. It exploits the guess m t+1 of g t+1 to get w t+1, since the vector h t+1 uses m t+1. Notice that the gradient vector is computed at w t instead of w t− 1 2 and the moving average of gradients is used to update w t+ 1 2. One might want to combining line 8 and line 9and getting a single line: DISPLAYFORM0. From this, we see that w t+1 is updated from w t− 1 2 instead of w t. Therefore, while NEW-OPTIMISTIC-AMSGRAD looks like just doing an additional update compared to AMSGRAD, the difference of update is subtle. We also want to emphasize that although the learning rate on line 9 contains 4 1−β1 factor. We suspect that it is due to the artifact of our theoretical analysis. In our experiments, the learning rate on line 9 does not have the factor of 4 1−β1. That is, in practice, we implement line 9 as w t+1 = w t+ DISPLAYFORM1 We leave closing the gap between theory and practice as a future work. We see that NEW-OPTIMISTIC-AMSGRAD inherits three properties• Adaptive learning rate of each dimension as ADAGRAD BID7 ). (line 8)• Exponentially moving average of the past gradients as NESTEROV'S METHOD BID19 ) and the HEAVY-BALL method BID20 The first property helps acceleration when the gradient has sparse structure. The second one is the well-recognized idea of momentum which can achieve acceleration. The last one, perhaps less known outside the ONLINE LEARNING community, can actually achieve acceleration when the prediction of the next gradient is good. We are going to elaborate this property in the later section where we give the theoretical analysis of NEW-OPTIMISTIC-AMSGRAD.To obtain m t, we use the extrapolation algorithm of . Extrapolation studies estimating the limit of sequence using the last few iterates BID1 ). Some classical works include Anderson acceleration (Walker & Ni. ), minimal polynomial extrapolation BID2 ), reduced rank extrapolation BID8 ). These method typically assumes that the sequence {x t} ∈ R d has a linear relation DISPLAYFORM2 for an unknown matrix A ∈ R d×d (not necessarily symmetric). The goal is to use the last few iterates {x t} to estimate the fixed point x * on. adapt the classical extrapolation methods to the iterates/updates of Get mini-batch stochastic gradient vector g t ∈ R d at w t.5: DISPLAYFORM3 6: DISPLAYFORM4 DISPLAYFORM5 Obtain z by solving (U U + λI)z = 1. Get c = z/(z 1) 5: Output: Σ r−1 i=0 c i x i, the approximation of the fixed point x *.an optimization algorithm and propose an algorithm that produces a solution that is better than the last iterate of the underlying optimization algorithm in practice. The algorithm of (Scieur et al. FORMULA0) (shown in Algorithm 3) allows the iterates {x t} to be nonlinear DISPLAYFORM0 where e t is a second order term (namely, satisfying e t 2 = O( x t−1 − x * 2 2). Some theoretical guarantees regarding the distance between the output and x * is provided in .In NEW-OPTIMISTIC-AMSGRAD, we use Algorithm 3 to get m t. Specifically, m t is obtained by• Call Algorithm 3 with input being a sequence of some past r + 1 gradients, {g t, g t−1, g t−2, . . ., g t−r} to obtain m t, where r is a parameter.• Set m t:= Σ r−1 i=0 c i g t−r+i from the output of Algorithm 3.If the past few gradients can be modeled by FORMULA9 approximately, the extrapolation method should be expected to work well in predicting the gradient. In practice, it helps to achieve faster convergence. NEW-OPTIMISTIC-ADAM By removing line 7 in Algorithm 2, the step of making monotone weighted second moment, we obtain an algorithm which we call it NEW-OPTIMISTIC-ADAM, as the ing algorithm can be viewed as an OPTIMISTIC-variant of ADAM. We provide the regret analysis here. We denote the Mahalanobis norm · H = ·, H· for some PSD matrix H. For the PSD matrix diag{v t}, where diag{v t} represents the diagonal matrix such that its i th diagonal element isv t [i] in Algorithm 2, we define the the corresponding Mahalanobis norm · ψt:= ·, diag{v t} 1/2 ·, where we use the notation ψ t to represent the matrix diag{v t} 1/2. We can also define the the corresponding dual norm DISPLAYFORM0 We assume that the model parameter w is in d-dimensional space. That is, w ∈ R d. Also, the analysis of NEW-OPTIMISTIC-AMSGRAD is for unconstrained optimization. Thus, we assume that the constraint K of the benchmark in the regret definition, min w∈K T t=1 t (w), is a finite norm ball that contains the optimal solutions to the underlying offline unconstrained optimization problem. Now we can conduct our analysis. First of all, we can decompose the regret as follows. DISPLAYFORM1 where the first inequality is by assuming that the loss function t (·) is convex and that we use the notation g t:= ∇ t (w t) which we adopt throughout the following proof for brevity. Given the decomposition, let us analyze the first term DISPLAYFORM2 Astute readers may realize that the bound in Lemma 1 is actually the bound of AMSGRAD. Indeed, since in online learning setting the loss vectors g t come adversarially, it does matter how g t is generated. Therefore, the regret of DISPLAYFORM3 − w *, g t can be bounded in the same way as AMSGRAD. In Appendix B, we provide the detail proof of Lemma 1. Now we switch to bound the other sums DISPLAYFORM4, h t in. The proof is available in Appendix C. DISPLAYFORM5 Combining and Lemma 1 and 2 leads to DISPLAYFORM6 Now we can conclude the following theorem. The proof is in Appendix D. DISPLAYFORM7 One should compare the bound with that of AMSGRAD , which is DISPLAYFORM8 where η t = η/ √ t in their setting. We need to compare the last two terms in with the last term in. We are going to show that, under certain conditions, the bound is smaller than that of AMSGRAD. Let us suppose that g t is close to m t so that DISPLAYFORM9 is much smaller than the last term of. Yet, the last term of, DISPLAYFORM10 ψt−1) *, might be actually o(√ T) and consequently might also be smaller than the last term of. To see this, let us rewrite DISPLAYFORM11 Assume that if each DISPLAYFORM12 in the inner sum is bounded by a constant c, we will get DISPLAYFORM13 Yet, the denominator v t−1 is non-decreasing so that we can actually have a smaller bound. That is, in practice, v t−1 [i] might grow over time, and the growth rate is different for different dimension i. DISPLAYFORM14 then the last term is just O(log T), which might be better than that of the last term on. One can also get a data dependent bound of the last term of. To summarize, when m t is close to g t, NEW-OPTIMISTIC-AMSGRAD can have a smaller regret (thus better convergence rate) than ADAM (Kingma & Ba FORMULA0) and AMSGRAD . Algorithm 4 ADAM-DISZ BID5 1: Required: parameter β 1, β 2, and η t. 2: Init: DISPLAYFORM0 Get mini-batch stochastic gradient vector g t ∈ R d at w t.5: DISPLAYFORM1 6: DISPLAYFORM2 7: DISPLAYFORM3 We are aware of Algorithm 1 in BID5, which was also motivated by OPTIMISTIC ONLINE LEARN-ING 2. For comparison, we replicate ADAM-DISZ in Algorithm 4. We are going to describe the differences of the algorithms and the differences of the contributions between our work and their work. FORMULA0 ) showing that if both players use some kinds of OPTIMISTIC-update, then acceleration to the convergence of the minimax value of the game is possible. BID5 was inspired by these related works and showed that OPTIMISTIC-MIRROR-DESCENT can avoid the cycle behavior in a bilinear zero-sum game, which accelerates the convergence. Our work is about solving min x f (x) (e.g. empirical risk) quickly. We also show that ADAM-DISZ suffers the non-convergence issue as ADAM. The proof is available in Appendix E.Theorem 2. There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e. One might wonder if the non-convergence issue can be avoided if one let the weighted second moment of ADAM-DISZ be monotone by adding the stepv t = max(v t−1, v t) as AMSGRAD, which we call it AMSGRAD-DISZ. Unfortunately, we are unable to prove if the step guarantees convergence or not. To demonstrate the effectiveness of our proposed method, we test its performance with various neural network architectures, including fully-connected neural networks, convolutional neural networks (CNN's) and recurrent neural networks (RNN's). The illustrate that NEW-OPTIMISTIC-AMSGRAD is able to speed up the convergence of state-of-art AMSGRAD algorithm, making the learning process more efficient. For AMSGRAD algorithm, we set the parameter β 1 and β 2, respectively, to be 0.9 and 0.999, as recommended in. We tune the learning rate η over a fine grid and report the under best-tuned parameter setting. For NEW-OPTIMISTIC-AMSGRAD and AMSGRAD-DISZ, we use same β 1, β 2 and learning rate as those for AMSGRAD to make a fair comparison of the enhancement brought by the optimistic step. We use the same weight initialization for all algorithms. The remaining tuning parameter of NEW-OPTIMISTIC-AMSGRAD is r, the number of previous gradients that we use to predict the next move. We conduct NEW-OPTIMISTIC-AMSGRAD with different values of r and observe similar performance (See Appendix A). Hence, we report r = 15 for all experiments for tidiness of plots. To follow previous works of and BID13, we compare different methods on MNIST, CIFAR10 and IMDB datasets in our experiment. For MNIST, we use a noisy version named as MNIST-back-rand in BID14 to increase the training difficulty. Our experiments start with fully connected neural network for multi-class classification problems. MNIST-back-rand dataset consists of 12000 training samples and 50000 test samples, where random is inserted in original MNIST hand written digit images. The input dimension is 784 (28×28) and the number of classes is 10. We investigate a multi-layer neural networks with input layer followed by a hidden layer with 200 cells, which is then connected to a layer with 100 neurons before the output layer. All hidden layer cells are rectifier linear units (ReLu's). We use mini-batch size 128 to calculate stochastic gradient in each iteration. Model performance is evaluated by multi-class cross entropy loss. The training loss with respect to number of iterations is reported in FIG0. On this dataset, we empirically observe obvious improvement of NEW-OPTIMISTIC-AMSGRAD in terms of both convergence speed and training loss. On the other hand, AMSGRAD-DISZ performs similarly to AMSGRAD in general. Convolutional Neural Networks (CNN) have been widely studied and is important in various deep learning applications such as computer vision and natural language processing. We test the effectiveness of NEW-OPTIMISTIC-AMSGRAD in deep CNN's with dropout. We use the CIFAR10 dataset, which includes 60,000 images (50,000 for training and 10,000 for testing) of size 32 × 32 in 10 different classes. ALL-CNN architecture proposed in Springenberg et al. FORMULA0 is implemented with two blocks of 3 × 3 convolutional filter, 3 × 3 convolutional layer with stride 2 and dropout layer with keep probability 0.5. Another block of 3 × 3, 1 × 1 convolutional layer and a 6 × 6 global averaging pooling is added before the output layer. We apply another dropout with keep probability 0.8 on the input layer. The cost function is multi-class cross entropy. The batch size is 128. The images are all whitened. The training loss is provided in FIG5. The shows that NEW-OPTIMISTIC-AMSGRAD accelerates the learning process significantly and gives lowest training cost after 10000 iterations. For this dataset, the performance of AMSGRAD-DISZ is worse than original AMSGRAD. As another important application of deep learning, natural language processing tasks often benefit from considering sequence dependency in the models. Recurrent Neural Networks (RNN's) achieves this goal by adding hidden state units that act as "memory". Long-Short Term Memory (LSTM) is the most popular structure in building RNN's. We use IMDB movie review dataset from BID16 to test the performance of NEW-OPTIMISTIC-AMSGRAD in RNN's under the circumstance of high data sparsity. IMDB is a binary classification dataset with 25000 training and test samples respectively. Our model includes a word embedding layer with 5000 input entries representing most frequent words in the dataset and each word is embedded into a 32 dimensional space. The output of embedding layer is passed to 100 LSTM units, which is then connected to 100 fully connected ReLu's before reaching the output layer. Binary cross-entropy loss is used and the batch size is 128. We provide the in figure 3. We observe a considerable improvement in convergence speed. In the first epoch, the is more exciting. At epoch 0.5, NEW-OPTIMISTIC-AMSGRAD already achieves the training loss that vanilla AMSGRAD can produce with more than 1 epoch. The sample efficiency is significantly improved. On this dataset, AMSGRAD-DISZ performs less effectively and may be trapped in local minimum. We remark that in each iteration, only a small portion of gradients in embedding layer is non-zero. Thus, this experiment demonstrates that NEW-OPTIMISTIC-AMSGRAD could also perform well with sparse gradient. (See additional experiments in Appendix A.) In this paper, we propose NEW-OPTIMISTIC-AMSGRAD, which combines optimistic learning and AMSGRAD to strengthen the learning process of optimization problems, in particular, deep neural networks. The idea of adding optimistic step can be easily extended to other optimization algorithms, e.g ADAM and ADAGRAD. We provide OPTIMISTIC-ADAGRAD algorithm and theoretical in Appendix F. A potential direction based on this work is to improve the method for predicting next gradient. We expect that optimistic acceleration strategy could be widely used in various optimization problems. In NEW-OPTIMISTIC-AMSGRAD, parameter r, the number of previous gradients used, may affect the performance dramatically. If we choose r too small (e.g r < 10), the optimistic updates will start at very early stage, but the information we collect from the past is limited. This may make our training process off-track at first several iterations. On the other hand, if r is chosen to be too large (e.g r >= 30), although we may get a better prediction of next gradient, the optimistic step will start late so NEW-OPTIMISTIC-AMSGRAD may miss great chances to improve the learning performance at early stages. Additionally, we need more room to store past gradients, hence the operational time will increase. The empirical impact of different r value is reported in FIG7. We suggest 10 ≤ r ≤ 20 as an ideal range, and r = 15 tend to perform well in most training tasks. Actually, we may make the algorithm more flexible by "early start". For example, when we set r = 20, we can instead start adding optimistic step at iteration 10, and gradually increase the number of past gradients we use to predict next move from 10 to 20 in next 10 iterations. After iteration 21, we fix the moving window size for optimistic prediction as 20. This may bring enhancement to NEW-OPTIMISTIC-AMSGRAD because it can seek opportunities to accelerate learning in first several iterations, which is critical for indicating a better direction towards the minimum loss. We also conduct experiments on NEW-OPTIMISTIC-ADAM and ADAM-DISZ. The experiment setting is similar to NEW-OPTIMISTIC-AMSGRAD, where we fix β 1, β 2 and compare the performance using the best learning rate with respect to ADAM. We provide a brief summary of the in FIG8, with r = 15. The improvement brought by NEW-OPTIMISTIC-ADAM is obvious, indicating that adding optimistic step could also enhance the performance of ADAM optimizer. B PROOF OF LEMMA 1 DISPLAYFORM0 Proof. The proof of this lemma basically follows that of . We have that DISPLAYFORM1 By rearranging the terms above and summing it from t = 1,..., T, DISPLAYFORM2 where the first inequality is due to Young's inequality and the second one is due to the constraint 0 < β 1 ≤ 1 so that DISPLAYFORM3 where the first inequality is due to.Continue the analysis, we have DISPLAYFORM4 where the last equality is by telescoping sum. DISPLAYFORM5 Proof. From the update, w t = w t− 1 2 DISPLAYFORM6 Therefore, we have that DISPLAYFORM7 where (a) is by Hölder's inequality and (b) is by Young's inequality. D PROOF OF THEOREM 1 DISPLAYFORM8 Proof. Recall that from FORMULA20, we show that DISPLAYFORM9 To proceed, let us analyze DISPLAYFORM10 above. Notice that DISPLAYFORM11 and DISPLAYFORM12 where we use the update rule, w t = w t− 1 2 DISPLAYFORM13. Therefore, we have that DISPLAYFORM14 where the second inequality is due to that the sequence {v t [i]} is non-decreasing and the last equality is because DISPLAYFORM15 To summarize, DISPLAYFORM16 E PROOF OF THEOREM 2Theorem 2 There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e. We basically follow the same setting as Theorem 1 of . In each round, the loss function t (w) is linear and the learner's decision space is DISPLAYFORM0 where C ≥ 4. For this loss function sequences, the point w = −1 achieves the minimum regret, i.e. − 1 = arg min w∈K T t=1 f t (w) when T → ∞. Consider the execution of ADAM-DISZ with DISPLAYFORM1 β 2 < 1 in this case, which satisfies the conditions of BID13.The goal is to show that for all t, w 3t+1 = 1, and w t > 0. Let us denoteŵ t+1: DISPLAYFORM2 Assume the initial point is 1. As pointed it out by , the assumption of the initial point is without loss of generality. If the initial point is not 1, then one can translate the coordinate system to a new coordinate system so that the initial point is 1 and then choose the loss function sequences for the new coordinate system. Therefore, the base case x 1 = 1 is true. Now assume that for some t > 0, x 3t+1 = 1 and x s > 0 for all s ≤ 3t + 1. Observe that.∇f t (w) = C, for t mod 3 = 1 −1, otherwise, According to the update of ADAM-DISẐ DISPLAYFORM3 So,ŵ 3t+2 = w 3t+2. DISPLAYFORM4 Forŵ 3t+4, let us first consider the case thatŵ 3t+3 < 1 DISPLAYFORM5 (3t + 1)(1 − β 2)) + η (3t + 1)(β 2 C 2 + (1 − β 2)) + 2η (3t + 3)(β 2 C 2 + (1 − β 2)) is a decreasing function of t and has its largest value at t = 1 for any t > 0, which is 1 − (DISPLAYFORM6) ≤ 0.12. To summarize, we have that w 3t+4 = 1. Now if it is the case thatŵ 3t+3 ≥ 1 w 3t+4 = min(ŵ 3t+3, 1) + 2η (3t + 3)(β 2 v 3t+2 + (1 − β 2)) − η (3t + 3)(β 2 v 3t+1 + (1 − β 2)) = 1 + 2η (3t + 3)(β 2 v 3t+2 + (1 − β 2)) − η (3t + 3)(β 2 v 3t+1 + (1 − β 2)) > 1where the last inequality is because 2 β 2 v 3t+1 + (1 − β 2) ≥ β 2 v 3t+2 + (1 − β 2)To see this, β 2 v 3t+2 + (1 − β 2) = β 2 (β 2 v 3t+1 + (1 − β 2)) + (1 − β 2) = (β 2 2)v 3t+1 + 1 − β 2 2 So, the above inequality is equivalent to DISPLAYFORM7 which is true. This means that w 3t+4 = 1. Therefore, we have completed the induction. To summarize, we have 3t+1 (w 3t+1) + 3t+2 (w 3t+2) + 3t+3 (w 3t+3) − 3t+1 (−1) − 3t+2 (w −1) − 3t+3 (−1) ≥ 2C − 4. That is, for every 3 steps, the algorithms suffers regret at least 2C − 4, this means that the regret over T rounds would be (2C − 4)T /3, which is not sublinear to T. Now we have completed the proof. One may follow the analysis of Theorem 2 of to generalize the so that ADAM-DISZ does not converge for any β 1, β 2 ∈ such that β 1 < √ β 2. The optimistic update step can also be extended to other algorithms as well. For example, based on ADAGRAD, we propose OPTIMISTIC-ADAGRAD by including the optimistic update step. Algorithm 5 OPTIMISTIC-ADAGRAD (UNCONSTRAINED) Get current gradient g t at w t. − ηdiag(G t) −1/2 m t+1, where m t+1 is the guess of g t+1. 7: end for We provide the regret analysis here. Let us recall the notations and assumptions first. We denote the Mahalanobis norm · H = ·, H· for some PSD matrix H. We let ψ t (x):= x, diag{v t} 1/2 x for the PSD matrix diag{v t}, where diag{v t} represents the diagonal matrix such that its i th diagonal element isv t [i] in Algorithm 2. Consequently, ψ t (·) is 1-strongly convex with respect to the norm · ψt:= ·, diag{v t} 1/2 ·. Namely, ψ t (·) satisfies ψ t (u) ≥ ψ t (v) + ψ t (v), u − v + (u, v) is defined as B ψt (u, v):= ψ t (u)−ψ t (v)− ψ t (v), u−v and ψ t is called the distance generating function. We can also define the associate dual norm as ψ * t (x):= x, diag{v t} −1/2 x. We assume that the model parameter w is in d-dimensional space. That is, w ∈ Rd. It suffices to analyze Algorithm 6, which holds for any convex set K. The algorithm reduces to Algorithm 5 when K = R d.

This paper proves that skinny neural networks cannot approximate certain functions, no matter how deep they are.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Contrary to previous beliefs, the training performance of deep networks, when measured appropriately, is predictive of test performance, consistent with classical machine learning theory.