Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks. Large deep neural networks have enabled breakthroughs in fields such as computer vision BID22, speech recognition, and reinforcement learning BID28. In most successful applications, these neural networks share two commonalities. First, they are trained as to minimize their average error over the training data, a learning rule also known as the Empirical Risk Minimization (ERM) principle BID35. Second, the size of these state-of-theart neural networks scales linearly with the number of training examples. For instance, the network of BID31 used 10 6 parameters to model the 5 · 10 4 images in the CIFAR-10 dataset, the network of BID30 Strikingly, a classical in learning theory BID36 tells us that the convergence of ERM is guaranteed as long as the size of the learning machine (e.g., the neural network) does not increase with the number of training data. Here, the size of a learning machine is measured in terms of its number of parameters or, relatedly, its VC-complexity BID16.This contradiction challenges the suitability of ERM to train our current neural network models, as highlighted in recent research. On the one hand, ERM allows large neural networks to memorize (instead of generalize from) the training data even in the presence of strong regularization, or in classification problems where the labels are assigned at random. On the other hand, neural networks trained with ERM change their predictions drastically when evaluated on examples just outside the training distribution BID33, also known as adversarial examples. This evidence suggests that ERM is unable to explain or provide generalization on testing distributions that differ only slightly from the training data. However, what is the alternative to ERM?The method of choice to train on similar but different examples to the training data is known as data augmentation BID29, formalized by the Vicinal Risk Minimization (VRM) principle BID3. In VRM, human knowledge is required to describe a vicinity or neighborhood around each example in the training data. Then, additional virtual examples can be drawn from the vicinity distribution of the training examples to enlarge the support of the training distribution. For instance, when performing image classification, it is common to define the vicinity of one image as the set of its horizontal reflections, slight rotations, and mild scalings. While data augmentation consistently leads to improved generalization BID29, the procedure is dataset-dependent, and thus requires the use of expert knowledge. Furthermore, data augmentation assumes that the examples in the vicinity share the same class, and does not model the vicinity relation across examples of different classes. Contribution Motivated by these issues, we introduce a simple and data-agnostic data augmentation routine, termed mixup (Section 2). In a nutshell, mixup constructs virtual training examples DISPLAYFORM0 where x i, x j are raw input vectors y = λy i + (1 − λ)y j, where y i, y j are one-hot label encodings (x i, y i) and (x j, y j) are two examples drawn at random from our training data, and λ ∈. Therefore, mixup extends the training distribution by incorporating the prior knowledge that linear interpolations of feature vectors should lead to linear interpolations of the associated targets. mixup can be implemented in a few lines of code, and introduces minimal computation overhead. Despite its simplicity, mixup allows a new state-of-the-art performance in the CIFAR-10, CIFAR-100, and ImageNet-2012 image classification datasets (Sections 3.1 and 3.2). Furthermore, mixup increases the robustness of neural networks when learning from corrupt labels (Section 3.4), or facing adversarial examples (Section 3.5). Finally, mixup improves generalization on speech (Sections 3.3) and tabular (Section 3.6) data, and can be used to stabilize the training of GANs (Section 3.7). The source-code necessary to replicate our CIFAR-10 experiments is available at:https://github.com/facebookresearch/mixup-cifar10.To understand the effects of various design choices in mixup, we conduct a thorough set of ablation study experiments (Section 3.8). The suggest that mixup performs significantly better than related methods in previous work, and each of the design choices contributes to the final performance. We conclude by exploring the connections to prior work (Section 4), as well as offering some points for discussion (Section 5). In supervised learning, we are interested in finding a function f ∈ F that describes the relationship between a random feature vector X and a random target vector Y, which follow the joint distribution P (X, Y). To this end, we first define a loss function that penalizes the differences between predictions f (x) and actual targets y, for examples (x, y) ∼ P. Then, we minimize the average of the loss function over the data distribution P, also known as the expected risk: DISPLAYFORM0 Unfortunately, the distribution P is unknown in most practical situations. Instead, we usually have access to a set of training data DISPLAYFORM1, where (x i, y i) ∼ P for all i = 1,..., n. Using the training data D, we may approximate P by the empirical distribution DISPLAYFORM2 where δ(x = x i, y = y i) is a Dirac mass centered at (x i, y i). Using the empirical distribution P δ, we can now approximate the expected risk by the empirical risk: DISPLAYFORM3 Learning the function f by minimizing is known as the Empirical Risk Minimization (ERM) principle BID35. While efficient to compute, the empirical risk monitors the behaviour of f only at a finite set of n examples. When considering functions with a number parameters comparable to n (such as large neural networks), one trivial way to minimize is to memorize the training data. Memorization, in turn, leads to the undesirable behaviour of f outside the training data BID33. However, the naïve estimate P δ is one out of many possible choices to approximate the true distribution P. For instance, in the Vicinal Risk Minimization (VRM) principle BID3, the distribution P is approximated by DISPLAYFORM4 where ν is a vicinity distribution that measures the probability of finding the virtual feature-target pair (x,ỹ) in the vicinity of the training feature-target pair (x i, y i). In particular, BID3 considered Gaussian vicinities ν(x,ỹ|x i, DISPLAYFORM5, which is equivalent to augmenting the training data with additive Gaussian noise. To learn using VRM, we sample the vicinal distribution to construct a dataset DISPLAYFORM6, and minimize the empirical vicinal risk: DISPLAYFORM7 The contribution of this paper is to propose a generic vicinal distribution, called mixup: DISPLAYFORM8 where λ ∼ Beta(α, α), for α ∈ (0, ∞). In a nutshell, sampling from the mixup vicinal distribution produces virtual feature-target vectorsx DISPLAYFORM9 where (x i, y i) and (x j, y j) are two feature-target vectors drawn at random from the training data, and λ ∈. The mixup hyper-parameter α controls the strength of interpolation between feature-target pairs, recovering the ERM principle as α → 0.The implementation of mixup training is straightforward, and introduces a minimal computation overhead. FIG2 shows the few lines of code necessary to implement mixup training in PyTorch. Finally, we mention alternative design choices. First, in preliminary experiments we find that convex combinations of three or more examples with weights sampled from a Dirichlet distribution does not provide further gain, but increases the computation cost of mixup. Second, our current implementation uses a single data loader to obtain one minibatch, and then mixup is applied to the same minibatch after random shuffling. We found this strategy works equally well, while reducing I/O requirements. Third, interpolating only between inputs with equal label did not lead to the performance gains of mixup discussed in the sequel. More empirical comparison can be found in Section 3.8.What is mixup doing? The mixup vicinal distribution can be understood as a form of data augmentation that encourages the model f to behave linearly in-between training examples. We argue that this linear behaviour reduces the amount of undesirable oscillations when predicting outside the training examples. Also, linearity is a good inductive bias from the perspective of Occam's razor, since it is one of the simplest possible behaviors. FIG2 shows that mixup leads to decision boundaries that transition linearly from class to class, providing a smoother estimate of uncertainty. FIG4 illustrate the average behaviors of two neural network models trained on the CIFAR-10 dataset using ERM and mixup. Both models have the same architecture, are trained with the same procedure, and are evaluated at the same points in-between randomly sampled training data. The model trained with mixup is more stable in terms of model predictions and gradient norms in-between training samples. We evaluate mixup on the ImageNet-2012 classification dataset BID27. This dataset contains 1.3 million training images and 50,000 validation images, from a total of 1,000 classes. For training, we follow standard data augmentation practices: scale and aspect ratio distortions, random crops, and horizontal flips BID13. During evaluation, only the 224 × 224 central crop of each image is tested. We use mixup and ERM to train several state-of-the-art ImageNet-2012 classification models, and report both top-1 and top-5 error rates in For all the experiments in this section, we use data-parallel distributed training in Caffe2 1 with a minibatch size of 1,024. We use the learning rate schedule described in BID13. Specifically, the learning rate is increased linearly from 0.1 to 0.4 during the first 5 epochs, and it is then divided by 10 after 30, 60 and 80 epochs when training for 90 epochs; or after 60, 120 and 180 epochs when training for 200 epochs. For mixup, we find that α ∈ [0.1, 0.4] leads to improved performance over ERM, whereas for large α, mixup leads to underfitting. We also find that models with higher capacities and/or longer training runs are the ones to benefit the most from mixup. For example, when trained for 90 epochs, the mixup variants of ResNet-101 and ResNeXt-101 obtain a greater improvement (0.5% to 0.6%) over their ERM analogues than the gain of smaller models such as ResNet-50 (0.2%). When trained for 200 epochs, the top-1 error of the mixup variant of ResNet-50 is further reduced by 1.2% compared to the 90 epoch run, whereas its ERM analogue stays the same. We conduct additional image classification experiments on the CIFAR-10 and CIFAR-100 datasets to further evaluate the generalization performance of mixup. In particular, we compare ERM and mixup training for: PreAct ResNet-18 as implemented in BID25, WideResNet-28-10 (a) as implemented in BID40, and DenseNet BID20 as implemented in BID37. For DenseNet, we change the growth rate to 40 to follow the DenseNet-BC-190 specification from BID20. For mixup, we fix α = 1, which in interpolations λ uniformly distributed between zero and one. All models are trained on a single Nvidia Tesla P100 GPU using PyTorch 2 for 200 epochs on the training set with 128 examples per minibatch, and evaluated on the test set. Learning rates start at 0.1 and are divided by 10 after 100 and 150 epochs for all models except WideResNet. For WideResNet, we follow BID39 and divide the learning rate by 10 after 60, 120 and 180 epochs. Weight decay is set to 10. We do not use dropout in these experiments. We summarize our in FIG5. In both CIFAR-10 and CIFAR-100 classification problems, the models trained using mixup significantly outperform their analogues trained with ERM. As seen in FIG5, mixup and ERM converge at a similar speed to their best test errors. Note that the DenseNet models in BID20 were trained for 300 epochs with further learning rate decays scheduled at the 150 and 225 epochs, which may explain the discrepancy the performance of DenseNet reported in FIG5 and the original of BID20. extract normalized spectrograms from the original waveforms at a sampling rate of 16 kHz. Next, we zero-pad the spectrograms to equalize their sizes at 160 × 101. For speech data, it is reasonable to apply mixup both at the waveform and spectrogram levels. Here, we apply mixup at the spectrogram level just before feeding the data to the network. For this experiment, we compare a LeNet BID23 ) and a VGG-11 BID30 architecture, each of them composed by two convolutional and two fully-connected layers. We train each model for 30 epochs with minibatches of 100 examples, using Adam as the optimizer BID21. Training starts with a learning rate equal to 3 × 10 DISPLAYFORM0 and is divided by 10 every 10 epochs. For mixup, we use a warm-up period of five epochs where we train the network on original training examples, since we find it speeds up initial convergence. TAB6 shows that mixup outperforms ERM on this task, specially when using VGG-11, the model with larger capacity. Following, we evaluate the robustness of ERM and mixup models against randomly corrupted labels. We hypothesize that increasing the strength of mixup interpolation α should generate virtual examples further from the training examples, making memorization more difficult to achieve. In particular, it should be easier to learn interpolations between real examples compared to memorizing interpolations involving random labels. We adapt an open-source implementation BID42 to generate three CIFAR-10 training sets, where 20%, 50%, or 80% of the labels are replaced by random noise, respectively. All the test labels are kept intact for evaluation. Dropout BID32 ) is considered the state-of-the-art method for learning with corrupted labels BID1. Thus, we compare in these experiments mixup, dropout, mixup + dropout, and ERM. For mixup, we choose α ∈ {1, 2, 8, 32}; for dropout, we add one dropout layer in each PreAct block after the ReLU activation layer between two convolution layers, as suggested in BID39. We choose the dropout probability p ∈ {0.5, 0.7, 0.8, 0.9}. For the combination of mixup and dropout, we choose α ∈ {1, 2, 4, 8} and p ∈ {0.3, 0.5, 0.7}. These experiments use the PreAct ResNet-18 model implemented in BID25. All the other settings are the same as in Section 3.2.We summarize our in TAB3, where we note the best test error achieved during the training session, as well as the final test error after 200 epochs. To quantify the amount of memorization, we also evaluate the training errors at the last epoch on real labels and corrupted labels. As the training progresses with a smaller learning rate (e.g. less than 0.01), the ERM model starts to overfit the corrupted labels. When using a large probability (e.g. 0.7 or 0.8), dropout can effectively reduce overfitting. mixup with a large α (e.g. 8 or 32) outperforms dropout on both the best and last epoch test errors, and achieves lower training error on real labels while remaining resistant to noisy labels. Interestingly, mixup + dropout performs the best of all, showing that the two methods are compatible. One undesirable consequence of models trained using ERM is their fragility to adversarial examples BID33. Adversarial examples are obtained by adding tiny (visually imperceptible) perturbations to legitimate examples in order to deteriorate the performance of the model. The adversarial noise is generated by ascending the gradient of the loss surface with respect to the legitimate example. Improving the robustness to adversarial examples is a topic of active research. Among the several methods aiming to solve this problem, some have proposed to penalize the norm of the Jacobian of the model to control its Lipschitz constant BID9 BID6 BID2 BID18. Other approaches perform data augmentation by producing and training on adversarial examples BID12. Unfortunately, all of these methods add significant computational overhead to ERM. Here, we show that mixup can significantly improve the robustness of neural networks without hindering the speed of ERM by penalizing the norm of the gradient of the loss w.r.t a given input along the most plausible directions (e.g. the directions to other training points). Indeed, FIG4 shows that mixup in models having a smaller loss and gradient norm between examples compared to vanilla ERM.To assess the robustness of mixup models to adversarial examples, we use three ResNet-101 models: two of them trained using ERM on ImageNet-2012, and the third trained using mixup. In the first set of experiments, we study the robustness of one ERM model and the mixup model against white box attacks. That is, for each of the two models, we use the model itself to generate adversarial examples, either using the Fast Gradient Sign Method (FGSM) or the Iterative FGSM (I-FGSM) methods BID12, allowing a maximum perturbation of = 4 for every pixel. For I-FGSM, we use 10 iterations with equal step size. In the second set of experiments, we evaluate robustness against black box attacks. That is, we use the first ERM model to produce adversarial examples using FGSM and I-FGSM. Then, we test the robustness of the second ERM model and the mixup model to these examples. The of both settings are summarized in TAB4.For the FGSM white box attack, the mixup model is 2.7 times more robust than the ERM model in terms of Top-1 error. For the FGSM black box attack, the mixup model is 1.25 times more robust than the ERM model in terms of Top-1 error. Also, while both mixup and ERM are not robust to white box I-FGSM attacks, mixup is about 40% more robust than ERM in the black box I-FGSM setting. Overall, mixup produces neural networks that are significantly more robust than ERM against adversarial examples in white box and black settings without additional overhead compared to ERM. ERM GAN mixup GAN (α = 0.2) Figure 5: Effect of mixup on stabilizing GAN training at iterations 10, 100, 1000, 10000, and 20000. To further explore the performance of mixup on non-image data, we performed a series of experiments on six arbitrary classification problems drawn from the UCI dataset BID24. The neural networks in this section are fully-connected, and have two hidden layers of 128 ReLU units. The parameters of these neural networks are learned using Adam BID21 with default hyper-parameters, over 10 epochs of mini-batches of size 16. TAB6 shows that mixup improves the average test error on four out of the six considered datasets, and never underperforms ERM. Generative Adversarial Networks, also known as GANs, are a powerful family of implicit generative models. In GANs, a generator and a discriminator compete against each other to model a distribution P. On the one hand, the generator g competes to transform noise vectors z ∼ Q into fake samples g(z) that resemble real samples x ∼ P. On the other hand, the discriminator competes to distinguish between real samples x and fake samples g(z). Mathematically, training a GAN is equivalent to solving the optimization problem DISPLAYFORM0 where is the binary cross entropy loss. Unfortunately, solving the previous min-max equation is a notoriously difficult optimization problem BID10, since the discriminator often provides the generator with vanishing gradients. We argue that mixup should stabilize GAN training because it acts as a regularizer on the gradients of the discriminator, akin to the binary classifier in FIG2. Then, the smoothness of the discriminator guarantees a stable source of gradient information to the generator. The mixup formulation of GANs is: DISPLAYFORM1 ), λ). Figure 5 illustrates the stabilizing effect of mixup the training of GAN (orange samples) when modeling two toy datasets (blue samples). The neural networks in these experiments are fullyconnected and have three hidden layers of 512 ReLU units. The generator network accepts twodimensional Gaussian noise vectors. The networks are trained for 20,000 mini-batches of size 128 using the Adam optimizer with default parameters, where the discriminator is trained for five iterations before every generator iteration. The training of mixup GANs seems promisingly robust to hyper-parameter and architectural choices. mixup is a data augmentation method that consists of only two parts: random convex combination of raw inputs, and correspondingly, convex combination of one-hot label encodings. However, there are several design choices to make. For example, on how to augment the inputs, we could have chosen to interpolate the latent representations (i.e. feature maps) of a neural network, and we could have chosen to interpolate only between the nearest neighbors, or only between inputs of the same class. When the inputs to interpolate come from two different classes, we could have chosen to assign a single label to the synthetic input, for example using the label of the input that weights more in the convex combination. To compare mixup with these alternative possibilities, we run a set of ablation study experiments using the PreAct ResNet-18 architecture on the CIFAR-10 dataset. Specifically, for each of the data augmentation methods, we test two weight decay settings (10 which works well for ERM). All the other settings and hyperparameters are the same as reported in Section 3.2.To compare interpolating raw inputs with interpolating latent representations, we test on random convex combination of the learned representations before each residual block (denoted Layer 1-4) or before the uppermost "average pooling + fully connected" layer (denoted Layer 5). To compare mixing random pairs of inputs (RP) with mixing nearest neighbors (KNN), we first compute the 200 nearest neighbors for each training sample, either from the same class (SC) or from all the classes (AC). Then during training, for each sample in a minibatch, we replace the sample with a synthetic sample by convex combination with a random draw from its nearest neighbors. To compare mixing all the classes (AC) with mixing within the same class (SC), we convex combine a minibatch with a random permutation of its sample index, where the permutation is done in a per-batch basis (AC) or a per-class basis (SC). To compare mixing inputs and labels with mixing inputs only, we either use a convex combination of the two one-hot encodings as the target, or select the one-hot encoding of the closer training sample as the target. For label smoothing, we follow BID34 and use 10 as the target for incorrect classes, and 1 − 9 10 as the target for the correct class. Adding Gaussian noise to inputs is used as another baseline. We report the median test errors of the last 10 epochs. Results are shown in TAB8.From the ablation study experiments, we have the following observations. First, mixup is the best data augmentation method we test, and is significantly better than the second best method (mix input + label smoothing). Second, the effect of regularization can be seen by comparing the test error with a small weight decay. For example, for ERM a large weight decay works better, whereas for mixup a small weight decay is preferred, confirming its regularization effects. We also see an increasing advantage of large weight decay when interpolating in higher layers of latent representations, indicating decreasing strength of regularization. Among all the input interpolation methods, mixing random pairs from all classes (AC + RP) has the strongest regularization effect. Label smoothing and adding Gaussian noise have a relatively small regularization effect. Finally, we note that the SMOTE algorithm BID4 does not lead to a noticeable gain in performance. Data augmentation lies at the heart of all successful applications of deep learning, ranging from image classification BID22 to speech recognition BID14 BID0. In all cases, substantial domain knowledge is leveraged to design suitable data transformations leading to improved generalization. In image classification, for example, one routinely uses rotation, translation, cropping, resizing, flipping BID23 BID30, and random erasing BID43 to enforce visually plausible invariances in the model through the training data. Similarly, in speech recognition, noise injection is a prevalent practice to improve the robustness and accuracy of the trained models BID0.More related to mixup, BID4 propose to augment the rare class in an imbalanced dataset by interpolating the nearest neighbors; BID8 show that interpolation and extrapolation the nearest neighbors of the same class in feature space can improve generalization. However, their proposals only operate among the nearest neighbors within a certain class at the input / feature level, and hence does not account for changes in the corresponding labels. Recent approaches have also proposed to regularize the output distribution of a neural network by label smoothing BID34, or penalizing high-confidence softmax distributions BID26. These methods bear similarities with mixup in the sense that supervision depends on multiple smooth labels, rather than on single hard labels as in traditional ERM. However, the label smoothing in these works is applied or regularized independently from the associated feature values.mixup enjoys several desirable aspects of previous data augmentation and regularization schemes without suffering from their drawbacks. Like the method of , it does not require significant domain knowledge. Like label smoothing, the supervision of every example is not overly dominated by the ground-truth label. Unlike both of these approaches, the mixup transformation establishes a linear relationship between data augmentation and the supervision signal. We believe that this leads to a strong regularizer that improves generalization as demonstrated by our experiments. The linearity constraint, through its effect on the derivatives of the function approximated, also relates mixup to other methods such as Sobolev training of neural networks BID7 or WGAN-GP BID15. We have proposed mixup, a data-agnostic and straightforward data augmentation principle. We have shown that mixup is a form of vicinal risk minimization, which trains on virtual examples constructed as the linear interpolation of two random examples from the training set and their labels. Incorporating mixup into existing training pipelines reduces to a few lines of code, and introduces little or no computational overhead. Throughout an extensive evaluation, we have shown that mixup improves the generalization error of state-of-the-art models on ImageNet, CIFAR, speech, and tabular datasets. Furthermore, mixup helps to combat memorization of corrupt labels, sensitivity to adversarial examples, and instability in adversarial training. In our experiments, the following trend is consistent: with increasingly large α, the training error on real data increases, while the generalization gap decreases. This sustains our hypothesis that mixup implicitly controls model complexity. However, we do not yet have a good theory for understanding the'sweet spot' of this bias-variance trade-off. For example, in CIFAR-10 classification we can get very low training error on real data even when α → ∞ (i.e., training only on averages of pairs of real examples), whereas in ImageNet classification, the training error on real data increases significantly with α → ∞. Based on our ImageNet and Google commands experiments with different model architectures, we conjecture that increasing the model capacity would make training error less sensitive to large α, hence giving mixup a more significant advantage.mixup also opens up several possibilities for further exploration. First, is it possible to make similar ideas work on other types of supervised learning problems, such as regression and structured prediction? While generalizing mixup to regression problems is straightforward, its application to structured prediction problems such as image segmentation remains less obvious. Second, can similar methods prove helpful beyond supervised learning? The interpolation principle seems like a reasonable inductive bias which might also help in unsupervised, semi-supervised, and reinforcement learning. Can we extend mixup to feature-label extrapolation to guarantee a robust model behavior far away from the training data? Although our discussion of these directions is still speculative, we are excited about the possibilities mixup opens up, and hope that our observations will prove useful for future development.

Training on convex combinations between random training examples and their labels improves generalization in deep neural networks


Question: Represent the next text

Question: We present a novel approach to spike sorting for high-density multielectrode probes using the Neural Clustering Process (NCP), a recently introduced neural architecture that performs scalable amortized approximate Bayesian inference for efficient probabilistic clustering. To optimally encode spike waveforms for clustering, we extended NCP by adding a convolutional spike encoder, which is learned end-to-end with the NCP network. Trained purely on labeled synthetic spikes from a simple generative model, the NCP spike sorting model shows promising performance for clustering multi-channel spike waveforms. The model provides higher clustering quality than an alternative Bayesian algorithm, finds more spike templates with clear receptive fields on real data and recovers more ground truth neurons on hybrid test data compared to a recent spike sorting algorithm. Furthermore, NCP is able to handle the clustering uncertainty of ambiguous small spikes by GPU-parallelized posterior sampling. The source code is publicly available. Large-scale neuronal population recordings using high-density multi-electrode arrays (MEA) are at the forefront of current progress in understanding neural circuit dynamics. In MEA recordings, each electrode channel reads extracellular signals from many neurons, and each neuron is recorded by multiple nearby electrodes. A key step in the analysis of MEA data is spike sorting, which converts the raw electrical signal into a set of neural spike trains belonging to individual neurons. As MEAs grow in scale and popularity, there is a new urgency in improving spike sorting performance. A typical spike sorting pipeline consists of three steps. The spike detection step extracts putative spike events from noisy recordings. The clustering step groups similar spike waveforms into clusters, each representing a putative neuron. To resolve colliding waveforms, a deconvolution step is often performed. Spike clustering is at the core of the pipeline, as the clustering performance determines both the accuracy of spike assignment and the quality of spike templates used for deconvolution. Spike clustering, however, poses significant challenges: Spike waveforms form highly nonGaussian clusters in spatial and temporal dimensions, and it is unclear what are the optimal features for clustering. It is unknown a priori how many clusters there are. Although existing methods perform well on spikes with high signal-to-noise ratios (SNR), there remain significant challenges in the lower-SNR regime with increased clustering uncertainty. Fully-Bayesian approaches proposed to handle this uncertainty do not scale to large datasets due to expensive Gibbs sampling. To address these challenges, we propose a novel approach to spike clustering using the recently introduced Neural Clustering Process (NCP) (Figure 1). NCP is based on a neural architecture that performs scalable amortized approximate Bayesian clustering. Rather than selecting arbitrary features for clustering, the spike waveforms are encoded with a convolutional neural network (ConvNet), which is learned end-to-end jointly with the NCP network to ensure optimal feature encoding. Using a variable-input softmax function, NCP is able to compute full posterior distributions on cluster labels and the number of clusters, without assuming a fixed or maximum number of clusters. NCP allows for efficient probabilistic clustering by GPU-parallelized posterior sampling, which is particularly useful for handling the clustering uncertainty of ambiguous small spikes. The computational cost of NCP training can be highly amortized, since neuroscientists often sort spikes form many statistically similar datasets. We trained NCP for spike clustering using synthetic spikes from a simple yet effective generative model that mimics the distribution of real spikes, and evaluated the performance on labeled synthetic data, unlabeled real data, and hybrid test data with partial ground truth. We show that using NCP for spike sorting provides high clustering quality, matches or outperforms a recent spike sorting algorithm, and handles clustering uncertainty by efficiently producing multiple plausible clustering configurations. These show substantial promise for incorporating NCP into a production-scale spike sorting pipeline.. The model is composed by the deep networks h, g, q, f. Bottom left: After assigning the cluster labels c 1:n−1, each possible discrete value k for c n gives a different symmetry-invariant encoding of x 1:n into the vector G k, using the functions h and g. The remaining, yet-unassigned points x n+1:N are encoded by q and summed into the vector Q. Bottom right: Each pair G k, Q is mapped by f into a real number (logit), which in turn is mapped into the multinomial distribution q θ (c n |c 1:n−1, x) via a variable-input softmax. 2 Spike Sorting using the Neural Clustering Process Data preprocessing. Training and test data come from the retinal recordings in using a 512-channel 2D hexagonal MEA with 20 kHz sampling rate. After spike detection, each multi-channel spike waveform was assigned to the channel where the waveform has the maximum peak-to-peak (PTP) amplitude (i.e. the center channel, ch0). This partitioned the recording data by channel such that each center-channel-based partition only contains multi-channel spike waveforms centered at that channel. Each spike waveform is represented as a 7 × 32 array containing the 32 time steps surrounding the peak from the center channel and the same time window from the 6 immediate neighbor channels (Figure 1 top). These 7 × 32 arrays are the spikes on which clustering was performed. Neural architecture for NCP spike sorting. The NCP architecture contains four neural networks, h, q, g, f, as shown in Figure 1 (bottom). We refer to for the detailed formulation and notations of NCP. To extract useful features from the spatial-temporal patterns of spike waveforms, we use a 1D ConvNet as the h and q encoder functions. The convolution is applied along the time axis, with each electrode channel treated as a feature dimension. The ConvNet uses a ResNet architecture with 4 residual blocks, each having 32, 64, 128, 256 feature maps (kernel size = 3, stride =). The last block is followed by an averaged pooling layer and a final linear layer. The outputs of the ResNet encoder are the h i and q i vectors of NCP, i.e. The other two functions, g and f, are multilayer perceptrons identical to those in the 2D Gaussian example in. Training NCP using synthetic data. To train NCP for spike clustering, we created synthetic labeled training data (Figure 2) using a mixture of finite mixtures (MFM) generative model of noisy spike waveforms that mimics the distribution of real spikes: Here, N is the number of spikes between. The number of clusters K is sampled from a shifted Poisson distribution with λ = 2 so that each channel has on average 3 clusters. π 1:K represents the proportion of each cluster and is sampled from a Dirichlet distribution with α 1:K = 1. The training spike templates µ k ∈ R 7×32 are sampled from a reservoir of 957 ground-truth templates not present in any test data, with the temporal axis slightly jittered by random resampling. Finally, each waveform x i is obtained by adding to µ ci Gaussian noise with covariance given by the Kronecker product of spatial and temporal correlation matrices estimated from the training data. This method creates spatially and temporally correlated noise patterns similar to real data (Figure 2). We trained NCP for 20000 iterations on a GPU with a batch size of 32 to optimize the NLL loss by the Adam optimizer. A learning rate of 0.0001 was used (reduced by half at 10k and 17k iterations). Probabilistic spike clustering using NCP. At inference time, we fed the 7 x 32 arrays of spike waveforms to NCP, and performed GPU-parallelized posterior sampling of cluster labels (Figure 1). Using beam search with a beam size of 150, we were able to efficiently sample 150 high-likelihood clustering configurations for 2000 spikes in less than 10 seconds on a single GPU. After clustering, we obtained a spike template for each cluster as the average shape of the spike waveforms. The clustering configuration with the highest probability was used in most experiments. We compared NCP spike sorting against two other methods: Variational inference on a Gaussian Mixture of Finite Mixtures (vGMFM), which is an alternative Bayesian clustering algorithm, and Kilosort, a state-of-the-art spike sorting pipeline described in. For vGMFM, the first 5 principal components of the spike waveforms from each channel were used as the input features. For Kilosort, we run the entire pipeline using the Kilosort2 package. Synthetic Data. We run NCP and vGMFM on 20 sets of synthetic test data each with 500, 1000, and 2000 spikes. As the ground-truth cluster labels are known, we compared the clustering quality using Adjusted Mutual Information (AMI). As shown in Figure 3, The AMI of NCP is on average 11% higher than vGMFM, showing better performance of NCP on synthetic data. Real Data. We run NCP, vGMFM and Kilosort on a 49-channel, 20-minute retina recording with white noise stimulus, and extracted the averaged spike template of each cluster (i.e. putative neuron). For NCP and vGMFM, we performed clustering on 2000 randomly sampled spikes from each channel (clusters containing less than 20 spikes were discarded), and assigned all remaining spikes to a cluster based on the L2 distance to the cluster centers. Then, a final set of unique spike templates were computed, and each detected spike was assigned to one of the templates. Example clustering of NCP and vGMFM in Figure 4 (top and bottom-left) show that NCP produces clean clusters with visually more distinct spike waveforms compared to vGMFM. As real data do not come with ground-truth cluster labels, we compared the receptive fields (RFs) extracted by NCP and Kilosort; the RF is computed for each cluster as the spike-triggered average of the stimulus (spatiotemporal white noise in this experiment). A clearly demarcated RF provides encouraging evidence that the spike template corresponds to a real neuron. After extracting spike templates and RFs from each pipeline, we matched pairs of templates from different methods by L-infinity distance and pairs of RFs by cosine distance. Side-by-side comparisons of 5 example RF pairs are shown in Figure 4 (bottom-right). See Figure 7 in Appendix for more examples. Overall, the NCP pipeline found 103 templates with clear RFs, among which 48 were not found in Kilosort. Kilosort found 72 and 17 of them were not found by NCP (Figure 4 bottom-right). This shows that NCP performs at least as well as Kilosort, and finds many additional spike templates with clear RFs. Hybrid Data. We compared NCP against vGMFM and Kilosort on a hybrid recording with partial ground truth as in. 20 ground-truth spike templates were manually selected from a 49-channel test recording and injected into another test dataset according to the original spike times. This approach tests the clustering performance on realistic recordings with complex noise and colliding spikes. As shown in Figure 5, NCP recovered 13 of the 20 injected ground-truth templates, outperforming both Kilosort and vGMFM, which recovered 8 and 6, respectively. Probabilistic clustering of ambiguous small spikes. Spike sorting of small-amplitude waveforms has been challenging due to the low SNR and increased uncertainty of cluster assignment. Traditional methods could not handle the uncertainty and previous fully-Bayesian approaches do not scale. By efficient GPU-parallelized sampling of cluster labels from the posterior, NCP is able to handle the clustering uncertainty by producing multiple plausible clustering . Figure 6 shows examples where NCP separates spike clusters with amplitude as low as 3-4× the standard deviation of the noise into plausible units that are not mere scaled version of each other but have distinct shapes on different channels. Conclusions. Our show that NCP spike sorting achieves high clustering quality, matches or outperforms a state-of-the-art method, and is able to handle clustering uncertainty by efficient posterior sampling. Future directions include more realistic generative models, better spike encoders that utilize information from distant channels, and integrating NCP into a standard spike sorting pipeline.

Represent the TLDR for finding the corresponding scientific article

We present a novel approach to spike sorting using the Neural Clustering Process (NCP), a recently introduced neural architecture that performs scalable amortized approximate Bayesian inference for efficient probabilistic clustering.

We show that it is possible to recover the parameters of a 1-layer ReLU generative model from looking at samples generated by it


Question: Represent the article for finding a single-sentence TLDR

Question: The goal of the paper is to propose an algorithm for learning the most generalizable solution from given training data. It is shown that Bayesian approach leads to a solution that dependent on statistics of training data and not on particular samples. The solution is stable under perturbations of training data because it is defined by an integral contribution of multiple maxima of the likelihood and not by a single global maximum. Specifically, the Bayesian probability distribution of parameters (weights) of a probabilistic model given by a neural network is estimated via recurrent variational approximations. Derived recurrent update rules correspond to SGD-type rules for finding a minimum of an effective loss that is an average of an original negative log-likelihood over the Gaussian distributions of weights, which makes it a function of means and variances. The effective loss is convex for large variances and non-convex in the limit of small variances. Among stationary solutions of the update rules there are trivial solutions with zero variances at local minima of the original loss and a single non-trivial solution with finite variances that is a critical point at the end of convexity of the effective loss in the mean-variance space. At the critical point both first- and second-order gradients of the effective loss w.r.t. means are zero. The empirical study confirms that the critical point represents the most generalizable solution. While the location of the critical point in the weight space depends on specifics of the used probabilistic model some properties at the critical point are universal and model independent. Finding a generalizable solution is a critical problem for any machine learning task. The ultimate goal of learning from the available ground truths is to make a good prediction for new data. The Bayesian method is a very powerful approach that gives a probabilistic measure of the ability of a proposed model to predict by estimating how well the model predicts known data. The accuracy of the predictions depends on how the found solution is able to overcome a sampling bias to avoid overfitting for given particular samples of training data. Specifically, in Bayesian method predictions of labels y for an input x are made by using a probabilistic model, for certainty a neural network, which defines a function parametrized by weights w that allows computing probabilities P (y|x, w) for each weight point. Each weight point contributes to predicted probabilities of labels P rob(y|x) in accordance with probability distribution of weights. The distribution of weights is learned from a known training data set {x n, y n ; n = 1..N} and its prior probability distribution P 0 (w) in the following way: P rob(y|x) = w P (y|x, w)P 0 (w) N n=1 P (y n |x n, w)/ w P 0 (w) N n=1 P (y n |x n, w)Here the predicted probability P rob(y|x) is an average of the model probability P (y|x, w) at a weight w over the learned weight distribution. To make predictions we are only interested in a method that allows to find averages in eq. FORMULA0 and not absolute values of the integrals. According to mean value theorem (Cauchy, also in Encyclopedia of ) values of the averages can be represented by a single point, which in our case means that there is a single point in the weight space w 0 that represents a of computing the integrals, so P rob(y|x) = P (y|x, w 0). That point w 0 is a solution of the training of the neural network. A standard approach to get the solution is a maximum likelihood method that finds a maximum of the integrand. However, there are some cases when the maximum likelihood fails to represent main contribution to the integral by weights. Consider this example: if log-likelihood for N data samples has a maximum at some weight point w 1, then in general its first derivative by weights is zero, second derivative is negative and proportional to N, so corresponding Gaussian integral by the weights is proportional to N −d/2, where d is number of weights. This will change if there is a flat maximum, which has not only first but also second and third derivatives equal to zero. In this case the integral is proportional to N −d/4. For large number of samples the flat maximum makes the most significant contribution to the integral by weights: DISPLAYFORM0 and DISPLAYFORM1. For a typical case when the number of weights d ∼ N and average sample probabilities at maxima are comparable O(P 1) ∼ O(P 2) the integral around flat maximum I 2 is always bigger than the integral around narrow maximum I 1, unless P 2 is zero. While in general a likelihood has a number of regular local maxima and no flat maximum the effect of integration over multiple frequent local maxima can in an effective flat maximum that defines a solution. We argue that any local or global maximum of likelihood gives a wrong solution that is not generalizable and so makes inaccurate predictions, because the locations for the global maximum and local maxima depend on specific samples in the training data and any modification of it by adding or removing samples will change the solution BID6. Instead we will show that there is another solution that more associated with properties of the distribution of training data and less with particular samples. The purpose of this paper is to show that the effective flat maximum always exists for specific parameters of prior weight distribution P 0 (w) (regularization parameters) and corresponding solution is the most generalizable solution that can be found in training. We show that the solution is a critical point in an effective loss that represents the of integration over the weights. In the next sections we derive the algorithm for the optimizer for finding the critical point solution and analyze properties of the solutions. The empirical study is outside of the scope of the paper and will be presented separately. For simplicity we use same notations for a vector of weights and its components, as well as corresponding parameters of distributions of weights because all weight components are independent in our consideration and it is clear from context when it is a vector or its component. We use a recurrent approach for estimating the integrals. First, we represent a probability of each training sample as a product of factors close to one, P (y|x, w) = (1 + 1/T ln P (y|x, w))T, where free parameter T 1 is a number of epochs: DISPLAYFORM0 then model each factor as a product of Gaussian distributions Q(w|µ, σ) one for each component of weight vector. For each iteration a running prior distribution of weights is updated by absorbing a single factor to produce a new prior DISPLAYFORM1, with Q 0 (w) = P 0 (w) and Q(w|µ, σ) = e DISPLAYFORM2 Under review as a conference paper at ICLR 2019Specifically, we do the following:First, let's enumerate all factors for all data samples DISPLAYFORM3 Then, under the integral by weights we use an identical re-writing for a product of a prior Q t (w) and one of the factors and a new prior Q t+1 (w) and normalization factor N t DISPLAYFORM4 where normalization factor DISPLAYFORM5 and distribution DISPLAYFORM6 Finally, we make an approximation by replacing ratio of distributions R t (w)/Q t+1 (w) by its mean for distribution Q t+1 which is equal to 1. Then the iterations are repeated until all factors from probabilities of data are replaced by a single final Gaussian distribution and some normalization factor. To minimize the introduced error on each iteration we select the means and variances of the new prior Q t+1 to minimize variance of the ratio R t /Q t+1. The variance is equal to DISPLAYFORM7 The lower bound of the variance is expressible via KL divergence BID3 ) of R t and Q t+1 DISPLAYFORM8 Finding the minimum of KL divergence is equivalent to minimizing the lower bound of the variance of the ratio of R t /Q t+1 which leads to equations DISPLAYFORM9 Then solving the above equations gives the update rules for means and variances of each weight component DISPLAYFORM10 where averages are defined as A(w) µ,σ = w Q(w|µ, σ)A(w) and averages of gradients by weights are equal to gradients of averages by means DISPLAYFORM11 Another useful identity allows to replace second gradient of (µ, σ)-average w.r.t mean on first gradient by variance DISPLAYFORM12 Under review as a conference paper at ICLR 2019For a full batch the log of probability of data in eqs. ln P (w) = N n ln P (y n |x n, w), while for a minibatch the sum goes over the size of the minibatch. In eqs. we used rescaled variances σ 2 → σ 2 /N so all gradients are normalized per data sample. With the rescaled variances the prior distribution of weights is a product over all weight dimensions d DISPLAYFORM13 The index t enumerates iterations over all minibatches in all epochs. Then for a minibatch size one the total number of iterations equals to a number of epochs T times number of data samples N in a training set. By recursively applying the update rules in eqs. FORMULA13 over N samples and T epochs we obtain the approximation of Bayesian integrals that allows to compute averages DISPLAYFORM14 It is important to emphasize that the averages are defined by distribution Q tmax (w) after a finite number of iterations t max, which for the minibatch one is equal to a product N × T and not by distribution at the infinite number of iterations Q ∞ (w). Number of epochs T controls the accuracy of the factorized representation and in practical computing any T larger than 10 or 100 is good enough. The prediction probability in eq. FORMULA0 is defined by weight w 0 that is a mean µ tmax of distribution Q tmax(w) P rob(y|x) ≈ P (y|x, µ tmax).That mean µ tmax is a final point of iterations in eqs.. The trajectory in mean-variance space is defined by starting point (µ 0, σ 0), the mean and variance of a prior distribution of weights in eq. which are regularization parameters: P 0 (w) = Q(w|µ 0, σ 0). Before going into detailed analysis of eqs. FORMULA13 let's formulate the in the form of the following statements. The update rules above are solving SGD-type optimization problem for an effective loss given by Gaussian average L(µ, σ) = − w Q(w|µ, σ) n ln P (y n |x n, w) for each mean-variance point (µ, σ).The following statements have been proved:1. The effective loss L(µ, σ) is convex for large variances σ 2. In particular, it is true for any neural network with ReLU activations and L1, L2 or cross entropy loss functions. For a convex effective loss its second-order gradient by mean is positive.2. The effective loss is converging to the original loss in the limit of small variances where it is generally non-convex (excluding completely trivial linear cases).3. For each mean there is a critical variance σ 2 c that separates convex effective loss from non-convex. At the critical variance second-order gradient of effective loss by mean is zero.4. There are trivial stationary solutions of the update rules that correspond to zero variances and zero gradients of loss w.r.t. weights. These solutions are unstable when training set is modified because they correspond to narrow minima that are changing drastically as data change BID6. Because second-order gradients by weights are large and positive, changes in loss are second-order by weights and changes in solutions are at least linear and often in jumps to a new location.5. There is a non-trivial stationary solution at critical point at the end of convexity where both firstorder and second-order gradients of effective loss are zero. That solution is much less sensitive to changes in data. It is responding by cubic changes in loss and quadratic changes in solutions and due to convexity there is no jumps. For that reason the critical point solution is the most generalizable solution as well as most stable against adversarial perturbations BID1.6. Trajectories in mean-variance space that follow from update rules show universal behavior in vicinity of the critical point. Analysis shows that not all trajectories are converging to the critical point. Typical trajectory that starts in convex area moves toward the critical point until it may cross to non-convex area then it moves away from critical point and finally ends as a trivial solution in a local minimum.7. Approximating Gaussian averaging by sampling in weight space in dropout method also known as "fast dropout" BID5 and "dropconnect" BID4. The update rules completely define the dropout rate for each weight component. With averaging via dropout the critical point and the convexity of effective loss exist only in an unreachable limit, nevertheless it allows to find a solution that is close to the critical point. To understand better an effect of averaging let's consider a simple polynomial case where loss l(w) as a function of weights has two symmetrical minima at points (−a, +a): DISPLAYFORM0 Then the effective loss is Gaussian average of the loss above L(µ, σ) = l(w) µ,σ, specifically DISPLAYFORM1 And the first and second gradients of the effective loss are DISPLAYFORM2 When variance is large, σ 2 > a 2 /3, second gradient of the effective loss L w.r.t. µ is positive and there is only one minimum at µ = 0. When variance is small σ 2 < a 2 /3, the effective loss has a maximum at µ = 0 and two minima at µ = ± √ a 2 − 3σ 2.When σ 2 = a 2 /3 there is a critical point at µ = 0 where both first and second gradients of the effective loss w.r.t. µ are zero: DISPLAYFORM3 ∂µ 2 = 0. By using the update rules from eqs. FORMULA13 where average log of probability is equal to the negative effective loss L = − ln P with first and second gradients defined above in eqs. FORMULA20 we can consider trajectories in the mean-variance space: DISPLAYFORM4 We can see that the critical point is a saddle point in the mean-variance space. Any trajectory that is missing a critical point after an infinite number of iterations ends in a local minimum of an effective loss with zero variance σ 2, which is an original local minimum. There are only two trajectories starting with µ = 0 from convex area (3σ 2 > a 2) and non-convex area (3σ 2 < a 2) that after a large enough number of iterations will always converge to the critical point. However, for a finite number of iterations there is an area of starting points (µ 0, σ 0) that defines trajectories with end points arbitrary close to a critical point. Let's consider a multilayered neural network where predictions y are computed via ReLU activations and loss function l(y) is a convex function of predictions. For each weight w predictions y are piecewise functions of the weight. Then second gradient of loss function by weight is DISPLAYFORM0 There are two terms in second gradient of loss by weight: first is always positive due to convexity of l(y) and second contains second gradient of predictions. Average of the first term is always positive. Average of the second term could be positive or negative. Due to piecewise weight dependency second gradient of predictions ∂ 2 y/∂w 2 is singular and not zero, however for any network with finite number of layers predictions y are linear functions of the weight when the value of the weight goes to infinity. For that reason second gradient of predictions for large values of the weight is zero and second gradient of loss by weight is positive for large weight values. If variance of Gaussian distribution of the weight is very large then averages are defined by contributions of large weight values and then average of the second gradient of loss by weight is positive. DISPLAYFORM1 That proves the convexity of the effective loss for large variances. On other side when the variance goes to zero the effective loss converges to original loss which generally non-convex. Because an effective loss is a continuous function of the variance there exists a critical value of the variance where second gradient of the effective loss may have zero at some mean. That means the mean-variance plane could be divided on two convex and non-convex areas. These areas are separated by the line on mean-variance plane where second gradient of the effective loss w.r.t. means is zero. The general properties of the critical point are universal due to the identity in eq.. Let's consider an expansion of the effective loss L(µ, σ) in a vicinity of a point µ c, σ c where both first gradients of L by mean and variance are zero: DISPLAYFORM2 Then using the identity in equation FORMULA15 we can see that 1 2 ∂ 2 L ∂µ 2 = a 2 + 3a 3 (µ − µ c) and a 2 = 0 because zero-order term by (µ − µ c) in ∂L ∂σ 2 is zero. Also it gives c = 3a 3. Due to the identity the universal structure of the effective loss at the critical point is as follows DISPLAYFORM3 Essentially, the condition makes a regular minimum of the effective loss to be a critical point. At critical point both first and second gradients by means of the effective loss are zero. At critical point the effective loss is a flat function of the means. That makes the solution stable when training data set is modified unlike any maximum log-likelihood solutions that changes when data changes. For that reason the critical point solution is most generalizable. Any available data set is a collection of samples from an unknown true distribution. The empirical loss is defined as loss per sample for sampled data for a given weight point. The expected loss is average over unknown true distribution of data at a given weight point. In a two-minimum model loss for a data point is l(w) = (w 2 − a 2) 2, where a is a sample specific parameter. The empirical loss per sample for m samples is DISPLAYFORM0 while the expected loss per data point for the same weight point is DISPLAYFORM1 The generalization error is a difference between an expected loss and empirical loss for the same solution w: DISPLAYFORM2 For a given weight point the expectation of the generalization error is zero: DISPLAYFORM3 However, we only can find solution for a minimum of the empirical loss. DISPLAYFORM4 Due to sampling a minimum loss solution for the empirical loss depends on specific samples. For that reason the expectation of the generalization error for a minimum of the empirical loss solution is not zero and is equal to DISPLAYFORM5 The expectation of the generalization error is positive which reflects overfitting and goes to zero when number of samples m goes to infinity. Now, let's compute generalization error for the effective loss for m data samples. The solution for empirical effective loss are found from equations DISPLAYFORM6 There are trivial solutions with zero variance σ 0 = 0, µ DISPLAYFORM7 And there is a critical point solution with non-zero variance: µ c = 0, 3σ DISPLAYFORM8 n. That solution gives the expected generalization error that is 3 times smaller than the expected generalization error for trivial solutions: m E (a 2 − E a 2) 2 /3 = E [GErr ef f (µ 0, σ 0)] /3.This example supports the claim of the paper that the critical point solution is more generalizable than a trivial minimum loss solution. In the paper we consider a learning of a predictive model from training data by approximately computing Bayesian integral over weights -the parameters of the model. By using recurrent variational approximations with Gaussian weight distributions we are able to find a solution -a single point in weight space that represents an effect of averaging over distribution of weights in the Bayesian integrals. We show that this approach leads to SGD-type optimization problem for an effective loss in meanvariance space. For each mean-variance point the effective loss is defined by average of the loglikelihood over Gaussian distribution at same mean-variance point. Due to averaging the effective loss and its gradients of any order are continuous function of means even for ReLU based neural networks. The recurrent update rules define trajectories in mean-variance space. Starting points of the trajectories are defined by regularization parameters, which are parameters of the Gaussian weight prior in Bayesian integrals. It is shown that there are two types of stationary solutions of the update rules. First solution type corresponds to local minima of the original loss or maxima of the log-likelihood. Second solution type is a critical point in mean-variance space that is a of the integration over multiple maxima of the log-likelihood. At the critical point both first and second gradient of the effective loss are zero. That leads to stability of the solution against perturbations of the training data set due to addition or removal data samples or via creation of adversarial examples.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Proposed method for finding the most generalizable solution that is stable w.r.t. perturbations of trainig data.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Humans rely on episodic memory constantly, in remembering the name of someone they met 10 minutes ago, the plot of a movie as it unfolds, or where they parked the car. Endowing reinforcement learning agents with episodic memory is a key step on the path toward replicating human-like general intelligence. We analyze why standard RL agents lack episodic memory today, and why existing RL tasks don't require it. We design a new form of external memory called Masked Experience Memory, or MEM, modeled after key features of human episodic memory. To evaluate episodic memory we define an RL task based on the common children's game of Concentration. We find that a MEM RL agent leverages episodic memory effectively to master Concentration, unlike the baseline agents we tested. From a neurobiological perspective, episodic memory is a key component of human life -remembering the name of a new acquaintance, recalling the plot of a movie as it unfolds, or realizing where the car is parked, are all examples of how we use episodic memory 1 to store and recall novel information. If a person's ability to form and retrieve new episodic memories is lost, as in advanced Alzheimer's disease, the person is severely incapacitated as a . Although today's standard Reinforcement Learning (RL) agents possess forms of procedural and semantic memory BID10, they lack any functional equivalent of episodic memory. Our motivation is to expand the general intelligence of RL agents by imbuing them with a useful form of episodic memory. Human episodic memories appear to be records of experience that are re-experienced when associatively recalled BID8. In RL, fundamental experiences are termed observations. Accordingly, we propose the following working definition: Episodic memory for an RL agent is the ability to leverage details of a past observation that is similar to the current observation. This definition implies that an agent would exercise episodic memory by doing certain things at specific points in time, including 1. At the time of the old observation, the details of that observation must be stored somewhere in the agent. This stored record is the episodic memory. 2. Later, when another observation arrives, it must somehow be compared with the stored observations. If one of those is sufficiently similar, then the details of the old observation must be retrieved from memory. There are different implementations of similarity and retrieval. We will propose a concrete one later. 3. After retrieving the details of the old observation that is similar to the new one, the agent must be able to utilize that information to benefit it's pursuit of reward. Designing an RL agent with episodic memory is one challenge, and designing an RL task to evaluate episodic memory in an agent is another. The main difficulty is that unless the task is very carefully designed, the RL agent may find a way to solve the task using other learning abilities besides episodic memory. To illustrate, we briefly introduce the RL task that we will present later in detail. To evaluate an agent's episodic memory ability, we introduce the Concentration task based on the card game of the same name. Concentration is a memory game with the goal of identifying matching pairs of cards among a large set of face-down cards. During play, one card at a time is temporarily revealed to the player who must correctly memorize and recall the locations of each pair. Concentration tests episodic memory by requiring an agent to leverage past observations of cards and their locations in order to succeed. In our variant of Concentration, cards are not limited to the standard deck and are instead randomly generated for each game, so each card pair is unique and never before seen in the agent's lifetime. Unique cards test the agent's ability to use episodic memory to reason about the identities and locations of the cards that are seen within the current episode, rather than learning to recognize specific cards. Recently, the capabilities of intelligent agents have greatly expanded through the combination of deep learning and reinforcement learning. Deep RL agents have achieved notable success outperforming humans on Atari games BID15. However, many of the hardest tasks in which RL agents still fail to surpass humans are fraught with the difficulties of sparse rewards, partial observability, and a limited amount of samples. Equipping an RL agent with memory is a promising approach to tackling some of these challenges, and has attracted a growing amount of interest in the research community. Recurrent neural networks such as LSTMs are commonly used as controllers BID13. LSTMs can be trained to maintain and use information on timescales of tens of steps, but have trouble learning over longer sequences. Additionally, LSTMs do not store observations as discrete entities, so it is unclear how an LSTM could compare a never-before-seen observation (such as a unique card) with detailed instances of past observations, which also may have occurred only once. Memory augmented neural networks provide storage capabilities beyond those of an LSTM. One such architecture, the differentiable neural computer (DNC) has been shown to be capable of handling several different memory-based tasks. We evaluate the DNC on Concentration, but discover that it has difficulty reusing elements of its memory matrix. The key contributions of this paper are:• We propose a working definition of episodic memory for RL agents.• We introduce the Concentration task for evaluating episodic memory.• We present the Masked Experience Memory (MEM) architecture, a new type of external memory designed to provide an RL agent with human-inspired episodic memory, and incorporating a novel improvement over cosine similarity for content-based addressing.• We empirically demonstrate that MEM successfully enables an RL agent to solve the Concentration task by remembering the identities and locations of cards it has seen only once.• We show that baseline RL agents (LSTM-based and DNC-based) fail to solve the task. Neither neural network weights nor activation states support the storage and associative retrieval of discrete, one-shot experiences necessary for episodic memory. Neural network weights change too slowly to store samples as individual experiences. Fast weights are one approach to storing information from discrete samples in weights, but we find no published evaluations of fast weights on episodic memory tasks. Many external memory architectures have been proposed for augmenting the capabilities of neural networks in the supervised learning setting. Some of these were evaluated on data samples that occur a relatively small number of times BID28 BID19 BID23 ). Adapting these architectures to RL tasks is non-trivial. For instance, the memory module of requires ground truth output labels to create and modify memories. A few augmented memory architectures have been applied to the more challenging setting of deep reinforcement learning BID29 BID17 BID20. These external memory schemes were shown to improve learning on certain tasks, typically by increasing sample efficiency. But to our knowledge none of them were evaluated on episodic memory tasks. Despite the growing body of literature on active memory, most environments do not capture the diversity of observations required to test episodic memory. For instance, in maze tasks such as the the T-Maze studied by BID17, the agent must remember a color seen at the start of the episode, then use that information later to move in the correct direction at the T-junction. But since only two colors are ever displayed at the start of the maze, the agent can learn to associate the floor tile color with the correct actions using neural network weights. In contrast, an episodic memory task like Concentration presents many previously unseen observations which must be handled correctly without prior exposure. With the differentiable neural computer (DNC), showed than an RL agent could use a memory matrix to buffer and use data to complete a moving blocks puzzle (Mini-SHRDLU). We show that Mini-SHRDLU can be decomposed into separate data buffering and problem-solving subtasks (Appendix B), neither of which requires human-like episodic memory. Episodic memory tasks can be viewed as a special type of transfer learning. Transfer and multitask learning BID26 ) involve evaluating the agent on a novel task (e.g. with new observations, rewards, transition dynamics). Episodic memory tasks such as Concentration feature an endless stream of novel observations but unchanged rewards and dynamics. Prior work on transfer learning has relied on techniques from DeepRL and model compression BID18 BID22 BID9.Concentration can be viewed as an episodic one-shot learning task in which novel observations must be correctly memorized and recalled after a single viewing. Prior work on few-shot image classification BID28 BID21 has used learned metric spaces and siamese networks . The Masked Experience Memory (MEM) architecture imbues an RL agent with the ability to leverage details of a past observation that is similar to the current observation. MEM's focus on observations differentiates it from DNC, which writes vectors to memory which are abstract in the sense that they are not bound to agent observations. Similarly, while MEM's read operation compares past observations with the current observation, DNC's read operation compares previously written memory vectors to an abstract read vector having no necessary connection to any observation. In these respects, DNC's memory mechanism is strictly more general than that of MEM, possessing more freedom of representation, as well as more potential challenges in training. This makes DNC a valuable baseline for comparison to MEM.Each MEM memory write operation copies the last observation into a fixed-size memory store, while the oldest memory is dropped from the store. Other external memory implementations share this general method of writing memories BID24 BID17. This corresponds to the rapid forgetting of human episodic memories BID8. Despite its simplicity, we view this design as applying the useful prior assumption that the most recent history is often the most relevant to selecting a good next action. Each memory read operation compares the current observation with all past observations in memory, and returns a vector calculated as a weighted sum of all memories. This part of the read operation is the same as that used by most content-based addressing memory architectures, given by: DISPLAYFORM0, where the memory matrix M contains N memories, each implemented as a real-valued vector of D dimensions (elements); the read vector R (also of length D) is a weighted average of all memories; the read weighting vector b is a normalized probability distribution over memories; finally, each memory's weight b i is a function of that memory's similarity (Q i) to the current read key vector. There are various choices for the similarity function Q, such as the popular cosine function. Here, we propose to use vector quadrance scaled by an explicit mask over vector elements: DISPLAYFORM1 where each memory's similarity (Q i) to the current read key (state vector s) is the squared Euclidean distance (quadrance) between them scaled by the corresponding element of the mask vector a; the mask vector a is a learned distribution over memory dimensions; the mask weight vector w and attention distribution sharpness parameter z are trained by gradient descent. The mask weight vector w is intended to learn which memory dimensions should be used as the lookup key for memory read operations, which in turn determines the attention distribution over memories. For instance, as a particular mask weight w i increases, the corresponding mask element a i will also increase, causing that element of each memory to contribute more to all the similarity calculations. Figure 1: Pathological Example of Cosine Similarity: Although memory vector 1 is identical to the completed (non-zero) portion of the key vector, cosine similarity judges memory vector 2 to be more similar to the key vector. MEM's usage of a mask vector in calculating vector similarity is designed to avoid a potential source of noise associated with a commonlyused similarity calculation, cosine similarity: DISPLAYFORM2 Since cosine similarity measures the angular similarity between two vectors by normalizing out their magnitudes, it is ideally suited for comparing word count vectors from documents of different lengths, for instance. But in the general case of content-based addressing, it is often intuitive to view the read key as partially specified, with zeros in the unspecified elements. From that perspective the read operation replaces the zeros with values from a memory or memories that best match the read key. This avoids the complexity of separate key and value vectors. However, applying cosine similarity in this way can add noise to the similarity calculation, as illustrated in Figure 1. Since cosine similarity normalizes the dot product by the magnitudes of both vectors being compared, the supposedly masked-out elements of the memory vector can still affect the . This noise becomes large as the non-zero portion of the key vector becomes small. MEM avoids this problem by using an explicit mask to select which vector elements will participate in the similarity measurement. In this section, we discuss how to use MEM in an RL agent. An RL agent aims to maximize its expected long-term return by acting in an initially unknown environment BID25. In each step, it makes an observation about the environment, takes an action, then receives an immediate reward and next observation. While there exist many algorithms in the literature, for concreteness, we use one of the most effective algorithms known as Asynchronous Advantage Actor-Critic (A3C) to explain how to incorporate MEM, and run experiments with this instantiation. The use of our memory architecture is similar for other RL algorithms. Since the environment is only partially observable in many real-world problems as well as in the game of Concentration, the observations received in individual time steps are not Markovian BID25. Information collected from past observations should therefore be remembered and used to make a decision at each step. One possibility is to use an LSTM to compress past observations into a fixed-length vector, which is used to approximate a Markovian state of the environment. This approach makes use of a limited form of memory, and is illustrated in FIG0 (left panel) where LSTM is used inside both the actor and critic networks of A3C.The more general DNC-based agent uses separate actor and critic DNCs, each containing its own LSTM controller and memory matrix, as shown in the right panel of FIG0. Note that the same approach is taken by for the Mini-SHRDLU task. Our proposed architecture, based on MEM, is given in the middle panel of FIG0. It uses separate actor and critic LSTM controllers which share the same memory store. For episodic RL tasks, we clear MEM's memory store at the beginning of each episode, although other possibilities exist. For all agents on every time step, each LSTM controller receives as inputs the current observation vector from the environment, concatenated with a one-hot vector representing the last action taken, plus the reward just received. Each LSTM also receives as input the most recent output from the memory store (whether MEM or DNC). In the case of MEM, a memory similarity strength value is also passed as an additional feature to the LSTMs. In the case of DNC, the memory store's output is immediately concatenated with the output from its LSTM running to the output layer of its network. To the best of our knowledge, no existing RL benchmark task could unambiguously evaluate episodic memory in RL agents. We therefore designed a new task for this purpose, derived from the common children's memory game of Concentration, as described in Wikipedia. BID0 The game is played with a deck of cards in which each card face appears twice. At the start of each game, the cards are arranged face down on a flat surface. A player's turn consists of turning over any two of the cards. If their faces are found to match, the player wins those two cards and removes them from the table, then plays again. If the two cards do not match, the player turns them face down again, then play passes to the next player. The game proceeds until all cards have been matched and removed from the table. The winning strategy is to remember the locations of the cards as their faces are revealed, then use those memories to find matching pairs. We convert the Concentration game into a single-player, episodic RL task. The agent occupies one cell at a time within a square grid of cells, each of which may be empty or may contain one card. The grid is just large enough to hold all the cards. Each card may be either face up or face down on any given time step. The agent's available actions are to take one step in any of the four directions, or to flip over the card at the current location. Whenever two cards fail to match, they automatically turn face down on the next time step. Whenever two cards match, the agent is rewarded, and the two cards are automatically removed from the grid on the next time step. The episode terminates when the last two cards are matched and removed, or when enough time steps have passed for all cards to have been removed. The agent receives a small penalty for each card it flips over. The major issue in designing this as a test of episodic memory is how to represent the agent's observation of a card face. The simplest scheme would be to represent each card face by a onehot vector of length N/2, where N is the number of cards in the deck. But this would allow the agent to solve the task without relying on episodic memory, because the total number of card faces and card positions could be small enough for the agent's network (over many training episodes) to dedicate a different unit to every possible card-plus-position combination. Then in the course of play, whenever a card was revealed, the network would only need to toggle the activation state of the unit representing that particular card and position. If two units corresponding to a matching pair were both active, the agent would know their locations and could then proceed to flip both of them over. Instead of using one-hot vectors, each card face could be represented by a complex image, such as an Omniglot character, to be processed by a convolutional neural network. But the network could still employ the strategy described above, only at a higher embedding level in the network, after learning the fixed identities of the cards. So to make this an unambiguous test of episodic memory, we generate new images for all card faces at the start of each episode. This is equivalent to playing just one game of Concentration with a deck of cards, then replacing it with a new deck of cards having different face images for the next game, etc. This ensures that each image appears in no more than one episode, making it impractical for an agent's neural network to learn each image as a persistent entity from game to game. Instead of using images composed of pixels, we define each card face to be a random real-valued vector of length 6. This can be thought of as an image embedding vector learned at some upper level of a CNN. We did not try any other image sizes. Each card face image (to appear on two cards) is generated by randomly selecting six real numbers in the range. If the ing vector is too close to an already generated vector, based on a fixed Euclidean distance threshold, the vector is randomly regenerated. The agent's performance is evaluated in terms of card-pair matches per card flip, which is closely tied to the reward received. Agents are not directly penalized for spending time wandering around the grid, but reward per time step is maximized by clearing the board quickly. For the experiments reported here, all tasks used 8 cards on a 3 × 3 grid. We tested the following six agents on the Concentration task: During training, counters keep track of the number of cards flipped by all 16 worker agents, as well as the number of card-pair matches obtained by the agents. After every 100, 000 training steps these counts are used to calculate the matches per flip for that period, then reset to zero. This produces a trailing estimate (including exploratory actions) of the agent's performance on the task. Table 1: Hyper-parameter settings agent using its optimal settings on a number of additional runs. Table 1 gives the hyper-parameter settings used on Concentration, both tuned and fixed. Key are collected in FIG2. The optimal mean performance attainable by an agent with perfect episodic memory is shown at the top of FIG2 BID27. Only the MEM agent learned a near-optimal policy. The baseline LSTM-A3C agent's were overlapped with those of its colorblind version 3b, demonstrating that the LSTM-A3C agent never learned to remember the locations of the cards it saw. The Sonnet LSTM agent performed consistently better than the TensorFlow LSTM agent 3b, though not by a large amount. Both implementations claim to be based on BID30, so the difference in behavior is unexpected. Despite being unable to see the card faces, the colorblind MEM agent 3b still performed a bit better than any of the LSTM agents, indicating that it found some other strategy (not based on card faces) to derive a small amount of gain from its external memory. Even after dozens of trial settings over a wide range of hyper-parameters, the DNC agent performed only very slightly better than the LSTM-A3C agent, and noticeably worse than its own recurrent controller alone, the Sonnet LSTM agent. We did not attempt curriculum learning. Appendix A presents a detailed investigation into the causes of DNC's poor performance on this type of task. Performing ablation studies on the MEM architecture, we found that using the mask (instead of cosine similarity) and Euclidean distance squared were both essential to scoring above the LSTM-A3C baseline. Adaptation of the sharpness term turned out to be essential for stable . On the other hand, the similarity strength feature provided no measurable benefit. As intended, MEM's most positive learned mask weights were the ones for the six card face dimensions. At convergence of the best MEM model, 83% of the mask's mass was concentrated on those six elements, even though they constitute only 11% of the observation vector's 54 elements. We have defined episodic memory for RL agents, provided an unambiguous test for evaluating it, and presented an implementation of episodic memory that corrects a problem with current content-based addressing methods. Our show that this MEM architecture, designed to emulate specific aspects of human episodic memory, is able to use that memory effectively in the Concentration task by remembering the locations of cards it has seen only once before. This is in sharp contrast to the other agents tested, which never learned to remember card locations. The code to replicate this work will be made public prior to the conference. MEM represents the initial step on a path towards more robust and powerful episodic memory for RL agents. We plan to extend MEM in several significant ways:1. Making the mask weights context-sensitive so that read key vectors can quickly shift to cover different aspects of experience depending on the situation. 2. Expanding the memory dimensions beyond the current observation to also include recurrent network activations, so that an agent's internal thought vectors can themselves be stored as experiences for later recall, and can be used as read keys. 3. Rendering memory deletion a function of memory importance, so that certain experiences can be remembered longer than others. 4. Introducing an additional mask over dimensions for write operations, so that memories need not cover all available dimensions. The human mind offers a remote, shining existence proof of general intelligence still beyond our reach. Despite the distance, it lights our path, and grows brighter with each step we take toward it. The differentiable neural computer ) employs a differentiable memory matrix as external memory, which shares certain sequential addressing features with computer random access memory, with the objective of combining the advantages of neural and computational processing in one trainable system. To explore why DNC never learned to use its memory matrix on the Concentration task, we applied DNC to a series of simpler tests of associative recall. The general pattern was that DNC performance seemed to deteriorate with episode length, as if DNC had difficulty reusing previously allocated locations in its memory matrix. We found the simplest demonstration of this problem to be a small modification of the copy task included in DNC's GitHub repository. For Figures 4 and 5 the memory matrix was of size 16x16, exactly large enough to store the 16x16 random input data in each copy round, even if none of the data was stored in the LSTM (size 64). But when given two copy rounds in a sequence with no reset in between, the only way to achieve zero error was to reuse some locations in the memory matrix. This happened on two of our runs, but not on the other three runs. The instabilities in FIG4 demonstrate the difficulty that DNC had in learning to reuse its memory matrix. B MINI-SHRDLU DNC was evaluated on multiple supervised tasks and one RL task: Mini-SHRDLU. The Mini-SHRDLU task was actually composed of two separate sub-tasks: data buffering and puzzle solving. The constraints defining the problem, along with many other decoy constraints, were fed to the RL agent once while the agent was not allowed to work on the puzzle. Only after termination of the constraint presentation phase was the agent allowed to reposition the blocks to solve the puzzle. The demonstrated that DNC used its external memory to buffer the incoming constraint information in the memory matrix, then used that data to achieve significantly better on the combined buffer-puzzle task than did a baseline LSTM-based RL agent without external memory. We considered using the Mini-SHRDLU task as a test of an RL agent's episodic memory. The databuffering stage of the task did not seem relevant to this goal, since human memory seems ill-suited for memorizing long lists of data seen only once. Since the buffer and puzzle sub-tasks were not Under review as a conference paper at ICLR 2018 evaluated separately, we couldn't be sure whether DNC's external memory helped in the puzzlesolving component of the combined task. We investigated this question by implementing Mini-SHRDLU without the data-buffering subtask, and giving an LSTM-based A3C RL agent access to a simple, non-differentiable, circular array of constraints. This allowed the agent to read the instructions at its own pace using 3 additional actions (move to next element, move to previous element, stay at current element). Figure 6: Mini-SHRDLU . The DNC were copied from. The LSTM-A3C were achieved by an RL agent with no external memory that read the constraints from a simple array at its own pace. After training on a lesson, the LSTM agent was tested on that lesson using 10,000 random problems. Direct numeric comparisons between these two performance curves are not meaningful for various reasons: The LSTM agent was advanced to the next lesson only after its performance on the previous lesson had plateaued, which gave it more training problems than used by DNC. The LSTM agent was always given problems using the full 6 blocks on the first 6 lessons, which is why those are not shown. Twelve of the subsequent lessons were actually skipped during training, and the LSTM agent was tested on those lessons using the model from the next trained lesson. As shown in Figure 6, the LSTM-based RL agent learns to solve Mini-SHRDLU problems as well as DNC, but without using a differentiable memory matrix. These demonstrate that external memory was not required for the problem-solving stage of the task.

Implementing and evaluating episodic memory for RL.

Meta-learning curiosity algorithms by searching through a rich space of programs yields novel mechanisms that generalize across very different reinforcement-learning domains.