Question: Represent the article for finding a single-sentence TLDR

Question: Catastrophic forgetting poses a grand challenge for continual learning systems, which prevents neural networks from protecting old knowledge while learning new tasks sequentially. We propose a Differentiable Hebbian Plasticity (DHP) Softmax layer which adds a fast learning plastic component to the slow weights of the softmax output layer. The DHP Softmax behaves as a compressed episodic memory that reactivates existing memory traces, while creating new ones. We demonstrate the flexibility of our model by combining it with existing well-known consolidation methods to prevent catastrophic forgetting. We evaluate our approach on the Permuted MNIST and Split MNIST benchmarks, and introduce Imbalanced Permuted MNIST — a dataset that combines the challenges of class imbalance and concept drift. Our model requires no additional hyperparameters and outperforms comparable baselines by reducing forgetting. A key aspect of human intelligence is the ability to continually adapt and learn in dynamic environments, a characteristic which is challenging to embed into artificial intelligence. Recent advances in machine learning (ML) have shown tremendous improvements in various problems, by learning to solve one complex task very well, through extensive training on large datasets with millions of training examples or more. Most of the ML models that we use during deployment assume that the real-world is stationary, where in fact it is non-stationary and the distribution of acquired data changes over time. Therefore, after learning is complete, and these models are fine-tuned with new data, performance degrades with respect to the original data. This phenomenon *Work done during an internship at Uber AI. † Work done while at Google Brain. known as catastrophic forgetting or catastrophic interference BID17 BID7 serves to be a crucial problem for deep neural networks (DNNs) that are tasked with continual learning BID26 or lifelong learning . In this learning paradigm, the goal is to adapt and learn consecutive tasks without forgetting how to perform previously learned tasks. Some of the real-world applications that typically require this kind of learning include perception for autonomous vehicles, recommender systems, fraud detection, etc. In most supervised learning methods, DNN architectures require independent and identically distributed (iid) samples from a stationary training distribution. However, for ML systems that require continual learning in the real-world, the iid assumption is easily violated when: There is concept drift or class imbalance in the training data distribution. Data representing all scenarios in which the learner is expected to perform are not initially available. In such situations, DNNs face the "stability-plasticity dilemma" BID6 BID0. This presents a continual learning challenge for models that need to balance plasticity (integrate new knowledge) and stability (preserve existing knowledge).Two major theories have been proposed to explain a human's ability to perform continual learning. The first theory is inspired by synaptic consolidation in the mammalian neocortex BID5 where a subset of synapses are rendered less plastic and therefore preserved for a longer timescale. The second theory is the complementary learning systems (CLS) theory BID16 BID23 BID12, which suggests that humans extract high-level structural information and store it in a different brain area while retaining episodic memories. Here, we extend the work on differentiable plasticity BID18 BID19 to a continual learning setting and develop a model that is capable of adapting quickly to changing environments as well as consolidating previous knowledge by selectively adjusting the plasticity of synapses. We modify the traditional softmax layer and propose to augment the slow weights with a set of plastic weights implemented using Differentiable Hebbian Plasticity (DHP). The model's slow weights learn deep representations of data and the fast weights implemented with DHP learn to quickly "auto-associate" the class labels to representations. We also demonstrate the flexibility of our model by combining it with recent task-specific synaptic consolidation based methods to overcoming catastrophic forgetting such as elastic weight consolidation BID11 BID28, synaptic intelligence and memory aware synapses. Our model unifies core concepts from Hebbian plasticity, synaptic consolidation and CLS theory to enable rapid adaptation to new unseen data, while consolidating synapses and leveraging compressed episodic memories to remember previous knowledge and mitigate catastrophic forgetting. Plastic Neural Networks: One of the major theories that have been proposed to explain a human's ability to learn continually is Hebbian learning BID8, which suggests that learning and memory are attributed to weight plasticity, that is, the modification of the strength of existing synapses according to variants of Hebb's rule BID24; BID22.Recent approaches in the meta-learning literature have shown that we can incorporate fast weights into a neural network BID21 BID25. BID21 augmented fully-connected (FC) layers preceding the softmax with a matrix of fast weights. Here, the fast weights were implemented with non-trainable Hebbian learning-based associative memory. BID25 proposed a softmax layer that can improve learning of rare classes by interpolating between Hebbian updates and stochastic gradient descent (SGD) updates on the output layer using an arbitrarily engineered scheduling scheme. BID19 proposed differentiable plasticity, which uses SGD to optimize the plasticity of each synaptic connection composed of a slow weight and a plastic (fast) weight. Although this approach served to be a powerful new method for training neural networks, it was mainly demonstrated on RNNs for solving simple tasks. Overcoming Catastrophic Forgetting: This work leverages two biologically inspired strategies to overcome the catastrophic forgetting problem: 1) Task-specific Synaptic Consolidation -Protecting old knowledge by dynamically adjusting the synaptic strengths to consolidate and retain memories. 2) CLS Theory -A dual memory system where, structural knowledge is acquired through slow learning via the neocortex and rapid learning via the hippocampus. There have been several notable works inspired by taskspecific synaptic consolidation for overcoming catastrophic forgetting BID11; BID1. All of these approaches propose a method to estimate the importance of each parameter or synapse, Ω k, where the least plastic synapses can retain memories for a long time and the more plastic synapses are considered less important. The Ω k and network parameters θ k are updated online or after learning task T n. Therefore, when learning new task T n, a regularizer is added to the original loss function L n (θ), so that we dynamically adjust the plasticity w.r.t. Ω k and prevent any changes to the important parameters of previously learned tasks: DISPLAYFORM0 where θ n−1 k are the learned network parameters after training on the previous n − 1 tasks and λ is a hyperparameter for the regularizer to control the amount of forgetting. In Elastic Weight Consolidation (EWC), BID11 use the diagonal values of an approximated Fisher information matrix for Ω k, and it is computed offline after training on a task is completed. BID28 proposed an online variant of EWC to improve scalability by ensuring the computational cost of the regularization term does not grow with the number of tasks. proposed an online method called Synaptic Intelligence (SI) for computing the parameter importance where, Ω k is the cumulative change in individual synapses over the entire training trajectory on a given task. Memory Aware Synapses (MAS) from BID1 measures Ω k by the sensitivity of the learned function to a perturbation in the parameters and use the cumulative change in individual synapses on the squared L2-norm of the penultimate layer. There have been numerous approaches based on CLS principles involving pseudo-rehersal (; BID2 BID3, episodic replay BID15 BID14 and generative replay . However, in our work, we are primarily interested in neuroplasticity techniques inspired from CLS theory for representing memories. showed how each synaptic connection can be composed of a fixed weight where slow learning stores long-term knowledge and a fast weight for temporary associative memory. Recent research in this vein has included replacing soft attention mechanism with fast weights in RNNs BID4, the Hebbian Softmax layer BID25, augmenting the FC layer with a fast weights matrix BID21, differentiable plasticity BID19 and neuromodulated differentiable plasticity BID20. However, all of these methods were focused on rapid learning on simple tasks or meta-learning over a distribution of tasks. Furthermore, they did not examine learning a large number of new tasks while, alleviating catastrophic forgetting in continual learning. In our model, each synaptic connection in the softmax layer has two weights: 1) The slow weights, θ ∈ R m×d, where m is the number of units in the final hidden layer. 2) A Hebbian plastic component of the same cardinality as the slow weights, composed of the plasticity coefficient, α, and the Hebbian trace, Hebb. The α is a scaling parameter for adjusting the magnitude of the Hebb. Hebb accumulates the mean activations of the penultimate layer for each target label in the mini-batch {y 1:B} of size B which are denoted byh ∈ R 1×m (refer to Algorithm 1). Given the activation of each neuron in h at the pre-synaptic connection i, the unnormalized log probabilities z at the post-synaptic connection j can be more formally computed using Eq. 2. Then, the softmax function is applied on z to obtain the desired logitsŷ thus,ŷ = softmax(z). The η parameter in Eq. 3 is a "learning rate" that learns how quickly to acquire new experiences into the plastic component. The η parameter also acts as a decay term to prevent instability caused by a positive feedback loop in the Hebbian traces. DISPLAYFORM0 The network parameters α i,j, η and θ i,j are optimized by gradient descent as the model is trained sequentially on different tasks in the continual learning setup. Hebb is initialized to zero only at the start of learning the first task T 1 and is automatically updated based on Algorithm 1 in the forward pass during training. Specifically, the Hebbian update for the active class c in y 1:B is computed on line 6. This Hebbian update BID8, where w i,j is the change in weight at connection i, j and a k i, a k j denote the activation levels of neurons i and j, respectively, for the k th input. Therefore, in our model, w =h the Hebbian weight update, a i = h the hidden activations of the last hidden layer, a j = y the active target class in y 1:B and N = s the number of inputs for the corresponding class in y 1:B (see Algorithm 1). Across the model's lifetime, we only update Hebb during training and during test time, we use the most recent Hebbian traces to make predictions. The plastic component learns rapidly and performs sparse parameter updates to quickly store memory traces for each recent experience without interference from other similar recent experiences. Furthermore, the hidden activations corresponding to the same active class are accumulated into one vectorh, thus forming a compressed episodic memory in the Hebb to reflect individual episodic memory traces. This method improves learning of rare classes and speeds up binding of class labels to deep representations of the data. Updated Loss: Following the existing work for overcoming catastrophic forgetting such as EWC, Online EWC, SI and MAS (see Eq. 1), we regularize the loss L n (θ, α, η) and update the synaptic importance parameters of the network in an online manner. We rewrite Eq. 1 to obtain Eq. 4 and show that the network parameters θ i,j are the weights of the connections between pre-and post-synaptic activity, as seen in Eq. 2. DISPLAYFORM1 We adapt these existing consolidation approaches to our model and only compute the synaptic importance parameters on the slow weights of the network. The plastic part of our model can alleviate catastrophic forgetting of learned classes by optimizing the plasticity of the synaptic connections. We tested our continual learning approach on the Permuted MNIST, Imbalanced Permuted MNIST and Split MNIST benchmarks. We evaluated the methods based on the average classification accuracy on all previously learned tasks. To establish a baseline for comparison of well-known synaptic consolidation methods, we trained neural networks with Online EWC, SI and MAS, respectively, on all tasks in a sequential manner. In the Permuted MNIST and Imbalanced Permuted benchmarks we trained a multi-layered perceptron (MLP) network on a sequence of 10 tasks using plain SGD. Detailed descriptions of the hyperparameters and training setups for all benchmarks can be found in Appendix A.Permuted MNIST: In this benchmark, all of the MNIST pixels are permuted differently for each task with a fixed random permutation. Although the output domain is constant, the input distribution changes between tasks thus, there exists a concept drift. Figure 1 shows the average test accuracy as new tasks are learned. The network with DHP Softmax alone showed significant improvement in its ability to alleviate catastrophic forgetting across all tasks compared to the baseline finetuned vanilla MLP network we refer to as Finetune in Figure 1. Then we compared the performance with and without DHP Softmax using the synaptic consolidation methods. We find our DHP Softmax with synaptic consolidation maintains a higher test accuracy after T 10 tasks than without DHP Softmax for all variants. Figure 1. The average test accuracy on a sequence of Permuted MNIST tasks Tn=1:10. The average test accuracy after T10 tasks is given in the legend. Error bars correspond to SE on 10 trials. This benchmark is identical to the Permuted MNIST benchmark but, now each task is an imbalanced distribution. The statistics of the class distribution in each task are presented in Appendix A.2, Table 1. Figure 2 shows the average test accuracy as new tasks are learned. We see that DHP Softmax achieves 80.85% after learning 10 tasks, thus providing significant improvement over the standard neural network baseline of 76.4%. The significance of the compressed episodic memory mechanism in the Hebbian traces is more apparent in this benchmark because the plastic component allows rare classes that are encountered infrequently to be remembered for a longer period of time. We find that DHP Softmax with MAS achieves 88.8%; outperforming all other methods and across all tasks. Split MNIST: A sequence of T n=1:5 tasks are generated by splitting the original MNIST training dataset into binary classification problems (0/1, 2/3, 4/5, 6/7, 8/9), making the output spaces disjoint between tasks. Similar to , we trained a multi-headed MLP network on a sequence of 5 tasks. We compute the cross entropy loss at the softmax output layer only for the digits present in the current task, T n. We observe that DHP Softmax provides a 4.7% improvement on test performance compared to a finetuned MLP network (Figure 3). Also, combining DHP Softmax with task-specific consolidation consistently improves performance across all tasks T n=1:5. Figure 3. The average test accuracy on a sequence of 5 binary classification problems (0/1, 2/3, 4/5, 6/7, 8/9) from the original MNIST dataset. The average test accuracy after learning T5 tasks is given in the legend. Error bars refer to the SE on 10 trials. We have shown that the problem of catastrophic forgetting in continual learning environments can be alleviated by adding compressed episodic memory in the softmax layer through DHP. DHP Softmax alone showed noticeable improvement across all benchmarks when compared to a neural network with a traditional softmax layer. We demonstrated the flexibility of our model where, in addition to DHP Softmax, we can regularize the slow weights using EWC, SI or MAS to improve a model's ability to alleviate catastrophic forgetting. The approach where we combine DHP Softmax and MAS consistently leads to overall superior compared to other baseline methods on several benchmarks. This gives a strong indication that Hebbian plasticity enables neural networks to learn continually and remember distant memories, thus reducing catastrophic forgetting when learning from sequential datasets in dynamic environments. For the Imbalanced Permuted MNIST experiments shown in Figure 2, the regularization hyperparameter λ for each of the task-specific consolidation methods is λ = 400 for Online EWC BID28, λ = 1.0 for SI and λ = 0.1 for MAS. In SI, the damping parameter, ξ, was set to 0.1. Similar to the Permuted MNIST benchmark, to find the best hyperparameter combination for each of these synaptic consolidation methods, we performed a grid search using a task sequence determined by a single seed. Across all experiments, we maintained the the same random probabilities detemined by a single seed to artificially remove training samples from each class. The hyperparameters of the synaptic consolidation methods (i.e. Online EWC, SI and MAS) remain the same with and without DHP Softmax, and the plastic components are not regularized.

Represent the TLDR for finding the corresponding scientific article

Hebbian plastic weights can behave as a compressed episodic memory storage in neural networks; improving their ability to alleviate catastrophic forgetting in continual learning.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.


Question:
represent the scientific article to find a short tldr

Given We propose a robust Bayesian deep learning algorithm to infer complex posteriors with latent variables. Inspired by dropout, a popular tool for regularization and model ensemble, we assign sparse priors to the weights in deep neural networks (DNN) in order to achieve automatic “dropout” and avoid over-fitting. By alternatively sampling from posterior distribution through stochastic gradient Markov Chain Monte Carlo (SG-MCMC) and optimizing latent variables via stochastic approximation (SA), the trajectory of the target weights is proved to converge to the true posterior distribution conditioned on optimal latent variables. This ensures a stronger regularization on the over-fitted parameter space and more accurate uncertainty quantification on the decisive variables. Simulations from large-p-small-n regressions showcase the robustness of this method when applied to models with latent variables. Additionally, its application on the convolutional neural networks (CNN) leads to state-of-the-art performance on MNIST and Fashion MNIST datasets and improved resistance to adversarial attacks. Bayesian deep learning, which evolved from Bayesian neural networks (; BID4, provides an alternative to point estimation due to its close connection to both Bayesian probability theory and cutting-edge deep learning models. It has been shown of the merit to quantify uncertainty BID6, which not only increases the predictive power of DNN, but also further provides a more robust estimation to enhance AI safety. Particularly, BID5 BID3 described dropout as a variational Bayesian approximation. Through enabling dropout in the testing period, the randomly dropped neurons generate some amount of uncertainty with almost no added cost. However, the dropout Bayesian approximation is variational inference (VI) based thus it is vulnerable to underestimating uncertainty. MCMC, known for its asymptotically accurate posterior inference, has not been fully investigated in DNN due to its unscalability in dealing with big data and large models. Stochastic gradient Langevin dynamics (SGLD) , the first SG-MCMC algorithm, tackled this issue by adding noise to a standard stochastic gradient optimization, smoothing the transition between optimization and sampling. Considering the pathological curvature that causes the SGLD methods inefficient in DNN models, BID15 proposed combining adaptive preconditioners with SGLD (pSGLD) to adapt to the local geometry and obtained state-of-the-art performance on MNIST dataset. To avoid SGLD's random-walk behavior, BID3 proposed using stochastic gradient Hamiltonian Monte Carlo (SGHMC), a second-order Langevin dynamics with a large friction term, which was shown to have lower autocorrelation time and faster convergence BID2. used SGHMC with GANs BID8 ) to achieve a fully probabilistic inference and showed the Bayesian GAN model with only 100 labeled images was able to achieve 99.3% testing accuracy in MNIST dataset.;; provided theoretical interpretations of SGLD from the perspective of non-convex optimization, echoing the empirical fact that SGLD works well in practice. When the number of predictors exceeds the number of observations, applying the spike-and-slab priors is particularly powerful and efficient to avoid over-fitting by assigning less probability mass on We denote the decaying learning rate at time k by (k), the entire data by DISPLAYFORM0, where d i = (x i, y i), the log of posterior by L(β), ∇ as the gradient of any function in terms of β. The minibatch of data B is of size n with indices S = {s 1, s 2, ..., s n}, where s i ∈ {1, 2, ..., N}. Stochastic gradient ∇L(β) from a mini-batch of data B randomly sampled from D is used to approximate the true gradient ∇L(β):∇L(β) = ∇ log P(β) + N n i∈S ∇ log P(d i |β).The stochastic gradient Langevin dynamics (no momentum) is formed as follows: DISPLAYFORM1 where τ > 0 denotes the temperature, G is a positive definite matrix to precondition the dynamics. It has been shown that SGLD asymptotically converges to a stationary distribution π(β|D) ∝ e τ L(β) . As τ increases, the algorithm tends towards optimization with underestimated uncertainty. Another variant of SG-MCMC, SGHMC BID3 ), proposes to use second-order Langevin dynamics to generate samples: DISPLAYFORM2 where r is the momentum item, M is the mass,B is an estimate of the error variance from the stochastic gradient, C is a user-specified friction term to counteracts the noisy gradient. Dropout has been proven successful, as it alleviates over-fitting and provides an efficient way of making bagging practical for ensembles of countless sub-networks. Dropout can be interpreted as assigning the Gaussian mixture priors on the neurons BID7. To mimic Dropout in our Bayesian CNN models, we assign the spike-and-slab priors on the most fat-tailed weights in FC1 FIG0. From the Bayesian perspective, the proposed robust algorithm distinguishes itself from the dropout approach in treating the priors: our algorithm keeps updating the priors during posterior inference, rather than fix it. The inclusion of scaled mixture priors in deep learning models were also studied in BID1; BID15 with encouraging . However, to the best of our knowledge, none of the existing SG-MCMC methods could deal with complex posterior with latent variables. Intuitively, the Bayesian formulation with model averaging and the spike-and-slab priors is expected to obtain better predictive performance through averages from a "selective" group of "good" submodels, rather than averaging exponentially many posterior probabilities roughly. For the weight priors of the rest layers (dimension u), we just assume they follow the standard Gaussian distribution, while the biases follow improper uniform priors. Similarly to the hierarchical prior in the EM approach to variable selection (EMVS) (Rořková and), we assume the weights β ∈ R p in FC1 follow the spike-and-slab mixture prior DISPLAYFORM0 where DISPLAYFORM1 for each j and 0 < v 0 < v 1. By introducing the latent variable γ j = 0 or 1, the mixture prior is represented as DISPLAYFORM2 The interpretation is: if γ j = 0, then β j is close to 0; if γ j = 1, the effect of β j on the model is intuitively large. The likelihood of this model given a mini-batch of data {( DISPLAYFORM3 where ψ(x i ; β) can be a mapping for logistic regression or linear regression, or a mapping based on a series of nonlinearities and affine transformations in the deep neural network. In the classification formulation, y i ∈ {1, . . ., K} is the response value of the i-th example. In addition, the variance σ 2 follows an inverse gamma prior DISPLAYFORM4 The i.i.d. Bernoulli prior is used since there is no structural information in the same layer. DISPLAYFORM5 Finally, our posterior density follows DISPLAYFORM6 The EMVS approach is efficient in identifying potential sparse high posterior probability submodels on high-dimensional regression (Rořková and) and classification problem . These characteristics are helpful for large neural network computation, thus we refer the stochastic version of the EMVS algorithm as Expectation Stochastic-Maximization (ESM). Due to the existence of latent variables, optimizing π(β, σ 2, δ|B) directly is difficult. We instead iteratively optimize the "complete-data" posterior log π(β, σ 2, δ, γ|B), where the latent indicator γ is treated as "missing data".More precisely, the ESM algorithm is implemented by iteratively increasing the objective function DISPLAYFORM0 where E γ|· denotes the conditional expectation DISPLAYFORM1 ) at the k-th iteration, we first compute the expectation of Q, then alter (β, σ, δ) to optimize it. For the conjugate spike-slab hierarchical prior formulation, the objective function Q is of the form DISPLAYFORM2 where DISPLAYFORM3 spike-and-slab priors in the "sparse" layer, e.g. FC1 DISPLAYFORM4 Gaussian priors in other layers DISPLAYFORM5 and DISPLAYFORM6 The physical meaning of E γ|· γ j in Q 2 is the probability ρ j, where ρ ∈ R p, of β j having a large effect on the model. Formally, we have DISPLAYFORM0 where DISPLAYFORM1 Bernoulli prior enables us to use P(γ j = 1|δ DISPLAYFORM2 The other conditional expectation comes from a weighted average κ j, where κ ∈ R p . DISPLAYFORM3 Since there is no closed-form optimal solution for β here, to optimize Q 1 with respect to β, we use Adam , a popular algorithm with adaptive learning rates, to train the model. In order to optimize Q 1 with respect to σ, by denoting DISPLAYFORM0 as V, following the formulation in and Rořková and we have: DISPLAYFORM1 To optimize Q 2, a closed-form solution can be derived from Eq. and Eq.. DISPLAYFORM2 Algorithm 1 SGLD-SA with spike and slab priors DISPLAYFORM3 Initialize: DISPLAYFORM4 end for The EMVS algorithm is designed for linear regression models, although the idea can be extended to nonlinear models. However, when extending to nonlinear models, such as DNNs, the M-step will not have a closed-form update anymore. A trivial implementation of the M-step will likely cause a local-trap problem. To tackle this issue, we replace the E-step and the M-step by SG-MCMC with the prior hyperparameters tuned via stochastic approximation BID0: DISPLAYFORM0 where DISPLAYFORM1 is the gradient-like function in stochastic approximation (see details in Appendix A.2), g θ (·) is the mapping detailed in Eq., Eq., Eq. and Eq. to derive the optimal θ based on the current β, the step size ω (k) can be set as A(k + Z) −α with α ∈ (0.5, 1]. The interpretation of this algorithm is that we sample β (k+1) fromL(β (k), θ (k) ) and adaptively optimize θ (k+1) from the mapping g θ (k). We expect to obtain an augmented sequence as follows: DISPLAYFORM2 We show the (local) L 2 convergence rate of SGLD-SA below and present the details in Appendix B.Theorem 1 (L 2 convergence rate). For any α ∈ and any compact subset Ω ∈ R 2p+2, under assumptions in Appendix B.1, the algorithm satisfies: there exists a constant λ such that DISPLAYFORM3 where t(Ω) = inf{k : DISPLAYFORM4 Corollary 1. For any α ∈ and any compact subset Ω ∈ R 2p+2, under assumptions in Appendix B.2, the distribution of β (k) in FORMULA1 converges weakly to the invariant distribution e L(β,θ *) as → 0.The key to guaranteeing the consistency of the latent variable estimators is from stochastic approximation and the fact of DISPLAYFORM5 The non-convex optimization of SGLD , in particular the ability to escape shallow local optima , ensures the robust optimization of the latent variables. Furthermore, the mappings of Eq., Eq., Eq. and Eq. all satisfy the assumptions on g in a compact subset Ω (Appendix B), which enable us to apply theorem 1 to SGLD-SA. Because SGHMC proposed by BID3 is essentially a second-order Langevin dynamics and yields a stationary distribution given an accurate estimation of the error variance from the stochastic gradient, the property of SGLD-SA also applies to SGHMC-SA. Corollary 2. For any α ∈ and any compact subset Ω ∈ R 2p+2, under assumptions in Appendix B.2, the distribution of β (k) from SG-MCMC-SA converges weakly to the invariant distribution π(β, ρ *, κ *, σ *, δ * |D) as → 0. The posterior average given decreasing learning rates can be approximated through the weighted DISPLAYFORM0 to avoid over-emphasizing the tail end of the sequence and reduce the variance of the estimator.; BID2 showed a theoretical optimal learning rate (k) ∝ k −1/3 for SGLD and k −1/5 for SGHMC to achieve faster convergence for posterior average, which are used in Sec. 4.1 and Sec. 4.2 respectively. 4.1 SIMULATION OF LARGE-P-SMALL-N REGRESSION SGLD-SA can be applied to the (logistic) linear regression cases, as long as u = 0 in Eq.. We conduct the linear regression experiments with a dataset containing n = 100 observations and p = 1000 predictors. N p (0, Σ) is chosen to simulate the predictor values X (training set) where DISPLAYFORM0 Response values y are generated from Xβ + η, where β = (β 1, β 2, β 3, 0, 0, ..., 0) and η ∼ N n (0, 3I n). To make the simulation in Rořková and more challenging, we assume DISPLAYFORM1 We introduce some hyperparameters, but most of them are uninformative, e.g. ν ∈ {10 −3, 1, 10 3} makes little difference in the test set performance. Sensitivity analysis shows that three hyperparameters are important: v 0, a and σ, which are used to identify and regularize the over-fitted space. We fix τ = 1, λ = 1, ν = 1, v 1 = 100, δ = 0.5, b = p and set a = 1. The learning rates for SGLD-SA and SGLD are set to (k) = 0.001 × k To implement SGLD-SA, we perform stochastic gradient optimization by randomly selecting 50 observations and calculating the corresponding gradient in each iteration. We simulate 500, 000 samples from the posterior distribution and at the same time keep optimizing the latent variables. EMVS is implemented with β directly optimized each time. We also simulate a group of the test set with 1000 observations (display 50 in FIG5 (e)) in the same way as generating the training set to evaluate the generalizability of our algorithm. Tab.1 shows that EMVS frequently fails given bad initializations, while SGLD-SA is fairly robust to the hyperparameters. In addition, from, we can see SGLD-SA is the only algorithm among the three that quantifies the uncertainties of β 1, β 2 and β 3 reasonably and always gives more accurate posterior average (FIG5); by contrast, the estimated response values y from SGLD is close to the true values in the training set (FIG5), but are far away from them in the testing set (2(e)), indicating the over-fitting problem of SGLD without proper regularization. For the simulation of SGLD-SA in logistic regression to demonstrate the advantage of SGLD-SA over SGLD and ESM, we leave the in Appendix C. We implement all the algorithms in Pytorch and run the experiments on GeForce GTX 1080 GPUs. The first DNN we use is a standard 2-Conv-2-FC CNN model (FIG0) of 670K parameters (see details in Appendix D.1). The first set of experiments is to compare methods on the same model without using other complication, such as data augmentation (DA) or batch normalization (BN) BID12. We refer to the general CNN without dropout as Vanilla, with 50% dropout rate applied to the green neurons (FIG0 as Dropout. Vanilla and Dropout models are trained q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q with Adam with Pytorch default parameters (with learning rate 0.001). We use SGHMC as a benchmark method as it is also sampling-based and has a close relationship with the popular momentum based optimization approaches in DNN. SGHMC-SA differs from SGHMC in that SGHMC-SA applies the spike-and-slab priors to the FC1 layer while SGHMC just uses the standard normal priors. The hyperparameters v 1 = 1, v 0 = 1 × 10 −3 and σ = 0.1 in SGHMC-SA are used to regularize the over-fitted space, and a, b are set to p to obtain a moderate "sparsity" to resemble dropout, the step size is ω (k) = 0.1 × (k + 1000) DISPLAYFORM0. We use training batch size 1000 and a thinning factor 500 to avoid a cumbersome system, and the posterior average is applied to each Bayesian model. Temperatures are tuned to achieve better (Appendix D.2).The four CNN models are tested on the classical MNIST and the newly introduced Fashion MNIST (FMNIST) dataset. Performance of these models is shown in Tab.2. Compared with SGHMC, our SGHMC-SA outperforms SGHMC on both datasets. We notice the posterior averages from SGHMC-SA and SGHMC obtain much better performance than Vanilla and Dropout. Without using either DA or BN, SGHMC-SA achieves 99.60% which even outperforms some state-of-the-art models, such as stochastic pooling (99.53%) , Maxout Network (99.55%) BID9 and pSGLD (99.55%) BID15. In F-MNIST, SGHMC-SA obtains 93.01% accuracy, outperforming all other competing models. To further test the maximum performance of SGHMC-SA, we apply DA and BN to the following experiments (see details in Appendix D.3) and refer the datasets with DA as aMNIST and aFMNIST. All the experiments are conducted using a 2-Conv-BN-3-FC CNN of 490K parameters. Using this model, we obtain 99.75% on aMNIST (300 epochs) and 94.38% on aFMNIST (1000 epochs). The are noticeable because posterior model averaging is essentially conducted on a single Bayesian neural network. We also conduct the experiments based on the ensemble of five networks and refer them as aMNIST-5 and aFMNIST-5 in Tab. 2. We achieve 99.79% on aMNIST-5 using 5 small Bayesian neural networks each with 2 thinning samples (4 thinning samples in aFMNIST-5), which is comparable with the state-of-the-art performance . Continuing with the setup in Sec. 4.2, the third set of experiments focus on evaluating model robustness. We expect less robust models perform considerably well on a certain dataset due to over-tuning; however, as the degree of adversarial attacks increases, the performance decreases sharply. In contrast, more robust models should be less affected by these adversarial attacks. We apply the Fast Gradient Sign method BID10 to generate the adversarial examples with one single gradient step as in's study: DISPLAYFORM0 where ζ ranges from 0.1, 0.2,..., 0.5 to control the different levels of adversarial attacks. Similar to the setup in the adversarial experiments by BID16, we normalize the adversarial images by clipping to the range. As shown in FIG6 and FIG6, there is no significant difference among all the four models in the early phase. As the degree of adversarial attacks arises, the images become vaguer as shown in FIG6 and FIG6. In this scenario the performance of Vanilla decreases rapidly, reflecting its poor defense against adversarial attacks, while Dropout performs better than Vanilla. But Dropout is still significantly worse than the sampling based methods SGHMC-SA and SGHMC. The advantage of SGHMC-SA over SGHMC becomes more significant when ζ > 0.25.In the case of ζ = 0.5 in MNIST where the images are hardly recognizable, both Vanilla and Dropout models fail to identify the right images and their predictions are as worse as random guesses. However, SGHMC-SA model achieves roughly 11% higher than these two models and 1% higher than SGHMC, which demonstrates the strong robustness of our proposed SGHMC-SA. Overall, SGHMC-SA always yields the most robust performance. We propose a mixed sampling-optimization method called SG-MCMC-SA to efficiently sample from complex DNN posteriors with latent variables and prove its convergence. By adaptively searching and penalizing the over-fitted parameter space, the proposed method improves the generalizability of deep neural networks. This method is less affected by the hyperparameters, achieves higher prediction accuracy over the traditional SG-MCMC methods in both simulated examples and real applications and shows more robustness towards adversarial attacks. Interesting future directions include applying SG-MCMC-SA towards popular large deep learning models such as the residual network BID11 on CIFAR-10 and CIFAR-100, combining active learning and uncertainty quantification to learn from datasets of smaller size and proving posterior consistency and the consistency of variable selection under various shrinkage priors concretely. The Fokker-Planck equation (FPE) can be formulated from the time evolution of the conditional distribution for a stationary random process. Denoting the probability density function of the random process at time t by q(t, β), where β is the parameter, the stochastic dynamics is given by DISPLAYFORM0 Let q(β) = lim t→∞ q(t, β). If lim t→∞ ∂ t q(t, β) = 0, then DISPLAYFORM1. In other words, lim t→∞ q(t, β) ∝ e L(β), i.e. q(t, β) gradually converges to the Bayesian posterior e L(β). Robbins-Monro algorithm is the first stochastic approximation algorithm to deal with the root finding problem. Given the random output of H(θ, β) with respect to β, our goal is to find θ * such that DISPLAYFORM0 where E θ * denotes the expectation with respect to the distribution of β given parameter θ *. To implement the Robbins-Monro Algorithm, we can generate iterates of the form as follows: DISPLAYFORM1 Note that in this algorithm, H(θ, β) is the unbiased estimator of h(θ), that is DISPLAYFORM2 If there exists an antiderivative Q(θ, β) that satisfies H(θ, β) = and E θ [Q(θ, β)] is concave, it is equivalent to solving the stochastic optimization problem max θ∈Θ E θ [Q(θ, β)]. In contrast to Eq. and Eq., the general stochastic approximation algorithm is intent on solving the root of the integration equation DISPLAYFORM0 where θ ∈ Θ, β ∈ B, for a subset B ∈ B, the transition kernel Π θ (β (k), B), which converges to the invariant distribution f θ (β), satisfies that DISPLAYFORM1. The stochastic approximation algorithm is an iterative recursive algorithm consisting of two steps: DISPLAYFORM2 is not equal to 0 but decays to 0 as k → ∞.To summarise, H(θ, β) is a biased estimator of h(θ) in finite steps, but as k → ∞, the bias decreases to 0. In the SG-MCMC-SA algorithm FORMULA1, DISPLAYFORM3 B CONVERGENCE ANALYSIS The stochastic gradient Langevin Dynamics with a stochastic approximation adaptation (SGLD-SA) is a mixed half-optimization-half-sampling algorithm to handle complex Bayesian posterior with latent variables, e.g. the conjugate spike-slab hierarchical prior formulation. Each iteration of the algorithm consists of the following steps: Sample β (k+1) using SGLD based on the current θ (k), i.e. DISPLAYFORM0 Optimize θ (k+1) from the following recursion DISPLAYFORM1 where g θ (k) (·) is some mapping to derive the optimal θ based on the current β. DISPLAYFORM2. In this formulation, our target is to find θ * such that h(θ DISPLAYFORM3 as k → ∞, this algorithm falls to the category of the general stochastic approximation. To provide the local L 2 upper bound for SGLD-SA, we first lay out the following assumptions:Assumption 1 (Step size and Convexity). {ω (k) } k∈N is a positive decreasing sequence of real numbers such that DISPLAYFORM0 There exist constant δ > 0 and θ DISPLAYFORM1 with additionally DISPLAYFORM2 Then for any α ∈ and suitable A and B, a practical ω (k) can be set as DISPLAYFORM3 Assumption 2 (Existence of Markov transition kernel). For any θ ∈ Θ, in the mini-batch sampling, there exists a family of noisy Kolmogorov operators {Π θ} approximating the operator (infinitesimal) of the Ito diffusion, such that every Kolmogorov operator Π θ corresponds to a single stationary distribution f θ, and for any Borel subset A of β, we have DISPLAYFORM4 Assumption 3 (Compactness). For any compact subset Ω of Θ, we only consider θ ∈ Ω such that DISPLAYFORM5 Note that the compactness assumption of the latent variable θ is not essential, the assumption that the variable is in the compact domain is not only reasonable, but also simplifies our proof. In addition, there exists constants C 1 (Ω) and C 2 (Ω) so that DISPLAYFORM6 Assumption 4 (Solution of Poisson equation). For all θ ∈ Θ, there exists a function µ θ on β that solves the Poisson equation DISPLAYFORM7 For any compact subset Ω of Θ, there exist constants C 3 (β, Ω) and C 4 (β, Ω) such that for all θ, θ ∈ Ω, DISPLAYFORM8 Remark: For notation simplicity, we write C 1 (Ω) as C 1, C 2 (Ω) as C 2,... in the following context. Lemma 1 is a restatement of Lemma 25 (page 447) from BID0. Lemma 1. Suppose k 0 is an integer which satisfies with DISPLAYFORM9 Then for any k > k 0, the sequence {Λ K k} k=k0,...,K defined below is increasing and bounded by 2ω DISPLAYFORM10 Lemma 2 is an extension of Lemma 23 (page 245) from BID0 Lemma 2. There exist λ 0 and k 0 such that for all λ ≥ λ 0 and k ≥ k 0, the sequence u DISPLAYFORM11 From FORMULA2, we can denote a positive constant ∆ + as lim k→∞ inf 2δω DISPLAYFORM12. Then can be simplified as DISPLAYFORM13 There exist λ 0 and k 0 such that for all λ > λ 0 and k > k 0, holds. Note that in practical case when C 1 is small, finding a suitable λ 0 will not be a problem. Theorem 1 (L 2 convergence rate). Suppose that Assumptions 1-4 hold, for any compact subset Ω ∈ Θ, the algorithm satisfies: there exists a constant λ such that DISPLAYFORM14 where t(Ω) = inf{k : DISPLAYFORM15 Proof. Denote T (k) = θ (k) − θ *, with the help of FORMULA2 and Poisson equation FORMULA2, we deduce that DISPLAYFORM16 First of all, according to FORMULA1 and FORMULA2, we have DISPLAYFORM17 DISPLAYFORM18 Conduct the decomposition of D3 similar to Theorem 24 (p.g. 246) from BID0 and Lemma A.5 . DISPLAYFORM19 (ii) From FORMULA3 and FORMULA3 respectively, we deduce that DISPLAYFORM20 Thus there exists C 4 = 2C 4 C 0 such that DISPLAYFORM21 (iii) D3-3 can be further decomposed to D3-3a and D3-3b DISPLAYFORM22 ) with a constant C 3 = 4C 0 C 3 which satisfies that DISPLAYFORM23 Finally, add all the items D1, D2 and D3 together, for some C 1 = C 3 + C 4, we have DISPLAYFORM24 Moreover, from FORMULA3 and FORMULA3, there exists a constant C 5 such that DISPLAYFORM25 Lemma 3 is an extension of Lemma 26 (page 248) from BID0.Lemma 3. Let {u (k) } k≥k0 as a sequence of real numbers such that for all k ≥ k 0, some suitable constants C 1 and C 2 DISPLAYFORM26 and assume there exists such k 0 that DISPLAYFORM27 Then for all k > k 0, we have DISPLAYFORM28 Proof of Theorem 1 (Continued). From Lemma 2, we can choose λ 0 and k 0 which satisfy the conditions and DISPLAYFORM29 From Lemma 3, it follows that for all k > k 0 DISPLAYFORM30 From and the increasing property of Λ k j in Lemma 1, we have DISPLAYFORM31 Therefore, given the sequence u (k) = λ 0 ω (k) that satisfies conditions FORMULA3, FORMULA5 and Lemma 3, for any k > k 0, from FORMULA1 and FORMULA2, we have DISPLAYFORM32 where λ = λ 0 + 6C 5. In addition to the previous assumptions, we make one more assumption on the stochastic gradients to guarantee that the samples converge to the posterior conditioned on the optimal latent variables: Assumption 5 (Gradient Unbiasedness and Smoothness). For all β ∈ B and θ ∈ Θ, the mini-batch of data B, the stochastic noise ξ, which comes from∇L(β, θ) − ∇L(β, θ), is a white noise and independent with each other. In addition, there exists a constant l ≥ 2 such that the following conditions hold: DISPLAYFORM0 For all θ, θ ∈ Θ, there exists a constant M > 0 such that the gradient is M-smooth: DISPLAYFORM1 Corollary 1. For all α ∈, under assumptions 1-5, the distribution of β (k) converges weakly to the invariant distribution e L(β,θ *) as → 0.Proof. The proof framework follows from section 4 of. In the context of stochastic noise ξ (k), we ignore the subscript of and only consider the case of τ = 1. Since θ DISPLAYFORM2 converges to θ * in SGLD-SA and the gradient is M-smooth FORMULA5, we transform the stochastic gradient from ∇L(DISPLAYFORM3, therefore Eq. FORMULA2 can be written as DISPLAYFORM4 Using Eq. FORMULA3, the characteristic function of DISPLAYFORM5 Then the characteristic function of DISPLAYFORM6 Rewrite β (k+1) as β (k+), the characteristic function of β (t+) is the characteristic function of DISPLAYFORM7 With the fact exp(x) = 1 + x + O(x 2), we can get DISPLAYFORM8 Therefore, For any integrable function f, set F as the Fourier transform defined by DISPLAYFORM9 DISPLAYFORM10 The inverse Fourier transform of F[f (x)] and the l-th order derivatives of f (l) (x) is DISPLAYFORM11 Combine Eq. FORMULA5, Eq. FORMULA6 and Eq. FORMULA1, we arrive at the following simplified equation: DISPLAYFORM12 Since DISPLAYFORM13 Finally, we have proved that the distribution of β (k) converges weakly to the invariant distribution e L(β,θ *) as → 0. Now we conduct the experiments on binary logistic regression. The setup is similar as before, except n is set to 500, Σ i,j = 0.3 |i−j| and η ∼ N (0, I/2). We set the learning rates in SGLD-SA and SGLD to 0.01 × k FIG11, FIG11 and FIG11 demonstrate the posterior distribution of SGLD-SA is significantly better than that of SGLD. As shown in FIG11, SGLD-SA is the best method to regulate the over-fitting space and provides the most reasonable posterior mean. Table. 3 illustrates the predictive power of SGLD-SA is overall better than the other methods and robust to different initializations. FIG11 and FIG11 show that the over-fitting problem of SGLD when p > n in logistic regression and the algorithm fails to regulate the over-fitting space; We observe SGLD-SA is able to resist over-fitting and always yields reproducible . q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q The first DNN we use is a standard 2-Conv-2-FC CNN: it has two convolutional layers with a 2 × 2 max pooling after each layer and two fully-connected layers. The filter size in the convolutional layers is 5 × 5 and the feature maps are set to be 32 and 64, respectively BID13 ). The fully-connected layers (FC) have 200 hidden nodes and 10 outputs. We use the rectified linear unit (ReLU) as activation function between layers and employ a cross-entropy loss. The second DNN is a 2-Conv-BN-3-FC CNN: it has two convolutional layers with a 2 × 2 max pooling after each layer and three fully-connected layers with batch normalization applied to the first FC layer. The filter size in the convolutional layers is 4 × 4 and the feature maps are both set to 64. We use 256 × 64 × 10 fully-connected layers. In practice, we observe a suitable temperature setting is helpful to improve the classification accuracy. For example, by setting τ = 100 in the second DNN (see Appendix D.1) we obtain 99.70% on aMNIST. To account for the scale difference of weights in different layers, we apply different temperatures to different layers based on different standard deviations of the gradients in each layer and obtain the in Tab. 2. The MNIST dataset is augmented by randomCrop: randomly crop each image with size 28 and padding 4, random rotation: randomly rotate each image by a degree in [−15•, +15•], normalization: normalize each image with empirical mean 0.1307 and standard deviation 0.3081.The FMNIST dataset is augmented by randomCrop: same as MNIST, randomHorizontalFlip: randomly flip each image horizontally, normalization: same as MNIST., a positive would be a robust Bayesian deep learning algorithm to infer complex posteriors with latent variables & a negative would be This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.

Question: While real brain networks exhibit functional modularity, we investigate whether functional mod- ularity also exists in Deep Neural Networks (DNN) trained through back-propagation. Under the hypothesis that DNN are also organized in task-specific modules, in this paper we seek to dissect a hidden layer into disjoint groups of task-specific hidden neurons with the help of relatively well- studied neuron attribution methods. By saying task-specific, we mean the hidden neurons in the same group are functionally related for predicting a set of similar data samples, i.e. samples with similar feature patterns. We argue that such groups of neurons which we call Functional Modules can serve as the basic functional unit in DNN. We propose a preliminary method to identify Functional Modules via bi- clustering attribution scores of hidden neurons. We find that first, unsurprisingly, the functional neurons are highly sparse, i.e., only a small sub- set of neurons are important for predicting a small subset of data samples and, while we do not use any label supervision, samples corresponding to the same group (bicluster) show surprisingly coherent feature patterns. We also show that these Functional Modules perform a critical role in discriminating data samples through ablation experiment. While real brain networks exhibit functional modularity, we investigate whether functional modularity also exists in Deep Neural Networks (DNN) trained through back-propagation. Under the hypothesis that DNN are also organized in task-specific modules, in this paper we seek to dissect a hidden layer into disjoint groups of task-specific hidden neurons with the help of relatively wellstudied neuron attribution methods. By saying task-specific, we mean the hidden neurons in the same group are functionally related for predicting a set of similar data samples, i.e. samples with similar feature patterns. We argue that such groups of neurons which we call Functional Modules can serve as the basic functional unit in DNN. We propose a preliminary method to identify Functional Modules via biclustering attribution scores of hidden neurons. We find that first, unsurprisingly, the functional neurons are highly sparse, i.e., only a small subset of neurons are important for predicting a small subset of data samples and, while we do not use any label supervision, samples corresponding to the same group (bicluster) show surprisingly coherent feature patterns. We also show that these Functional Modules perform a critical role in discriminating data samples through ablation experiment. Also, these modules learn rich representations and are able to detect certain feature patterns demonstrated in a visual classification example. Modularity is generally encountered across a broad range of networks, including real brain neuronal networks, which means that the entire population of neurons can be parcellated into internally dense and externally sparse groups called modules or communities. And since that, researchers naturally think artificial neural networks also exhibits modularity, for example, Hinton et al. put forward neuron co-adaption that some hidden neurons in the same layer co-adapt together as a module for prediction. Co-adaption is only discussed in thought experiments supported by some biological inspirations and we do not know how to identify such co-adaptive neurons. However this phenomenon itself inspires Dropout [3; 7] which is arguably the most robust regularization technique for DNN. Identifying modularity in DNN remains difficult as community detection methods are generally not directly applicable on densely-connected acyclic graphs. We turn to the relatively well-studied neuron attribution methods and biclustering algorithm for help. Biclustering. is a data mining technique that simultaneously clusters the rows and columns of a matrix, and is especially popular in bioinformatics. We favor biclustering instead the standard clustering methods, which only cluster the rows or the columns of a matrix, for two reasons: 1) Practically, standard clustering methods can easily fail because of the curse of dimensionality. 2) From the hypothesis we really do not expect two neurons in the same neuron group to be similar for every stimulus (input data sample). We only expect neurons in the same group are functionally related for predicting a subset of similar data samples. Hopefully, this subset of samples should share similar feature patterns. That being said, there are many biclustering algorithms for different purposes as it is a very active research field. We choose to use spectral co-clustering because it produces biclusters of strong connections with no overlaps, which is the simplest case for us to start with. Attribution methods for hidden neurons. Neuron attribution, assigning a importance score to a neuron, is easier to do for artificial neural networks than real neural nets. Basically, the goal of neuron attribution is to assign a score a to a hidden neuron n that represents how much important this neuron is for predicting a sample x to a class y. In a sense, the attribution score measures the per-sample importance of a hidden neuron. While for real neurons this can be computed as, for example, Pearson correlation to some task, for DNN we can utilize the internal weights and the feedforward structure of DNN to compute attribution scores. There is a variety of neuron attribution methods. To name a few, Shrikumar et al. proposed DeepLIFT, originally designed to assign scores to input nodes and can be generalized to assign a score of importance to a specific hidden neuron. Sundararajan et al. proposed integrated gradients for attributing input neurons and later generalized it to total conductance [6; 1] for attribution hidden neurons. Leino et al. proposed an influence-based attribution method. Such methods have been demonstrated to be capable of identifying important hidden neurons that are relevant to a specific prediction on a class y given a data sample x. However, these methods do not consider that hidden neurons may be functionally related. At a high-level, our approach for finding Functional Modules in hidden neurons goes in two phases: 1) we first construct a neuron-sample matrix where each entry is a attribution score of a hidden neuron (row) for a input sample (column) and then 2) based on this matrix, we apply spectral coclustering, a biclustering algorithm to simultaneously group neurons and samples that have consistent high attribution values. Given a dataset X = {x} N of data samples, a pre-trained DNN f parameterized by θ such that y = f θ (x) is the prediction of sample x, and an attribution function a which can be any of [1; 4; 5; 6], the attribution score a(n, x, y) measures how much a hidden neuron n contributes to the prediction of sample x into class y. Construct the neuron-sample attribution matrix. We first specify the neurons to be from a given layer l and construct the neuron-sample matrix M N l * N data where N = |l| is the number of neurons in that layer and N data = |X| be the size of dataset X = x 1, x 2,..., x N. Each entry in the matrix represents e ij = e(n i, x j), where e is a embedding function measuring the contribution of neuron n i for predicting data sample x j. We compute e(n i, x j) = e ij = a(n i, x j,ŷ) − 1 |C| c a(n i, x j, c) whereŷ = f θ (x j) is the predicted class of x j and a can be any of the attritbution function among [1; 4; 5; 6]. We choose to use DeepLIFT score, and in experiments we find these attribution methods do not differ much. Spectral Co-clustering. Given the constructed matrix, we use spectral co-clustering to find biclusters with values higher than those in the corresponding other rows and columns. Given a predefined number k of biclusters, spectral coclustering algorithm treats the input data matrix as a bipartite graph and approximates the normalized cut of this graph to find heavy subgraphs. In the ed biclusters, each row (neuron) and each column (sample) belongs to exactly one bicluster with no overlaps. We first inspect DNNs for visual classification on MNIST. We fetch a pre-trained model that is used and evaluated in the DeepLIFT paper. The architecture is a 4 layer feedforward rectified model where the first two layers are convolutional layer followed by two fully-connected layer. (We also test on other DNN models with rectified neurons trained such as multilayer perceptron which has no convolutional layer, and the are very similar.) We use 10000 data samples from the test set and choose l to be the second convolutional layer of shape. The total number of hidden neurons in l is 1600. The choice of layer is rather arbitray, we choose it to be the second convolutional layer because we want the rows and columns to be roughly of the same magnitude, so that the neuron-sample matrix would have a good visualization, and that the biclustering algorithm can work properly. We compute the DeepLIFT score and construct the neuron-sample matrix. We set k = 10 in spectral co-clustering as is the number of classes for MNIST. Different attribution methods do not differ much in the qualitive observations, so we only present the comparision of different attribution methods on the ablation study in Figure 5b. The neuron-sample matrix is highly sparse. The constructed neuron-sample matrix is very sparse, as only a few entries have attribution score above zero (see Figure 1a). Rearranging the display of the matrix by bicluster label, we can observe a checkerboard pattern in Figure 1b, indicating that certain groups of hidden neurons are indeed highly-correlated for predicting certain data samples. (a) raw neuron-sample matrix (b) neuron-sample matrix rearranged with bicluster labels Figure 1: Visualization of the neuron-sample matrix (Better see in digital version) Each row corresponds to one of the 1600 neurons in the second convolutional layer (5 * 5 * 64). Each column represents one of 10000 data samples. We further report the distribution of attribution scores in Figures 2a and 2b, where the x-axis represents the attribute score and the y-axis represents the frequency. The overal distribution is centered around zero but inside one bicluster, the mass is concentrated on the positive side. This indicates that the spectral coclustering algorithm does find good subgraphs that a subset of hidden neurons typically have higher attribution scores on a subset of samples. Samples corresponding to the same bicluster show coherent feature patterns. We find that the samples in one bicluster, i.e., samples corresponding to the same Functional Module, show coherent and similar feature patterns. We pick a bicluster and randomly gather samples from that bicluster and find these samples show very similar patterns (Figures 3a and 3b) and belong to the same class. We also show the distribution of true labels in the total k = 10 biclusters (Section 4). Note that despite Ground-truth label distribution in biclusters not using any label supervision in our approach, we can achieve a very good job of unsupervised discriminating. Spatial relationship of the neurons in a module: Is this modularity a of convolution?. Surprisingly, the neurons in a bicluster do not seem to have any spatial relationships, such as being in the same channel or centered in some position. We also test on multilayer perceptrons where there is no convolutional layers and the above phenomenon can still be observed. Ablation Study: Functional Modules are critical for discriminating samples. We check the performance of DNN by removing those hidden neurons by bicluster. In this experiment, we split the 10000 images X further into two sets where the first 5000 samples as validation set are processed by our approach and the rest 5000 samples are held-out for testing. We compare the accuracy when hidden neurons in layer l are gradually ablated on the test set with several baselines for ablating neurons, such as ablation of random neurons, ablation of top-important neurons greedily, and ablation of neurons by module (bicluster) (ours). Ablation of a neuron is implemented by setting the activation value of this neuron to zero, as convention in previous studies [1; 6]. Note that we set k larger in order to get a smoother curve. As shown in Figure 5a, ablation by module achieves the most significant accuracy drop compared with greedy ablation of top-k neurons (all use DeepLIFT score) and random ablation. It demonstrates that these Functional Module s found by spectral coclustering do play an important role in discriminating data samples. We also compare the effect of abaltion with different attribution methods in Figure 5b and observe no significant difference among different choices of attributions methods. We develop an approach to parcellate a hidden layer into functionally related groups which we call Functional Modules, by applying spectral coclustering on the attribution scores of hidden neurons. We find the Functional Modules identifies functionally-related neurons in a layer and play an important role in discriminating data samples. One major limitation of this short paper is that we have not tested on more general cases, such as different layers, different activation function, different models trained on more diverse datasets, etc. In order to gain generalizable insights, such a massive investigation is neccessary.

represent the tldr for finding the corresponding scientific article

We develop an approach to parcellate a hidden layer in DNN into functionally related groups, by applying spectral coclustering on the attribution scores of hidden neurons.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: Power-efficient CNN Domain Specific Accelerator (CNN-DSA) chips are currently available for wide use in mobile devices. These chips are mainly used in computer vision applications. However, the recent work of Super Characters method for text classification and sentiment analysis tasks using two-dimensional CNN models has also achieved state-of-the-art through the method of transfer learning from vision to text. In this paper, we implemented the text classification and sentiment analysis applications on mobile devices using CNN-DSA chips. Compact network representations using one-bit and three-bits precision for coefficients and five-bits for activations are used in the CNN-DSA chip with power consumption less than 300mW. For edge devices under memory and compute constraints, the network is further compressed by approximating the external Fully Connected (FC) layers within the CNN-DSA chip. At the workshop, we have two system demonstrations for NLP tasks. The first demo classifies the input English Wikipedia sentence into one of the 14 classes. The second demo classifies the Chinese online-shopping review into positive or negative. Power-efficient CNN Domain Specific Accelerator (CNN-DSA) chips are currently available for wide use. Sun et al. BID5;a) designed a two-dimensional CNN-DSA accelerator which achieved a power consumption of less than 300mW and an ultra power-efficiency of 9.3TOPS/Watt. All the processing is in internal memory instead of external DRAM. Demos on mobile and embedded systems show its applications in real-world implemen-Preliminary work. Under review by the International Conference on Machine Learning (ICML). Do not distribute. Figure 1. Efficient On-Device Natural Language Processing system demonstration. The CNN-DSA chip is connected to Raspberry Pi through the USB interface. Keyboard sends the typing text input to Raspberry Pi through USB. A monitor is connected to Raspberry Pi through HDMI for display. On the monitor, it shows the introduction for the demo (zoom in to see details). There are two demos. The first demo classifies the input English Wikipedia sentence into one of the 14 ontologies. The second demo classifies the Chinese online-shopping review into positive or negative.tations. The 28nm CNN-DSA accelerator attains a 140fps for 224x224 RGB image inputs at an accuracy comparable to that of the VGG BID2.For Natural Language Processing tasks, RNN and LSTM models BID8 BID1 are widely used, which are different network architectures from the twodimensional CNN. However, the recent work of Super Characters method BID4 BID7 using twodimensional word embedding achieved state-of-the-art in text classification and sentiment analysis tasks, showcasing the promise of this new approach. The Super Characters method is a two-step method. In the first step, the characters of the input text are drawn onto a blank image, so that an image of the text is generated with each of its characters embedded by the pixel values in the two-dimensional space. The ing image is called the Super Characters image. In the second step, the generated Super Characters image is fed into a twodimensional CNN models for classification. The two- dimensional CNN models are trained for the text classification task through the method of Transfer Learning, which finetunes the pretrained models on large image dataset, e.g. ImageNet BID0, with the labeled Super Characters images for the text classsification task. In this paper, we implemented NLP applications on mobile devices using the Super Characters method on a CNN-DSA chip as shown in Figure 1. It takes arbitrary text input from keyboard connecting to a mobile device (e.g. Raspberry Pi). And then the text is pre-processed into a Super Characters image and sent to the CNN-DSA chip to classify. After post-processing at the mobile device, the final will be displayed on the monitor. As shown in FIG0, the keyboard text input is preprocessed by the Raspberry Pi (or other mobile/embedded devices) to convert into a Super Characters image. This pre-processing is only a memory-write operation, which requires negligible computation and memory resources. The Super Characters method works well for Asian languages which has characters in squared shapes, such as Chinese, Japanese, and Korean. These glyphs are easier for CNN models to recognize than Latin languages such as English, which is alphabets-based in a rectangular shape and may have to break the words at line-changing. To improve the performance for English, a method of Squared English Word (SEW) is proposed to cast English word in a squared shape as a glyph BID6. FIG1 shows an example of this method. Basically, each word takes the same size of a square space lxl. Words with longer alphabets will have smaller space for each alphabet. Within the lxl space, the word with N alphabets will have each of its alpha in the square area of {l/ceil[sqrt(N)]} 2, where sqrt stands for square root, and ceil[.] is rounding to the top. The CNN-DSA chip receives the Super Characters im- The two-dimensional embedding in this image corresponds to the text of "The tiger is the largest species among the Felidae and classified in the genus Panthera. It is most recognizable for its dark vertical stripes on reddish-orange fur with a lighter underside." age through the USB connection to the mobile device. It outputs the classification scores for the 14 classes in the Wikipedia text classification demo. The classification scores mean the probabilities for classification but before softmax. The mobile device only calculate the argmax to display final classification on the monitor, which is also negligible computations. The CNN-DSA chip completes the complex CNN computations with low power less than 300mw. The CNN-DSA chip is a fast and low-power coprocessor. However, it does not directly support inner-product operations of the FC layers. It only supports 3x3 convolution, Relu, and max pooling. If the FC layers are executed on the mobile device, there will be increased requirements for memory, computation, and storage for the FC coefficients. And it will also spend more interface time with the CNN-DSA chip for transmitting the activation map from the chip, and also cost relative high power consumption for the mobile device to execute the inner-product operations. In order to address this problem, we proposed the GnetFC model, which approximates the FC layers using multiple layers of 3x3 convolutions. This is done by adding a sixth major layer with three sub-layers as shown in Figure 4. The Figure 4. Model architecture. The input is of size 224x224 with multiple channels, and the output is of size 14x1x1. The architecture within blue dashed square is the model loaded into the CNN-DSA chip.model is similar to VGG architecture except that it has six major layers, and the channels in the fifth major layer is reduced to 256 from the original 512 in order to save memory for the sixth layer due to the limitation of the on-chip memory. The sub-layers in each major layer has the same color. Each sub-layer name is followed by the detailed information in brackets, indicating the number of channels, bits-precision, and padding. The first five major layers has zero paddings at the image edge by one-pixel. But the sixth major layer has no padding for the three sublayers, which reduces the activation map from 7x7 through 5x5 and 3x3 and finally to 1x1. The output is of size 14x1x1, which is equal to an array of 14 scalars. The final classification can be simply obtained by an argmax operation on the 14 scalars. This reduces the system memory footprint on the mobile device and accelerate the inference speed. The memory of the CNN-DSA chip is built within the accelerator, so it is very power-efficient without wasting the energy for moving the bits from external DDR into internal SRAM. Thus the on-chip memory is very limited, which supports maximum 9MB for coefficients and activation map. As shown in Figure 4, the first two major layers uses 3-bits precision and the other four major layers uses 1-bit precision. All activations are presented by 5-bits in order to save on-chip data memory. The representation mechanism inside the accelerator supports up to four times compression with the 1-bit precision, and two times compression with the 3-bits precision. Due to the high compression rate, the convolutional layers in VGG16 with 58.9MB coefficients in floating precision could be compressed into only about 5.5MB within the chip. This is a more than 10x compression of the convolution layers. This compact representation has been proved to be successful on ImageNet BID0 ) standard training and testing data and achieved the same level of accuracy as floating point models with 71% Top1 accuracy. The compact CNN representation without accuracy loss is because of the redundancy in the original network. To efficiently use the on-chip memory, the model coefficients from the third major layers are only using 1-bit precision. For the first two major layers, 3-bits model coefficients are used as fine-grained filters from the original input image. And the cost on memory is only a quarter for the first major layer and a half for the second major layer if using the same 3-bits precision. The total model size is 2.8MB, which is more than 200x compression from the original VGG model with FC layers. It completes all the convolution and FC processing within the CNN-DSA chip for the classification task with little accuracy drop. The GnetFC model on the CNN-DSA chip on the Wikipedia demo obtains an accuracy of 97.4%, while the number for the original VGG model is 97.6%. The accuracy drop is mainly brought by the approximation in GnetFC model, and also partially because of the bit-precision compression. The accuracy drop is very little, but the savings on power consumption and increasing on the inference speed is significant. It consumes less than 300mw on the CNN-DSA chip, and the power for pre/postprocessing is negligible. The CNN-DSA chip processing time is 15ms, and the pre-processing time on mobile device is about 6ms. The time for post-processing is negligible, so the total text classification time is 21ms. It can process nearly 50 sentences in one second, which satisfies more than real-time requirement for NLP applications. We implemented efficient on-device NLP applications on a 300mw CNN-DSA chip by employing the twodimensional embedding used in the Super Characters method. The two-dimensional embedding converts text into images, which is ready to be fed into CNN-DSA chip for two-dimensional CNN computation. The demonstration system minimizes the power consumption of deep neural networks for text classification, with less than 0.2% accuracy drop from the original VGG model. The potential use cases for this demo system could be the intension recognition in a local-processing smart speaker or Chatbot.

Deploy text classification and sentiment analysis applications for English and Chinese on a 300mW CNN accelerator chip for on-device application scenarios.

Represent the TLDR for finding the corresponding scientific article

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.