Task Agnostic Continual Learning Using Online Variational Bayes
Chen Zeno Itay Golan Elad Hoffer Daniel Soudry


[
Task Agnostic Continual Learning Using Online Variational Bayes














equal*


Chen Zenoequal,technion
Itay Golanequal,technion
Elad Hoffertechnion
Daniel Soudrytechnion


technionTechnion - Israel Institute of Technology, Haifa, Israel

Daniel Soudrydaniel.soudry@gmail.com




Machine Learning, ICML

0.3in
]










 


Catastrophic forgetting is the notorious vulnerability of neural networks to the change of the data distribution while learning. This phenomenon has long been considered a major obstacle for allowing the use of learning agents in realistic continual learning settings. A large body of continual learning research assumes that task boundaries are known during training. However, research for scenarios in which task boundaries are unknown during training has been lacking. In this paper we present, for the first time, a method for preventing catastrophic forgetting (BGD) for scenarios with task boundaries that are unknown during training — task-agnostic continual learning. Code of our algorithm is available at https://github.com/igolan/bgdhttps://github.com/igolan/bgd.




§ INTRODUCTION





A continual learning algorithm is one that is faced with sequentially-arriving tasks, with no access to samples from previous or future tasks. Special measures are needed to prevent a deep neural network (DNN) from adapting only to the latest task and forgetting past knowledge — a phenomenon known as catastrophic forgetting <cit.>.
In this paper we present a new continual learning algorithm for training DNNs, Bayesian Gradient Descent (BGD).
Unlike previous algorithms, our algorithm can be applied to scenarios with task boundaries that are unknown or undefined during both training and testing.


Various continual learning methods have been suggested in the literature. 
Due to the lack of concrete scenario definitions, many works present an inconsistent mixture of experimental results, without an explicit definition of the tested scenarios.
Recently, the authors of <cit.> identified three distinct commonly-used experimental scenarios.
They differ in the knowledge that they assume during testing, in the number of  outputs (“heads”, i.e. output dimension), and in difficulty level.
All these scenarios share a common assumption: task boundaries are known to the algorithm during training.
This is a significant disadvantage, as in many realistic applications we may not possess the knowledge about the tasks schedule — i.e., a “task-agnostic continual learning" setting.

To allow a concrete comparison between different methods, in section <ref> we provide four questions for continual learning scenario definition, expanding that of <cit.>. It yields two new scenarios where the task schedule is unknown, which were previously unaddressed.
In section <ref>, we show empirically that all scenarios are fundamentally different, and present first results on scenarios with no knowledge on tasks schedule.


The motivation for well-defined scenarios can also be found in <cit.>.
The authors explain why some scenarios are more interesting than others, and provide a list of desiderata for a continual learning algorithm. Our algorithm, BGD, fits most of those desiderata.
They also urge the community to focus on a specific scenario called “class learning”. 
In section <ref> we analyze this scenario, and present a method (“labels trick”) to improve the performance of “class learning”. For example, on Split MNIST it improves average accuracy from ∼20% to ∼50% for all algorithms.
This is done by understanding the information available in such scenario.



  
Main contributions
Our main contributions are:



    
  * We present the first algorithm for continual learning (BGD) which is applicable to scenarios where task identity or boundaries are unknown during both training and testing — task-agnostic continual learning.
    
  * We suggest a fine-grain categorization for continual learning scenarios to allow well-defined comparison.
    
  * We present the “labels trick”, a way to improve performance in the “class learning” scenario.






§ CONTINUAL LEARNING SCENARIOS






In recent works, <cit.> defined three continual learning scenarios: “task learning”, “domain learning” and ”class learning”.
We review these scenarios, denoting T as the number of tasks, and C as the number of classes per task.[We assume all tasks have the same number of classes for simplicity, but this assumption may be omitted.]
The term “head” is used for the DNN output (the last linear classifier).

Knowledge about task identity means that a sample x_i is tagged with task number t_i, alongside the label y_i: (x_i, y_i, t_i).



  
Previous scenario definitions
In task, domain, and class learning, task boundaries are well-defined, and task identity is known during training.
The difference arises from knowledge on task identity during testing, and the number of heads.
In task learning, the task identity is known during testing. The best results are achieved by assigning a different set of heads for each task (T× C outputs), and using only the C heads of the current task during training and testing.
In both domain and class learning, the task identity is unknown during testing. Therefore, we cannot easily choose the correct set of heads on testing.
In domain learning, a single set of heads is shared among all tasks (C outputs). While in class learning a different set of heads represents each task classes (T× C outputs).
Specifically, in class learning, during testing one must use all T× C heads, which includes heads of other tasks, therefore the network needs to infer not only the correct label, but also the correct task.



  
Expanding scenario definitions
We expand those definitions, and suggest a fine-grain categorization of continual learning scenarios. 
To characterize a scenario, one should answer four questions:



     
    
  * Are the boundaries between tasks well defined?
    
  * Is the task identity known during training?
    
  * Is the task identity known during inference (test)?
    
  * Are tasks share the output head? (C vs. T× C outputs).



Additional questions that affect performance are:



    
     
    
  * For how long each task is trained?[When task boundaries are undefined we can not define an epoch, as the notion of “transition over the whole data set” is undefined.]
    
  * Is the number of tasks known? How many tasks exist?
    
  * General questions: network architecture, optimizer etc.



The tasks boundaries can be undefined in two different ways: (A) The transition between tasks occurs slowly over time, so the algorithm gets a mixture of samples from two different tasks during the transition (Figure <ref>); (B) The data itself changes continuously towards a new distribution (for example, images of cheetahs that slowly change into cats).
In all task-agnostic scenarios, the algorithm does not have any kind of knowledge on the distribution over the tasks.
Figure <ref> summarizes scenario characterization as a decision tree.





  
Task agnostic continual learning
Current methods for continual learning require to be informed when a task is switched during training in order to take some core action (e.g., changing parameters in the loss function).
When this knowledge is unavailable, such as in task-agnostic scenarios, those algorithms are inapplicable (see discussion in subsection <ref>).
Unlike previous approaches, BGD can be applied without any modification to scenarios where the task boundaries are unknown or undefined at all.



  
Labels trick
In class learning, the output heads are not shared, so different tasks do not share the same label space. In such case, we can infer the task identity during training even if it is not provided explicitly, solely by inspecting the labels of the current batch (as different tasks will have different labels due to the non-shared heads).
The basic idea is:




    Heads are not shared ⇒ Task identity known in training.




If task identity is known during training, we can train only the heads of the current task.
Using this notion, we suggest the “labels trick” for class learning.
The trick is training only the heads which correspond to labels that exist in the current batch (i.e. heads of the current task), in contrast to the common practice of training all heads in such scenario.
For example, if a batch consists of samples only with labels (3,4), we calculate the loss only over the heads corresponding to (3,4), instead of over all heads (0,1,2,3,4...T× C-1).
This way, we are able to train the network as in “task learning”. The error stems only from wrong task prediction during testing, since task identity is unavailable during testing.
This trick improves class learning results from ∼20% to ∼50% (see subsection <ref>). Note that the scenario setup is kept the same, and we do not use any new information — the labels were available during training also in previous papers results.



§ RELATED WORK
 



  
Approaches for continual learning
Approaches for continual learning can be generally divided into four main categories: (1) Architectural approach; (2) Rehearsal approach; (3) Regularization approach; (4) Bayesian approach. 

Architectural approaches alter the network architecture to adapt to new tasks (e.g. <cit.>).
Rehearsal approaches use memory[Rehearsal approaches include GANs which can be used as a sort of memory.] to allow re-training on examples from previous tasks (e.g. <cit.>).
Regularization approaches use some penalty on deviations from previous task weights.
Bayesian approaches use Bayes' rule with previous task posterior as the current prior.
Each approach has pros and cons. For example, architectural approaches may result in extremely big architecture for a large number of tasks, while rehearsal approaches may be problematic in terms of privacy.
For an extensive review on continual learning methods see <cit.>.
This paper focuses on regularization and Bayesian approaches – the architecture is fixed, and no memory is used to retrain on previous tasks.

In regularization approaches, a regularization term is added to the loss function. “Elastic weight consolidation” (EWC) proposed by <cit.>, slows the conversion of parameters important to the previous tasks by adding a quadratic penalty on the difference between the optimal parameters of the previous task and the current parameters. The importance of each parameter is measured using the diagonal of the Fisher information matrix of the previous tasks.
“Synaptic Intelligence” (SI) proposed by <cit.> also uses a quadratic penalty, 
however, the importance is measured by the path length of the updates on the previous task. <cit.> proposed a generalization of the online version of EWC and modified version of SI to achieve better performance. “Memory Aware Synapses” (MAS) proposed by <cit.> also uses a quadratic penalty, however, the importance is measured by the sensitivity of the output function.
“Learning without Forgetting” (LwF) proposed by <cit.> uses knowledge distillation so the network outputs of the new task enforced to be similar to the network of the previous tasks.


The Bayesian approach provides a solution to continual learning in the form of Bayes' rule. When data arrives sequentially, the posterior distribution of the parameters for the previous task is used as a prior for the new task. “Variational Continual Learning” (VCL) proposed by <cit.> used online variational inference combined with the standard variational Bayes approach developed by <cit.>, “Bayes By Backprop” (BBB) to reduce catastrophic forgetting by replacing the prior on a task switch. In the BBB approach, a “mean-field approximation” is applied, assuming weights are independent of each other (the covariance matrix is diagonal).

<cit.> suggest using Bayesian online learning with a Kronecker factored Laplace approximation to attain a non-diagonal method for reducing catastrophic forgetting, which allows the algorithm to take into account interactions between weights within the same layer.



  
Bayesian Neural Networks
In this work, we use the Bayesian approach for continual learning, specifically we use the online version of variational Bayes. Next we present a review on Bayesian neural networks and variational Bayes methods.
Bayesian inference for neural networks has been a subject of interest over many years. As exact Bayesian inference is intractable (for any realistic network size), much research has been focused on approximation techniques. Most modern techniques stemmed from previous seminal works which used either a Laplace approximation <cit.>, variational methods <cit.>, or Monte Carlo methods <cit.>. In the last years, many methods for approximating the posterior distribution have been suggested, falling into one of these categories. Those methods include assumed density filtering <cit.>, approximate power Expectation Propagation <cit.>, Stochastic Langevin Gradient Descent <cit.>, incremental moment matching <cit.> and variational Bayes <cit.>.

Practical variational Bayes for modern neural networks was first introduced by <cit.>, where parametric distribution is used to approximate the posterior distribution by minimizing the variational free energy. Calculating the variational free energy is intractable for general neural networks, and thus <cit.>
estimated its gradients using a biased Monte Carlo method, and used stochastic gradient descent (SGD)
to perform minimization. In a later work, <cit.> used a re-parameterization trick to introduce an unbiased estimator for the gradients. Variational Bayes methods were also used extensively on various probabilistic models including recurrent neural networks <cit.>, auto-encoder <cit.> and fully connected networks <cit.>. <cit.> suggested using the connection between natural gradient descent <cit.> and
variational inference to perform natural gradient optimization in deep neural networks.



  
Major differences The major differences between BGD and previous works are as follows:



    
  * Task agnostic scenarios: BGD (our method) is a task-agnostic algorithm — it  does not require any information on task identity. In contrast to previous methods (e.g. EWC, SI, VCL) which are based on some core action taken on task switch. 
    
  * Closed-form update rule: We update the posterior over the weights in closed-form, in contrast to BBB (and VCL which relies on BBB) which use SGD optimizer on the variational free energy.






§ THEORY


The process of Bayesian inference requires a full probability model
providing a joint probability distribution over the data and the model
parameters. The joint probability distribution can be written as a
product of two distributions:

    p(D,)=p(D|)p() ,

where p(D|) is the likelihood function of the data
set D, and p() is the prior distribution of
the parameters . The posterior distribution can be calculated
using Bayes' rule:

    p(|D)=p(D|)p()/p(D) ,

where p(D) is calculated using the sum rule. 

In this paper we will focus on the online version of Bayesian inference, in which the data arrives sequentially, and we do a sequential update of the posterior distribution each time that new data arrives. In each step, the previous posterior distribution is used as the new prior distribution. Therefore, according to Bayes' rule, the posterior distribution at time n is given by:

    p(|D_n)=p(D_n|)p(|D_n-1)/p(D_n) .

Unfortunately, calculating the posterior distribution is intractable
for most practical probability models;
therefore, we approximate the true posterior using variational methods.



 §.§ Online variational Bayes


In variational Bayes <cit.>, a parametric distribution q(|ϕ) 
is used for approximating the true posterior distribution p(|D)
by minimizing the Kullback-Leibler (KL) divergence with the true posterior
distribution.

    KL(q(|ϕ)||p(|D))=-𝔼_∼ q(|ϕ)[logp(|D)/q(|ϕ)]

The optimal variational parameters are the solution of the following optimization problem:

    ϕ^*    =min_ϕ∫ q(|ϕ)logq(|ϕ)/p(|D)d
        =min_ϕ∫ q(|ϕ)logq(|ϕ)/p(D|)p()d
        =min_ϕ𝔼_∼ q(|ϕ)[log(q(|ϕ)). 
    
        -log(p()) +L().],

where L()=-log(p(D|))
is the log-likelihood cost function.[Note that we define a cumulative log-likelihood cost function over the data.]

In online variational Bayes <cit.>, we aim to find the posterior in an online
setting, where the data arrives sequentially. Similar to Bayesian
inference we use the previous approximated posterior as the new prior
distribution. For example, at time n the optimal variational
parameters are the solution of the following optimization problem:

    ϕ^*    =min_ϕ∫ q_n(|ϕ)logq_n(|ϕ)/p(|D_n)d
        = min_ϕ∫ q_n(|ϕ)logq_n(|ϕ)/p(D_n|)q_n-1()d
        =min_ϕ𝔼_∼ q_n(|ϕ)[log(q_n(|ϕ)).  
     
       -log(q_n-1())+L_n().],

where L_n()=-log(p (D_n|))
is the log-likelihood cost function.



 §.§ Diagonal Gaussian approximation


A standard approach is to define the parametric distribution q(|ϕ) so that all the components of the parameter vector θ would be factorized, i.e. independent (“mean-field approximation”). In addition, in this paper we will focus on the case
in which the parametric distribution q(|ϕ) and
the prior distribution are Gaussian. Therefore:

    q_n(|ϕ)    =∏_i𝒩(θ_i|μ_i,σ_i^2) 
     q_n-1()   =∏_i𝒩(θ_i|m_i,v_i^2) .


In order to solve the optimization problem in (<ref>), we use
the unbiased Monte Carlo gradients, similarly to <cit.>. We define a deterministic transformation:

    θ_i   =μ_i+ε_iσ_i
    ε_i   ∼𝒩(0,1) 
    ϕ   =(μ,σ) .

The following holds:

    ∂/∂ϕ𝔼_[f(,ϕ)]=𝔼_ε[∂ f(,ϕ)/∂∂/∂ϕ+∂ f(,ϕ)/∂ϕ],

where f(,ϕ)=log(q_n(|ϕ))-log(q_n-1())+L().
Therefore, we can find a critical point of the objective function
by solving the following set of equations:

    𝔼_ε[∂ f(,ϕ)/∂∂/∂ϕ+∂ f(,ϕ)/∂ϕ]=0 .

Substituting (<ref>) and (<ref>)
into (<ref>) we obtain (see Appendix <ref> for additional
details):

    μ_i=m_i-v_i^2𝔼_ε[∂L_n()/∂θ_i] ,


    σ_i   =v_i√(1+(1/2v_i𝔼_ε[∂L_n()/∂θ_iε_i])^2)
       -1/2v_i^2𝔼_ε[∂L_n()/∂θ_iε_i] .

Note that (<ref>) and (<ref>) are implicit equations,
since the derivative ∂L_n()/∂θ_i
is a function of μ_i and σ_i.
We approximate the solution using a single explicit iteration of this equation, i.e. evaluate the derivative ∂L_n()/∂θ_i using the prior parameters.



  
BGD algoirthm
The approximation above results in an explicit closed-form update rule for μ and σ.
This approximation becomes more accurate when we are near the solution of (<ref>). If we are not near the solution, this can lead to a slow rate of convergence. To compensate for this we added a "learning rate" hyper-parameter η to adjust the convergence rate.  
Also, note that in the theoretical derivation we assumed that the data arrives sequentially, in an online setting. In practice, however, the log-likelihood cost function does not converge after one epoch since we use a single explicit iteration. Therefore, in Algorithm <ref> we repeatedly go over the training set until the convergence criterion is met.
In addition, the expectations are approximated using Monte Carlo sampling method (we use K Monte Carlo samples). Those Monte Carlo samples are the main difference from SGD in terms of computation complexity, see Appendix <ref> for complexity analysis.
The full algorithm is described in Algorithm <ref>.





 §.§ Theoretical properties of Bayesian Gradient Descent


Bayesian Gradient Descent (BGD) consists of a gradient descent algorithm
for μ, and a recursive update rule for σ, both result from an approximation to the online Bayes update in (<ref>), hence the name BGD. The learning rate of μ_i is proportional to the uncertainty in the parameter θ_i according to the prior distribution. During the learning process, as more data is seen, the learning rate decreases for parameters with a high degree of certainty, while the learning rate increases for parameters with a high degree of uncertainty. Next, we establish this intuitive idea more precisely.

It is easy to verify that the update rule for σ is a strictly monotonically decreasing
function of 𝔼_ε[∂L_n()/∂θ_iε_i].
Therefore:

    E_ε[∂ L()/∂θ_iε_i]     >0⟹σ_i(n)<σ_i(n-1)
    E_ε[∂ L()/∂θ_iε_i]     <0⟹σ_i(n)>σ_i(n-1)
    E_ε[∂ L()/∂θ_iε_i]     =0⟹σ_i(n)=σ_i(n-1)  .

Next, using a Taylor expansion, we show that for small values of σ, the quantity 𝔼_ε[∂L_n()/∂θ_iε_i], approximates the curvature of the loss:

    𝔼_ϵ[∂L_n()/∂θ_iε_i]
        = 
     𝔼_ε[(∂L_n(μ)/∂θ_i+∑_j∂^2L_n(μ)/∂θ_i∂θ_jϵ_jσ_j+O(‖σ‖ ^2))ε_i]
        =∂^2L_n(μ)/∂^2θ_iσ_i+O(‖σ‖ ^2) ,

where we used 𝔼_ε[ε_i]=0 and 𝔼_ε[ε_iε_j]=δ_ij in the last line. Thus, in this case, 𝔼_ε[∂L_n()/∂θ_iε_i] is a finite difference approximation to the component-wise product of the diagonal of the Hessian of the loss, and the vector σ. Therefore, we expect that the uncertainty (learning rate) would decrease in areas with positive curvature (e.g., near local minima), or increase in areas with high negative curvature (e.g., near maxima, or saddles). This seems like a “sensible” behavior of the algorithm, since we wish to converge to local minima, and escape saddles. This is in contrast to many common optimization methods, which are either insensitive to the sign of the curvature, or use it the wrong way <cit.>.

In the case of strongly convex loss, we can make a more rigorous statement, which we prove in Appendix <ref>.

 
We examine BGD with a diagonal Gaussian distribution for . If L_n() is a strongly convex function with parameter m_n>0 and a continuously
differentiable function over ℝ^n, then 
𝔼_ε[∂L_n()/∂θ_iε_i]≥ m_nσ_i>0.


 
 If L_n() is strongly convex (concave) function for all n∈ℕ, then the sequence {σ_i(n)}_n=1^∞
is strictly monotonically decreasing (increasing).


Furthermore, one can generalize these results and show that if a restriction of L_n() to an axis θ_i is strongly convex (concave) for all n∈ℕ, then {σ_i(n)}_n=1^∞
is monotonic decreasing (increasing).

Therefore, in the case of a strongly convex loss function, σ_i=0 is the only stable point of (<ref>), which means that we collapse to a point estimation similar to SGD. However, for neural networks σ_i does not generally converge to zero. In this case, the stable point σ_i=0 is generally not unique, since 𝔼_ε[∂L()/∂θ_iε_i] implicitly depends on σ_i. 

In Appendix <ref> we show the histogram of STD values on MNIST when training for 5000 epochs and demonstrate we do not collapse to a point estimation.



  
BGD in continual learning In the case of over-parameterized models and continual learning, only a part of the weights is essential for each task. We hypothesize that if a weight θ_i is important to the current task, this implies that, near the minimum, the function  L_i=L()|_()_i = θ_i  is locally convex. Corollary <ref> suggests that in this case σ_i would be small. In contrast, the loss will have a flat curvature in the direction of weights which are not important to the task. Therefore, these unimportant weights may have a large uncertainty σ_i.
Since BGD introduces the linkage between learning rate and the uncertainty (STD), the training trajectories in the next task would be restricted along the less important weights leading to a good performance on the new task, while retaining the performance on the current task. The use of BGD to continual learning exploits the inherent features of the algorithm, and it does not need any explicit information on tasks — it is completely unaware of the notion of tasks.



§ EXPERIMENTS


We show various empirical results to support the claims made throughout this paper.
Description of the used datasets can be found in Appendix <ref>, and full implementation details can be found in Appendix <ref>. In all experiments, we conducted a hyper-parameter search. The code used for BGD experiments is available at https://github.com/igolan/bgdhttps://github.com/igolan/bgd.
We aimed to provide as a broad variety of comparisons as possible, however, due to the lack of open-source implementations / architecture-specific implementations of other algorithms, not all algorithms are evaluated in all experiments.



 §.§ Continuous task agnostic scenario


First, we present results on the continuous task-agnostic scenario, where the transition between tasks is done slowly over time (as in Figure <ref>), so the task boundaries are undefined.
The output heads are shared among all tasks, task duration is 9380× T iterations (corresponds to 20 epochs per task), and the algorithms are not given with the number of tasks.
Results of this scenario on permuted MNIST for different numbers of tasks presented in Figure <ref>.
Reported accuracy is the average test accuracy over all tasks. BGD is able to maintain high average accuracy.




  
Inapplicability of other algorithms
In task-agnostic scenarios, previous methods of continual learning are generally inapplicable, as they rely on taking some action on task switch.
The task switch cannot be easily guessed because we assume no knowledge on the number of tasks.
Nevertheless, one trivial adaptation is to take the core action every iteration instead of every task switch. 
Doing so is impractical for most algorithms due to computational complexity, but we have succeeded to run both Online EWC and MAS with such an adaptation.






 §.§ Discrete task agnostic scenario


We use discrete task-agnostic scenario on Permuted MNIST to support our hypothesis of how BGD works in continual learning (subsection <ref>).

Figure <ref> shows the histogram of STD values at the end of the training process of each task.[BGD is unaware to the task identity, but we as the programmers know when a task is switched and recorded the STD values on that point.]
The results show that after the first task, a large portion of the weights have STD value which is close to the initial value of 0.06, while a small fraction of them have a much lower value. As training progresses,
more weights are assigned with STD values much lower than the initial value, but higher by at least a factor of 10 then the minimal value of σ, which is ∼ 10^-4.
These results support our hypothesis in subsection <ref> that only a small part of the weights is essential for each task, and as training progresses the percentage of weights with small STD increases as more tasks are seen. It also shows that we do not collapse to point estimation. 

In terms of performance, BGD is compared with two diagonal methods: SI and VCL.[In <cit.> the author compared SI and EWC on permuted MNIST and showed similar performance.]
Average accuracy on seen tasks is reported in Figure <ref>.
SI and VCL are evaluated in a domain learning scenario, since they use the knowledge on task switch during training.
BGD is evaluated in a discrete task-agnostic learning scenario, which means it has no information on task schedule.
Despite being evaluated on a more challenging scenario, BGD performs on-par with SI and VCL.
We find this highly encouraging, as naively one would expect task-agnostic algorithms to perform significantly worse, as they have less (important) information.

Another observation from this experiment is that training duration affects the performance of the algorithms.
The results reported in Figure <ref> are with 300 epochs per task. When training for a shorter duration, 20 epochs per task, SI achieves better performance than others (but, they all remain on-par), see Appendix <ref> for 20 epochs experiment results.







 §.§ “Labels trick” for class learning


In this experiment, we show that by applying the “labels trick” presented in this paper, we improve baseline results of “class learning”. Task identity is known during training (using the labels), but not during testing, and the tasks do not share the output head (T × C heads, or more specifically 5 × 2 heads). We followed <cit.> setup, and trained each task for four epochs (relatively short duration). Results on Split MNIST are reported in Table <ref>. Labels trick improves results of all tested algorithms in such scenario.





 §.§ Additional experiments

We provide additional evaluation of BGD using various continual learning experiments conducted in other works.
Those experiments are all done in one of the three basic scenarios, and some use more complex datasets such as CIFAR10/100. The algorithms which BGD is compared with are using the information on task-switch. Nevertheless, BGD results are on par in most cases. Those experiments can be found in Appendix <ref>, and includes:



     
    
  * Task, domain and class learning on Split MNIST and Permuted MNIST as in <cit.>.
    
  * Task learning on “vision datasets” mixture (MNIST, notMNIST, FashionMNIST, SVHN and CIFAR10), as in <cit.>.
    
  * Task learning on CIFAR10 and subsets of CIFAR100, as in <cit.>.



In addition, as a sanity check to BGD, in Appendix <ref> we show classification (single task) results on MNIST and on CIFAR10, using networks with Batch-Normalization <cit.>.
BGD can be easily applied to such networks. Lastly, in Appendix <ref>, we show that BGD does not experience underfitting and over-pruning, as shown by  <cit.> for BBB. We show that by 5000 epochs training on MNIST and CIFAR10.



§ DISCUSSION AND FUTURE WORK

In this work, we address a new scenario of continual learning — task-agnostic continual learning. We show that our algorithm is able to mitigate the notorious catastrophic forgetting phenomena, without being aware of task-switch. This important property can allow future models to better adapt to new tasks without explicitly instructed to do so, enabling them to learn in realistic continual learning settings. 

Our method, Bayesian Gradient Descent (BGD), show competitive results with various state-of-the-art continual learning algorithms on basic scenarios, while being able to work also in task-agnostic scenarios. It relies on solid theoretical foundations, and can be applied easily to advanced DNN architectures, including convolution layers and Batch-Normalization, and works on a broad variety of datasets.

Besides being a continual learning method, BGD had better classification accuracy than previous Bayesian methods on MNIST, and was on par with SGD on CIFAR10 (without Batch-Normalization).
Despite being an online version of the variational Bayes approach of <cit.>, it does not seem to have the underfitting and over-pruning issues previously observed in the variational Bayes approach of <cit.>. This allowed us to scale this approach beyond MNIST, to CIFAR10. The BGD approach is also closely related to the assumed density filtering approach of <cit.>. However, these methods rely on certain analytic approximations which are not easily applicable to different neural architectures (e.g. convnets).  In contrast,  it is straightforward to implement BGD for any neural architecture. This approach is also somewhat similar to the Stochastic Gradient Langevin Dynamics (SGLD) approach <cit.>, in the sense that we use multiple copies of the network during training. However, in contrast to the SGLD approach, we are not required to store all copies of the networks for inference, but only two parameters for each weight (μ and σ).

There are many possible extensions and uses of BGD, which were not explored in this work. In this work, our variational approximation used a diagonal Gaussian distribution. This assumption may be relaxed in the future to non-diagonal or non-Gaussian (e.g., mixtures) distributions, to allow better flexibility during learning, however, such an extension is not trivial.
Lastly, the “labels trick” presented in this paper, places foundations for future works to improve performance in a class learning scenario. By proposing a method for accurate task prediction during testing, one can completely close the performance gap between class and task learning.




  §.§.§ Acknowledgments


The authors are grateful to R. Amit, M. Shpigel Nacson, N. Merlis, O. Linial, B. Shuval, and O. Rabinovich for helpful comments on the manuscript. This research was supported by the Israel Science foundation (grant No. 31/1031), and by the Taub foundation. A Titan Xp used for this research was donated by the NVIDIA Corporation.

40
	
	
	urlstyle[Aljundi et al.(2018)Aljundi, Babiloni, Elhoseiny, Rohrbach, and
	Tuytelaars]aljundi2018memory
	Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T.
	Memory aware synapses: Learning what (not) to forget.
	In European Conference on Computer Vision, pp.  144–161.
	Springer, 2018.
	
	[Amari(1998)]amari1998natural
	Amari, S.-I.
	Natural gradient works efficiently in learning.
	Neural computation, 100 (2):0 251–276, 1998.
	
	[Balan et al.(2015)Balan, Rathod, Murphy, and
	Welling]balan2015bayesian
	Balan, A. K., Rathod, V., Murphy, K. P., and Welling, M.
	Bayesian dark knowledge.
	In Advances in Neural Information Processing Systems, pp.	3438–3446, 2015.
	
	[Blundell et al.(2015)Blundell, Cornebise, Kavukcuoglu, and
	Wierstra]blundell2015weight
	Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D.
	Weight uncertainty in neural network.
	In International Conference on Machine Learning, pp.	1613–1622, 2015.
	
	[Broderick et al.(2013)Broderick, Boyd, Wibisono, Wilson, and
	Jordan]broderick2013streaming
	Broderick, T., Boyd, N., Wibisono, A., Wilson, A. C., and Jordan, M. I.
	Streaming variational bayes.
	In Advances in Neural Information Processing Systems, pp.	1727–1735, 2013.
	
	[Chaudhry et al.(2018)Chaudhry, Dokania, Ajanthan, and
	Torr]chaudhry2018riemannian
	Chaudhry, A., Dokania, P. K., Ajanthan, T., and Torr, P. H.
	Riemannian walk for incremental learning: Understanding forgetting
	and intransigence.
	arXiv preprint arXiv:1801.10112, 2018.
	
	[Dauphin et al.(2014)Dauphin, Pascanu, Gulcehre, Cho, Ganguli, and
	Bengio]Dauphin2014
	Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y.
	Identifying and attacking the saddle point problem in
		high-dimensional non-convex optimization.
	NIPS, pp.  1–9, 2014.
	ISSN 10495258.
	
	[Farquhar & Gal(2018)Farquhar and Gal]farquhar2018towards
	Farquhar, S. and Gal, Y.
	Towards robust evaluations of continual learning.
	arXiv preprint arXiv:1805.09733, 2018.
	
	[Glorot & Bengio(2010)Glorot and Bengio]glorot2010understanding
	Glorot, X. and Bengio, Y.
	Understanding the difficulty of training deep feedforward neural
	networks.
	In Proceedings of the thirteenth international conference on
		artificial intelligence and statistics, pp.  249–256, 2010.
	
	[Graves(2011)]graves2011practical
	Graves, A.
	Practical variational inference for neural networks.
	In Advances in Neural Information Processing Systems, pp.	2348–2356, 2011.
	
	[Hernández-Lobato & Adams(2015)Hernández-Lobato and
	Adams]hernandez2015probabilistic
	Hernández-Lobato, J. M. and Adams, R.
	Probabilistic backpropagation for scalable learning of bayesian
	neural networks.
	In International Conference on Machine Learning, pp.	1861–1869, 2015.
	
	[Hernández-Lobato et al.(2016)Hernández-Lobato, Li, Rowland,
	Hernández-Lobato, Bui, and Turner]hernandez2016black
	Hernández-Lobato, J. M., Li, Y., Rowland, M., Hernández-Lobato, D.,
	Bui, T., and Turner, R. E.
	Black-box α-divergence minimization.
	2016.
	
	[Hinton & Camp(1993)Hinton and Camp]Hinton1993
	Hinton, G. E. and Camp, D. V.
	Keeping the neural networks simple by minimizing the description
		length of the weights.
	In COLT '93, 1993.
	URL <http://dl.acm.org/citation.cfm?id=168306>.
	
	[Hsu et al.(2018)Hsu, Liu, and Kira]hsu2018re
	Hsu, Y.-C., Liu, Y.-C., and Kira, Z.
	Re-evaluating continual learning scenarios: A categorization and case
	for strong baselines.
	arXiv preprint arXiv:1810.12488, 2018.
	
	[Ioffe & Szegedy(2015)Ioffe and Szegedy]ioffe2015batch
	Ioffe, S. and Szegedy, C.
	Batch normalization: Accelerating deep network training by reducing
	internal covariate shift.
	arXiv preprint arXiv:1502.03167, 2015.
	
	[Khan et al.(2018)Khan, Nielsen, Tangkaratt, Lin, Gal, and
	Srivastava]khan2018fast
	Khan, M. E., Nielsen, D., Tangkaratt, V., Lin, W., Gal, Y., and Srivastava, A.
	Fast and scalable bayesian deep learning by weight-perturbation in
	adam.
	arXiv preprint arXiv:1806.04854, 2018.
	
	[Kingma & Welling(2013)Kingma and Welling]kingma2013auto
	Kingma, D. P. and Welling, M.
	Auto-encoding variational bayes.
	arXiv preprint arXiv:1312.6114, 2013.
	
	[Kirkpatrick et al.(2017)Kirkpatrick, Pascanu, Rabinowitz, Veness,
	Desjardins, Rusu, Milan, Quan, Ramalho, Grabska-Barwinska,
	et al.]kirkpatrick2017overcoming
	Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu,
	A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.
	Overcoming catastrophic forgetting in neural networks.
	Proceedings of the National Academy of Sciences, 1140
	(13):0 3521–3526, 2017.
	
	[Krizhevsky & Hinton(2009)Krizhevsky and
	Hinton]krizhevsky2009learning
	Krizhevsky, A. and Hinton, G.
	Learning multiple layers of features from tiny images.
	2009.
	
	[LeCun et al.(1998)LeCun, Bottou, Bengio, and
	Haffner]lecun1998gradient
	LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P.
	Gradient-based learning applied to document recognition.
	Proceedings of the IEEE, 860 (11):0
	2278–2324, 1998.
	
	[Lee et al.(2017)Lee, Kim, Ha, and Zhang]Lee2017
	Lee, S., Kim, J., Ha, J., and Zhang, B.
	Overcoming catastrophic forgetting by incremental moment matching.
	2017.
	
	[Li & Hoiem(2017)Li and Hoiem]li2017learning
	Li, Z. and Hoiem, D.
	Learning without forgetting.
	IEEE Transactions on Pattern Analysis and Machine
		Intelligence, 2017.
	
	[MacKay(1992)]Mackay1992
	MacKay, D. J. C.
	A practical Bayesian framework for backpropagation networks.
	Neural computation, 4720 (1):0 448–472, 1992.
	
	[Martens & Grosse(2015)Martens and Grosse]martens2015optimizing
	Martens, J. and Grosse, R.
	Optimizing neural networks with kronecker-factored approximate
	curvature.
	In International conference on machine learning, pp.	2408–2417, 2015.
	
	[McCloskey & Cohen(1989)McCloskey and
	Cohen]mccloskey1989catastrophic
	McCloskey, M. and Cohen, N. J.
	Catastrophic interference in connectionist networks: The sequential
	learning problem.
	In Psychology of learning and motivation, volume 24, pp.	109–165. Elsevier, 1989.
	
	[Neal(1994)]Neal94
	Neal, R. M.
	Bayesian Learning for Neural Networks.
	PhD thesis, Dept. of Computer Science, University of Toronto, 1994.
	
	[Netzer et al.(2011)Netzer, Wang, Coates, Bissacco, Wu, and
	Ng]netzer2011reading
	Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y.
	Reading digits in natural images with unsupervised feature learning.
	In NIPS workshop on deep learning and unsupervised feature
		learning, volume 2011, pp.  5, 2011.
	
	[Nguyen et al.(2017)Nguyen, Li, Bui, and Turner]nguyen2017variational
	Nguyen, C. V., Li, Y., Bui, T. D., and Turner, R. E.
	Variational continual learning.
	arXiv preprint arXiv:1710.10628, 2017.
	
	[Parisi et al.(2018)Parisi, Kemker, Part, Kanan, and
	Wermter]parisi2018continual
	Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S.
	Continual lifelong learning with neural networks: A review.
	arXiv preprint arXiv:1802.07569, 2018.
	
	[Ritter et al.(2018)Ritter, Botev, and Barber]ritter2018online
	Ritter, H., Botev, A., and Barber, D.
	Online structured laplace approximations for overcoming catastrophic
	forgetting.
	arXiv preprint arXiv:1805.07810, 2018.
	
	[Rusu et al.(2016)Rusu, Rabinowitz, Desjardins, Soyer, Kirkpatrick,
	Kavukcuoglu, Pascanu, and Hadsell]rusu2016progressive
	Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J.,
	Kavukcuoglu, K., Pascanu, R., and Hadsell, R.
	Progressive neural networks.
	arXiv preprint arXiv:1606.04671, 2016.
	
	[Shin et al.(2017)Shin, Lee, Kim, and Kim]shin2017continual
	Shin, H., Lee, J. K., Kim, J., and Kim, J.
	Continual learning with deep generative replay.
	In Advances in Neural Information Processing Systems, pp.	2990–2999, 2017.
	
	[Simonyan & Zisserman(2014)Simonyan and Zisserman]simonyan2014very
	Simonyan, K. and Zisserman, A.
	Very deep convolutional networks for large-scale image recognition.
	arXiv preprint arXiv:1409.1556, 2014.
	
	[Soudry et al.(2014)Soudry, Hubara, and Meir]soudry2014expectation
	Soudry, D., Hubara, I., and Meir, R.
	Expectation backpropagation: Parameter-free training of multilayer
	neural networks with continuous or discrete weights.
	In Advances in Neural Information Processing Systems, pp.	963–971, 2014.
	
	[Trippe & Turner(2018)Trippe and Turner]trippe2018overpruning
	Trippe, B. and Turner, R.
	Overpruning in variational bayesian neural networks.
	arXiv preprint arXiv:1801.06230, 2018.
	
	[van de Ven & Tolias(2018)van de Ven and Tolias]van2018generative
	van de Ven, G. M. and Tolias, A. S.
	Generative replay with feedback connections as a general strategy for
	continual learning.
	arXiv preprint arXiv:1809.10635, 2018.
	
	[Welling & Teh(2011)Welling and Teh]welling2011bayesian
	Welling, M. and Teh, Y. W.
	Bayesian learning via stochastic gradient langevin dynamics.
	In Proceedings of the 28th International Conference on Machine
		Learning (ICML-11), pp.  681–688, 2011.
	
	[Xiao et al.(2017)Xiao, Rasul, and Vollgraf]xiao2017fashion
	Xiao, H., Rasul, K., and Vollgraf, R.
	Fashion-mnist: a novel image dataset for benchmarking machine
	learning algorithms.
	arXiv preprint arXiv:1708.07747, 2017.
	
	[Zenke et al.(2017)Zenke, Poole, and Ganguli]zenke2017continual
	Zenke, F., Poole, B., and Ganguli, S.
	Continual learning through synaptic intelligence.
	In International Conference on Machine Learning, pp.	3987–3995, 2017.
	
	[Zhang et al.(2017)Zhang, Sun, Duvenaud, and Grosse]zhang2017noisy
	Zhang, G., Sun, S., Duvenaud, D., and Grosse, R.
	Noisy natural gradient as variational inference.
	arXiv preprint arXiv:1712.02390, 2017.
	

icml2019





§ DERIVATION OF (<REF>) AND (<REF>)


In this section we provide additional details on the derivation of (<ref>) and (<ref>). The objective function is f(,ϕ)=log(q_n(|ϕ))-log(q_n-1())+L_n(),
where:

    log(q_n(|ϕ))     =-N/2log(2π)-∑_klog(σ_k) 
          -∑_k1/2σ_k^2(θ_k-μ_k)^2,


    log(q_n-1())     =-N/2log(2π)-∑_klog(v_k)
         -∑_k1/2v_k^2(θ_k-m_k)^2.

We derive (<ref>) by using the first-order necessary conditions for the optimal μ_i:

    𝔼_ε[∂ f(,ϕ)/∂θ_i∂θ_i/∂μ_i+∂ f(,ϕ)/∂μ_i]=0 .

Substituting the derivatives, we obtain:

    𝔼_ε[-1/σ_i^2(θ_i-μ_i)+1/v_i^2(θ_i-m_i)
             +∂L_n()/∂θ_i+1/σ_i^2(θ_i-μ_i)] 
         = 1/v_i^2(μ_i-m_i)+E_ε[∂L_n()/∂θ_i]  =0 .

And so we obtained (<ref>).

Next, we derive (<ref>), using the first-order necessary conditions for optimal σ_i:

    𝔼_ε[∂ f(,ϕ)/∂θ_i∂θ_i/∂σ_i+∂ f(,ϕ)/∂σ_i]=0 .

Substituting the derivatives we obtain:

    𝔼_ε[(-1/σ_i^2(θ_i-μ_i)+1/v_i^2(θ_i-m_i)+∂L_n()/∂θ_i)ε_i
            -1/σ_i+1/σ_i^3(θ_i-μ_i)^2] 
          =-1/σ_i+σ_i/v_i^2+𝔼_ε[∂L_n()/∂θ_iε_i]  =0 .

We get a quadratic equation for σ_i

    σ_i^2+σ_iv_i^2E_q(ε)[∂L_n()/∂θ_iε_i]-v_i^2=0 .

Since σ_i>0, the solution is (<ref>).



§ PROOF OF THEOREM <REF>



We define θ_j=μ_j+ε_jσ_j where  ε_j∼𝒩(0,1).
According to the smoothing theorem, the following holds

    𝔼_ε[∂L_n()/∂θ_iε_i]=𝔼_ε_j≠ i[𝔼_ε_i[.∂L_n()/∂θ_iε_i|ε_j≠ i]] .

The conditional expectation is:

    𝔼_ε_i[.∂L_n()/∂θ_iε_i|ε_j≠ i]=_-∞^∞∂L_n()/∂θ_iε_if_ε_i(ε_i)dε_i ,

where f_ε_i is the probability density function of a standard normal distribution. Since f_ε_i is an even function

    𝐄_ε_i[.∂L_n()/∂θ_iε_i|ε_j≠ i] 
          =  _0^∞∂L_n(μ_i+ε_iσ_i,θ_-i)/∂θ_iε_if_ε_i(ε_i)dε_i
            -_0^∞∂L_n(μ_i-ε_iσ_i,θ_-i)/∂θ_iε_if_ε_i(ε_i)dε_i .


Now, since L_n() is strongly convex function
with parameter m_n>0 and continuously differentiable function over
ℝ^d, the following holds ∀_1,_2∈ℝ^d:

    (∇L_n(_1)-∇L_n(_2))^T(_1-_2)≥ m_n_1-_2_2^2 .

For _1,_2 such that

    (_1)_j=(_2)_j,     j≠ i
    μ_i+ε_iσ_i,     j=i ,


    (_2)_j=(_1)_j,     j≠ i
    μ_i-ε_iσ_i,     j=i ,

the following holds:

    (∂L(_1)/∂θ_i-∂L(_2)/∂θ_i)ε_i≥ 2m_nσ_iε_i^2 .

Therefore, substituting this inequality into (<ref>), we obtain:

    𝔼_ε[∂L()/∂θ_iε_i]≥ m_nσ_i>0 .






§ DATASETS



MNIST is a database of handwritten digits, which has a training set of 60,000
examples, and a test set of 10,000 examples — each a 28 × 28 image.
The image is labeled with a number in the range of 0 to 9.

Permuted MNIST
is a set of tasks constructed by a random
permutation of MNIST pixels. Each task has a different permutation of
pixels from the previous one. 

Split MNIST
is a set of tasks constructed by taking pairs of digits from MNIST dataset. For example: the first task is classifying between the digits 0 and 1, the second task is classifying between the digits 2 and 3 etc. The total number of tasks in this experiment is five.

CIFAR10 
is a dataset which consists of 60000 32 × 32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. 

CIFAR100 
is a dataset which consists of 60000 32 × 32 color images in 100 classes, with 600 images per class <cit.>. There are 50000 training images and 10000 test images. 
The 100 classes in the CIFAR-100 are grouped into 20 superclasses. Each image comes with a “fine” label (the class to which it belongs) and a “coarse” label (the superclass to which it belongs). 

FashionMNIST is a dataset comprising of 28 × 28 grayscale images of 70,000 fashion products from 10 categories. The training set has 60,000 images and the test set has 10,000 images.

notMNIST is a dataset with 10 classes, with letters A-J taken from different fonts. It has similar characteristics like MNIST - same pixel size (28 × 28), grayscale images and comprised of 10-class.

SVHN is a real-world image dataset with 10 classes, 1 for each digit. It has RGB images of size 32 × 32 obtained from house numbers in Google Street View images.




§ IMPLEMENTATION DETAILS



We initialize the mean of the weights μ by sampling from a Gaussian
distribution with a zero mean and a variance of 2/(n_input+n_output) as in <cit.>, unless stated otherwise.
We use 10 Monte Carlo samples to estimate the expected gradient during training, and average the accuracy of 10 sampled networks during testing, unless stated otherwise.



  
Continuous permuted MNIST
We use a fully connected network with 2 hidden layers of width 200.
Following <cit.>, the original MNIST images are padded with zeros to match size of 32 × 32. The batch size is 128, and we sample with replacement due to the properties of the continuous scenario (no definition for epoch as task boundaries are undefined).
We conducted a hyper-parameter search for all algorithms and present the best, see 
Figure <ref>.
Results are averaged over three different seeds (2019, 2020, and 2021).
STD init for BGD is 0.06 and η = 1.
For Online EWC and MAS we used the following combinations of hyper-parameters: LR of 0.01 and 0.0001, regularization coefficient of 10, 0.1, 0.001, 0.0001, and optimizer SGD and Adam. The best results for online EWC were achieved using LR of 0.01, regularization coefficient of 0.1 and SGD optimizer.
For MAS, best results achieved using LR of 0.01, regularization coefficient of 0.001 and SGD optimizer.





  
Discrete permuted MNIST
We use a fully connected neural network
with 2 hidden layers of 200 width, ReLUs as activation functions
and softmax output layer with 10 units. We trained the network using
Bayesian Gradient Descent. The preprocessing is the same as with MNIST classification, but we used a training set of 60,000 examples.
BGD was trained with mini-batch of size 128 and η=1 for 300 epochs. 
We run an hyper-parameters tuning for SI using c=(0.3,0.1,0.05,0.01,0.001) and select the best one (c = 0.05) as a baseline (See Figure <ref>).





  
Labels trick for class learning on Split MNIST
We followed <cit.> setup. The network is a fully connected network with 2 hidden layers of width 400. The batch size is 128, and each task is trained for 4 epochs. The original MNIST images are padded with zeros. STD init for BGD is 0.017 and η = 1. Results are averaged over ten different seeds (2019-2028).



  
CIFAR10/CIFAR100 continual learning
Batch size is 256. On BGD we used MAP for inference method, initial STD was 0.019 and η=2. For SGD we used constant learning rate of 0.01 and for SI we used c=0.01. Hyper parameters for all algorithms were chosen after using grid search and taking the optimal value.



  
Vision datasets mix
We followed the experiment as described in <cit.>. We use a batch size of 64, and we normalize the datasets to have zero mean and unit variance. The network architecture is LeNet like with 2 convolution layers with 20 and 50 channels and kernel size of 5, each convolution layer is followed by a Relu activations function and max pool, and the two layers are followed with fully connected layer of size 500 before the last layer.

SGD baseline was trained with a constant learning rate of 0.001 and ADAM used ϵ=10^-8, LR of 0.001 and (β_1,β_2)=(0.9, 0.999). BGD trained with initial STD of 0.02, η set to 1 and batch size 64.



  
MNIST classification
We use a fully
connected neural network with two hidden layers of various widths, ReLU's
as activation functions and softmax output layer with 10 units. We
preprocessed the data to have pixel values in the range of [0,1].
We use mini-batch of size 128 and η=1. In addition, we used a validation set to find
the initialization value for σ. In order to compare the results of Bayesian Gradient Descent (BGD) with the results of Bayes By Backprop

(BBB) algorithm by <cit.>, we used a training set of 50,000 examples and validation set of 10,000 examples and present results of BBB with the same prior as BGD used. We train the network for 600 epochs. 



  
CIFAR10 classification
SGD was trained with learning rate starting at 0.01 and divided by 10 every 100 epochs,  momentum was set to 0.9.
On both BGD and SGD we used data augmentation of random cropping and flipping. Both experiments use a batch size of 128 and 10 Monte Carlo iterations.
In the experiment with Batch-Normalization (BN) initial STD is 0.011, η set to 8. μ is initialized using PyTorch 0.3.1 for convolution layers, bias is initialized to 0 and μ of BN layers scaling and shifting parameters are initialized to 1 and 0.
In the experiment without Batch-Normalization (BN) initial std is 0.015, η set to 10. μ of convolution layers is initialized using He initialization.



§ COMPLEXITY


BGD requires × 2 more parameters compared to SGD, as it stores both the mean and the STD per weight.
In terms of time complexity, the major difference between SGD and BGD arises from the estimation of the expected gradients using Monte Carlo samples during training.

Since those Monte Carlo samples are completely independent the algorithm is embarrassingly parallel.

Specifically, given a mini-batch: for each Monte Carlo sample, BGD generates a random network using μ and σ, then making a forward-backward pass with the randomized weights.

Two main implementation methods are available (using 10 Monte Carlo samples as an example):

    
  * Producing the (10) Monte Carlo samples sequentially, thus saving only a single randomized network in memory at a time (decreasing memory usage, increasing runtime).
    
  * Producing the (10) Monte Carlo samples in parallel, thus saving (10) randomized networks in memory (increasing memory usage, decreasing runtime).


We analyzed how the number of Monte Carlo iterations affects the runtime on CIFAR-10 using the first method of implementation (sequential MC samples). The results, reported in Table <ref>, show that runtime is indeed a linear function of the number of MC iterations.
In the experiments we used a single GPU (GeForce GTX 1080 Ti).




§ ADDITIONAL EXPERIMENTS




 §.§ Spilt MNIST and permuted MNIST


We followed <cit.> setup, and compared BGD to several continual learning algorithms, on all three basic scenarios (task, domain, and class learning).
Results for Split MNIST reported on Table <ref>, and for permuted MNIST on Table <ref>.
Note that those experiments are with relatively short duration of training.

For Permuted MNIST experiments we used STD init of 0.02.
For task learning on Split MNIST we used STD init of 0.02, for domain learning we used 0.05, and 0.017 for class learning.



 §.§ Continual learning on vision datasets


We followed <cit.> and challenged our algorithm with the vision datasets experiment. In this experiment, we train sequentially on MNIST, notMNIST, [Originally published at 
 http://yaroslavvb.blogspot.co.uk/2011/09/notmnist-dataset.html and downloaded from https://github.com/davidflanagan/notMNIST-to-MNIST] FashionMNIST, SVHN and CIFAR10 <cit.>.
Training is done in a sequential way with 20 epochs per task — in epochs 1-20 we train on MNIST (first task), and on epochs 81-100 we train on CIFAR10 (last task).
All five datasets consist of about 50,000 training images from 10 different classes, but they differ from each other in various ways: black and white vs. RGB, letters and digits vs. vehicles and animals etc. See Appendix <ref> for further details about the datasets.
We use the exact same setup as in <cit.> for the comparison — LeNet-like <cit.> architecture with separated last layer for each task as in CIFAR10/CIFAR100 experiment. Results are reported on Table <ref>.



 §.§ Continual learning on CIFAR10/CIFAR100


We follow SI <cit.> experiment on CIFAR10/CIFAR100 — training sequentially on six tasks from CIFAR10 and CIFAR100.
The first task is the full CIFAR10 dataset, and the next five tasks are subsets of CIFAR100 with ten classes, where we train each task for 150 epochs.
As a reference, we present results of SGD and SI. We used the same network as in the original experiment, which consists of four convolutional layers followed by a fully connected layer with dropout shared by all six tasks.

The results (Figure <ref>) show that the network is able attain a good balance between retaining reasonable accuracy on previous tasks and achieving high accuracy on newer ones. 

Figure <ref> shows the histogram of STD values at the end of the training process of each task. Similar to the results for permuted MNIST, as more tasks are seen the percentage of weights with STD values smaller than initial STD value (of 0.019) increases (the minimal value of σ is ∼ 10^-4).







 §.§ Discrete permuted MNIST



We ran an additional simulation with the same architecture on exactly the configuration as in <cit.>, see Figure <ref>.





 §.§ Classification


As a sanity check, we compare BGD performance on single-task classification to recent Bayesian methods used for classification along with SGD and ADAM.



  
MNIST classification




We compare BGD on MNIST with SGD and BBB <cit.>.
Table <ref> summarizes test accuracy[Results of SGD and BBB in Table <ref> for MNIST are as reported on <cit.>.] of networks with various widths. Overall, BGD performs better than BBB and SGD.




  
CIFAR10 classification
We use CIFAR10 to evaluate BGD's performance on a more complex dataset and a larger network.
In this experiment, besides comparison to SGD and ADAM, we compare BGD to a group of recent advanced variational inference algorithms, including both diagonal and non-diagonal methods — K-FAC <cit.>, Noisy K-FAC and Noisy ADAM <cit.>.

To do so, we followed the experiment described in <cit.>. There, a VGG16-like[The detailed network architecture is 32-32-M-64-64-M-128-128-128-M-256-256-256-M-256-256-256-M-FC10, where each number represents the number of filters in a convolutional layer, and M denotes max-pooling - same as <cit.>.] <cit.> architecture was used and data augmentation is applied. We present results both with Batch Normalization <cit.> and without it in Table <ref>.


A relaxed version of BBB was used, with λ=0.1 as described in <cit.>.



 §.§ 5000 epochs training




  
MNIST
We use the MNIST classification experiment to demonstrate the convergence of the log-likelihood cost function and the histogram of STD values. We train a fully connected neural network with two hidden layers and layer width of 400 for 5000 epochs.



Figure <ref> shows the log-likelihood cost function of the training set and the test set. As can be seen, the log-likelihood cost function on the training set decreases during the training process and converges to a low value. Thus, BGD does not experience underfitting and over-pruning as was shown by <cit.> for BBB.



Figure <ref> shows the histogram of STD values during the training process.
As can be seen, the histogram of STD values converges. This demonstrates that σ_i does not collapse to zero even after 5000 epochs. 



Figure <ref> shows the learning curve of the train set and the test set. As can be seen, the test accuracy does not drop even if we continue to train for 5000 epochs.



  
CIFAR10
To further show that BGD does not experience underfitting and over-pruning, we trained VGG11. [We trained a full VGG11. The detailed network architecture is 64-M-128-M-256-256-M-512-512-M-512-512-M-FC10] Figure <ref> show the test accuracy during the 5000 epochs.













