Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Score matching provides an effective approach to learning flexible unnormalized models, but its scalability is limited by the need to evaluate a second-order derivative. In this paper,we connect a general family of learning objectives including score matching to Wassersteingradient flows. This connection enables us to design a scalable approximation to theseobjectives, with a form similar to single-step contrastive divergence. We present applications in training implicit variational and Wasserstein auto-encoders with manifold-valued priors. Unnormalized models define the model distribution as q(x; θ) ∝ exp(−E(x; θ)), where E(x; θ) is an energy function that can be parameterized by e.g. DNNs. Unnormalized models can be used directly for density estimation, but another important application is in gradient estimation for implicit variational inference, where we can use score estimation in latent space to approximate an intractable learning objective. This approach leads to improved performance in training implicit auto-encoders . Maximum likelihood estimation for unnormalized models is intractable, and score matching (Hyvärinen, 2005) is a popular alternative. Score matching optimizes the Fisher divergence where we denote the data distribution as p. Hyvärinen shows D F is equivalent to E p(x) ∆ log q(x; θ) + 1 2 ∇ log q(x; θ) 2, where ∆ = i ∂ 2 i is the Laplacian; the equivalent form can be estimated using samples from p. So far, when E has a complex parameterization, calculating the equivalent objective is still difficult, as it involves the second-order derivatives; and in practice, people turn to scalable approximations of the score matching objective (; ;) or other objectives such as the kernelized Stein discrepancy (KSD; b;). However, these approximations are developed on a case-by-case basis, leaving important applications unaddressed; for example, there is a lack of scalable learning methods for models on manifolds . In this work, we present a unifying perspective to this problem, and derive scalable approximations for a variety of objectives including score matching. We start by interpreting these objectives as the initial velocity of certain distribution-space gradient flows, which are simulated by common samplers. This novel interpretation leads to a scalable approximation algorithm for all such objectives, reminiscent to single-step contrastive divergence (CD-1). We refer to any objective bearing the above interpretation as above as a "minimum velocity learning objective", a term coined in the unpublished work. Our formulation is a distribution-space generalization of their work, and applies to different objectives as the choice of distribution space varies. Another gap we fill in is the development of a practically applicable algorithm: while the idea of approximating score matching with CD-1 is also explored in , previously the approximation suffers from an infinite variance problem, and is thus believed to be impractical ; we present a simple fix to this issue. Additionally, we present an approximation to the objective function instead of its gradient, thus enabling the use of regularization like early-stopping. Other related work will be reviewed in Appendix C. One important application of our framework is in learning unnormalized models on manifolds. This is needed in areas such as image analysis , geology and bioinformatics . Moreover, as we present an approximation to the Riemannian score matching objective, it enables flexible inference for VAEs and WAEs with manifold-valued latent variables, as it enables gradient estimation for implicit variational distributions on manifolds. It is believed that auto-encoders with a manifold-valued latent space can capture the distribution of certain types of data better (; ;). As we will see in Section 3, our method leads to improved performance of VAEs and WAEs. We now present our framework, which concerns all learning objectives of the following form: where q θ is the model distribution, {p t} is the gradient flow of KL q in a suitable distribution space (e.g. the 2-Wasserstein space), and KL q is the exclusive KL divergence functional, p → KL(p q θ). We refer to these objectives as "minimum velocity learning (MVL) objectives", since will show that they correspond to the initial velocity of the gradient flow. subsumes the Fisher divergence for score matching as a special case, since from the properties of the 2-Wasserstein space (please refer to Appendix A for the necessary preliminary knowledge), we have D F (p|q) = 1 2 grad p KL q 2, where the gradient and norm are defined in the 2-Wasserstein space, and the data manifold X is endowed with the Euclidean metric. Rearranging terms, we get i.e., score matching is a special case of the MVL objective, when the space of distributions is chosen as the 2-Wasserstein space P(X). In certain cases, the gradient flow of KL q corresponds to common samplers, and can be efficiently simulated: e.g., the gradient flow in P(X) is the (Riemannian) Langevin dynamics. Now we utilize this connection to design a scalable approximation to these objectives. First, note holds regardless of the chosen space of distributions. Denote As the first term in is independent of θ, the MVL objective is always equivalent to the second term. We approximate it by simulating a modified gradient flow: letp t be the distribution obtained by running the sampler targeting q 1/2. Then can be approximated by replacing the limit with a fixed, and running the corresponding sampler starting from a mini-batch of training data. The approximation becomes unbiased when → 0. A Control Variate The approximation to will have small bias. However, it suffers from high variance when the sampler consists of Itô diffusion (e.g. when it is Langevin dynamics). Fortunately, we can solve this problem with a control variate. To illustrate this, suppose the MVL objective is defined using Langevin dynamics (LD). Without loss of generality, we assume we use a batch size of 1 in the approximation, so approximation isL, where x + is sampled from the training data, and Z ∼ N (0, I). By Taylor expansion 1, and as → 0, VarL = Θ −1 → ∞. Thus a control variate is needed. In this LD example, the control variate is More generally, the control variate is always the inner product of ∇ x E(x +) and the diffusion term in the sampler. As a side product of our work, we note that similar control variate can be obtained for CD-1 and denoising score matching, and it solves their problem. See Appendix B.2. As an application, let us consider learning unnormalized models on Riemannian manifolds. In this case, the gradient flow of KL q in the 2-Wasserstein space becomes the Riemannian Langevin dynamics , and the approximate MVL objective becomes is a sample from the Riemannian LD, and z ∼ N (0, G −1 (y)). This is the Riemannian score matching objective , which also has the form of, but the norm is defined by the Riemannian metric of X. From this example, we can see the power of our framework, which enables us to approximate new objectives with ease. We now apply our approximation to learning implicit variational auto-encoders (VAEs) and Wasserstein auto-encoders (WAEs) with manifold-valued prior. First we review the use of score estimator in learning implicit auto-encoding models. We use VAE as an example; for WAE with KL penalty, the derivation is similar, see. The VAE objective is E p(x) E q(z|x;φ) log p(z)p(x|z;θ) q(z|x;φ), where q(z|x; φ) is the push-forward measure of N (0, I) by f (·; φ). The objective is intractable, as the entropy term H[q(z|x; φ)] is intractable; however, we can show that Thus to approximate the objective, it suffices to approximate the score function ∇ z log q(z). This can be implemented by learning an unnormalized model using score matching. As the score matching objective directly aligns the learnt score function ∇ z E to the data score, it can be viewed as score estimation using conservative fields, thus it leads to a better approximation to the gradient ∇ φ H[q(z)] compared to indirect approximations such as the adversarial density ratio estimators. As we turn to the case where the latent space is an embedded manifold, the original score matching objective can no long be used, since q(z) no longer has a density w.r.t. the Lebesgue measure in the embedded space. However, we can still do score estimation on the manifold, i.e. estimate the log derivative of the density w.r.t. the manifold Hausdorff measure. This can be done by fitting an unnormalized model on manifold, using the approximate objective developed in Section 2.2. We note that in this case, still holds, and we can still estimate the gradient of the ELBO using the score estimate. See Appendix E.4 for details. Empirical Evaluations We apply our method to train implicit hyperspherical VAEs with implicit encoders and WAEs, on the MNIST dataset. Our experiment setup follows , with the exception that we parameterize an energy network E(z; ψ) and uses its gradient as the score estimate, instead of parameterizing a score network. Detailed setup and additional synthetic experiments are in Appendix D. For the VAE experiment, we compare with VAEs with explicit variational posteriors, as well as Euclidean VAEs, and report negative log likelihood estimated with annealed importance sampling ; for the WAE experiment, we compare with WAE-GAN , and report the FID score . The are summarized in Figure 1. We can see that in all cases, hyperspherical prior outperforms the Euclidean prior, and our method leads to improved performance. Interestingly, for VAEs with explicit encoders, hyperspherical VAE could not match the performance of Euclidean VAE in high dimensions; this is consistent with the in , who incorrectly conjectured that hyperspherical prior is inadequate in high dimensions; we can see that the problem is actually the lack of flexibility in inference, which our method addresses. In this section, we review knowledge needed in this work, most importantly Wasserstein gradient flow and its connection to sampling algorithms. A (differential) manifold M is a topological space locally diffeomorphic to an Euclidean or Hilbert space. A manifold is covered by a set of charts, which enables the use of coordinates locally, and specifies a set of basis {∂ i} in the local tangent space. A Riemannian manifold further possesses a Riemannian structure, which assigns to each tangent space T p M an inner product structure. The Riemannian structure can be described using coordinates w.r.t. local charts. The manifold structure enables us to differentiate a function along curves. Specifically, consider a curve c: [0, T] → M, and a smooth function f: M → R. At c(t) ∈ M, a tangent vector A tangent vector field assigns to each p ∈ M a tangent vector V p ∈ T p M. It determines a flow, a set of curves {φ p (t): p ∈ M} which all have V φp(t) as their velocity. On Riemannian manifolds, the gradient of a smooth function f is a tangent vector field p → grad p f such that It determines the gradient flow, which generalizes the Euclidean-space notion dx = ∇ x f (x)dt. We will work with two types of manifolds: the data space X when we apply our method to manifold-valued data, and the space of probability distributions over X. On the space of distributions, we are mostly interested in the 2-Wasserstein space P(X), a Riemannian manifold. The following properties of P(X) will be useful for our purposes : 1. Its tangent space T p P(X) can be identified as a subspace of the space of vector fields on X; the Riemannian metric of P(X) is defined as for all p ∈ P(X), X, Y ∈ T p P(X); the inner product on the right hand side above is determined by the Riemannian structure of X. We will also consider a few other spaces of distributions, including the Wasserstein-Fisher-Rao space , and the H-Wasserstein space introduced in . On the data space, we need to introduce the notion of density, i.e. the Radon-Nikodym derivative w.r.t. a suitable base measure. The Hausdorff measure is one such choice; it reduces to the Lebesgue measure when X = R n. In most cases, distributions on manifolds are specified using their density w.r.t. the Hausdorff measure; e.g. "uniform" distributions has constant densities in this sense. Finally, the data space X will be embedded in R n; we refer to real-valued functions on the space of distributions as functionals; we denote the functional q → KL(q p) as KL p; we adopt the Einstein summation convention, and omit the summation symbol when an index appears both as subscript and superscript on one side of an equation, e.g. Now we review the sampling algorithms considered in this work. They include diffusion-based MCMC, particle-based variational inference, and other stochastic interacting particle systems. Riemannian Langevin Dynamics Suppose our target distribution has density p(x) w.r.t. the Hausdorff measure of X. In a local chart U ⊂ X, let G: U → R m×m be the coordinate matrix of its Riemannian metric. Then the Riemannian Langevin dynamics corresponds to the following stochastic differential equation in the chart 2: where and (g ij) is the coordinate of the matrix G −1. It is known that the Riemannian Langevin dynamics is the gradient flow of the KL functional KL p (q):= KL(q p) in the 2-Wasserstein space P(X). Particle-based Samplers A range of samplers approximate the gradient flow of KL p in various spaces, using deterministic or stochastic interacting particle systems. 3 For instance, Stein variational gradient descent (SVGD;) simulates the gradient flow in the so-called H-Wasserstein space , which replaces the Riemannian structure in P(X) with the RKHS inner product. Birth-death accelerated Langevin dynamics is a stochastic interacting particle system that simulates to the gradient flow of KL p in the Wasserstein-Fisher-Rao space. Finally, the stochastic particle-optimization sampler (SPOS; ;) combines the dynamics of SVGD and Langevin dynamics; as we will show in Appendix E.2, SPOS also has a gradient flow structure. In this section we present additional discussions about our framework. In Section B.1 we discuss other objectives that can be derived from our framework; in Section B.2 we show 2. differs from the form in some literature (e.g.), as in our case, the density of the target measure is defined w.r.t. the Hausdorff measure of X, instead of the Lebesgue measure in. See (, eq ) or (, Section 1.5). 3. There are other particle-based samplers (b,a;) corresponding to accelerated gradient flows. However, as we will be interested in the initial velocity of the flow, they do not lead to new objectives in our framework. our control variate could be applied to CD-1 and denoising score matching; finally, while readers familiar with Riemannian Brownian motion may be concerned about the use of local coordinates in our Riemannian score matching approximation, we show in Section B.3 it does not lead to issues. As our derivation is independent of the distribution space of choice, we can derive approximations to other learning objectives using samplers other than Langevin dynamics, as reviewed in Section A.2. An example is Riemannian Langevin dynamics which we have discussed in the main text; another example is when we choose the sampler as SVGD. In this case, we will obtain an approximation to the kernelized Stein discrepancy, generalizing the derivation in . When the sampling algorithm is chosen as SPOS, the corresponding MVL objective will be an interpolation between KSD and the Fisher divergence. See Appendix E.3 for derivations. Finally, the use of birth-death accelerated Langevin dynamics leads to a novel learning objective. In terms of applications, our work focuses on learning neural energy-based models, and we find these objectives do not improve over score matching in this aspect. However, these derivations generalize previous discussions, and establish new connections between sampling algorithms and learning objectives. It is also possible that these objectives could be useful in other scenarios, such as learning kernel exponential family models , direct estimation of the score function , and improving the training of GANs or amortized variational inference methods . As a side product of our work, we show in this section that our variance analysis explains the pitfall of two well-known approximations to the score matching objective: CD-1 and denoising score matching . Both approximations become unbiased as a step-size hyper-parameter → 0, but could not match the performance of exact score matching in practice, as witnessed in;;. Our analysis leads to novel control variates for these approximators. As we will show in Section D.1, the variance-reduced versions of the approximations have comparable performance to the exact score matching objective. The first two terms inside the norm represent a noise corrupted sample, and ψ θ represents a "single-step denoising direction" . It is proved that the optimal ψ satisfies ψ = σ 2 ∇ logp, wherep is the density of the corrupted distribution . Consider the stochastic estimator of. We assume a batch size of 1, and denote the data sample as x. To keep notations consistent, denote = σ 2, ψ θ (x) = ∇ x E(x; θ). Then the stochastic estimator iŝ Denotex:= x + √ z. By Taylor expansion we havê As which is known as the Hutchinson's trick , But V ar(B) = O, so as → 0, the rescaled estimator −2L dsm becomes unbiased with infinite variance; and subtracting (B) from (A) in a finite-variance estimator. Proposed as an approximation to the maximum likelihood estimate, the K-step contrastive divergence (CD-K) learning rule updates the model parameter with where ν is the learning rate, and p K is obtained from p by running K steps of MCMC. does not define a valid objective, since p K also depends on θ; however, proved that when K = 1 and the sampler is the Langevin dynamics, recovers the gradient of the score matching objective. Using the same derivation as in the main text, we can see that as the step-size of the sampler approaches 0 (and ν is re-scaled appropriately), the gradient produced by CD-1 also suffers from infinite variance, and this can be fixed using the same control variate. However, practical utility of CD-1 is still hindered by the fact that it does not correspond to a valid learning objective; consequently, it is impossible to monitor the training process for CD-1, or introduce regularizations such as early stopping 4. Readers familiar with Riemannian Brownian motion will notice that we used local coordinates when deriving the MPF objective, and this is only valid until the particle exits the local chart. In this section, we show that this does not affect the validity of our method; specifically, we prove in Proposition 3 that the local coordinate representation lead to valid approximation to the MVL objective in the compact case. We also argue in Remark 4 that the use of local coordinate does not lead to numerical instability. 4. In practice, the term EpE − Ep K E is often used to tract the training process of CD-K. It is not a proper loss; we can see from that when K = 1 and → 0, EpE − Ep K E is significantly different from the proper score matching (MVL) loss, by a term of Remark 1 While a more general than Proposition 3 is likely attainable (e.g. by replacing compactness of X with quadratic growth of the energy), this is out of the scope of our work; for our purpose, it is sufficient to note that the proposition covers manifolds like S n, and the local coordinate issue will not exist in manifolds possessing a global chart, such as H n. Lemma 2 (Theorem 3.6.1 in ) For any manifold M, x ∈ M, and a normal neighborhood B of x, there exists constant C > 0 such that the first exit time τ from B, of the Riemannian Brownian motion starting from x, satisfies for any L ≥ 1. Proposition 3 Assume the data manifold X is compact, and for all θ, E(·; θ) is in C 1. Let L mvl_rld be defined as in, X t following the true Riemannian Langevin dynamics targeting i.e. recovers true WMVL objective. Proof By the tower property of conditional expectation, it suffices to prove the when P (X 0 = x) = 1 for some x. Choose a normal neighborhood B centered at x such that B is contained by our current chart, and has distance from the boundary of the chart bounded by some δ > 0. Let C,τ be defined as in Lemma 2. Recall the Riemannian LD is the sum of a drift and the Riemannian BM. Since X is compact and E is in C 1, the drift term in the SDE will have norm bounded by some finite C. Thus the first exit time of the Riemannian LD is greater than min(τ, δ/C) =: τ. Let X t follow the true Riemannian LD,X t = X t when t < τ, and be such that E(X t) = 0 afterwards. 5 , until τ,X t follows the local coordinate representation of Riemannian LD, thus on the event {≤ τ},X would correspond to y − in. As X is compact, the continuous energy function E is bounded by |E(·)| ≤ A for some finite A. Then for sufficiently small, In the above the first term converges to d dt E(E(X t)) t=0 as → 0, and Hence the proof is complete. 5. This is conceptually similar to the standard augmentation used in stochastic process texts; from a algorithmic perspective it can be implemented by modifying the algorithm so that in the very unlikely event when y − escapes the chart, we return 0 as the corresponding energy. We note that this is unnecessary for manifolds like S n, since the charts can be extended to R d and hence τ = ∞. Remark 4 It is argued that simulating diffusion-based MCMC in local coordinates leads to numeric instabilities (; a). We stress that in our setting of approximating MVL objectives, this is not the case. The reason is that we only need to do a single step of MCMC, with arbitrarily small step-size. Therefore, we could use different step-size for each sample, based on the magnitude of g and log q in their locations. We can also choose different local charts for each sample, which is justified by the proposition above. Our work concerns scalable learning algorithms for unnormalized models. This is a longstanding problem in literature, and some of the previous work is discussed in Section 1. Apart from those work, it is worth mentioning , which also designed a CD-like algorithm to approximate the kernelized Stein discrepancy; as we have discussed in Section B.1, in our framework there exists a similar algorithm, as well as a slight generalization when we replace SVGD with SPOS. Other notable work includes noise contrastive estimation (Gutmann and Hyvärinen, 2010) and Parzen score matching . However, to our knowledge, they have not been applied to complex unnormalized models such as those parameterized by DNNs, and a comparison would fall out of the scope of this work. Apart from the MVL formulation used in this work, there also exists other work on the connection between learning objectives of unnormalized model and infinitesimal actions of sampling dynamics. The minimum probability flow framework studies the slightly different objective lim →0 1 KL(p 0 p), where {p t} is the trajectory of the sampler. It also recovers score matching as a special instance; however, it does not lead to scalable learning algorithms for continuous-state unnormalized models as our method does; instead, its main application is in discrete-state models. Many of the MVL objectives we have derived are also instances of the Stein discrepancy . This interpretation is helpful in establishing theoretical properties, but it does not lead to scalable implementations of these objectives, that do not depend on higher-order derivatives. To demonstrate the proposed estimators have small bias and variance, we first evaluate them on low-dimensional synthetic data. We also verify the claim in Section B.2 that our control variate improves the performance of CD-1 and DSM. In this section, we evaluate our MVL approximation to the Euclidean score matching objective, as well as the variance-reduced DSM objective 6. 6. An experiment confirming the effectiveness of our control variate on CD-1 is presented in Appendix D.1.3. We evaluate the bias and variance of our estimators by comparing them to sliced score matching (SSM), an unbiased estimator for the score matching objective. We choose the data distribution p as the 2-D banana dataset from , and the model distribution q θ as an EBM trained on that dataset. We estimate the squared bias with a stochastic upper bound, using 5,000,000 samples. More specifically, denote the two methods as, respectively. We estimate the squared bias as, where Observe that this expectation of the estimate upper bounds the true squared bias following Cauchy's inequality; and the bias → 0 as K, M → 0. We choose K = 100, M = 50000 and plot the confidence interval. We also use these samples to estimate the variance of our estimator. For the model distribution q, we choose an EBM as stated in the main text. The energy of the model is parameterized as follows: we parameterize a d-dimensional vector ψ(x; θ) using a feed-forward network, then return x ψ(x; θ) as the energy function. This is inspired by the "score network" parameterization in ; we note that this choice has little influence on the synthetic experiments (and is merely chosen here for consistency), but leads to improved performance in the AE experiments. Finally, ψ(x; θ) is parameterized with 2 hidden layers and Swish activation , and each layer has 100 units. We apply spectral normalization to the intermediate layers. We train the EBM for 400 iterations with our approximation to the score matching objective, using a batch size of 200 and a learning rate of 4 × 10 −3. The choice of training objective is arbitrary; changing it to sliced score matching does not lead to any notable difference, as is expected from this experiment. The are shown in Figure 2, in which we plot the (squared) bias and variance for both estimators, with varying step-size. The bias plot is shown in the left. We can see that for both estimators, the bias is negligible at ≤ 10 −2. We further use a z-test to compare the mean of the two estimators (for = 6 × 10 −5) with the mean of SSM. The p value is 0.48 for our estimator and 0.19 for DSM, indicating there is no significant difference in either case. The variance of the estimators, with and without our control variate, are shown in Figure 2 right. As expected, the variance grows unbounded in absence of the control variate, and is approximately constant when it is added. From the scale of the variance, we can see that it is exactly this variance problem that causes the failure of the original DSM estimator. We now evaluate our approximation to the Riemannian score matching objective, by learning an unnormalized model. The data distribution is chosen as a mixture of two von Mises distributions on S 1: p(x) = 0.7p vM (x|, 2) + 0.3p vM (x|(0.5, −0.5), 3), where p vM is the von Mises density The energy function in the model is parameterized with a feed-forward network, using the same score-network-inspired parameterization as in the last experiment. The network uses tanh activation and has 2 hidden layers, each layer with 100 units. We generate 50,000 samples from p(x) for training. We use full batch training and train for 6,000 iterations, using a learning rate of 5 × 10 −4. The step-size hyperparameter in the MVL approximation is set to 10 −5. Results We plot the log densities of the ground truth distribution as well as the learned model in Figure 3. We can see the two functions matches closely, suggesting our method is suitable for density estimation on manifolds. To verify our control variate also solves the variance issue in CD-1, we train EBMs using CD-1 with varying step-size, with and without our control variate, and compare the score matching loss to EBMs trained with our method as well as sliced score matching. We use a separate experiment for CD-1 since it only estimates the gradient of the score matching loss. The score matching loss is calculated using SSM on training set, and averaged over 3 separate runs. We use the cosine dataset in ; the energy parameterization is the same as in Section D.1.1. The are shown in Figure 4. We can see that with the introduction of the control variate, CD-1 performs as well as other score matching methods. In all auto-encoder experiments, setup follows from whenever they applies. The only difference is that for score estimation, we parameterize the energy function, and use its gradient as the score estimate, as opposed to directly parameterizing the score function as done in . This modification makes our method applicable; essentially, it corrects the score estimation in so that it constitute a conservative field, which is a desirable property since score functions should be conservative. For this reason, we re-implement all experiments for Euclidean-prior auto-encoders to ensure a fair comparison. The are slightly worse than for the VAE experiment, but significantly better for WAE experiments. It should be noted that for the VAE experiment, our implicit hyperspherical VAE is still better than the implicit Euclidean VAE reported in . The (conditional) energy function in this experiment is parameterized using the score-net-inspired method described in Appendix D.1, with a feed-forward network. The network has 2 hidden layers, each with 256 hidden units. We use tanh activation for the network, and do not apply spectral normalization. When training the energy network, we add a L2 regularization term for the energy scale, with coefficient 10 −4. The coefficient is determined by grid search on {10 −3, 10 −4, 10 −5}, using AIS-estimated likelihood on a heldout set created from the training set. The step-size of the MVL approximation is set to 10 −3; we note that the performance is relatively insensitive w.r.t. the step-size inside the range of [10 −4, 10 −2], as suggested by the synthetic experiment. Outside this range, using a smaller step-size makes the worse, presumably due to floating point errors. For implicit models, the test likelihood is computed with annealed importance sampling, using 1,000 intermediate distributions, following . The transition operator in AIS is HMC for Euclidean-space latents, and Riemannian LD for hyperspherical latents. The training setup follows from : for all methods, we train for 100,000 iterations using RMSProp use a batch size of 128, and a learning rate of 10 −3. WAE Experiment on MNIST For our method, the energy network is parameterized in the same way as in the VAE experiments. When training the energy network, we use a step-size of 10 −4, and apply L2 regularization on the energy scale with coefficient 10 −5. For the WAE-GAN baseline, we use the Wasserstein GAN , and parameterize its critic as a feed-forward network with 2 hidden layers, each with 256 units. We experimented with both the standard parameterization (i.e. put a linear layer after the last hidden layer that outputs scalar) as well as the score-network-like parameterization used in our energy network, and found the to be similar. We use tanh activation, apply spectral normalization and a L2 regularization with coefficient 10 −4. The rest of the training setup follows from : training for 100,000 iterations using RMSProp, a batch size of 128, and a learning rate of 10 −3. The Lagrange multiplier hyperparameter λ in the WAE objective is fixed at 10. The energy network is parameterized in the same way as in . For our method, we use a step-size of 10 −4. For the GAN baseline, we use the standard parameterization for the critic, i.e. the final linear layer outputs a scalar; the previous layers follow the same architecture of ours. In both methods we use a L2 regularization with coefficient 10 −5. Following , we train for 100,000 iterations, using RMSProp and a learning rate of 10 −4. FID scores are calculated using the implementation in . Notations In this section, let the parameter space be d-dimensional, and define While in the main text, we identified the tangent space of P(X) as a subspace of L 2 (ρX → R d) for clarity, here we use the equivalent definition T ρ (P(X)):= {s ∈ L 2 (ρX → R): E ρ s = 0} following . The two definition are connected by the transform In this section, we give a formal derivation of SPOS as the gradient flow of the KL divergence functional, with respect to a new metric. Recall the SPOS sampler targeting distribution (with density) φ corresponds to the following density evolution: ∂ t ρ t = −∇ · (ρ t (x) (φ * ρt,φ (x) + α∇ log(φ/ρ)) νt(x) ) where α > 0 is a hyperparameter, and φ * ρt,φ (x):= E ρt(x) (S φ ⊗ k)(x, x):= E ρt(x) [(∇ x log φ(x))k(x, x) + ∇ x k(x, x)] is the SVGD update direction . Fix ρ, define the integral operator and define the tensor product operator K ⊗d which we will derive shortly at the end of this subsection. Subsequently, we have The rest of our derivation follows : consider the function space H ρ,α:= {(αId + K ⊗d ρt)[∇h]}, where h: X → R is any square integrable and differentiable function. It connects to the tangent space of P(X) if we consider s = −∇ · (ρp) for anỹ p ∈ H ρ,α. Define on H ρ,α the inner product f, g Hρ,α:= f, (αId + K It then determines a Riemannian metric on the function space. Forp ∈ H ρ,α and s = −∇·(ρp), by we have ν t,p Hρ,α = E ρt(x) ∇ log(φ/ρ t)(x),p(x) = − log φ ρ t (∇ · (pρ))dx = −(dKL φ)(s), i.e. with respect to the metric, SPOS is the gradient flow of the (negative) KL divergence functional. Derivation of let (λ i, ψ i) ∞ i=1 be its eigendecomposition (i.e. the Mercer representation). For j ∈ [d] let ψ i,j:= ψ i e j where {e j} d j=1 is the coordinate basis in R d, so {λ −1/2 i ψ i,j} becomes an orthonormal basis in H ⊗d. Now we calculate the coordinate of φ * ρ,φ in this basis. S φ is known to satisfy the Stein's identity for all g ∈ H. Thus, we can subtract E ρ S ρ (K ρ [ψ i,j]) from the right hand side of without changing its value, and it becomes As the equality holds for all i, k, we completed the derivation of. By and, the MVL objective derived from SPOS is Hρ,α = ∇ log(φ/ρ t), (αId + K ⊗d)∇ log(φ/ρ t) L 2 (ρX →R d). In the right hand side above, the first term in the summation is the Fisher divergence, and the second is the kernelized Stein discrepancy (b, Definition 3.2). We note that a similar for SVGD has been derived in , and our derivations connect to the observation that Langevin dynamics can be viewed as SVGD with a Dirac function kernel (thus SPOS also corresponds to SVGD with generalized-function-valued kernels). In this section we derive, when the latent-space distribution q φ (z) is defined on a pdimensional manifold embedded in some Euclidean space, and H[q φ (z)] is the relative entropy w.r.t. the Hausdorff measure. The derivation is largely similar to the Euclidean case, and we only include it here for completeness. holds because where (i) follows from Theorem 2.10.10 in , and (ii) follows from the same theorem as well as the fact that E q φ (z) [∇ φ log q φ (z)] = ∇ φ q φ (z)dz = 0.

represent the short summary for finding the corresponding scientific article (long)

We present a scalable approximation to a wide range of EBM objectives, and applications in implicit VAEs and WAEs

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question:
represent the text to find a very short summary (1-2 sentences max)

E.g. During the last years, a remarkable breakthrough has been made in AI domain thanks to artificial deep neural networks that achieved a great success in many machine learning tasks in computer vision, natural language processing, speech recognition, malware detection and so on. However, they are highly vulnerable to easily crafted adversarial examples. Many investigations have pointed out this fact and different approaches have been proposed to generate attacks while adding a limited perturbation to the original data. The most robust known method so far is the so called C&W attack. Nonetheless, a countermeasure known as fea- ture squeezing coupled with ensemble defense showed that most of these attacks can be destroyed. In this paper, we present a new method we call Centered Initial Attack (CIA) whose advantage is twofold: first, it insures by construc- tion the maximum perturbation to be smaller than a threshold fixed beforehand, without the clipping process that degrades the quality of attacks. Second, it is robust against recently introduced defenses such as feature squeezing, JPEG en- coding and even against a voting ensemble of defenses. While its application is not limited to images, we illustrate this using five of the current best classifiers on ImageNet dataset among which two are adversarialy retrained on purpose to be robust against attacks. With a fixed maximum perturbation of only 1.5% on any pixel, around 80% of attacks (targeted) fool the voting ensemble defense and nearly 100% when the perturbation is only 6%. While this shows how it is difficult to defend against CIA attacks, the last section of the paper gives some guidelines to limit their impact. Since the skyrocketing of data volumes and parallel computation capacities with GPUs during the last years, deep neural networks (DNN) have become the most effective approaches in solving many machine learning problems in several domains like computer vision, speech recognition, games playing etc. They are even intended to be used in critical systems like autonomous vehicle BID17, BID18. However, DNN as they are currently built and trained using gradient based methods, are very vulnerable to attacks a.k.a. adversarial examples BID1. These examples aim to fool a classifier to make it predict the class of an input as another one, different from the real class, after bringing only a very limited perturbation to this input. This can obviously be very dangerous when it comes to systems where human life is in stake like in self driven vehicles. Companies IT networks and plants are also vulnerable if DNN based intrusion detection systems were to be deployed BID20.Many approaches have been proposed to craft adversarial examples since the publication by Szegedy et al. of the first paper pointing out DNN vulnerability issue BID4. In their work, they generated adversarial examples using box-constrained L-BFGS. Later in BID1, a fast gradient sign method (FGSM) that uses gradients of a loss function to determine in which direction the pixels intensity should be changed is presented. It is designed to be fast not optimize the loss function. Kurakin et al. introduced in BID13 a straightforward simple improvement of this method where instead of taking a single step of size in the direction of the gradient-sign, multiple smaller steps are taken, and the is clipped in the end. Papernot et al. introduced in BID3 an attack, optimized under L0 distance, known as the Jacobian-based Saliency Map Attack (JSMA). Another simple attack known as Deepfool is provided in. It is an untargeted attack technique optimized for the L2 distance metric. It is efficient and produces closer adversarial examples than the L-BFGS approach discussed earlier. Evolutionary algorithms are also used by authors in BID14 to find adversarial example while maintaining the attack close to the initial data. More recently, Carlini and Wagner introduced in BID0 the most robust attack known to date as pointed out in BID5. They consider different optimization functions and several metrics for the maximum perturbation. Their L2-attack defeated the most powerful defense known as distillation BID7. However, authors in BID6 showed that feature squeezing managed to destroy most of the C&W attacks. Many other defenses have been published, like adversarial training BID3, gradient masking BID9, defenses based on uncertainty using dropout BID10 as done with Bayesian networks, based on statistics BID11, BID12, or principal components BID22, BID23. Later, while we were carrying out our investigation, paper BID16 showed that not less than ten defense approaches, among which are the previously enumerated defenses, can be defeated by C&W attacks. It also pointed out that feature squeezing also can be defeated but no thorough investigation actually was presented. Another possible defense but not investigated is based on JPEG encoding when dealing with images. It has never been explicitly attacked even after it is shown in BID13 that most attacks are countered by this defense. Also, to our knowledge, no investigation has been conducted when dealing with ensemble defenses. Actually, attacks transferability between models that is well investigated and demonstrated in BID19 in the presence of an oracle (requesting defense to get labels to train a substitute model) is not guaranteed at all when it is absent. Finally, when the maximum perturbation added to the original data is strictly limited, clipping is needed at the end of training (adversarial crafting) even if C&W attacks are used. The quality of crafted attacks is therefore degraded as the brought perturbation during the training is brutally clipped. We tackle all these points in our work while introducing a new attack we call Centered Initial Attack (CIA). This approach considers the perturbation limits by construction and consequently no alteration is done on the CIA ing examples. To make it clearer for the reader, an example is given below to illustrate the clipping issue. FIG0 shows a comparison between CIA and C&W L2 attack before and after clipping on an example, a guitar targeted as a potpie with max perturbation equal to 4.0 (around 1.5%). The same number of iterations FORMULA4 is considered for both methods. As can be seen on FIG0, CIA generates the best attack with 96% confidence. C&W is almost as good with a score of 95% but it is degraded to 88% after applying the clipping to respect the imposed max perturbation. Avoiding this degradation due to clipping is the core motivation of our investigation. The remaining of this paper is organized as follows. Section I presents some mathematical formulations and the principle of CIA strategy. Then Section II investigates the application of CIA against ensemble defense, feature squeezing and JPEG encoding defenses. Then Section III provides some guidelines to limit the impact of CIA attacks. Finally, we give some possible future investigations in the . Before entering into details of CIA, let us give some useful formulations. A neural network can be seen as a function F (x) = y that accepts an input x and produces an output y. The function F depends actually on some model parameters often called weights an biases. These are the variables that are adjusted during the learning process to fit the training data on one hand and generalize well to unseen data on the other hand. Since they do not change in our models, we omit them in our notations. The input x can be a vector or an array of any dimension. So, without loss of generality, we consider x ∈ n as it can be flattened in any case. So, the i th component of x is noted x i with integer i ∈ [1, n].Since we consider m-class classifiers, the output is calculated using the sof tmax function. The output y = F (x) can be seen as a vector of m probabilities p j with j ∈ [1, m]. The component with the biggest value gives the predicted class C(x). This can be written as: DISPLAYFORM0 We note the output corresponding to the correct class as C c (x). An adversarial examplex is crafted as a non targeted attack so as to get C(x) = C c (x) or a targeted one to get C(x) = t where t is the target class. Crafting an example can then be formulated using a loss function L(F, x) to maximize the probability of getting a class different from the correct one. Cross entropy is used in our work. The adversarial examplex can be written asx = x + δ where δ is the added perturbation. In our work we constrain δ to be within domain [−∆, ∆] as considered for instance in Google Brain-Kaggle competition, ∆ being the maximum perturbation. With the existing approaches, adversarial examples are generated through some iterations then clipped in the end to respect this constraint. With C&W attacks for instance, the loss function includes a norm term δ to minimize perturbation δ but the clipping is still needed as we saw above. The main idea behind Centered Initial Attack is to find for each component i the center x * i of the domain in whichx i is allowed to be (green segment on Figure. 2). This is not trivial since we have to insure at the same time the componentx i to be within another domain [α i, β i] to be valid, α i and β i are respectively the minimum and maximum values that can be taken by the i To find the center of domain definition ofx, three cases are to be considered actually, not four since ∆ i is much smaller than (β i − α i), as can be seen on FIG2. For recall, ∆ i is the i th component of ∆.The three cases are: DISPLAYFORM1 DISPLAYFORM2 Now if we consider a continuous differentiable function g such that: g: → [−1, +1], then we can write every componentx i as: DISPLAYFORM3 This equation can be rewritten using arrays as: DISPLAYFORM4 where operator is the elementwise product and r is a new variable on which we optimize the loss function. Finally, the loss function can be written as: DISPLAYFORM5 No constraint is to be considered on the variable r since it is well defined in domain (−∞, +∞). Any initialization of r is possible but we consider it as zero for simplicity. So, the initial attackx is therefore different from x whereas it is centered in its domain of definition (green segment). This explains the CIA attack name. Regarding g, many continuous functions can be used. For instance we tried three functions: DISPLAYFORM6 Obviously other functions can be considered. In our experiments, as they all lead to similar , we always considered tanh. It is interesting to note that with CIA, we can define a different maximum perturbation from a component x i to another x j. Likewise, it is easy for instance to limit the crafting to only a portion of an image by considering a zero max perturbation on the other regions, without changing anything in the training algorithm. This is an advantage with regard to existing approaches as it is difficult with the current machine learning frameworks to select from the same array only some variables to optimize on. Gradients masking would be a solution but not desired as it is a clipping operation. An example of such partial crafting is displayed on Figure. 3 where only a 50px band on the top and right sides is modified. We generated it using ∆ = 32 to make the difference visible on the paper but the image (spider) is also classified as the target (dog) even with smaller values. Also, any gradient descent optimizer can be used to craft attacks as the case with BID0. Adam BID15 turns out to be the fastest in our experiments. We used it for training and considered 20 iterations, a good compromise between computation time and attacks crafting convergence, for all the adversarial examples crafting. To reproduce the , the Adam hyperparameters to be considered are {learning rate = 0.2, DISPLAYFORM0 Finally, it is worth noting that CIA is not limited to images. It can be used for any type of data with bounded continuous features. In order to check the effectiveness of CIA attacks, we consider mainly targeted attacks as they are more difficult to craft against three different strategies of defense; ensemble defense with many classifiers, feature squeezing, and JPEG encoding. A combination of these defenses is also considered as we will see later. In this paper, we consider only white box attacks where we have full access to defense models parameters. Other works BID19 pointed out the transferability property between models when it is possible to quest the defense classifier and get the labels back to train a substitute model to be used for crafting attacks. When it is the case, an attack generated using the substitute is likely to remain an attack on the defense. When it is not the case however, this transferability is inexistent as we will see below. So, attacking as many models as possible at once is required. In order to check the robustness of CIA in attacking many classifiers at once, we consider the five best classifiers on ImageNet dataset : Inception V3 (IncV3 a), Inception V4 (IncV4), InceptionResnet V2 (IncRes a), adversarialy trained Inception V3(IncV3 b) and adversarialy InceptionResnet V2 (IncRes b). The accuracy of these classifiers is around 80% on the whole ImageNet dataset. The accuracies on 1000 images dataset we consider are showed in TAB0. In this experiment, we attack IncV3 b classifier and present the success rate of targeted attacks and the miss-classification rate of each classifier. The are showed in TAB1. As can be seen on TAB1, the transferability is inexistent whatever the maximum perturbation used when looking at the targeted attacks success rate. However, a small increase in misclassification rate is noticed especially with IncV3 a, raising from 3.9% to 7%. This was verified when attacking any other classifier alone and checking the impact on the others. This demonstrates clearly the need to attack many classifiers at once. To do so, we considered an optimization using a sum of losses, each loss being related to one classifier: DISPLAYFORM0 where F i is the function relative to the i th classifier. A weighed version can be considered to target a classifier more aggressively than another. In our experiments, they are all attacked equally. The are displayed in TAB2. As we can see on TAB2, the success rate of the targeted attacks against the voting ensemble is high, around 80% for ∆ = 4.0 and approaching 100% for ∆ = 16.0. It is also interesting to notice that the success rate of attacking IncV3 b has decreased compared to the case when it was attacked alone. With ∆ = 4.0 for instance, it went from 97.6% to 92,5%. This can be explained by the fact that the gradients are balanced in a way to change the input in a direction that minimizes all the losses at the same time. As a of this section, attacking an ensemble defense using CIA is effective when we have complete access to all defense models (white box attacks). Section 3.3 will show that the transferability to an anknown model, while attacking four among the five, is limited but not negligible (more than 30%). This defense approach has been developed to counter attacks using smoothing filters BID6. The intuition behind this idea was that smoothing removes the sharp changes brought while crafting adversarial examples. There are different feature squeezing possibilities but we consider spatial smoothing in the current study. The other ones will be addressed in future experiments. While adding a filter, one should care about the possible loss of accuracy of the defense classifier. Authors in BID6 showed that a 3x3 filter is a good compromise that gives an effective defense while limiting the loss of accuracy. We noticed it too in the current investigation and therefore considered this kernel size. Also, different smoothing strategies are possible like Gaussian, diagonal, mean, etc. As the are quite similar in our experiments we carried out, we consider the mean filter in the sequel for its calculation simplicity. The filter used for defense can actually be replaced by a convolution layer before the neural network as shown in Figure. 4. Figure 4: spatial smoothing modeling using a convolution layer. As can be seen on Figure 4, adding a convolution layer in a new network that could be represented using a new function F. We can therefore simply craft an adversarial example using this new function. As a start, we conducted an attack against only one network (IncV3 a). The are shown in TAB3. As we can notice in TAB3, the success rate of targeted attacks and the miss-classification rate are nearly 100%. This means that the spatial smoothing based defense is not effective. Once again, the transferability between models is inexistent. An interesting point is to check the success of the same attacks when the defense is not actually using spatial smoothing for sure. Said in other words, we suspect the defense to use spatial smoothing but we are not sure about it. So, we craft attacks as before for a guarantee. Or not! The are showed in TAB4.As we can clearly see, the attacks success rate depleted to only 3.2% and the misclassification rate to 18.2%. The attack is therefore not effective in case of filter based defense uncertainty. This disagrees with the intuition behind spatial smoothing as an efficient defense against attacks presented in BID6. As demonstrated indeed, the filtered adversarial example can be an effective attack but not the unfiltered one! Then, how to overcome this issue for a more robust attack? To answer this question, we consider a hybrid network where both filtered and non filtered inputs are used for optimization as represented on Figure. 5. The loss function to be used is given as a sum of two terms (weighing would be useful for more robustness) as follows: DISPLAYFORM0 where a, b are real positive numbers. We conducted a new experiment using this hybrid loss function and the are given in TAB5. 6.3% TAB5 shows that the success rate is this time very high in both cases: 98.4% in filter based defense and nearly 100% in no filter defense. We conclude that this attack is robust whether filtering is used or not in defense. Another question arises from previous given the lack of transferability of attacks between models. What if an ensemble defense is used and filter use is uncertain? Once again, we consider the sum of all losses, but use the hybrid losses this time as follows: DISPLAYFORM1 where L Hi is the hybrid loss relative to the i th classifier. The are presented in TAB6 Even with ∆ = 4.0 and considering only targeted attacks, the success and miss-classification rates are high, around 50% when all classifiers use filters and much higher (around 80%) when no filters are used. Other experiments we conducted showed that attacks rate for filter based defense can be improved by assigning a greater weight b (twice the weight a) in the hybrid loss equation FORMULA9. An investigation conducted by authors in BID13 showed that most adversarial examples are countered if they are JPEG encoded before classifying them. Lets check if it is the case with CIA attacks. We conducted an attack against IncV3 a and classified the adversarial examples after being JPEG encoded. The encoding uses different compression quality values Q. A higher Q means a better quality of image with a bigger size however since it undergoes less loss and compression. The are displayed on TAB7. TAB7 shows that CIA is robust when performing non targeted attacks as almost 100% of them are successful with Q = 80 and around 50% with Q = 20. The targeted attacks are less successful with the highest score of 20% when Q = 80 and 0% when Q = 20.The is somehow mitigated with regard to targeted attaks . Indeed, one has to keep in mind that a Q = 20 would not be a reasonable defense as this will degrade highly the accuracy of the classifier. Nonetheless, we tried to improve the attacks success score by finding a suitable approximation JPEG transformations. Obviously, JPEG encoding cannot be modeled accurately using a differentiable function that can be included in crafting examples process as we did before with spatial smoothing. As a brief recall, this encoding implies the passage from RGB space to another color space called YCbCr where Y is brightness, Cb and Cr components represent the chrominance. Actually, humans can see considerably more fine detail in the brightness of an image (the Y component) than in the hue and color saturation of an image (the Cb and Cr components). Considering this fact, Cb and Cr can be downsampled by a factor of 2 or 3 without sensitive change of receptivity to human eye. Another fact is the eye not being sensitive to sharp changes in images. So, removing the high frequencies from the spectral space after a DCT (Discrete Cosine Transform) of an image would not affect its quality remarkably too. These are the important facts used when making a JPEG compression. Other steps like dividing the image into blocks, the quantization of frequency components, the encoding of these components and so on are thoroughly well documented for interested readers BID21. We do not take them into account. Our idea for approximating JPEG is represented in FIG5.As shown on FIG5, we first transform RGB images to YCrCb space using a function T (product an sum operations) then we filter each component using a convolution layer. Given the facts enumerated before about filtering high frequencies and down sampling the chrominance, we consider a mean 3x3 kernel for brightness component and a 6x6 kernel for Cr and Cb. Once filtered, the is brought back to RGB space using T −1 before feeding the neural network. The of crafting attacks against IncV3 a using this approximation are given in TAB8. As can be noticed, the are almost the same as those of TAB7. This is a bit untriguing as if all the filters used in the approximation are inexistent. However, a similar remark as in the filterbased defense case can be made. The attacks crafted using JPEG encoding are no longer attacks if JPEG is not used by the defense. This means that the considered approximation of JPEG encoding is not that bad but not enough accurate to give strong attacks. It has obviously to be improved. We are working on it. As we saw previously, defending against attacks is hard as along as the defense can be modeled using a function that could be added to form a new network that can be used for crafting examples. As a , one has to find a transformation that is hardly represented or approximated using simple functions. This is obviously not trivial as we saw with JPEG defense. Another, and more short term realizable defense, is to consider a big number of well performing classifiers including limited accuracy variance transformations like spatial smoothing. As we saw in TAB6, the success rate decreased to around 50% when using five classifiers including filters for defense. This is not guaranteed but it worth being investigated. For instance, when we attack all the classifiers except IncV4, the transferability is somehow limited. Indeed, the success rate of attacks is almost 100% against the attacked classifiers whereas it is only around 34% for IncV4 as can be noticed on TAB0. In this paper we presented a new strategy called CIA for crafting adversarial examples while insuring the maximum perturbation added to the original data to be smaller than a fixed threshold. We demonstrated also its robustness against some defenses, feature squeezing, ensemble defenses and even JPEG encoding. For future work, it would be interesting to investigate the transferability of CIA attacks to the physical world as it is shown in BID13 that only a very limited amount of FGDM attacks, around 20%, survive this transfer. Another interesting perspective is to consider partial crafting attacks while selecting regions taking into account the content of the data. With regard to images for instance, this would be interesting to hide attacks with big but imperceptible perturbations. ==  In this paper, a new method we call Centered Initial Attack (CIA) is provided. It insures by construction the maximum perturbation to be smaller than a threshold fixed beforehand, without the clipping process. != This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.

Question: This paper develops variational continual learning (VCL), a simple but general framework for continual learning that fuses online variational inference (VI) and recent advances in Monte Carlo VI for neural networks. The framework can successfully train both deep discriminative models and deep generative models in complex continual learning settings where existing tasks evolve over time and entirely new tasks emerge. Experimental show that VCL outperforms state-of-the-art continual learning methods on a variety of tasks, avoiding catastrophic forgetting in a fully automatic way. Continual learning (also called life-long learning and incremental learning) is a very general form of online learning in which data continuously arrive in a possibly non i.i.d. way, tasks may change over time (e.g. new classes may be discovered), and entirely new tasks can emerge BID43 BID47 BID39. What is more, continual learning systems must adapt to perform well on the entire set of tasks in an incremental way that avoids revisiting all previous data at each stage. This is a key problem in machine learning since real world tasks continually evolve over time (e.g. they suffer from covariate and dataset shift) and the size of datasets often prohibits frequent batch updating. Moreover, practitioners are often interested in solving a set of related tasks that benefit from being handled jointly in order to leverage multi-task transfer. Continual learning is also of interest to cognitive science, being an intrinsic human ability. The ubiquity of deep learning means that it is important to develop deep continual learning methods. However, it is challenging to strike a balance between adapting to recent data and retaining knowledge from old data. Too much plasticity leads to the infamous catastrophic forgetting problem BID34 BID36 BID13 and too much stability leads to an inability to adapt. Recently there has been a resurgence of interest in this area. One approach trains individual models on each task and then carries out a second stage of training to combine them BID28. A more elegant and more flexible approach maintains a single model and uses a single type of regularized training that prevents drastic changes in the parameters which have a large influence on prediction, but allows other parameters to change more freely BID29 BID26 BID50. The approach developed here follows this venerable work, but is arguably more principled, extensible and automatic. This paper is built on the observation that there already exists an extremely general framework for continual learning: Bayesian inference. Critically, Bayesian inference retains a distribution over model parameters that indicates the plausibility of any setting given the observed data. When new data arrive, we combine what previous data have told us about the model parameters (the previous posterior) with what the current data are telling us (the likelihood). Multiplying and renormalizing yields the new posterior, from which point we can recurse. Critically, the previous posterior constrains parameters that strongly influence prediction, preventing them from changing drastically, but it allows other parameters to change. The wrinkle is that exact Bayesian inference is typically intractable and so approximations are required. Fortunately, there is an extensive literature on approximate inference for neural networks. We merge online variational inference (VI) BID11 BID42 BID4 with Monte Carlo VI for neural networks BID3 to yield variational continual learning (VCL). In addition, we extend VCL to include a small episodic memory by combining VI with the coreset data summarization method BID0 BID19. We demonstrate that the framework is general, applicable to both deep discriminative models and deep generative models, and that it yields excellent performance. Consider a discriminative model that returns a probability distribution over an output y given an input x and parameters θ, that is p(y|θ, x). Below we consider the specific case of a softmax distribution returned by a neural network with weight and bias parameters, but we keep the development general for now. In the continual learning setting, the goal is to learn the parameters of the model from a set of sequentially arriving datasets {x DISPLAYFORM0 where, in principle, each might contain a single datum, N t = 1. Following a Bayesian approach, a prior distribution p(θ) is placed over θ. The posterior distribution after seeing T datasets is recovered by applying Bayes' rule: DISPLAYFORM1 Here the input dependence has been suppressed on the right hand side to lighten notation. We have used the shorthand D t = {y DISPLAYFORM2 . Importantly, a recursion has been identified whereby the posterior after seeing the T -th dataset is produced by taking the posterior after seeing the (T − 1)-th dataset, multiplying by the likelihood and renormalizing. In other words, online updating emerges naturally from Bayes' rule. In most cases the posterior distribution is intractable and approximation is required, even when forming the first posterior p(θ|D 1) ≈ q 1 (θ) = proj(p(θ)p(D 1 |θ)). Here q(θ) = proj(p * (θ)) denotes a projection operation that takes the intractable un-normalized distribution p * (θ) and returns a tractable normalized approximation q(θ). The field of approximate inference provides several choices for the projection operation including i) Laplace's approximation, ii) variational KL minimization, iii) moment matching, and iv) importance sampling. Having approximated the first posterior distribution, subsequent approximations can be produced recursively by combining the approximate posterior distribution with the likelihood and projecting, that is p(θ|D 1:T) ≈ q T (θ) = proj(q T −1 (θ)p(D T |θ)). In this way online updating is supported. This general approach leads, for the four projection operators previously identified, to i) Laplace propagation BID46, ii) online VI BID11 BID42 ) also known as streaming variational Bayes BID4, iii) assumed density filtering BID33 and iv) sequential Monte Carlo BID30. In this paper the online VI approach is used as it typically outperforms the other methods for complex models in the static setting ) and yet it has not been applied to continual learning of neural networks. Variational continual learning employs a projection operator defined through a KL divergence minimization over the set of allowed approximate posteriors Q, DISPLAYFORM0 The zeroth approximate distribution is defined to be the prior, q 0 (θ) = p(θ). Z t is the intractable normalizing constant of p * t (θ) = q t−1 (θ) p(D t |θ) and is not required to compute the optimum. VCL will perform exact Bayesian inference if the true posterior is a member of the approximating family, p(θ|D 1, D 2, . . ., D t) ∈ Q at every step t. Typically this will not be the case and we might worry that performing repeated approximations may accumulate errors causing the algorithm to forget old tasks, for example. Furthermore, the minimization at each step may also be approximate (e.g. due to employing an additional Monte Carlo approximation) and so additional information may be lost. In order to mitigate this potential problem, we extend VCL to include a small representative set of data from previously observed tasks that we call the coreset. The coreset is analogous to an episodic memory that retains key information (in our case, important training data points) from previous tasks which the algorithm can revisit in order to refresh its memory of them. The use of an episodic memory for continual learning has also been explored by BID31. Input: Prior p(θ). Output: Variational and predictive distributions at each step {qt(θ), p(y * |x *, D1:t)} T t=1. Initialize the coreset and variational approximation: C0 ← ∅,q0 ← p. for t = 1... T do Observe the next dataset Dt. Ct ← update the coreset using Ct−1 and Dt. Update the variational distribution for non-coreset data points: DISPLAYFORM0 Compute the final variational distribution (only used for prediction, and not propagation): DISPLAYFORM1 Perform prediction at test input x *: p(y * |x *, D1:t) = qt(θ)p(y * |θ, x *)dθ. end for Algorithm 1 describes coreset VCL. For each task, the new coreset C t is produced by selecting new data points from the current task and a selection from the old coreset C t−1. Any heuristic can be used to make these selections, e.g. K data points can be selected at random from D t and added to C t−1 to form an unbiased new coreset C t. Alternatively, the greedy K-center algorithm BID12 can be used to return K data points that are guaranteed to be spread throughout the input space. Next, a variational recursion is developed. Bayes' rule can be used to decompose the true posterior taking care to break out contributions from the coreset, DISPLAYFORM2 Here the variational distributionq t (θ) approximates the contribution to the posterior from the noncoreset data points. A recursion is identified by noting DISPLAYFORM3 Hence propagation is performed viaq t (θ) = proj(q t−1 (θ)p(D t ∪ C t−1 \ C t |θ)) with VCL employing the variational KL projection. A further projection step is needed before performing prediction q t (θ) = proj(q t (θ)p(C t |θ)). In this way the coreset is incorporated into the approximate posterior directly before prediction which helps mitigate any residual forgetting. From a more general perspective, coreset VCL is equivalent to a message-passing implementation of VI in which the coreset data point updates are scheduled after updating the other data. The VCL framework is general and can be applied to many discriminative probabilistic models. Here we apply it to continual learning of deep fully-connected neural network classifiers. Before turning to the application of VCL, we first consider the architecture of neural networks suitable for performing continual learning. In simple instances of discriminative continual learning, where data are arriving in an i.i.d. way or where only the input distribution p(x 1:T) changes over time, a standard single-head discriminative neural network suffices. In many cases the tasks, although related, might involve different output variables. Standard practice in multi-task learning BID1 uses networks that share parameters close to the inputs but with separate heads for each output, hence multi-head networks. Graphical models depicting the network architecture for deep discriminative and deep generative models are shown in FIG0. Recent work has explored more advanced structures for continual learning BID40 and multi-task learning more generally BID48 BID37. These architectural advances are complementary to the new learning schemes developed here and a synthesis of the two would be potentially more powerful. Moreover, a general solution to continual learning would perform automatic continual model building adding new bespoke structure to the existing model as new tasks are encountered. Although this is a very interesting research direction, here we make the simplifying assumption that the model structure is known a priori. VCL requires specification of q(θ) where θ in the current case is a D dimensional vector formed by stacking the network's biases and weights. For simplicity we use a Gaussian mean-field approximate posterior DISPLAYFORM0 Taking the most general case of a multi-head network, before task k is encountered the posterior distribution over the associated head parameters will remain at the prior and so q(θ DISPLAYFORM1 . This is convenient as it means the variational approximation can be grown incrementally, starting from the prior, as each task emerges. Moreover, only tasks present in the current dataset D t need to have their posterior distributions over head parameters updated. The shared parameters, on the other hand, will be constantly updated. Training the network using the VFE approach in eq. is equivalent to maximizing the negative online variational free energy or the variational lower bound to the online marginal likelihood DISPLAYFORM2 with respect to the variational parameters DISPLAYFORM3. Whilst the KL-divergence KL(q t (θ)||q t−1 (θ)) can be computed in closed-form, the expected log-likelihood requires further approximation. Here we take the usual approach of employing simple Monte Carlo and use the local reparameterization trick to compute the gradients BID41 BID23. At the first time step, the prior distribution, and therefore q 0 (θ) is chosen to be a multivariate Gaussian distribution (see e.g. BID15 BID3 . Deep generative models (DGMs) have garnered much recent attention. By passing a simple noise variable (e.g. Gaussian noise) through a deep neural network, these models have been shown to be able to generate realistic images, sounds and videos sequences BID8 BID25 BID49. Standard approaches for learning DGMs have focused on batch learning, i.e. the observed instances are assumed to be i.i.d. and are all available at the same time. In this section we extend the VCL framework to encompass variational auto-encoders (VAEs) BID23 BID38, a form of DGM. The approach could be extended to generative adversarial networks (GANs) BID14 for which continual learning is an open problem (see BID44 for an initial attempt).Consider a model p(x|z, θ)p(z), for observed data x and latent variables z. The prior over latent variables p(z) is typically Gaussian, and the distributional parameters of p(x|z, θ) are defined by a deep neural network. For example, if Bernoulli likelihood is used, then p(x|z, θ) = Bern(x; f θ (z)), where f θ denotes the deep neural network transform and θ collects all the weight matrices and bias vectors. In the batch setting, given a dataset DISPLAYFORM0, the standard VAE approach learns the parameters θ by approximate maximum likelihood estimation (MLE). This proceeds by maximizing the variational lower bound with respect to θ and φ: DISPLAYFORM1 where φ are the variational parameters of the approximate posterior or "encoder".The approximate MLE approach is unsuitable for the continual learning setting as it does not return parameter uncertainty estimates that are critical for weighting the information learned from old data. So, instead the VCL approach will approximate the full posterior distribution over parameters, q t (θ) ≈ p(θ|D 1:t), after observing the t-th dataset. Specifically, the approximate posterior q t is obtained by maximizing the full variational lower bound with respect to q t and φ: DISPLAYFORM2 where the encoder network q φ (z DISPLAYFORM3 is parameterized by φ which is task-specific. It is likely to be beneficial to share (parts of) these encoder networks, but this is not investigated in this paper. As was the case for multi-head discriminative models, we can split the generative model into shared and task-specific parts. There are two options: (i) the generative models share across tasks the network that generates observations x from the intermediate-level representations h, but have private "head networks" for generating h from the latent variables z (see FIG0), and (ii) the other way around. Architecture (i) is arguably more appropriate when data are composed of a common set of structural primitives (such as strokes in handwritten digits) that are selected by high level variables (character identities). Moreover, initial experiments on architecture (ii) indicated that information about the current task tended to be encoded entirely in the task-specific lower-level network negating multi-task transfer. For these reasons, we focus on architecture (i) in the experiments. Continual Learning for Deep Discriminative Models: Many neural network continual learning approaches employ regularized maximum likelihood estimation, optimizing objectives of the form: DISPLAYFORM0. Here the regularization biases the new parameter estimates towards those estimated at the previous step θ t−1. λ t is a user-selected hyper-parameter that controls the overall contribution from previous data and Σ t−1 is a matrix (normally diagonal in form) that encodes the relative strength of the regularization on each element of θ. We now discuss specific instances of this scheme:• Maximum-likelihood estimation and MAP estimation: maximum likelihood estimation is recovered when there is no regularization (λ t = 0). More generally, the regularization term can be interpreted as a Gaussian prior, q(θ|D 1:t−1) = N (θ; θ t−1, Σ t−1 /λ t). The optimization returns the maximum a posteriori estimate of the parameters, but this does not directly provide Σ t for the next stage. A simple fix is to set Σ t = I and use cross-validation to find λ t, but this approximation is often coarse and leads to catastrophic forgetting BID13 BID26 ).• Laplace Propagation (LP) BID46: applying Laplace's approximation at each step leads to a recursion for Σ −1 t, which is initialized using the covariance of the Gaussian prior, Σ −1 DISPLAYFORM1 To avoid computing the full Hessian of the likelihood, diagonal Laplace propagation retains only the diagonal terms of Σ −1 t.Published as a conference paper at ICLR 2018• Elastic Weight Consolidation (EWC) BID26 builds on diagonal Laplace propagation by approximating the average Hessian of the likelihoods using well-known identities for the Fisher information: DISPLAYFORM2 EWC also modifies the Laplace regularization, DISPLAYFORM3, introducing hyper-parameters, removing the prior and regularizing to intermediate parameter estimates, rather than just those derived from the last task, DISPLAYFORM4 These changes may be unnecessary BID20 BID21 and require storing θ 1:t−1, but may slightly improve performance (see BID27 and our experiments).• Synaptic Intelligence (SI) BID50: SI computes Σ −1 t using a measure of the importance of each parameter to each task. Practically, this is achieved by comparing the changing rate of the gradients of the objective and the changing rate of the parameters. VCL differs from the above methods in several ways. First, unlike MAP, EWC and SI, it does not have free parameters that need to be tuned on a validation set. This can be especially awkward in the online setting. Second, although the KL regularization penalizes the mean of the approximate posterior through a quadratic cost, a full distribution is retained and averaged over at training time and at test time. Third, VI is generally thought to return better uncertainty estimates than approaches like Laplace's method and MAP estimation, and we have argued this is critical for continual learning. There is a long history of research on approximate Bayesian training of neural networks, including extended Kalman filtering BID45 ), Laplace's approximation BID32, variational inference BID18 BID2 BID15 BID3 BID10, sequential Monte Carlo BID9, expectation propagation (EP) BID16, and approximate power EP. These approaches have focused on batch learning, but the framework described in section 2 enables them to be applied to continual learning. On the other hand, online variational inference has been previously explored BID11 BID4 BID6, but not for neural networks or in the context of sets of related complex tasks. Continual Learning for Deep Generative Models: A naïve continual learning approach for deep generative models would directly apply the VAE algorithm to the new dataset D t with the model parameters initialized at the previous parameter values θ t−1. The experiments show that this approach leads to catastrophic forgetting, in the sense that the generator can only generate instances that are similar to the data points from the most recently observed task. Alternatively, EWC regularization can be added to the VAE objective: DISPLAYFORM5 However computing Φ t requires the gradient of the intractable marginal likelihood ∇ θ log p(x|θ). Instead, we can approximate the marginal likelihood by the variational lower bound, i.e. DISPLAYFORM6 Similar variational lower-bound approximations apply when computing the Hessian matrices for LP and Σ −1 t for SI. An importance sampling estimate could also be used BID7. The experiments evaluate the performance and flexibility of VCL through three discriminative tasks and two generative tasks. Standard continual learning benchmarks are used where possible. Comparisons are made to EWC, diagonal LP and SI that employ tuned hyper-parameters λ whereas VCL's objective is hyper-parameter free. More details of the experiment settings and an additional experiment are available in the appendix. We consider the following three continual learning experiments for deep discriminative models. Permuted MNIST: This is a popular continual learning benchmark BID13 BID26 BID50. The dataset received at each time step D t consists of labeled MNIST images whose pixels have undergone a fixed random permutation. We compare VCL to EWC, SI, and diagonal LP. For all algorithms, we use fully connected single-head networks with two hidden layers, where each layer contains 100 hidden units with ReLU activations. We evaluate three versions of VCL: VCL with no coreset, VCL with a random coreset, and VCL with a coreset selected by the K-center method. For the coresets, we select 200 data points from each task. FIG1 compares the average test set accuracy on all observed tasks. From this figure, VCL outperforms EWC, SI, and LP by large margins, even though they benefited from an extensive hyperparameter search for λ. Diagonal LP performs slightly worse than EWC both when λ = 1 and when the values of λ are tuned. After 10 tasks, VCL achieves 90% average accuracy, while EWC, SI, and LP only achieve 84%, 86%, and 82% respectively. The also show that the coresets perform poorly by themselves, but combining them with VCL leads to a modest improvement: both random coresets and K-center coresets achieve 93% accuracy. We also investigate the effect of the coreset size. In FIG3, we plot the average test set accuracy of VCL with random coresets of different sizes. At the coreset size of 5,000 examples per task, VCL achieves 95.5% accuracy after 10 tasks, which is significantly better than the 90% of vanilla VCL. Performance improves with the coreset size although it asymptotes for large coresets as expected: if a sufficiently large coreset is employed, it will be fully representative of the task and thus training on the coreset alone can achieve a good performance. However, the experiments show that the combination of VCL and coresets is advantageous even for large coresets. Split MNIST: This experiment was used by BID50 to assess the SI method. Five binary classification tasks from the MNIST dataset arrive in sequence: 0/1, 2/3, 4/5, 6/7, and 8/9. We use fully connected multi-head networks with two hidden layers comprising 256 hidden units with ReLU activations. We compare VCL (with and without coresets) to EWC, SI, and diagonal LP. For the coresets, 40 data points from each task are selected through random sampling or the K-center method. FIG4 compares the test set accuracy on individual tasks (averaged over 10 runs) as well as the accumulated accuracy averaged over tasks (right). As an upper bound on the algorithms' performance, we compare to batch VI trained on the full dataset. From this figure, VCL significantly outperforms EWC and LP although it is slightly worse than SI. Again, unlike VCL, EWC and SI benefited from a hyper-parameter search for λ, but a value close to 1 performs well in both cases. After 5 tasks, VCL achieves 97.0% average accuracy on all tasks, while EWC, SI, and LP attain 63.1%, 98.9%, and 61.2% respectively. Adding the coreset improves VCL to around 98.4% accuracy. This experiment is similar to the previous one, but it uses the more challenging notMNIST dataset and deeper networks. The notMNIST dataset 2 here contains 400,000 images of the characters from A to J with different font styles. We consider five binary classification tasks: A/F, B/G, C/H, D/I, and E/J using deeper networks comprising four hidden layers of 150 hidden units with ReLU activations. The other settings are kept the same as the previous experiment. VCL is competitive with SI and significantly outperforms EWC and LP (see FIG5), although the SI and EWC baselines benefited from a hyper-parameter search. VCL achieves 92.0% average accuracy after 5 tasks, while EWC, SI, and LP attain 71%, 94%, and 63% respectively. Adding the random coreset improves the performance of VCL to 96% accuracy. We consider two continual learning experiments for deep generative models: MNIST digit generation and notMNIST (small) character generation. In both cases, ten datasets are received in sequence. For MNIST, the first dataset comprises exclusively of images of the digit zero, the second dataset ones and so on. For notMNIST, the datasets contain the characters A to J in sequence. The generative model consists of shared and task-specific components, each represented by a one hidden layer neural network with 500 hidden units (see FIG0). The dimensionality of the latent variable z and the intermediate representation h are 50 and 500, respectively. We use task-specific encoders that are neural networks with symmetric architectures to the generator. We compare VCL to naïve online learning using the standard VAE objective, LP, EWC and SI (with hyper-parameters λ = 1, 10, 100). For full details of the experimental settings see Appendix E. Samples from the generative models attained at different time steps are shown in fig. 6. The naïve online learning method fails catastrophically and so numerical are omitted. LP, EWC, SI and VCL remember previous tasks, with SI and VCL achieving the best visual quality on both datasets. The algorithms are quantitatively evaluated using two metrics in FIG7: an importance sampling estimate of the test log-likelihood (test-LL) using 5, 000 samples and a measure of quality we term "classifier uncertainty". For the latter, we train a discriminative classifier for the digits/alphabets to achieve high accuracy. The quality of generated samples can then be assessed by the KL-divergence from the one-hot vector indicating the task, to the output classification probability vector computed on the generated images. A well-trained generator will produce images that are correctly classified in high confidence ing in zero KL. We only report the best performance for LP, EWC and SI.We observe that LP and EWC perform similarly, most likely due to the fact that both LP and EWC use the same Σ t matrices. EWC achieves significantly worse performance than SI. VCL is on par with or slightly better than SI. VCL has a superior long-term memory of previous tasks which leads to better overall performance on both metrics even though it does not have tuned hyper-parameters in its objective function. For MNIST, the performance of LP and EWC deteriorate markedly when moving from task "digit 0" to "digit 1" possibly due to the large task differences. Also for all experimental settings we tried, SI fails to produce high test-LL after task "digit 7". Future work will investigate continual learning on a sequence of tasks that follows "adversarial ordering", i.e. the ordering that makes the next task maximally different from the current task. Approximate Bayesian inference provides a natural framework for continual learning. Variational Continual Learning (VCL), developed in this paper, is an approach in this vein that extends online variational inference to handle more general continual learning tasks and complex neural network models. VCL can be enhanced by including a small episodic memory that leverages coreset algorithms from statistics and connects to message-scheduling in variational message passing. We demonstrated how the VCL framework can be applied to both discriminative and generative models. Experimental showed state-of-the-art performance when compared to previous continual learning approaches, even though VCL has no free parameters in its objective function. Future work should explore alternative approximate inference methods using the same framework and also develop more sophisticated episodic memories. Finally, we note that VCL is ideally suited for efficient model refinement in sequential decision making problems, such as reinforcement learning and active learning. DISPLAYFORM0 Figure 6: Generated images from each of the generators after training. Each of the columns shows the images generated from a specific task's generator, and each of the lines shows the generations from generators of all trained tasks. Clearly the naive approach suffers from catastrophic forgetting, while other approaches successfully remember previous tasks. In this experiment, we use fully connected single-head networks with two hidden layers, where each layer contains 100 hidden units with ReLU activations. The metric used for comparison is the test set accuracy on all observed tasks. We train all the models using the Adam optimizer BID22 with learning rate 10 −3 since we found that it works best for all models. All the VCL algorithms are trained with batch size 256 and 100 epochs. For all the algorithms with coresets, we choose 200 examples from each task to include into the coresets. The algorithms that use only the coresets are trained using the VFE method with batch size equal to the coreset size and 100 epochs. We use the prior N (0, I) and initialize our optimizer for the first task at the mean of the maximum likelihood model and a very small initial variance (10 −6).We compare the performance of SI with hyper-parameters λ = 0.01, 0.1, 0.5, 1, 2 and select the best one (λ = 0.5) as our baseline (see fig. 8). Following BID50, we train these models with batch size 256 and 20 epochs. We also compare the performance of EWC with λ = 1, 10, 10 2, 10 3, 10 4 and select the best value λ = 10 2 as our baseline (see fig. 9). The models are trained without dropout and with batch size 200 and 20 epochs. We approximate the Fisher information matrices in EWC using 600 random samples drawn from the current dataset. For diagonal LP, we compare the performance of λ = 0.01, 0.1, 1, 10, 100 and use the best value λ = 0.1 as our baseline (see FIG0). The models are also trained with prior N (0, I), batch size 200, and 20 epochs. The Hessians of LP are approximated using the Fisher information matrices with 200 samples. In this experiment, we use fully connected multi-head networks with two hidden layers, each of which contains 256 hidden units with ReLU activations. At each time step, we compare the test set accuracy of the current model on all observed tasks separately. We also plot the average accuracy over all tasks in the last column of FIG4. All the for this experiment are the averages over 10 runs of the algorithms with different random seeds. We use the Adam optimizer with learning rate 10 −3 for all models. All the VCL algorithms are trained with batch size equal to the size of the training set and 120 epochs. We use the prior N (0, I) and initialize our optimizer for the first task at the mean of the maximum likelihood model and a very small initial variance (10 −6). For the coresets, we choose 40 examples from each task to include into the coresets. In this experiment, the final approximate posterior used for prediction in eq. is computed for each task separately using the coreset points corresponding to the task. The algorithms that use only the coresets are trained using the VFE method with batch size equal to the coreset size and 120 epochs. We compare the performance of SI with λ = 0.01, 0.1, 1, 2, 3 and use the best value λ = 1 as our baseline (see FIG0). We also compare EWC with both single-head and multi-head models and λ = 1, 10, 10 2, 10 3, 10 4 (see FIG0). We approximate the Fisher information matrices using 200 random samples drawn from the current dataset. The figure shows that the multi-head models work better than the single-head models for EWC, and the performance is insensitive to the choice of λ. Thus, we use the multi-head model with λ = 1 as the EWC baseline for our experiment. For diagonal LP, we also use the multi-head model with λ = 1, prior N (0, I), and approximate the Hessians using the Fisher information matrices with 200 samples. The settings for this experiment are the same as those in the Split MNIST experiment above, except that we use deeper networks with 4 hidden layers, each of which contains 150 hidden units. FIG0 show the performance of SI and EWC with different hyper-parameter values respectively. In the experiment, we choose λ = 10 4 for multi-head EWC, λ = 1 for multi-head LP, and λ = 0.1 for SI. Here we consider a small experiment on a toy 2D dataset to understand some of the properties of EWC with λ = 1 and VCL. The experiment comprises two sequential binary classification tasks. The first task contains two classes generated from two Gaussian distributions. The data points (with green and black classes) for this task are shown in the first column of FIG0. The second task contains two classes also generated from two Gaussian distributions. The green class for this task has the same input distribution as the first task, while the input distribution for the black class is different. The data points for this task are shown in the third columns of FIG0. Each task contains 200 data points with 100 data points in each class. We compare the multi-head models trained by VCL and EWC on these two tasks. In this experiment, we use fully connected networks with one hidden layer containing 20 hidden units with ReLU activations. The first column of FIG0 shows the contours of the prediction probabilities after observing the first task, and both methods perform reasonably well for this task. However, after observing the second task, the EWC method fails to learn the classifiers for both tasks, while the VCL method are still able to learn good classifiers for them. In the experiments on Deep Generative Models, the learning rates and numbers of optimization epochs are tuned on separate training of each tasks. This gives a learning rate of 10 −4 and the number of epochs 200 for MNIST (except for SI) and 400 for notMNIST. For SI we optimize for 400 epochs on MNIST. For the VCL approach, the parameters of q t (θ) are initialized to have the same mean as q t−1 (θ) and the log standard deviation is set to 10 −6.The generative model consists of shared and task-specific components, each represented by a one hidden layer neural network with 500 hidden units (see FIG0). The dimensionality of the latent variable z and the intermediate representation h are 50 and 500, respectively. We use task-specific encoders that are neural networks with symmetric architectures to the generator. In many probabilistic models with conjugate priors, the exact posterior of the parameters/latent variables can be obtained. For example, a Bayesian linear regression model with a Gaussian prior over the parameters and a Gaussian observation model has a Gaussian posterior. If we insist on using a diagonal Gaussian approximation to this posterior and use either the variational free energy method or the Laplace's approximation, we will end up at the same solution -a Gaussian distribution with the same mean as that of the exact posterior and the diagonal precisions being the diagonal precisions of the exact posterior. Consequently, the online variational Gaussian approximation will give the same to that given by the online Laplace's approximation. However, when a diagonal Gaussian approximation is used, the batch and sequential solutions are different. In the following, we will explicitly detail the sequential variational updates for a Bayesian linear regression model to associate random binary patterns to binary outcomes BID26, and show its relationship to the online Laplace's approximation and the EWC approach of BID26. The task consists of associating a random D-dimensional binary vector x t to a random binary output y t by learning a weight vector W. Note that the possible values of the features and outputs are 1 and −1, and not 0 and 1. We also assume that the model sees only one input-output pair {x t, y t} at the t-th time step and the previous approximate posterior q t−1 (W) = By further assuming that σ y = 1 and x t 2 = 1, the equations above become: DISPLAYFORM0 DISPLAYFORM1 wherex t = y t x t. When v −1 0,d = 0, i.e. the prior is ignored, the update for the mean above is exactly equation S4 in the supplementary material of BID26. Therefore, in this case, the memory of the network trained by online variational inference is identical to that of the online Laplace's method, provided in BID26. These methods differ, in practice, when the prior is not ignored or when the parameter regularization constraints are accumulated as discussed in the main text. This equivalence also does not hold in the general case, as discussed by BID35 where the Gaussian variational approximation can be interpreted as an averaged and smoothed Laplace's approximation.

Represent the short summary for finding the corresponding scientific article (long)

This paper develops a principled method for continual learning in deep models.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.