Question: Represent the scientific article to find a short TLDR:

Question: Deep generative models have achieved remarkable progress in recent years. Despite this progress, quantitative evaluation and comparison of generative models remains as one of the important challenges. One of the most popular metrics for evaluating generative models is the log-likelihood. While the direct computation of log-likelihood can be intractable, it has been recently shown that the log-likelihood of some of the most interesting generative models such as variational autoencoders (VAE) or generative adversarial networks (GAN) can be efficiently estimated using annealed importance sampling (AIS). In this work, we argue that the log-likelihood metric by itself cannot represent all the different performance characteristics of generative models, and propose to use rate distortion curves to evaluate and compare deep generative models. We show that we can approximate the entire rate distortion curve using one single run of AIS for roughly the same computational cost as a single log-likelihood estimate. We evaluate lossy compression rates of different deep generative models such as VAEs, GANs (and its variants) and adversarial autoencoders (AAE) on MNIST and CIFAR10, and arrive at a number of insights not obtainable from log-likelihoods alone. Generative models of images represent one of the most exciting areas of rapid progress of AI (; b; a). However, evaluating the performance of generative models remains a significant challenge. Many of the most successful models, most notably Generative Adversarial Networks (GANs) , are implicit generative models for which computation of log-likelihoods is intractable or even undefined. Evaluation typically focuses on metrics such as the Inception score or the Fréchet Inception Distance (FID) score , which do not have nearly the same degree of theoretical underpinning as likelihood-based metrics. Log-likelihoods are one of the most important measures of generative models. Their utility is evidenced by the fact that likelihoods (or equivalent metrics such as perplexity or bits-per-dimension) are reported in nearly all cases where it's convenient to compute them. Unfortunately, computation of log-likelihoods for implicit generative models remains a difficult problem. Furthermore, log-likelihoods have important conceptual limitations. For continuous inputs in the image domain, the metric is often dominated by the fine-grained distribution over pixels rather than the high-level structure. For models with low-dimensional support, one needs to assign an observation model, such as (rather arbitrary) isotropic Gaussian noise . Lossless compression metrics for GANs often give absurdly large bits-per-dimension (e.g. 10 14) which fails to reflect the true performance of the model . for more discussion of limitations of likelihood-based evaluation. Typically, one is not interested in describing the pixels of an image directly, and it would be sufficient to generate images close to the true data distribution in some metric such as Euclidean distance. For this reason, there has been much interest in Wasserstein distance as a criterion for generative models, since the measure exploits precisely this metric structure; ). However, Wasserstein distance remains difficult to approximate, and hence it is not routinely used to evaluate generative models. We aim to achieve the best of both worlds by measuring lossy compression rates of deep generative models. In particular, we aim to estimate the rate distortion function, which measures the number of bits required to match a distribution to within a given distortion. Like Wasserstein distance, it can exploit the metric structure of the observation space, but like log-likelihoods, it connects to the rich literature of probabilistic and information theoretic analysis of generative models. By focusing on different parts of the rate distortion curve, one can achieve different tradeoffs between the description length and the fidelity of reconstruction -thereby fixing the problem whereby lossless compression focuses on the details at the expense of high-level structure. It has the further advantage that the distortion metric need not have a probabilistic interpretation; hence, one is free to use more perceptually valid distortion metrics such as structural similarity (SSIM) or distances between hidden representations of a convolutional network . Algorithmically, computing rate distortion functions raises similar challenges to estimating loglikelihoods. We show that the rate distortion curve can be computed by finding the normalizing constants of a family of unnormalized probability distributions over the noise variables z. Interestingly, when the distortion metric is squared error, these distributions correspond to the posterior distributions of z for Gaussian observation models with different variances; hence, the rate distortion analysis generalizes the evaluation of log-likelihoods with Gaussian observation models. Annealed Importance Sampling (AIS) is currently the most effective general-purpose method for estimating log-likelihoods of implicit generative models, and was used by to compare log-likelihoods of a variety of implicit generative models. The algorithm is based on gradually interpolating between a tractable initial distribution and an intractable target distribution. We show that when AIS is used to estimate log-likelihoods under a Gaussian observation model, the sequence of intermediate distributions corresponds to precisely the distributions needed to compute the rate distortion curve. Since AIS maintains a stochastic lower bound on the normalizing constants of these distributions, it automatically produces an upper bound on the entire rate distortion curve. Furthermore, the tightness of the bound can be validated on simulated data using bidirectional Monte Carlo (BDMC) . Hence, we can approximate the entire rate distortion curve for roughly the same computational cost as a single log-likelihood estimate. We use our rate distortion approximations to study a variety of variational autoencoders (VAEs) , GANs and adversarial autoencoders (AAE) , and arrive at a number of insights not obtainable from log-likelihoods alone. For instance, we observe that VAEs and GANs have different rate distortion tradeoffs: While VAEs with larger code size can generally achieve better lossless compression rates, their performances drop at lossy compression in the low-rate regime. Conversely, expanding the capacity of GANs appears to bring substantial reductions in distortion at the high-rate regime without any corresponding deterioration in quality in the low-rate regime. We find that increasing the capacity of GANs by increasing the code size (width) has a qualitatively different effect on the rate distortion tradeoffs than increasing the depth. We also find that that different GAN variants with the same code size achieve nearly identical RD curves, and that the code size dominates the performance differences between GANs. Annealed importance sampling (AIS) is a Monte Carlo algorithm based on constructing a sequence of n intermediate distributions, where k ∈ {0, . . ., n}, between a tractable initial distribution p 0 (z) and the intractable target distribution p n (z). At the the k-th state (0 ≤ k ≤ n), the forward distribution q f and the un-normalized backward distributionq b are where T k is an MCMC kernel that leaves p k (z) invariant; andT k is its reverse kernel. We run M independent AIS chains, numbered i = 1,..., M. Let z i k be the k-th state of the i-th chain. The Bidirectional Monte Carlo. We know that the log partition function estimate logẐ is a stochastic lower bound on log Z (Jensen's inequality). As the , using the forward AIS distribution as the proposal distribution in a lower bound on the data log-likelihood. By running AIS in reverse, however, we obtain an upper bound on log Z. However, in order to run the AIS in reverse, we need exact samples from the true posterior, which is only possible on the simulated data. The combination of the AIS lower bound and upper bound on the log partition function is called bidirectional Monte Carlo (BDMC), and the gap between these bounds is called the BDMC gap . We note that AIS combined with BDMC has been used to estimate log-likelihoods for implicit generative models . In this work, we validate our AIS experiments by using the BDMC gap to measure the accuracy of our partition function estimators. Let x be a random variable that comes from the data distribution p d (x). Shannon's fundamental compression theorem states that we can compress this random variable losslessly at the rate of H(x). But if we allow lossy compression, we can compress x at the rate of R, where R ≤ H(x), using the code z, and have a lossy reconstructionx = f (z) with the distortion of D, given a distortion measure d(x,x) = d(x, f (z)). The rate distortion theory quantifies the trade-off between the lossy compression rate R and the distortion D. The rate distortion function R(D) is defined as the minimum number of bits per sample required to achieve lossy compression of the data such that the average distortion measured by the distortion function is less than D. Shannon's rate distortion theorem states that R(D) equals the minimum of the following optimization problem: min where the optimization is over the channel conditional distribution q(z|x). Suppose the datadistribution is p d (x). The channel conditional q(z|x) induces the joint distribution q(z, x) = p d (x)q(z|x), which defines the mutual information I(z; x). q(z) is the marginal distribution over z of the joint distribution q(z, x), and is called the output marginal distribution. We can rewrite the optimization of Eq. 5 using the method of Lagrange multipliers as follows: min The goal of generative modeling is to learn a model distribution p(x) to approximate the data distribution p d (x). Implicit generative models define the model distribution p(x) using a latent variable z with a fixed prior distribution p(z) such as a Gaussian distribution, and a decoder or generator network which computesx = f (z). In some cases (e.g. VAEs, AAEs), the generator explicitly parameterizes a conditional distribution p(x|z), such as a Gaussian observation model N (x; f (z), σ 2 I). But in implicit models such as GANs, the generator directly outputsx = f (z). In order to treat VAEs and GANs under a consistent framework, we ignore the Gaussian observation model of VAEs (thereby treating the VAE decoder as an implicit model), and use the squared error distortion of 2. However, we note that it is also possible to assume a Gaussian observation model with a fixed σ 2 for GANs, and use the Gaussian negative log-likelihood (NLL) as the distortion measure for both VAEs and GANs: d(x, f (z)) = − log N (x; f (z), σ 2 I). This is equivalent to squared error distortion up to a linear transformation. In this section, we describe the rate-prior distortion function, as a variational upper bound on the true rate distortion function. We must modify the standard rate distortion formalism slightly in order to match the goals of generative model evaluation. Specifically, we are interested in evaluating lossy compression with coding schemes corresponding to particular trained generative models, including the fixed prior p(z). For models such as VAEs, KL(q(z|x) p(z)) is standardly interpreted as the description length of z. Hence, we adjust the rate distortion formalism to use E p d (x) KL(q(z|x) p(z)) in place of I(x, z), and refer to this as the rate-prior objective. The rate-prior objective upper bounds the standard rate: In the context of variational inference, q(z|x) is the posterior, q(z) = p d (x)q(z|x)dx is the aggregated posterior , and p(z) is the prior. In the context of rate distortion theory, q(z|x) is the channel conditional, q(z) is the output marginal, and p(z) is the variational output marginal distribution. The inequality is tight when p(z) = q(z), i.e., the variational output marginal (prior) is equal to the output marginal (aggregated posterior). We note that the upper bound of Eq. 7 has been used in other algorithms such as the Blahut-Arimoto algorithm or the variational information bottleneck algorithm . Analogously to the rate distortion function, we define the rate-prior distortion function R p (D) as the minimum value of the rate-prior objective for a given distortion D. More precisely, We can rewrite the optimization of Eq. 8 using the method of Lagrange multipliers as follows: Conveniently, the Lagrangian decomposes into independent optimization problems for each x, allowing us to treat this as an optimization problem over q(z|x) for fixed x. We can compute the rate distortion curve by sweeping over β rather than by sweeping over D. Now we describe some of the properties of the rate-prior distortion function R p (D), which are straightforward analogues of well-known properties of the rate distortion function. Proposition 1. R p (D) has the following properties: is non-increasing and convex function of D. (c) The rate-prior distortion optimization of Eq. 9 has a unique global optimum which can be expressed as q * Proof. The proofs are provided in Appendix C.1. Prop. 1b states that for any prior p(z), R p (D) is a variational upper-bound on R(D). More specifically, we have R(D) = min p(z) R(D), which implies that for any given β, there exists a prior p * β (z), for which the variational gap between rate distortion and rate-prior distortion functions at β is zero. Fig. 1a shows a geometrical illustration of Prop. 1b for three values of β ∈ {0.25, 1, 4}. We can see in this figure that all R p (D) curves are upper bounds on R(D), and for any given β, R p * β (D) is tangent to both R p (D) and to the line with the slope of β passing through the optimal solution. If the decoder outputs a probability distribution (as in a VAE), we can define the distortion metric to coincide with the negative log-likelihood (NLL): d(x, f (z)) = − log p(x|z). We now describe some of the properties of the rate-prior distortion functions with NLL distortions. Proposition 2. The rate-prior distortion function R p (D) with NLL distortion of − log p(x|z) has the following properties: The global optimum of rate-prior distortion optimization (Eq. 9) can be expressed as q * (c) At β = 1, the negative summation of rate-prior and distortion is the true log-likelihood: Proof. The proofs are provided in Appendix C.2.. At β = 1, let L * and L p be the negative summation of rate and distortion on the rate distortion and rate-prior distortion curves respectively (shown in Fig. 1b). From Prop. 2c we know that L p is the true log-likelihood of the generative model. From Prop. 1b, we can conclude that L * = max p(z) L p. This reveals an important relationship between rate distortion theory and generative modeling that was also observed in: for a given generative model with a fixed conditional p(x|z), the best log-likelihood L p that can be achieved by optimizing the prior p(z) is the L *, which can be found by solving the rate distortion problem. Furthermore, the corresponding optimal prior p * (z) is the output marginal of the optimal channel conditional of the rate distortion problem at β = 1. Fig. 1b shows the rate-prior distortion function R p * (D) corresponding to p * (z). In a "good" generative model, where the model distribution is close to the data-distribution, the negative log-likelihood −L p is close to the entropy of data H d, and the variational gap between R p (D) and R(D) is tight. In the previous section, we introduced the rate-prior distortion function R p (D) and showed that it upper bounds the true rate distortion function R(D). However, evaluating R p (D) is also intractable. In this section, we show how we can upper bound R p (D) using a single run of the AIS algorithm. AIS Chain. We fix a temperature schedule 0 = β 0 < β 1 <... < β n = ∞. For the k-th intermediate distribution, we use the optimal channel conditional q k (z|x) and partition function Z k (x), corresponding to points along R p (D) and derived in Prop. 1c: Conveniently, this choice coincides with geometric averages, the typical choice of intermediate distributions for AIS, i.e, the k th step happens to be the optimal solutions for β k. This chain is shown in Fig. 2. For the transition operator, we use Hamiltonian Monte Carlo . At the k-th step, the rate-prior R k (x) and the distortion D k (x) are AIS Rate-Prior Distortion Curve. For each data point x, we run M independent AIS chains, numbered i = 1,..., M, in the forward direction. At the k-th state of the i-th chain, let z i k be the state, w i k be the AIS importance weights, andw i k be the normalized AIS importance weights. We denote the AIS distribution at the k-th step as the distribution obtained by first sampling from all, and then re-sampling the samples based on their normalized importance weightsw i k (see Section 2.1 and Appendix C.4 for more details). More formally q Using the AIS distribution q AIS k (z|x) defined in Eq. 12, we now define the AIS distortion D AIS k (x) and the AIS rate-prior R AIS k (x) as follows: We now define the AIS rate-prior distortion curve R Proposition 3. The AIS rate-prior distortion curve upper bounds the rate-prior distortion function: Proof. The proof is provided in Appendix C.4. Estimated AIS Rate-Prior Distortion Curve. Although the AIS distribution can be easily sampled from, its density is intractable to evaluate. As the , evaluating R Having found the estimatesD, we propose to estimate the rate as follows: We define the estimated AIS rate-prior distortion curveR Fig. 1b ) as an RD curve obtained by tracing pairs of rate distortion estimates. Proposition 4. The estimated AIS rate-prior distortion curve upper bounds the AIS rate-prior distortion curve in expectation: Proof. The proof is provided in Appendix C.4. In summary, from Prop. 1, Prop. 3 and Prop. 4, we can conclude that the estimated AIS rate-prior distortion curve upper bounds the true rate distortion curve in expectation (shown in Fig. 1b): In all the experiments, we plot the estimated AIS rate-prior distortion functionR Accuracy of AIS Estimates. While the above discussion focuses on obtaining upper bounds, we note that AIS is one of the most accurate general-purpose methods for estimating partition functions, and therefore we believe our AIS upper bounds to be fairly tight in practice. In theory, for large number of intermediate distributions, the AIS variance is proportional to 1/M K (; 2005), where M is the number of AIS chains and K is the number of intermediate distributions. For the main experiments of our paper, we evaluate the tightness of the AIS estimate by computing the BDMC gap, and show that in practice our upper bounds are tight (Appendix D). The Rate Distortion Tradeoff in the AIS Chain. Different values of β corresponds to different tradeoffs between the compression rate and the distortion in a given generative model. β = 0 corresponds to the case where q 0 (z|x) = p(z). In this case, the compression rate is zero, and the distortion would be large, since in order to reconstruct x, we simply sample from the prior and generate a randomx that is completely independent of x. In this case, the distortion would In the case of probabilistic decoders with NLL distortion, another interesting intermediate distribution is β = 1, where the optimal channel conditional is the true posterior of the generative model q (z|x) = p(z|x). In this case, as shown in Prop. 2c, the summation of the rate-prior and the distortion term is the negative of true log-likelihood of the generative model. As β n → ∞, the network cares more about the distortion and less about the compression rate. In this case, the optimal channel conditional would be q n (z|x) = δ(z − z ML (x)), where. In other words, since the network only cares about the distortion, the optimal channel condition puts all its mass on z ML which minimizes the distortion. However, the network would require infinitely many bits to precisely represent this delta function, and thus the rate goes to infinity. Evaluation of Implicit Generative Models. Quantitative evaluation of the performance of GANs has been a challenge for the field since the beginning. Many heuristic measures have been proposed, such as the Inception score and the Fréchet Inception Distance (FID) . One of the main drawbacks of the IS or FID is that a model that simply memorizes the training dataset would obtain a near-optimal score. Another, drawback of these methods is that they use a pretrained ImageNet classifier weights which makes them sensitive to the weights of the classifier, and less applicable to other domains and datasets. Another evaluation method that sometimes is being used is the Parzen window estimate, which can be shown to be an instance of AIS with zero intermediate distributions, and thus has a very large variance. Another evaluation method of GANs that was proposed in is measuiring the ability of the generator network to reconstruct the samples from the data distribution. This metric is similar to the distortion obtained at the high-rate regime of our rate distortion framework when β → ∞. Another related work is GILBO, which similar to our framework does not require the generative model to have a tractable posterior and thus allows direct comparison of VAEs and GANs. However, GILBO can only evaluate the performance of the generative model on the simulated data and not the true data distribution. Rate Distortion Theory and Generative Models. Perhaps the closest work to ours is "Fixing a Broken ELBO", which plots rate-prior distortion curves for VAEs. Our work is different than in two key aspects. First, in the rate-prior distortion function is evaluated by fixing the architecture of the neural network, and learning the distortion measure d(x, f (z)) in addition to learning q(z|x). Whereas, in our definition of rate distortion, we assumed the distortion measure is fixed and given by a trained generative model. As the , we plot the rate-prior distortion curve for a particular generative model, rather than a particular architecture. The second key difference is that, consistent with the Shannon's rate distortion Deep and Shallow GANs (a) MNIST Comparing GANs on CIFAR10 Figure 3: The rate distortion curves of GANs. theorem, we find the optimal channel conditional q * (z|x) by using AIS; while in, q(z|x) is a variational distribution that is restricted to a variational family. See Appendix A for a discussion of related works about practical compression schemes, distortionperception tradeoffs, and precision-recall tradeoffs. In this section, we use our rate distortion approximations to answer the following questions: How do different generative models such as VAEs, GANs and AAEs perform at different lossy compression rates? What insights can we obtain from the rate distortion curves about different characteristics of generative models? What is the effect of the code size (width), depth of the network, or the learning algorithm on the rate distortion tradeoffs? Rate Distortion Curves of GANs. Fig. 3 shows rate distortion curves for GANs trained on MNIST and CIFAR-10. We varied the dimension of the noise vector z, as well as the depth of the decoder. For the GAN experiments on MNIST (Fig. 3a), the label "deep" corresponds to three hidden layers of size 1024, and the label "shallow" corresponds to one hidden layer of size 1024. We trained shallow and deep GANs with Gradient Penalty (GAN-GP) with the code size d ∈ {2, 5, 10, 100} on MNIST. For the GAN experiments on CIFAR-10 (Fig. 3b), we trained the DCGAN , GAN with Gradient Penalty (GP) , SN-GAN , and BRE-GAN , with the code size of d ∈ {2, 10, 100}. In both the MNIST and CIFAR experiments, we observe that in general increasing the code size has the effect of extending the curve leftwards. This is expected, since the high-rate regime is effectively measuring reconstruction ability, and additional dimensions in z improves the reconstruction. We can also observe from Fig. 3a that increasing the depth pushes the curves down and to the left. In other words, the distortion in both high-rate and mid-rate regimes improves. In these regimes, increasing the depth increases the capacity of the network, which enables the network to make a better use of the information in the code space. In the low-rate regime, however, increasing the depth, similar to increasing the latent size, does not improve the distortion. Comparing GANs, VAEs and AAEs Figure 4: RD curves of VAEs, GANs, AAEs. Rate Distortion Curves of VAEs. Fig. 4 compares VAEs, AAEs and GP-GANs with the code size of d ∈ {2, 10, 100}, and the same decoder architecture on the MNIST dataset. In general, we can see that in the mid-rate to high-rate regimes, VAEs achieve better distortions than GANs with the same architecture. This is expected as the VAE is trained with the ELBO objective, which encourages good reconstructions (in the case of factorized Gaussian decoder). We can see from Fig. 4 increasing the capacity reduces the distortion at the high-rate regime, at the expense of increasing the distortion in the low-rate regime (or equivalently, increasing the rate required to adequately approximate the data). We believe the performance drop of VAEs in the low-rate regime is symptomatic of the "holes problem" in the code space of VAEs with large code size: because these VAEs allocate a large fraction of their latent spaces to garbage images, it requires many bits to get close to the image manifold. Interestingly, this trade-off could also help explain the well-known problem of blurry samples from VAEs: in order to avoid garbage samples (corresponding to large distortion in the low-rate regime), one needs to reduce the capacity, thereby increasing the distortion at the high-rate regime. By contrast, GANs do not suffer from this tradeoff, and one can train high-capacity GANs without sacrificing performance in the low-rate regime. Rate Distortion Curves of AAEs. The AAE was introduced by to address the holes problem of VAEs, by directly matching the aggregated posterior to the prior in addition to optimizing the reconstruction cost. Fig. 4 shows the RD curves of AAEs. In comparison to GANs, AAEs can match the low-rate performane of GANs, but achieve a better high-rate performance. This is expected as AAEs directly optimize the reconstruction cost as part of their objective. In comparison to VAEs, AAEs perform slightly worse at the high-rate regime, which is expected as the adversarial regularization of AAEs is stronger than the KL regularization of VAEs. But AAEs perform slightly better in the low-rate regime, as they can alleviate the holes problem to some extent. Since log-likelihoods constitute only a scalar value, they are unable to distinguish different aspects of a generative model which could be good or bad, such as the prior or the observation model. Here, we show that two manipulations which damage a trained VAE in different ways in very different behavior of the RD curves. Our first manipulation, originally proposed by , is to use a mixture of the VAE's density and another distribution concentrated away from the data distribution. As pointed out by , this in a model which achieves high log-likelihood while generating poor samples. Specifically, after training the VAE10 on MNIST, we "damage" its prior p(z) = N (0, I) by altering it to a mixture prior (1 − α)p(z) + αq(z), where q(z) = N (0, 10I) is a "poor" prior, which is chosen to be far away from the original prior p(z); and α is close to 1. This process would in a "poor" generative model that generates garbage samples most of the time (more precisely with the probability of α). Suppose p(x) and q(x) are the likelihood of the good and the poor generative models. It is straightforward to see that log q(x) is at most 4.6 nats worse that log p(x), and thus log-likelihood fails to tell these models apart: Fig. 5a plots the rate distortion curves of this model for different values of α. We can see that the high-rate and log-likelihood performance of the good and poor generative models are almost identical, whereas in the low-rate regime, the RD curves show significant drop in the performance and successfully detect this failure mode of log-likelihood. (b) VAE, GANs and AAEs Figure 6: The RD curves of GANs, VAEs and AAEs with MSE distortion on the deep feature space. The behavior is qualitatively similar to the for MSE in images (see Fig. 3 and Fig. 4), suggesting that the RD analysis is not particularly sensitive to the particular choice of metric. Our second manipulation is to damage the decoder by adding a Gaussian blur kernel after the output layer. Fig. 5b shows the rate distortion curves for different standard deviations of the Gaussian kernel. We can see that, in contrast to the mixture prior experiment, the high-rate performance of the VAE drops due to inability of the decoder to output sharp images. However, we can also see an improvement in the low-rate performance of the VAE. This is because (similarly to log-likelihoods with Gaussian observation models) the data distribution does not necessarily achieve the minimal distortion, and in fact, in the extremely low-rate regime, blurring appears to help by reducing the average Euclidean distance between low-rate reconstructions and the input images. Hence, our two manipulations in very different effects to the RD curves, suggesting that these curves provide a much richer picture of the performance of generative models, compared to scalar log-likelihoods. The experiments discussed above all used pixelwise MSE as the distortion metric. However, for natural images, one could use more perceptually valid distortion metrics such as SSIM . Fig. 6 shows the RD curves of GANs, VAEs, and AAEs on the MNIST dataset, using the MSE on the deep features of a CNN as distortion metric. In all cases, the qualitative behavior of the RD curves with this distortion metric closely matches the qualitative behaviors for pixelwise MSE. We can see from Fig. 6a that similar to the RD curves with MSE distortion, GANs with different depths and code sizes have the same low-rate performance, but as the model gets deeper and wider, the RD curves are pushed down and extended to the left. Similarly, we can see from Fig. 6b that compared to GANs and AAEs, VAEs generally have a better high-rate performance, but worse lowrate performance. The fact that the qualitative behaviors of RD curves with this metric closely match those of pixelwise MSE indicates that the of our analysis are not overly sensitive to the particular choice of distortion metric. In this work, we studied rate distortion approximations for evaluating different generative models such as VAEs, GANs and AAEs. We showed that rate distortion curves provide more insights about the model than the log-likelihood alone while requiring roughly the same computational cost. For instance, we observed that while VAEs with larger code size can generally achieve better lossless compression rates, their performances drop at lossy compression in the low-rate regime. Conversely, expanding the capacity of GANs appears to bring substantial reductions in distortion at the high-rate regime without any corresponding deterioration in quality in the low-rate regime. This may help explain the success of large GAN architectures (; a; b). We also discovered that increasing the capacity of GANs by increasing the code size (width) has a very different effect than increasing the depth. The former extends the rate distortion curves leftwards, while the latter pushes the curves down. We also found that different GAN variants with the same code size has almost similar rate distortion curves, and that the code size dominates the algorithmic differences of GANs. Overall, lossy compression yields a richer and more complete picture of the distribution modeling performance of generative models. The ability to quantitatively measure performance tradeoffs should lead to algorithmic insights which can improve these models. Practical Compression Schemes. We have justified our use of compression terminology in terms of Shannon's fundamental implying that there exist a rate distortion code for any rate distortion pair that is achievable according to the rate distortion function. Interestingly, for lossless compression with generative models, there is a practical compression scheme which nearly achieves the theoretical rate (i.e. the negative ELBO): bits-back encoding. The basic scheme was proposed by; , and later implemented by. Practical versions for modern deep generative models were developed by;. We do not currently know of an analogous practical scheme for lossy compression with deep generative models. Other researchers have developed practical coding schemes achieving variational rate distortion bounds for particular latent variable models which exploited the factorial structure of the variational posterior (Ballé et al., 2018;). These methods are not directly applicable in our setting, since we don't assume an explicit encoder network, and our variational posteriors lack a convenient factorized form. We don't know whether our variational approximation will lead to a practical lossy compression scheme, but the successes for other variational methods give us hope. Relationship with the Rate-Distortion-Perception Tradeoff. Our work is related to which incorporates a perceptual quality loss function in the rate-distortion framework and characterizes the triple tradeoff between rate distortion and perception. More specifically, defines the perceptual loss using a divergence between the marginal reconstruction distribution and the data distribution. This perceptual loss is then incorporated as an additional constraint in the rate-distortion framework to encourage the reconstruction distribution to perceptually look like the data distribution. It is shown that as the perceptual constraint becomes tighter, the rate-distortion function elevates more. In our rate-prior distortion framework, we are also enforcing a perceptual constraint on the reconstruction distribution by incorporating the regularization term of KL(q(z) p(z)) in the rate-distortion objective, which encourages matching the aggregated posterior to the prior . More precisely, let us define the reconstruction distribution r(x) as the the distribution obtained by passing the data distribution through the encoder and then the decoder: It can be shown that the regularization term KL(q(z) p(z)) upper bounds KL(r(x) p(x)): In other words, in the rate-prior distortion optimization, for a given distortion constraint, we are not only interested in minimizing the rate I(x; z), but also at the same time, we are interested in preserving the perceptual quality of the reconstruction distribution by matching it to the model distribution. In the low-rate regime, when the model is allowed to have large distortions, the model obtains small rates and at the same time preserves the perceptual distribution of the reconstruction samples. As the distortion constraint becomes tighter, the model starts to trade off the rate I(x; z) and the perceptual quality KL(q(z) p(z)), which in an elevated rate distortion curve. Relationship with the Precision-Recall Tradeoff. One of the main drawbacks of the IS or FID is that they can only provide a single scalar value that cannot distinguish the mode dropping behavior from the mode inventing behavior (generating outlier or garbage samples) in generative models. In order to address this issue,; propose to study the precision-recall tradoff for evaluating generative models. In this context, high precision implies that the samples from the model distribution are close to the data distribution, and high recall implies the generative model can reconstruct (or generate a sample similar to) any sample from the data distribution. The precision-recall curves enable us to identify both the mode dropping and the mode inventing behavior of the generative model. More specifically, mode dropping drops the precision of the model at the high-recall regime, and mode inventing drops the precision of the model in the low-recall regime. Our rate-prior distortion framework can be thought as the information theoretic analogue of the precision-recall curves, which extends the scalar notion of log-likelihood to rate distortion curves. More specifically, in our framework, mode dropping drops the distortion performance of the model at the high-rate regime, and mode inventing drops the distortion performance of the model at the low-rate regime. In Section 6, we will empirically study the effect of mode dropping and mode inventing on our rate-prior distortion curves. AIS Validation. We conducted several experiments to validate the correctness of our implementation and the accuracy of the AIS estimates. Firstly, we compared our AIS with the analytical solution of rate-prior distortion curve on a linear VAE (derived in Appendix D.3.1) trained on MNIST. As shown in Fig. 7, the RD curve estimated by AIS agrees closely with the analytical solution. Secondly, for the main experiments of the paper, we evaluated the tightness of the AIS estimate by computing the BDMC gap. The largest BDMC gap for VAEs and AAEs was 0.127 nats, and the largest BDMC gap for GANs was 1.649 nats, showing that our AIS upper bounds are tight. More details are provided in Appendix D.3.2. APPENDIX C PROOFS C.1 PROOF OF PROP. 1. ] is a linear function of the channel conditional distribution q(z|x). The mutual information is a convex function of q(z|x). The KL(q(z) p(z)) is also convex function of q(z), which itself is a linear function of q(z|x). Thus the rate-prior objective is a convex function of q(z|x). Suppose for the distortions D 1 and D 2, q 1 (z|x) and q 2 (z|x) achieve the optimal rates in Eq. 5 respectively. Suppose the conditional q λ (z|x) is defined as q λ (z|x) = λq 1 (z|x) + (1 − λ)q 2 (z|x). The rate-prior objective that the conditional q λ (z|x) achieves is I λ (z; x) + KL(q λ (z) p(z)), and the distortion D λ that this conditional achieves is which proves the convexity of R p (D). Alternative Proof of Prop. 1a. We know the rate-prior term E p d (x) KL(q(z|x) p(z)) is a convex function of q(z|x), and E q(x,z) [d(x, f (z))] is a linear and thus convex function of q(z|x). As the , the following optimization problem is a convex optimization problem. min The rate distortion function R p (D) is the perturbation function of the convex optimization problem of Eq. 22. The convexity of R p (D) follows from the fact that the perturbation function of any convex optimization problem is a convex function . Proof of Prop. 1b. We have min = min = min where in Eq. 24, we have used the fact that for any function f (x, y), we have min and in Eq. 25, we have used the fact that KL(q(z) p(z)) is minimized when p(z) = q(z). Proof of Prop. 1c. In Prop. 1a, we showed that the rate-prior term is a convex function of q(z|x), and that the distortion is a linear function of q(z|x). So the summation of them in Eq. 9 will be a convex function of q(z|x). The unique global optimum of this convex optimization can be found by rewriting Eq. 9 as where Z β (x) = p(z) exp(−βd(x, f (z)))dz. The minimum of Eq. 28 is obtained when the KL divergence is zero. Thus the optimal channel conditional is q * Proof of Prop. 2a. R(D) ≤ R p (D) was proved in Prop. 1b. To prove the first inequality, note that the summation of rate and distortion is where q * (x, z) is the optimal joint channel conditional, and q * (z) and q * (x|z) are its marginal and conditional. The equality happens if there is a joint distribution q(x, z), whose conditional q(x|z) = p(x|z), and whose marginal over x is p d (x). But note that such a joint distribution might not exist for an arbitrary p(x|z). Proof of Prop. 2b. The proof can be easily obtained by using d(x, f (z)) = − log p(x|z) in Prop. 1c. Proof of Prop. 2c. Based on Prop. 2b, at β = 1, we have AIS has the property that for any step k of the algorithm, the set of chains up to step k, and the partial computation of their weights, can be viewed as the of a complete run of AIS with target distribution q * k (z|x). Hence, we assume without loss of generality that we are looking at a complete run of AIS (but our analysis applies to the intermediate distributions as well). 2. Compute the importance weights and normalized importance weights of each chain using w 3. Select a chain index S with probability ofw 5. Keep the unselected chain values and re-label them as (z where −S denotes the set of all indices except the selected index S. 6. Return z = z 1 k . More formally, the AIS distribution is Using the AIS distribution q AIS k (z|x) defined as above, we define the AIS distortion D AIS k (x) and the AIS rate-prior R In order to estimate R We would like to prove that The proof of Eq. 40 is straightforward: Eq. 42 shows thatD We also know logẐ AIS k (x) obtained by Eq. 38 is the estimate of the log partition function, and by the Jenson's inequality lower bounds in expectation the true log partition function: ) In order to simplify notation, suppose z Using the above notation, Eq. 44 can be re-written aŝ Hence, where the inequality follows from the monotonicity of KL divergence. Rearranging terms, we bound the rate: Eq. 49 shows thatR AIS k (x) upper bounds the AIS rate-prior R AIS k (x) in expectation. We also showed D AIS k (x) is an unbiased estimate of the AIS distortion D AIS k (x). Hence, the estimated AIS rate-prior curve upper bounds the AIS rate-prior distortion curve in expectation: The code for reproducing all the experiments of this paper will be open sourced publicly. We used MNIST and CIFAR-10 datasets in our experiments. Real-Valued MNIST. For the VAE experiments on the real-valued MNIST dataset (Fig. 4a), we used the "VAE-50" architecture described in , and only changed the code size in our experiments. The decoder variance is a global parameter learned during the training. The network was trained for 1000 epochs with the learning rate of 0.0001 using the Adam optimizer . For the GAN experiments on MNIST (Fig. 3a), we used the "GAN-50" architecture described in . In order to stabilize the training dynamic, we used the gradient penalty (GP) . In our deep architectures, we used code sizes of d ∈ {2, 5, 10, 100} and three hidden Figure 8: The rate-prior distortion curves obtained by adaptively tuning the HMC parameters in the preliminary run, and pre-loading the HMC parameters in the second formal run. "rs" in the legend indicates the random seed used in the second run. layers each having 1024 hidden units to obtain the following GAN models: Deep-GAN2, Deep-GAN5, Deep-GAN10 and Deep-GAN100. The shallow GANs architectures are similar to the deep architectures but with one layer of hidden units. For the CIFAR-10 experiments (Fig. 3b), we experimented with different GAN models such as DCGAN , DCGAN with Gradient Penalty (GP-GAN) , Spectral Normalization (SN-GAN) , and DCGAN with Binarized Representation Entropy Regularization (BRE-GAN) . The numbers at the end of each GAN name in Fig. 3b indicate the code size. For each RD curve, there are 1999 points computed with only one AIS chain, for 999 β s spaced linearly from β max to 1 and another 999 β s spaced linearly from 1 to β min, and plus β = 1, thus 1999 points. β min = 1 12 for all models. β max = 1 0.0003 ≈ 3333 for 100 dimensional models such as GAN100, VAE100 or AAE 100, and β max = For the 2, 5 and 10 dimensional models, N = 40000, and the above procedure will in 60292 intemediate distributions in total. For 100 dimensional models, to ensure accuracy of our AIS estimator with a small BDMC gap, we used N = 1600000 and the above procedure will in 1611463 intermediate distributions in total. We used 20 leap frog steps for HMC, 40 independent chains, on a single batch of 50 images. On MNIST, we also tested with a larger batch of 500 MNIST images, but did not observe significant difference compared with a batch 50 images, thus we did all of our experiments with a single batch 50 images. On a P100 GPU, for Adaptive Tuning of HMC Parameters. While running the AIS chain, the parameters of the HMC kernel cannot be adaptively tuned, since it would violate the Markovian property of the chain. So in order to be able to adaptively tune HMC parameters such as the number of leapfrog steps and the step size, in all our experiments, we first do a preliminary run where the HMC parameters are adaptively tuned to yield an average acceptance probability of 65% as suggested in. Then in the second "formal" run, we pre-load and fix the HMC parameters found in the preliminary run, and start the chain with a new random seed to obtain our final . Interestingly, we observed that the difference in the RD curves obtained from the preliminary run and the formal runs with various different random seeds is very small, as shown in Fig. 8. This figure shows that the AIS with the HMC kernel is robust against different choices of random seeds for approximating the RD curve of VAE10. We conducted several experiments to validate the correctness of our implementation and the accuracy of the AIS estimates. We compared our AIS with the analytical solution of the rate-prior distortion optimization on a linear VAE trained on MNIST as shown in Fig. 7. In order to derive the analytical solution, we first find the optimal distribution q * β (z|x) from Prop. 2b. For simplicity, we assume a fixed identity covariance matrix I at the output of the conditional likelihood of the linear VAE decoder. In other words, the decoder of the VAE is simply: x = Wz + b +, where x is the observation, z is the latent code vector, W is the decoder weight matrix and b is the bias. The observation noise of the decoder is ∼ N (0, I). It's easy to show that the conditional likelihood raised to a power β is: p(x|z) β = N (x|Wz + b, For numerical stability, we can further simplify thw above by taking the SVD of W : let W = UDV, and then apply the Woodbury Matrix Identity to the matrix inversion, we can get: Where R β is a diagonal matrix with the i th diagonal entry being Where k is the dimension of the latent code z. With negative log-likelihood as the distortion metric, the analytically form of distortion term is: We evaluated the tightness of the AIS estimate by computing the BDMC gaps using the same AIS settings. Fig. 9, shows the BDMC gaps at diffrent compression rates for the VAE, GAN and AAE experiments on the MNIST dataset. The largest BDMC gap for VAEs and AAEs is 0.127 nats, and the largest BDMC gap for GANs is 1.649 nats, showing that our AIS upper bounds are tight. In this section, we visualize the high-rate (β ≈ 3500) and low-rate (β = 0) reconstructions of the MNIST images for VAEs, GANs and AAEs with different hidden code sizes. The qualitative are shown in Fig. 10 and Fig. 11, which is in consistent with the quantitative presented in experiment section of the paper. (h) High Rate VAE100 (i) High Rate AAE100 (j) High Rate GAN100 Figure 11: High-rate reconstructions (β max) of VAEs, GANs and AAEs on MNIST. β max = 3333 for 100 dimensional models, and β max = 36098 for the 2 and 10 dimensional models.

Represent the TLDR for finding the corresponding scientific article

We study rate distortion approximations for evaluating deep generative models, and show that rate distortion curves provide more insights about the model than the log-likelihood alone while requiring roughly the same computational cost.

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question: Represent the article for finding a single-sentence TLDR.

Question: Although reinforcement learning methods can achieve impressive in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot: We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads. Both model-based and model-free reinforcement learning (RL) methods generally operate in one of two regimes: all training is performed in advance, producing a model or policy that can be used at test-time to make decisions in settings that approximately match those seen during training; or, training is performed online (e.g., as in the case of online temporal-difference learning), in which case the agent can slowly modify its behavior as it interacts with the environment. However, in both of these cases, dynamic changes such as failure of a robot's components, encountering a new terrain, environmental factors such as lighting and wind, or other unexpected perturbations, can cause the agent to fail. In contrast, humans can rapidly adapt their behavior to unseen physical perturbations and changes in their dynamics BID6: adults can learn to walk on crutches in just a few seconds, people can adapt almost instantaneously to picking up an object that is unexpectedly heavy, and children that can walk on carpet and grass can quickly figure out how to walk on ice without having to relearn how to walk. How is this possible? If an agent has encountered a large number of perturbations in the past, it can in principle use that experience to learn how to adapt. In this work, we propose a meta-learning approach for learning online adaptation. Motivated by the ability to tackle real-world applications, we specifically develop a model-based meta-reinforcement learning algorithm. In this setting, data for updating the model is readily available at every timestep in the form of recent experiences. But more crucially, the meta-training process for training such an adaptive model can be much more sample efficient than model-free meta-RL approaches BID11 BID55. Further, our approach foregoes the episodic framework on which model-free meta-RL approaches rely on, where tasks are pre-defined to be different rewards or environments, and tasks exist at the trajectory level only. Instead, our method considers each timestep to potentially be a new "task, " where any detail or setting could have changed at any timestep. This view induces a more general meta-RL problem setting by allowing the notion of a task to represent anything from existing in a different part of the state space, to experiencing disturbances, or attempting to achieve a new goal. Learning to adapt a model alleviates a central challenge of model-based reinforcement learning: the problem of acquiring a global model that is accurate throughout the entire state space. Furthermore, even if it were practical to train a globally accurate dynamics model, the dynamics inherently change as a function of uncontrollable and often unobservable environmental factors, such as those mentioned above. If we have a model that can adapt online, it need not be perfect everywhere a priori. This property has previously been exploited by adaptive control methods BID2 BID45 BID38; but, scaling such methods to complex tasks and nonlinear systems is exceptionally difficult. Even when working with deep neural networks, which have been used to model complex nonlinear systems BID21, it is exceptionally difficult to enable adaptation, since such models typically require large amounts of data and many gradient steps to learn effectively. By specifically training a neural network model to require only a small amount of experience to adapt, we can enable effective online adaptation in complex environments while putting less pressure on needing a perfect global model. The primary contribution of our work is an efficient meta reinforcement learning approach that achieves online adaptation in dynamic environments. To the best knowledge of the authors, this is the first meta-reinforcement learning algorithm to be applied in a real robotic system. Our algorithm efficiently trains a global model that is capable to use its recent experiences to quickly adapt, achieving fast online adaptation in dynamic environments. We evaluate two versions of our approach, recurrence-based adaptive learner (ReBAL) and gradient-based adaptive learner (GrBAL) on stochastic and simulated continuous control tasks with complex contact dynamics (Fig. 2). In our experiments, we show a quadrupedal "ant" adapting to the failure of different legs, as well as a "half-cheetah" robot adapting to the failure off different joints, navigating terrains with different slopes, and walking on floating platforms of varying buoyancy. Our model-based meta RL method attains substantial improvement over prior approaches, including standard model-based methods, online model-adaptive methods, model-free methods, and prior meta-reinforcement learning methods, when trained with similar amounts of data. In all experiments, meta-training across multiple tasks is sample efficient, using only the equivalent of 1.5 − 3 hours of real-world experience, roughly 10× less than what model-free methods require to learn a single task. Finally, we demonstrate GrBAL on a real dynamic legged millirobot (see Fig 2). To highlight not only the sample efficiency of our meta model-based reinforcement learning approach, but also the importance of fast online adaptation in the real world, we show the agent's learned ability to adapt online to tasks such as a missing leg, novel terrains and slopes, miscalibration or errors in pose estimation, and new payloads to be pulled. Advances in learning control policies have shown success on numerous complex and high dimensional tasks BID48 BID27 BID32 BID49. While reinforcement learning algorithms provide a framework for learning new tasks, they primarily focus on mastery of individual skills, rather than generalizing and quickly adapting to new scenarios. Furthermore, model-free approaches BID39 require large amounts of system interaction to learn successful control policies, which often makes them impractical for realworld systems. In contrast, model-based methods attain superior sample efficiency by first learning a model of system dynamics, and then using that model to optimize a policy BID9 BID23 BID36 BID58. Our approach alleviates the need to learn a single global model by allowing the model to be adapted automatically to different scenarios online based on recent observations. A key challenge with model-based RL approaches is the difficulty of learning a global model that is accurate for the entire state space. Prior model-based approaches tackled this problem by incorporating model uncertainty using Gaussian Processes (GPs) BID18 BID8 BID10. However, these methods make additional assumptions on the system (such as smoothness), and does not scale to high dimensional environments. BID7 has recently showed that neural networks models can also benefit from incorporating uncertainty, and it can lead to model-based methods that attain model-free performance with a significant reduction on sample complexity. Our approach is orthogonal to theirs, and can benefit from incorporating such uncertainty. Prior online adaptation approaches BID51 BID3 have aimed to learn an approximate global model and then adapt it at test time. Dynamic evaluation algorithms BID42 BID20 BID14, for example, learn an approximate global distribution at training time and adapt those model parameters at test time to fit the current local distribution via gradient descent. There exists extensive prior work on online adaptation in modelbased reinforcement learning and adaptive control BID45. In contrast from inverse model adaptation BID17 BID54 BID38 BID40, we are concerned in the problem of adapting the forward model, closely related to online system identification BID28. Work in model adaptation BID24 BID16 BID15 BID56 has shown that a perfect global model is not necessary, and prior knowledge can be fine-tuned to handle small changes. These methods, however, face a mismatch between what the model is trained for and how it is used at test time. In this paper, we bridge this gap by explicitly training a model for fast and effective adaptation. As a , our model achieves more effective adaptation compared to these prior works, as validated in our experiments. Our problem setting relates to meta-learning, a long-standing problem of interest in machine learning that is concerned with enabling artificial agents to efficiently learn new tasks by learning to learn BID52 BID47 BID37 BID22. A meta-learner can control learning through approaches such as deciding the learner's architecture BID4, or by prescribing an optimization algorithm or update rule for the learner BID5 BID46 BID59 BID1 BID26 BID41. Another popular meta-learning approach involves simply unrolling a recurrent neural network (RNN) that ingests the data BID44 BID33 BID31 and learns internal representations of the algorithms themselves, one instantiation of our approach (ReBAL) builds on top of these methods. On the other hand, the other instantiation of our method (GrBAL) builds on top of MAML. GrBAL differs from the supervised version of MAML in that MAML assumes access to a hand-designed distribution of tasks. Instead, one of our primary contributions is the online formulation of meta-learning, where tasks correspond to temporal segments, enabling "tasks" to be constructed automatically from the experience in the environment. Meta-learning in the context of reinforcement learning has largely focused on model-free approaches BID11 BID55 BID50 BID0. However, these algorithms present even more (meta-)training sample complexity than non-meta model-free RL methods, which precludes them from real-world applications. Recent work BID43 ) has developed a model-based meta RL algorithm, framing meta-learning as a hierarchical latent variable model, training for episodic adaptation to dynamics changes; the modeling is done with GPs, and are shown on the cart-pole and double-pendulum agents. In contrast, we propose an approach for learning online adaptation of high-capacity neural network dynamics models; we present two instantiations of this general approach and show on both simulated agents and a real legged robot. In this section, we present model-based reinforcement learning, introduce the meta-learning formulation, and describe the two main meta-learning approaches. Reinforcement learning agents aim to perform actions that maximize some notion of cumulative reward. Concretely, consider a Markov decision process (MDP) defined by the tuple (S, A, p, r, γ, ρ 0, H). Here, S is the set of states, A is the set of actions, p(s |s, a) is the state transition distribution, r: S × A → R is a bounded reward function, ρ 0: S → R + is the initial state distribution, γ is the discount factor, and H is the horizon. A trajectory segment is denoted by τ (i, j):= (s i, a i, ..., s j, a j, s j+1). Finally, the sum of expected rewards from a trajectory is the return. In this framework, RL aims to find a policy π: S → A that prescribes the optimal action to take from each state in order to maximize the expected return. Model-based RL aims to solve this problem by learning the transition distribution p(s |s, a), which is also referred to as the dynamics model. This can be done using a function approximatorp θ (s |s, a) to approximate the dynamics, where the weights θ are optimized to maximize the log-likelihood of the observed data D. In practice, this model is then used in the process of action selection by either producing data points from which to train a policy, or by producing predictions and dynamics constraints to be optimized by a controller.3.2 META-LEARNING Meta-learning is concerned with automatically learning learning algorithms that are more efficient and effective than learning from scratch. These algorithms leverage data from previous tasks to acquire a learning procedure that can quickly adapt to new tasks. These methods operate under the assumption that the previous meta-training tasks and the new meta-test tasks are drawn from the same task distribution ρ(T) and share a common structure that can be exploited for fast learning. In the supervised learning setting, we aim to learn a function f θ with parameters θ that minimizes a supervised loss L T. Then, the goal of meta-learning is to find a learning procedure, denoted as DISPLAYFORM0 We can formalize this meta-learning problem setting as optimizing for the parameters of the learning procedure θ, ψ as follows: min DISPLAYFORM1 where DISPLAYFORM2 T are sampled without replacement from the meta-training dataset D T. Once meta-training optimizes for the parameters θ *, ψ *, the learning procedure u ψ (·, θ) can then be used to learn new held-out tasks from small amounts of data. We will also refer to the learning procedure u as the update function. Gradient-based meta-learning. Model-agnostic meta-learning (MAML) aims to learn the initial parameters of a neural network such that taking one or several gradient descent steps from this initialization leads to effective generalization (or few-shot generalization) to new tasks. Then, when presented with new tasks, the model with the meta-learned initialization can be quickly fine-tuned using a few data points from the new tasks. Using the notation from before, MAML uses gradient descent as a learning algorithm: DISPLAYFORM3 The learning rate α may be a learnable paramter (in which case ψ = α) or fixed as a hyperparameter, leading to ψ = ∅. Despite the update rule being fixed, a learned initialization of an overparameterized deep network followed by gradient descent is as expressive as update rules represented by deep recurrent networks.Recurrence-based meta-learning. Another approach to meta-learning is to use recurrent models. In this case, the update function is always learned, and ψ corresponds to the weights of the recurrent model that update the hidden state. The parameters θ of the prediction model correspond to the remainder of the weights of the recurrent model and the hidden state. Both gradient-based and recurrence-based meta-learning methods have been used for meta model-free RL BID11. We will build upon these ideas to develop a meta model-based RL algorithm that enables adaptation in dynamic environments, in an online way. In this section, we present our approach for meta-learning for online model adaptation. As explained in Section 3.2, standard meta-learning formulations require the learned model θ *, ψ * to learn using M data points from some new "task." In prior gradient-based and model-based meta-RL approaches BID43, the M has corresponded to M trajectories, leading to episodic adaptation. Our notion of task is slightly more fluid, where every segment of a trajectory can be considered to be a different "task," and observations from the past M timesteps (rather than the past M episodes) can be considered as providing information about the current task setting. Since changes in system dynamics, terrain details, or other environmental changes can occur at any time, we consider (at every time step) the problem of adapting the model using the M past time steps to predict the next K timesteps. In this setting, M and K are pre-specified hyperparameters; see appendix for a sensitivity analysis of these parameters. In this work, we use the notion of environment E to denote different settings or configurations of a particular problem, ranging from malfunctions in the system's joints to the state of external disturbances. We assume a distribution of environments ρ(E) that share some common structure, such as the same observation and action space, but may differ in their dynamics p E (s |s, a). We denote a trajectory segment by τ E (i, j), which represents a sequence of states and actions (s i, a i, ..., s j, a j, s j+1) sampled within an environment E. Our algorithm assumes that the environment is locally consistent, in that every segment of length j − i has the same environment. Even though this assumption is not always correct, it allows us to learn to adapt from data without knowing when the environment has changed. Due to the fast nature of our adaptation (less than a second), this assumption is seldom violated. We pose the meta-RL problem in this setting as an optimization over (θ, ψ) with respect to a maximum likelihood meta-objective. The meta-objective is the likelihood of the data under a predictive modelp θ (s |s, a) with parameters θ, where θ = u ψ (τ E (t − M, t − 1), θ) corresponds to model parameters that were updated using the past M data points. Concretely, this corresponds to the following optimization: DISPLAYFORM0 In that τ E (t − M, t + K) ∼ D corresponds to trajectory segments sampled from our previous experience, and the loss L corresponds to the negative log likelihood of the data under the model: DISPLAYFORM1 In the meta-objective in Equation 3, note that the past M points are used to adapt θ into θ, and the loss of this θ is evaluated on the future K points. Thus, we use the past M timesteps to provide insight into how to adapt our model to perform well for nearby future timesteps. As outlined in Algorithm 1, the update rule u ψ for the inner update and a gradient step on θ for the outer update allow us to optimize this meta-objective of adaptation. Thus, we achieve fast adaptation at test time by being able to fine-tune the model using just M data points. While we focus on reinforcement learning problems in our experiments, this meta-learning approach could be used for a learning to adapt online in a variety of sequence modeling domains. We present our algorithm using both a recurrence and a gradient-based meta-learner, as we discuss next. Gradient-Based Adaptive Learner (GrBAL). GrBAL uses a gradient-based meta-learning to perform online adaptation; in particular, we use MAML. In this case, our update rule is prescribed by gradient descent (5.) DISPLAYFORM2 Recurrence-Based Adaptive Learner (ReBAL). ReBAL, instead, utilizes a recurrent model, which learns its own update rule (i.e., through its internal gating structure). In this case, ψ and u ψ correspond to the weights of the recurrent model that update its hidden state. Sample E ∼ ρ(E) Collect τ E using Alg. 2 6: DISPLAYFORM0 end if 8: DISPLAYFORM1 end for 13: DISPLAYFORM2 DISPLAYFORM3 a ← controller(θ *, r, H, n A) Execute a, add to D 6: end for 7: Return rollout D Figure 2: Two real-world and four simulated environments on which our method is evaluated and adaptation is crucial for success (e.g., adapting to different slopes and leg failures) Now that we have discussed our approach for enabling online adaptation, we next propose how to build upon this idea to develop a model-based meta-reinforcement learning algorithm. First, we explain how the agent can use the adapted model to perform a task, given parameters θ * and ψ * from optimizing the meta-learning objective. Given θ * and ψ *, we use the agent's recent experience to adapt the model parameters: θ * = u ψ * (τ (t − M, t), θ * ). This in a modelp θ * that better captures the local dynamics in the current setting, task, or environment. This adapted model is then passed to our controller, along with the reward function r and a planning horizon H. We use a planning H that is smaller than the adaptation horizon K, since the adapted model is only valid within the current context. We use model predictive path integral control (MPPI) BID57, but, in principle, our model adaptation approach is agnostic to the model predictive control (MPC) method used. The use of MPC compensates for model inaccuracies by preventing accumulating errors, since we replan at each time step using updated state information. MPC also allows for further benefits in this setting of online adaptation, because the modelp θ E itself will also improve by the next time step. After taking each step, we append the ing state transition onto our dataset, reset the model parameters back to θ *, and repeat the entire planning process for each timestep. See Algorithm 2 for this adaptation procedure. Finally, in addition to test-time, we also perform this online adaptation procedure during the meta-training phase itself, to provide on-policy rollouts for meta-training. For the complete meta-RL algorithm, see Algorithm 1. Our evaluation aims to answer the following questions: Is adaptation actually changing the model? Does our approach enable fast adaptation to varying dynamics, tasks, and environments, both inside and outside of the training distribution? How does our method's performance compare to that of other methods? How do GrBAL and ReBAL compare? How does meta model-based RL compare to meta model-free RL in terms of sample efficiency and performance for these experiments? Can our method learn to adapt online on a real robot, and if so, how does it perform? We next present our set-up and , motivated by these questions. Videos are available online 2, and further analysis is provided in the appendix. We first conduct a comparative evaluation of our algorithm, on a variety of simulated robots using the MuJoCo physics engine BID53. For all of our environments, we model the transition probabilities as Gaussian random variables with mean parameterized by a neural network model (3 hidden layers of 512 units each and ReLU activations) and fixed variance. In this case, maximum likelihood estimation corresponds to minimizing the mean squared error. We now describe the setup of our environments (Fig. 2), where each agent requires different types of adaptation to succeed at run-time:Half-cheetah (HC): disabled joint. For each rollout during meta-training, we randomly sample a joint to be disabled (i.e., the agent cannot apply torques to that joint). At test time, we evaluate performance in two different situations: disabling a joint unseen during training, and switching between disabled joints during a rollout. The former examines extrapolation to out-of-distribution environments, and the latter tests fast adaptation to changing dynamics. HC: sloped terrain. For each rollout during meta-training, we randomly select an upward or downward slope of low steepness. At test time, we evaluate performance on unseen settings including a gentle upward slope, a steep upward slope, and a steep hill that first goes up and then down. HC: pier. In this experiment, the cheetah runs over a series of blocks that are floating on water. Each block moves up and down when stepped on, and the changes in the dynamics are rapidly changing due to each block having different damping and friction properties. The HC is meta-trained by varying these block properties, and tested on a specific (randomly-selected) configuration of properties. Ant: crippled leg. For each meta-training rollout, we randomly sample a leg to cripple on this quadrupedal robot. This causes unexpected and drastic changes to the underlying dynamics. We evaluate this agent at test time by crippling a leg from outside of the training distribution, as well as transitioning within a rollout from normal operation to having a crippled leg. In the following sections, we evaluate our model-based meta-RL methods (GrBAL and ReBAL) in comparison to several prior methods:• Model-free RL (TRPO): To evaluate the importance of adaptation, we compare to a model-free RL agent that is trained across environments E ∼ ρ(E) using TRPO BID48.• Model-free meta-RL (MAML-RL): We compare to a state-of-the-art model-free meta-RL method, MAML-RL ).• Model-based RL (MB): Similar to the model-free agent, we also compare to a single modelbased RL agent, to evaluate the importance of adaptation. This model is trained using supervised model-error and iterative model bootstrapping.• Model-based RL with dynamic evaluation (MB+DE): We compare to an agent trained with model-based RL, as above. However, at test time, the model is adapted by taking a gradient step at each timestep using the past M observations, akin to dynamic evaluation BID20. This final comparison evaluates the benefit of explicitly training for adaptability. All model-based approaches (MB, MB+DE, GrBAL, and ReBAL) use model bootstrapping, use the same neural network architecture, and use the same planner within experiments: MPPI BID57 for the simulated experiments and random shooting (RS) BID35 for the real-world experiments. First, we analyze the effect of the model adaptation, and show from test-time runs on three environments: HC pier, HC sloped terrain with a steep up/down hill, and ant crippled leg with the chosen leg not seen as crippled during training. FIG1 displays the distribution shift between the pre-update and post-update model prediction errors of three GrBAL runs, showing that using the past M timesteps to update θ * (pre) into θ * (post) does indeed reduce model error on predicting the following K timesteps. We first study the sample efficiency of the meta-training process. FIG2 shows the average return across test environments w.r.t. the amount of data used for metatraining. We (meta-)train the model-free methods (TRPO and MAML-RL) until convergence, using the equivalent of about two days of real-world experience. In contrast, we meta-train the model-based methods (including our approach) using the equivalent of 1.5-3 hours of real-world experience. Our methods in superior or equivalent performance to the model-free agent that is trained with 1000 times more data. Our methods also surpass the performance of the non-meta-learned model-based approaches. Finally, our performance closely matches the high asymptotic performance of the modelfree meta-RL method for half-cheetah disabled, and achieves a suboptimal performance for ant crippled but, again, it does so with the equivalent of 1000 times less data. Note that this suboptimality in asymptotic performance is a known issue with model-based methods, and thus an interesting direction for future efforts. The improvement in sample efficiency from using model-based methods matches prior findings BID8 BID35 BID21; the most important evaluation, which we discuss in more detail next, is the ability for our method to adapt online to drastic dynamics changes in only a handful of timesteps. In our second comparative evaluation, we evaluate final test time performance both GrBAL and Re-BAL in comparison to the aforementioned methods. In the interest of developing efficient algorithms for real-world applications, we operate all methods in the low data regime for all experiments: the amount of data available (meta-)training is fixed across methods, and roughly corresponds to 1.5-3 hours of real-world experience depending on the domain. We also provide the performance of a MB oracle, which is trained using unlimited data from only the given test environment (rather than needing to generalize to various training environments).In these experiments, note that all agents were meta-trained on a distribution of tasks/environments (as detailed above), but we then evaluate their adaptation ability on unseen environments at test time. We test the ability of each approach to adapt to sudden changes in the environment, as well as to generalize beyond the training environments. We evaluate the fast adaptation (F.A.) component on the HC disabled joint, ant crippled leg, and the HC pier. On the first two, we cause a joint/leg of the robot to malfunction in the middle of a rollout. We evaluate the generalization component also on the tasks of HC disabled joint and ant crippled leg, but this time, the leg/joint that malfunctions has not been seen as crippled during training. The last environment that we test generalization on is the HC slopped terrain for a hill, where the agent has to run up and down a steep slope, which is outside of the gentle slopes that it experienced during training. The , shown in FIG3, show returns that are normalized such that the MB oracle achieves a return of 1. In all experiments, due to low quantity of training data, TRPO performs poorly. Although MB+DE achieves better generalization than MB, the slow nature of its adaptation causes it to fall behind MB in the environments that require fast adaptation. On the other hand, our approach surpasses the other approaches in all of the experiments. In fact, in the HC pier and the fast adaptation of ant environments, our approach surpasses the model-based oracle. This showcases the importance of adaptation in stochastic environments, where even a model trained with a lot of data cannot be robust to unexpected occurrences or disturbances. ReBAL displays its strengths on scenarios where longer sequential inputs allow it to better asses current environment settings, but overall, GrBAL seems to perform better for both generalization and fast adaptation. To test our meta model-based RL method's sample efficiency, as well as its ability to perform fast and effective online adaptation, we applied GrBAL to a real legged millirobot, comparing it to model-based RL (MB) and model-based RL with dynamic evaluation (MB+DE). Due to the cost of running real robot experiments, we chose the better performing method (i.e., GrBAL) to evaluate on the real robot. This small 6-legged robot, as shown in Fig. 1 and Fig. 2, presents a modeling and control challenge in the form of highly stochastic and dynamic movement. This robot is an excellent candidate for online adaptation for many reasons: the rapid manufacturing techniques and numerous custom-design steps used to construct this robot make it impossible to reproduce the same dynamics each time, its linkages and other body parts deteriorate over time, and it moves very quickly and dynamically withThe state space of the robot is a 24-dimensional vector, including center of mass positions and velocities, center of mass pose and angular velocities, back-EMF readings of motors, encoder readings of leg motor angles and velocities, and battery voltage. We define the action space to be velocity setpoints of the rotating legs. The action space has a dimension of two, since one motor on each side is coupled to all three of the legs on that side. All experiments are conducted in a motion capture room. Computation is done on an external computer, and the velocity setpoints are streamed over radio at 10 Hz to be executed by a PID controller on the microcontroller on-board of the robot. We meta-train a dynamics model for this robot using the meta-objective described in Equation 3, and we train it to adapt on entirely real-world data from three different training terrains: carpet, styrofoam, and turf. We collect approximately 30 minutes of data from each of the three training terrains. This data was entirely collected using a random policy, in conjunction with a safety policy, whose sole purpose was to prevent the robot from exiting the area of interest. Our first group of (Table 1) show that, when data from a random policy is used to train a dynamics model, both a model trained with a standard supervised learning objective (MB) and a GrBAL model achieve comparable performance for executing desired trajectories on terrains from the training distribution. Next, we test the performance of our method on what it is intended for: fast online adaptation of the learned model to enable successful execution of new, changing, or out-of-distribution environments at test time. Similar to the comparisons above, we compare GrBAL to a model-based method (MB) that involves neither meta-training nor online adaptation, as well as a dynamic evaluation method that involves online adaptation of that MB model (MB+DE). Our (Fig. 6) demonstrate that GrBAL substantially outperforms MB and MB+DE, and, unlike MB and MB+DE, and that GrBAL can quickly 1) adapt online to a missing leg, 2) adjust to novel terrains and slopes, 3) account for miscalibration or errors in pose estimation, and 4) compensate for pulling payloads. Figure 6: GrBAL clearly outperforms both MB and MB+DE, when tested on environments that require online adaptation, and/or were never seen during training.these environments were seen during training time, but the agent's ability to learn how to learn enables it to quickly leverage its prior knowledge and fine-tune to adapt to new environments online. Furthermore, the poor performance of the MB and MB+DE baselines demonstrate not only the need for adaptation, but also the importance of good initial parameters to adapt from (in this case, meta-learned parameters). The qualitative of these experiments in Fig. 7 show that the robot is able to use our method to adapt online and effectively follow the target trajectories, even in the presence of new environments and unexpected perturbations at test time.bounding-style gaits; hence, its dynamics are strongly dependent on the terrain or environment at hand. Figure 7: The dotted black line indicates the desired trajectory in the xy plane. By effectively adapting online, our method prevents drift from a missing leg, prevents sliding sideways down a slope, accounts for pose miscalibration errors, and adjusts to pulling payloads (left to right). Note that none of these tasks/environments were seen during training time, and they require fast and effective online adaptation for success. In this work, we present an approach for model-based meta-RL that enables fast, online adaptation of large and expressive models in dynamic environments. We show that meta-learning a model for online adaptation in a method that is able to adapt to unseen situations or sudden and drastic changes in the environment, and is also sample efficient to train. We provide two instantiations of our approach (ReBAL and GrBAL), and we provide a comparison with other prior methods on a range of continuous control tasks. Finally, we show that (compared to model-free meta-RL approaches), our approach is practical for real-world applications, and that this capability to adapt quickly is particularly important under complex real-world dynamics. In this section, we show the effect of adaptation in the case of GrBAL. In particular, we show the histogram of the K step normalized error, as well as the per-timestep visualization of this error during a trajectory. Across all tasks and environments, the post-updated modelp θ * achieves lower prediction error than the pre-updted modelp θ *. To see how training distribution affects test performance, we ran an experiment that used GrBAL to train models of the 7-DOF arm, where each model was trained on the same number of datapoints during meta-training, but those datapoints came from different ranges of force perturbations. We observe (in the plot below) that 1. Seeing more during training is helpful during testing -a model that saw a large range of force perturbations during training performed the best 2. A model that saw no perturbation forces during training did the worst 3. The middle 3 models show comparable performance in the "constant force = 4" case, which is an out-of-distribution task for those models. Thus, there is not actually a strong restriction on what needs to be seen during training in order for adaptation to occur at train time (though there is a general trend that more is better) In this section we analyze how sensitive is our algorithm w.r.t the hyperparameters K and M. In all experiments of the paper, we set K equal to M. FIG6 shows the average return of GrBAL across meta-training iterations of our algorithm for different values of K = M. The performance of the agent is largely unaffected for different values of these hyperparameters, suggesting that our algorithm is not particularly sensitive to these values. For different agents, the optimal value for these hyperparameters depends on various task details, such as the amount of information present in the state (a fully-informed state variable precludes the need for additional past timesteps) and the duration of a single timestep (a longer timestep duration makes it harder to predict more steps into the future). terrain environments. The x-axis shows data aggreation iterations during meta-training, whereas the y-axis shows the average return achieved when running online adaptation with the meta-learned model from the particular iteration. The curves suggest that GrBAL performance is fairly robust to the values of these hyperparameters. For each MuJoCo agent, the same reward function is used across its various tasks. TAB2 shows the reward functions used for each agent. We denote by x t the x-coordinate of the agent at time t, ee t refers to the position of the end-effector of the 7-DoF arm, and g corresponds to the position of the desired goal.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

A model-based meta-RL algorithm that enables a real robot to adapt online in dynamic environments

Represent the TLDR for finding the corresponding scientific article.

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.