Question: Represent

Question: We make the following striking observation: fully convolutional VAE models trained on 32x32 ImageNet can generalize well, not just to 64x64 but also to far larger photographs, with no changes to the model. We use this property, applying fully convolutional models to lossless compression, demonstrating a method to scale the VAE-based'Bits-Back with ANS' algorithm for lossless compression to large color photographs, and achieving state of the art for compression of full size ImageNet images. We release Craystack, an open source library for convenient prototyping of lossless compression using probabilistic models, along with full implementations of all of our compression . Bits back coding (; Hinton & van) is a method for performing lossless compression using a latent variable model. In an ideal implementation, the method can achieve an expected message length equal to the variational free energy, often referred to as the negative evidence lower bound (ELBO) of the model. Bits back was first introduced to form a theoretical argument for using the ELBO as an objective function for machine learning (Hinton & van). The first implementation of bits back coding made use of first-infirst-out (FIFO) arithmetic coding (AC) . However, the implementation did not achieve optimal compression, due to an incompatibility between a FIFO coder and bits back coding, and its use was only demonstrated on a small dataset of 8×8 binary images. Compression' (HiLLoC). In our experiments (Section 4), we demonstrate that HiLLoC can be used to compress color images from the ImageNet test set at rates close to the ELBO, outperforming all of the other codecs which we benchmark. We also demonstrate the speedup, of nearly three orders of magnitude, ing from vectorization. We release an open source implementation based on'Craystack', a Python package which we have written for general prototyping of lossless compression with ANS. In this section we briefly describe the BB-ANS algorithm first introduced by. We begin by giving a high-level description of the ANS LIFO entropy coder , along with a new notation for describing the basic ANS operations. Throughout the rest of the paper we use log to mean the base two logarithm, usually denoted log 2, and we measure message lengths in bits. As an entropy coder, ANS was designed for compressing sequences of discretely distributed symbols. It achieves a compressed message length equal to the negative log-probability (information content) of the sequence plus an implementation dependent constant, which is usually less than 32 bits. For long sequences, the constant overhead has a negligible contribution to the overall compression rate. Thus, by Shannon's source coding theorem , ANS coding is guaranteed to be near-optimal for long sequences. There are two basic operations defined by ANS, which we will refer to as'push' and'pop'. Push encodes a symbol by adding it to an existing message. It has the signature push: (message, symbol) → message. Pop is the inverse of push, and may be used to decode a symbol and recover a message identical to that before pushing. pop: message → (message, symbol). When multiple symbols are pushed in sequence, they must be popped using the precise inverse procedure, which means popping the symbols in the opposite order. Hence why ANS is referred to as a last-in-first-out coder, or a stack. The push and pop operations require access to a probabilistic model of symbols, summarized by a probability mass function p over the alphabet of possible symbols. The way that symbols are encoded depends on the model, and pushing a symbol s according to p in an increase in message length of log 1 p(s). Popping s in an equal reduction in message length. For details on how the ANS operations are implemented, see. Note that any model/mass function can be used for the pop operation, i.e. there's no hard restriction to use the distribution that was used to encode the message. In this way, rather than decoding the same data that was encoded, pop can actually be used to sample a symbol from a different distribution. The pop method itself is deterministic, so the source of randomness for the sample comes from the data contained within the message. This sampling operation, which can be inverted by pushing the sample back onto the stack, is essential for bits back coding. For convenience, we introduce the shorthand notation s → p(·) for encoding (pushing) a symbol s according to p, and s ← p(·) for decoding (popping). Suppose we have a model for data x which involves a latent variable z. A sender and receiver wish to communicate a sample x. They have access to a prior on z, denoted p(z), a likelihood p(x | z) and a (possibly approximate) posterior q(z | x), but not the marginal distribution p(x). Without access to p(x), sender and receiver cannot directly code x using ANS. However, BB-ANS specifies an indirect way to push and pop x. It does not require access to the marginal p(x), but rather uses the prior, conditional, and posterior from the latent variable model. Table 1 (a) shows, in order from the top, the three steps of the BB-ANS pushing procedure which the sender can perform to encode x. The'Variables' column shows the variables known to the sender before each step. 1(b) shows the inverse steps which the receiver can use to pop x, with the'Variables' column showing what is known to the receiver after each step. After decoding x, the third step of popping, z → q(· | x), is necessary to ensure that BB-ANS pop is a precise inverse of push. Table 1: Indirectly pushing and popping x using BB-ANS. → and ← denote pushing and popping respectively. ∆L denotes the change in message length ing from each operation. The three steps to push/pop are ordered, starting at the top of the table and descending. Operation ∆L The change in message length from BB-ANS can easily be derived by adding up the quantities in the ∆L column of Table 1. For encoding we get Taking the expectation over z gives the expected message length for a datum x which is the negative evidence lower bound (ELBO), also known as the free energy. This is a commonly used training objective for latent variable models. The above equation implies that latent variable models trained using the ELBO are implicitly being trained to minimize the expected message length of lossless compression using BB-ANS. Note that, as Table 1 shows, the first step of encoding a data point, x, using BB-ANS is to, counterintuitively, decode (and thereby sample) a latent z ← q(· | x). This requires that there is already a buffer of random data pushed to the ANS coder, which can be popped. This data used to start the encoding process is recovered after the final stage of decoding, hence the name'bits back'. If we have multiple samples to compress, then we can use'chaining', which is essentially repeated application of the procedure in Table 1 . In Section 3.4 we describe how we build up an initial buffer of compressed data by using a different codec to code the first images in a sequence. We now discuss the techniques we introduce to scale up BB-ANS. When all of the layers in the generative and recognition networks of a VAE are either convolutional or elementwise functions (i.e. the VAE has no densely connected layers), then it is possible to evaluate the recognition network on images of any height and width, and similarly to pass latents of any height and width through the generative network to generate an image. Thus, such a VAE can be used as a (probabilistic) model for images of any size. We exploit this fact, and show empirically in Section 4 that, surprisingly, a fully convolutional VAE trained on 32 × 32 images can perform well (in the sense of having a high ELBO) as a model for 64 × 64 images as well as far larger images. This in turn corresponds to a good compression rate, and we implement lossless compression of arbitrary sized images by using a VAE in this way. The primary computational bottlenecks in the original BB-ANS implementation were loops over data and latent variables occurring in the Python interpreter. We have been able to vectorize these, achieving an implementation which can scale to large ImageNet images. The effect of vectorization on runtime is shown in Figure 4. A vectorized implementation of ANS was described in using SIMD instructions. This works by expanding the size of the ANS stack head, from a scalar to a vector, and interleaving the output/input bit stream. We implement this in our lossless compression library, Craystack, using Numpy. Please refer to the Craystack code and to for more detail. We ensure that the compression rate overhead to vectorization is low by using the BitKnit technique described in , see Appendix D for more detail. Having vectorized, we found that most of the compute time for our compression was spent in neural net inference, whether running on CPU or GPU, which we know to already be reasonably well optimized. In Craystack, we further generalize the ANS coder using Numpy's n-dimensional array view interface, allowing the stack head to be'shaped' like an n-dimensional array, or a nested Python data-structure containing arrays. We can then use a shape which fits that of the data that we wish to encode or decode. When coding data according to a VAE we use an ANS stack head shaped into a pair of arrays, matching the shapes of the observation x and the latent z. This allows for a straightforward implementation and clarifies the lack of data dependence between certain operations, such as the and z → p(·) during encoding, which can theoretically be performed concurrently. This vectorized encoding process is visualized in Figure 2. It is standard for state of the art latent variable models to use continuous latent variables. Since ANS operates over discrete probability distributions, if we wish to use BB-ANS with such models it is necessary to discretize the latent space so that latent samples can be communicated. described a static discretization scheme for the latents in a simple VAE with a single layer of continuous latent variables, and showed that this discretization has a negligible impact on compression rate. The addition of multiple layers of stochastic variables to a VAE has been shown to improve performance (; ; Maaløe et al., 2019; Sønderby et al., 2016). Motivated by this, we propose a discretization scheme for hierarchical VAEs with multiple layers of latent variables. The discretization described in is formed by dividing the latent space into intervals of equal mass under the prior p(z). For a hierarchical model, the prior on each layer depends on the previous layers: It isn't immediately possible to use the simple static scheme from , since the marginals p(z 1),..., p(z L−1) are not known. estimate these marginals by sampling, and create static bins based on the estimates. They demonstrate that this approach can work well. We propose an alternative approach, allowing the discretization to vary with the context of the latents we are trying to code. We refer to our approach as dynamic discretization. In dynamic discretization, instead of discretizing with respect to the marginals of the prior, we discretize according to the conditionals in the prior, p(z l | z l+1:L). Specifically, for each latent layer l, we partition each dimension into intervals which have equal probability mass under the conditional p(z l | z l+1:L). This directly generalizes the scheme used in BB-ANS . Dynamic discretization is more straightforward to implement because it doesn't require callibrating the discretization to samples. However it imposes a restriction on model structure, in particular it requires that posterior inference is done top-down. This precludes the use of Bit-Swap. In Section 3.3.1 we contrast the model restriction from dynamic discretization with the bottom-up, Markov restriction imposed by Bit-Swap itself. We give further details about the dynamic discretization implementation we use in Appendix A. Figure 3: Graphical models representing the generative and inference models with HiLLoC and Bit-Swap, both using a 3 layer latent hierarchy. The dashed lines indicate dependence on the fixed observation. The first stage of BB-ANS encoding is to pop from the posterior, z 1:L ← q(· | x). When using dynamic discretization, popping the layer z l requires knowledge of the discretization used for z l and thus of the conditional distribution p(z l | z l+1:L). This requires the latents z l+1:L to have already been popped. Because of this, latents in general must be popped (sampled) in'top-down' order, i.e. z L first, then z L−1 and so on down to z 1. The most general form of posterior for which top-down sampling is possible is This is illustrated, for a hierarchy of depth 3, in Figure 3b. The Bit-Swap technique requires that inference be done bottom up, and that generative and inference models must both be a Markov chain on z 1,..., z L, and thus cannot use skip connections. These constraints are illustrated in Figure 3c,d. Skip connections have been shown to improve model ELBO in very deep models (Sønderby et al., 2016; Maaløe et al., 2019). HiLLoC does not have this constraint, and we do utilize skip connections in our experiments. As discussed in Section 3.3, our dynamic discretization method precludes the use of Bit-Swap for reducing the one-time cost of starting a BB-ANS chain. We propose instead to use a significantly simpler method to address the high cost of coding a small number of samples with BB-ANS, namely we code the first samples using a different codec. The purpose of this is to build up a sufficiently large buffer of compressed data to permit the first stage of the BB-ANS algorithm -to pop a latent sample from the posterior. In our experiments we use the'Free Lossless Image Format' (FLIF) to build up the buffer. We chose this codec because it performed better than other widely used codecs, but in principal any lossless codec could be used. The amount of previously compressed data required to pop a posterior sample from the ANS stack (and therefore start the BB-ANS chain) is roughly proportional to the size of the image we wish to compress, since in a fully convolutional model the size of the latent space is determined by the image size. We can exploit this to allow us to obtain a better compression rate than FLIF as quickly as possible. We do so by partitioning the first images we wish to compress with HiLLoC into smaller patches. These patches require a smaller data buffer, and thus we can use the superior HiLLoC coding sooner than if we attempted to compress full images. We find experimentally that, generally, larger patches have a better coding rate than smaller patches. Therefore we increase the size of the image patches being compressed with HiLLoC as more images are compressed and the size of the data buffer grows, until we finally compress full images once the buffer is sufficiently large. For our experiments on compressing full ImageNet images, we compress 32×32 patches, then 64×64, then 128×128 before switching to coding the full size images directly. Note that since our model can compress any shape image, we can compress the edge patches which will have different shape if the patch size does not divide the image dimensions exactly. Using this technique means that our coding rate improves gradually from the FLIF coding rate towards the coding rate achieved by HiLLoC on full images. We compress only 5 ImageNet images using FLIF before we start compressing 32×32 patches using HiLLoC. Using Craystack, we implement HiLLoC with a ResNet VAE (RVAE) . This powerful hierarchical latent variable model achieves ELBOs comparable to state of the art autoregressive models 2. In all experiments we used an RVAE with 24 stochastic hidden layers. The RVAE utilizes skip connections, which are important to be able to effectively train models with such a deep latent hierarchy. See Appendix E for more details. We trained the RVAE on the ImageNet 32 training set, then evaluated the RVAE ELBO and HiLLoC compression rate on the ImageNet 32 test set. To test generalization, we also evaluated the ELBO and compression rate on the tests sets of ImageNet64, CIFAR10 and full size ImageNet. For full size ImageNet, we used the partitioning method described in 3.4. The are shown in Table 2. For HiLLoC the compression rates are for the entire test set, except for full ImageNet, where we use 2000 random images from the test set. Table 2 shows that HiLLoC achieves competitive compression rates on all benchmarks, and state of the art on full size ImageNet images. The fact that HiLLoC can achieve state of the art compression on ImageNet relative to the baselines, even under a change of distribution, is striking. This provides strong evidence of its efficacy as a general method for lossless compression of natural images. Naively, one might expect a degradation of performance relative to the original test set when changing the test distribution-even more so when the resolution changes. However, in the settings we studied, the opposite was true, in that the average per-pixel ELBO (and thus the compressed message length) was lower on all other datasets compared to the ImageNet 32 validation set. In the case of CIFAR, we conjecture that the reason for this is that its images are simpler and contain more redundancy than ImageNet. This theory is backed up by the performance of standard compression algorithms which, as shown in Table 2, also perform better on CIFAR images than they do on ImageNet 32. We find the compression rate improvement on larger images more surprising. We hypothesize that this is because pixels at the edge of an image are harder to model because they have less context to reduce uncertainty. The ratio of edge pixels to interior pixels is lower for larger images, thus we might expect less uncertainty per pixel in a larger image. To demonstrate the effect of vectorization we timed ANS of single images at different, fixed, sizes, using a fully vectorized and a fully serial implementation. The are shown in Figure 4, which clearly shows a speedup of nearly three orders of magnitude for all image sizes. We find that the run times for encoding and decoding are roughly linear in the number of pixels, and the time to compress an average sized ImageNet image of 500 × 374 pixels (with vectorized ANS) is around 29s on a desktop computer with 6 CPU cores and a GTX 1060 GPU. Our experiments demonstrate HiLLoC as a bridge between large scale latent variable models and compression. To do this we use simple variants of pre-existing VAE models. Having shown that bits back coding is flexible enough to compress well with large, complex models, we see plenty of work still to be done in searching model structures (i.e. architecture search), optimizing with a trade-off between compression rate, encode/decode time and memory usage. Particularly pertinent for HiLLoC is latent dimensionality, since compute time and memory usage both scale with this. Since the model must be stored/transmitted to use HiLLoC, weight compression is also highly relevant. This is a well-established research area in machine learning . Our experiments also demonstrated that one can achieve good performance on a dataset of large images by training on smaller images. This is promising, but future work should be done to discover what the best training datasets are for coding generic images. One question in particular is whether could be improved by training on larger images and/or images of varying size. We leave this to future work. Another related direction for improvement is batch compression of images of different sizes using masking, analogous to how samples of different length may be processed in batches by recurrent neural nets. Whilst this work has focused on latent variable models, there is also promise in applying state of the art fully observed auto-regressive models to lossless compression. We look forward to future work investigating the performance of models such as WaveNet (van den) for lossless audio compression as well as PixelCNN++ and the state of the art models in for images. Sampling speed for these models, and thus decompression, scales with autoregressive sequence length, and can be very slow. This could be a serious limitation, particularly in common applications where encoding is performed once but decoding is performed many times. This effect can be mitigated by using dynamic programming , and altering model architecture , but on parallel architectures sampling/decompression is still significantly slower than with VAE models. On the other hand, fully observed models, as well as the flow based models of and, do not require bits back coding, and therefore do not have to pay the one-off cost of starting a chain. Therefore they may be well suited to situations where one or a few i.i.d. samples are to be communicated. Similar to the way that we use FLIF to code the first images for our experiments, one could initially code images using a fully observed model then switch to a faster latent variable model once a stack of bits has been built up. We presented HiLLoC, an extension of BB-ANS to hierarchical latent variable models, and show that HiLLoC can perform well with large models. We open-sourced our implementation, along with the Craystack package for prototyping lossless compression. We have also explored generalization of large VAE models, and established that fully convolutional VAEs can generalize well to other datasets, including images of very different size to those they were trained on. We have described how to compress images of arbitrary size with HiLLoC, achieving a compression rate superior to the best available codecs on ImageNet images. We look forward to future work reuniting machine learning and lossless compression. After discretizing the latent space, the latent variable at layer l can be treated as simply an index i l into one of the intervals created by the discretization. As such, we introduce the following notation for pushing and popping according to a discretized version of the posterior. Where is the distribution over the intervals of the discretized latent space for z l, with interval masses equal to their probability under q(z l |z l+1:L, x). The discretization is created from splitting the latent space into equal mass intervals under p(z l |z l+1:L). The mass of a given interval under some distribution is the CDF at the upper bound of the interval minus the CDF at the lower end of the interval. We have usedz to indicate that these will be discrete z l values that are reconstructed from the indices i l. In practise we takez l (i l) to be the centre of the interval indexed by i l. It is important to note that the Q l has an implicit dependence on the previous prior distributions p(z k |z k+1:L) for k ≥ l, as these prior distributions are required to calculatez l+1:L and the discretization of the latent space. Since we discretize each latent layer to be intervals of equal mass under the prior, the prior distribution over the indices i l becomes a uniform distribution over the interval indices, U (i l), which is not dependent on i =l. Note that this allows us to push/pop the i l according to the prior in parallel. The full encoding and decoding procedures with a hierarchical latent model and the dynamic discretization we have described are shown in Table 3. Note that the operations in the two tables are ordered top to bottom. Variables Operation Table 3: The BB-ANS encoding and decoding operations, in order from the top, for a hierarchical latent model with l layers. The Q l are posterior distributions over the indices i l of the discretized latent space for the lth latent, z l. The discretization for the lth latent is created such that the intervals have equal mass under the prior. Here we describe a codec to compress a set of images of arbitrary size. The encoder now adds the dimensions of the image being coded to the stream of compressed data, such that the decoder knows what shape the image will be before decoding it. Since we are using a vectorized ANS coder, as described in Section 3.2, we resize the top of the coder in between each coding/decoding step such that the size of the top of the coder matches the sizes of the image and latents being coded. The codec is detailed in Table 4. To make the resizing procedure efficient, we resize via'folding' the top of the vectorized ANS coder such that we are roughly halving/doubling the number of individual ANS coders each time we fold. This makes the cost of the resize logarithmic with the size difference between the vectorized coder and the targeted size. The achieved compression rate on the entire ImageNet validation set is displayed in Table 5. The autoregressive component of the PixelVAE generative model leads to an asymmetry between the times required for compression and decompression. Compression with the PixelVAE model is readily parallelizable across pixels, since we already have access to the pixel values we wish to compress and thus also the conditional distributions on each pixel. However, decompression (equivalently, sampling) is not parallelizable across pixels, since we must decompress a pixel value in order to give us access to the conditional distribution on the next pixel. This means the time complexity of decompression is linear in the number of pixels, making it prohibitively slow for most image sizes. To ensure that the compression rate overhead from using vectorization is low, we use a technique from the BitKnit codec . When we reach the end of encoding, we could simply concatenate the integers in the (vector) stack head to form the final output message. However, this is inefficient because the stack head is not uniformly distributed. As discussed in , elements of the top of the stack have a probability mass roughly Equivalently, the length of h is approximately uniformly distributed. More detailed discussion and an empirical demonstration of this is given by. An efficient way to form the final output message at the end of decoding, is to fold the stack head vector by repeatedly encoding half of it onto the other half, until only a scalar remains, using the above distribution for the encoding. We implement this technique in Craystack and use it for our experiments. The number of (vectorized) encode steps required is logarithmic in the size (i.e. the number of elements) of the stack head. Some of the overhead from vectorization also comes at the start of encoding, when, in existing implementations, the elements of the stack head vector are initialized to copies of a fixed constant. Information from these copies ends up in the message and introduces redundancy which scales with the size of the head. This overhead can be removed by initializing the stack head to a vector of length 1 and then growing the length of the stack head vector gradually as more random data is added to the stack, by decoding new stack head vector elements according to the distribution. A full description of the RVAE architecture is given in , and a full implementation can be found in our repository https://github.com/hilloc-submission/hilloc, but we give a short description below. The RVAE is a hierarchical latent model, trained by maximization of the usual evidence lower bound (ELBO) on the log-likelihood: Take the latent hierarchy to be depth L, such that the latents are z 1:L. There are skip connections in both the generative model, p(x, z 1:L), and the inference model, q(z 1:L | x). Due to our requirement of using dynamic discretization, we use a top-down inference model 7. This means that we can write And the ELBO as Where D KL is the KL divergence. As in , the KL terms are individually clamped as max(D KL, λ), where λ is some constant. This is an optimization technique known as free bits, and aims to prevent latent layers in the hierarchy collapsing such that the posterior is equal to the prior. Each layer in the hierarchy consists of a ResNet block with two sets of activations. One set of activations are calculated bottom-up (in the direction of x to z L), and the other are calculated top-down. The bottom-up activations are used only within q(z 1:L | x), whereas the top-down activations are used by both q(z 1:L | x) and p(x, z 1:L). Every conditional distribution on a latent z l is parameterized as a diagonal Gaussian distribution, with mean and covariance a function of the activations within the ResNet block, and the conditional distribution on x is parameterized by a discretized logistic distribution. Given activations for previous ResNet blocks, the activations at the following ResNet block are a combination of stochastic and deterministic features of the previous latent layer, as well as from skip connections directly passing the previous activations. The features are calculated by convolutions. Note also that all latent layers are the same shape. Since we retained the default hyperparameters from the original implementation, each latent layer has 32 feature maps and spatial dimensions half those of the input (e.g. 7 Note that in , this is referred to as bidirectional inference.

We scale up lossless compression with latent variables, beating existing approaches on full-size ImageNet images.

We develop VAEs where the encoder takes a model parameter vector as input, so we can do rapid inference for many models


Question: Represent this text

Question: State of the art computer vision models have been shown to be vulnerable to small adversarial perturbations of the input. In other words, most images in the data distribution are both correctly classified by the model and are very close to a visually similar misclassified image. Despite substantial research interest, the cause of the phenomenon is still poorly understood and remains unsolved. We hypothesize that this counter intuitive behavior is a naturally occurring of the high dimensional geometry of the data manifold. As a first step towards exploring this hypothesis, we study a simple synthetic dataset of classifying between two concentric high dimensional spheres. For this dataset we show a fundamental tradeoff between the amount of test error and the average distance to nearest error. In particular, we prove that any model which misclassifies a small constant fraction of a sphere will be vulnerable to adversarial perturbations of size $O(1/\sqrt{d})$. Surprisingly, when we train several different architectures on this dataset, all of their error sets naturally approach this theoretical bound. As a of the theory, the vulnerability of neural networks to small adversarial perturbations is a logical consequence of the amount of test error observed. We hope that our theoretical analysis of this very simple case will point the way forward to explore how the geometry of complex real-world data sets leads to adversarial examples. There has been substantial work demonstrating that standard image models exhibit the following phenomenon: most randomly chosen images from the data distribution are correctly classified and yet are close to a visually similar nearby image which is incorrectly classified BID22. This is often referred to as the phenomenon of adversarial examples. These adversarially found errors can be constructed to be surprisingly robust, invariant to viewpoint, orientation and scale BID3. Despite some theoretical work and many proposed defense strategies BID6 BID18 BID20 ) the cause of this phenomenon is still poorly understood. There have been several hypotheses proposed regarding the cause of adversarial examples. We briefly survey some of them here. One common hypothesis is that neural network classifiers are too linear in various regions of the input space, BID17. Another hypothesis is that adversarial examples are off the data manifold BID2 a; BID16. BID6 argue that large singular values of internal weight matrices may cause the classifier to be vulnerable to small perturbations of the input. Alongside works endeavoring to explain adversarial examples, others have proposed defenses in order to increase robustness. Some works increase robustness to small perturbations by changing the non-linearities used BID14, distilling a large network into a small network BID20, or using regularization BID6. Other works explore detecting adversarial examples using a second statistical model BID7 BID0 BID11 BID19. However, many of these methods have been shown to fail BID4 BID7. Finally, adversarial training has been shown in many instances to increase robustness BID18 BID15 BID22. Despite some progress on increasing robustness to adversarial perturbations, local errors have still been shown to appear for distances just beyond what is adversarially trained for BID21. This phenomenon is quite intriguing given that these models are highly accurate on the test set. We hypothesize that this behavior is a naturally occurring of the high dimensional nature of the data manifold. In order to begin to investigate this hypothesis, we define a simple synthetic task of classifying between two concentric high dimensional spheres. This allows us to study adversarial examples in a setting where the data manifold is well defined mathematically and where we have an analytic characterization of the decision boundary learned by the model. Even more importantly, we can naturally vary the dimension of the data manifold and study the effect of the input dimension on the geometry of the generalization error of neural networks. Our experiments and theoretical analysis on this dataset demonstrate the following:• A similar behavior to that of image models occurs: most randomly chosen points from the data distribution are correctly classified and yet are "close" to an incorrectly classified input. This behavior occurs even when the test error rate is less than 1 in 10 million.• For this dataset, there is a fundamental tradeoff between the amount of generalization error and the average distance to the nearest error. In particular, we show that any model which misclassifies a small constant fraction of the sphere will be vulnerable to adversarial perturbations of size O(1 DISPLAYFORM0 • Neural networks trained on this dataset naturally approach this theoretical optimal tradeoff between the measure of the error set and the average distance to nearest error. This implies that in order to linearly increase the average distance to nearest error, the error rate of the model must decrease exponentially.• We also show that models trained on this dataset may become extremely accurate even when ignoring a large fraction of the input. We conclude with a detailed discussion about the connection between adversarial examples for the sphere and those for image models. The data distribution is mathematically described as two concentric spheres in d dimensions: we generate a random x ∈ R d where ||x|| 2 is either 1.0 or R, with equal probability assigned to each norm (for this work we choose R = 1.3). We associate with each x a target y such that y = 0 if ||x|| 2 = 1 and y = 1 if ||x|| 2 = R.Studying a synthetic high dimensional dataset has many advantages:• The probability density of the data p(x) is well defined and is uniform over all x in the support. We can also sample uniformly from p(x) by sampling z ∼ N (0, I) and then setting x = z/||z|| 2 or x = Rz/||z|| 2.• There is a theoretical max margin boundary which perfectly separates the two classes (the sphere with radius (R + 1)/2).• We can design machine learning models which provably can learn a decision boundary which perfectly separate the two spheres.• We can control the difficulty of the problem by varying d, and R.Our choice of R = 1.3 was a bit arbitrary and we did not explore in detail the relationship between adversarial examples and the distance between to two spheres. Additionally, our choice to restrict the data distribution to be the shells of the two spheres was made to simplify the problem further. In our experiments we investigate training on this dataset in two regimes. First, the online setting where each minibatch is a uniform sample from p(x) (N = ∞). Second, where there is a fixed training set of size N and the network is trained for many epochs on this finite sample. Our first experiment used an input dimensionality of d = 500. We then train a 2 hidden layer ReLU network with 1000 hidden units on this dataset. We applied batch normalization (Ioffe & Szegedy, Figure 1 : Visualizing a 2d slice of the input space where the subspace is spanned by: 2 randomly chosen directions (left), 1 random and 1 "adversarial direction" (center), and 2 orthogonal "adversarial directions" (right). The data manifold is indicated in black and the max margin boundary in red. The green area indicates points which are classified by the ReLU network as being on the inner sphere. In the last plot, the projection of the entire outer sphere is misclassified despite the fact that the error rate of the model is less than 1 in 10 million. 2015) to the two hidden layers, but not to the readout layer. We train with minibatch SGD, minimizing the sigmoid cross entropy loss. We use Adam optimizer for 1 million training steps with mini batch size 50 and learning rate 0.0001. Because this is training in the online setting with batch size 50 and 1 million training points, 50 million data points were used during training. We evaluated the final model on 10 million uniform samples from each sphere -20 million points in total and observed no errors on these finite samples. Thus the error rate of this model is unknown, we only have a statistical upper bound on the error rate. Despite this, we are able to adversarially find errors on the data manifold by performing gradient descent on the spheres (see Section 3.1). There are two types of adversarial examples we generate using this method, the first are worst-case examples, where we iterate the attack until the attack objective converges and do not restrict to a local region around the starting point. The second type are nearest neighbor examples, where we terminate the attack on the first misclassification found. In Figure 1 we visualize the decision boundary by taking different 2d projections of the 500 dimensional space. When we take a random projection, the model has closely approximated the max margin boundary on this projection. Note the model naturally interpolates between the two spheres despite only being trained on samples from the surfaces of the spheres. By contrast, when we take a 2d projection where one basis vector is a worst-case adversarial example, the model's decision boundary is highly warped along this "adversarial direction". There are points of norm > 2 for which the model is confident is on the inner sphere. We can also take a slice where the x and y axis are an orthogonal basis for the subspace spanned to two different worst-case examples. Although the last plot shows that the entire projection of the outer sphere is misclassified, the volume of this error region is exceedingly small due to the high dimensional space. Despite being extremely rare, these misclassifications appear close to randomly sampled points on the sphere. The mean L2 distance to the nearest error on the data manifold is 0.18, by comparison two randomly sampled points on the inner sphere are typically around √ 2 ≈ 1.41 distance from each other. If we look for the nearest point in between the two spheres which is classified as being on the outer sphere, then we get an average L2 distance of 0.0796, and an average norm of 1.07. Thus the nearest example of the other class is typically about half the distance to the theoretical margin. This phenomenon of very unlikely but local errors appears only when the spheres are high dimensional. In FIG0 (right), we visualize the same model trained on 100 samples in the case where d = 2. The model makes no errors on the data manifold. In our experiments the highest dimension we were able to train the ReLU net where no errors can be found adversarially (local or not) seems to be around d = 60. We consider the ReLU net trained on 50 million samples from two 500 dimensional spheres of radius 1.0 and 1.3. We evaluate the accuracy of this network on the entire space using a theoretical decision boundary of 1.15. For each norm considered we plot the accuracy among 10000 random samples. We see the accuracy rapidly increases as we move away from the margin. As we move far enough away we no longer observe errors on the random samples. However, we are able to adversarially find errors as far as norms.6 and 2.4. Right: We trained the same ReLU net on 100 samples from the data distribution when d = 2. By visualizing predictions on a dense subset of the entire space it appears that the model makes no errors on either circle. Several recent works have hypothesised that adversarial examples are off the data manifold BID2 a; BID16. We wanted to test if adversarial examples were off the data manifold. To that end we designed an attack which specifically produces adversarial examples on the data manifold which we call a manifold attack. Traditional attack methods for image models start with an input x and target classŷ and finds an inputx that maximizes P (ŷ |x) subject to the constraint ||x −x|| <, where || · || is often chosen to be the L ∞ norm. The manifold attack maximizes P (ŷ |x) subject to the constraint ||x|| 2 = ||x|| 2. This ensures that the produced adversarial example is of the same class as the starting point and lies in the support of the data distribution. We solve this constraint problem using projected gradient descent (PGD), only for the projection step we project back on the sphere by normalizing ||x|| 2. Because this attack only produces adversarial examples on the data manifold, their probability under the data distribution is identical to that of correctly classified points in that p(x) = p(x adv). It is difficult to reason about the learned decision boundary of the ReLU network. To obtain a more complete understanding of the decision boundary, we next study a simpler model. The network, dubbed "the quadratic network", is a single hidden layer network where the pointwise non-linearity is a quadratic function, σ(x) = x 2. There is no bias in the hidden layer, and the output simply sums the hidden activations, multiplies by a scalar and adds a bias. With hidden dimension h the network has d × h + 2 learn-able parameters. The logit is written aŝ DISPLAYFORM0 where W 1 ∈ R hxd, 1 is a column vector of h 1's. Finally, w and b are learned scalars. In the Appendix, we show that the output of this network can be rewritten in the form When we train the quadratic network with h = 1000 using the same setup as in Section 3 we arrive at the perfect solution: all of the α i ∈ [1/R 2, 1] and there are no adversarial examples. This again was in the online learning setup where each minibatch was an iid sample from the data distribution. The story is different, however, if we train on a finite sample from the data distribution. In particular if we sample N = 10 6 data points from p(x) as a fixed finite training set and train using the same setup we arrive at a model which empirically has a very low error rate -randomly sampling 10 million datapoints from each sphere in no errors, but for which there are adversarial examples. In fact, 394 out of 500 of the learned α i are incorrect in that α i ∈ [1/R 2, 1] (for a complete histogram see Figure 3). DISPLAYFORM1 We can use the Central Limit Theorem (CLT) to estimate the error rate of the quadratic network from the α i (Section 4.1). The estimated error rate of this particular model to be ≈ 10 −11. Note, we are applying the CLT at the tails of the distribution, so it is unclear how accurate this estimate is. However, we found the CLT closely approximates the error rate in the regime where it is large enough to estimate numerically. Next we augmented the above setup with a "perfect" initialization; we initialize the quadratic network at a point for which all of the α i are "correct" but there are non-zero gradients due to the sigmoid cross-entropy loss. The network is initialized at a point where the sigmoid probability of y = 1 for the inner sphere and outer spheres is.0016 and 0.9994 respectively. As shown in Figure 3 continued training from this initialization in a rapid divergence of the worst and average case loss. Although the average loss on the test set decreases with further training, the worst case rapidly increases and adversarial examples can once again be found after 1000 training steps. This behavior from the fact that the training objective (average sigmoid cross entropy loss) does not directly track the accuracy of the models. It also demonstrates how the worst and average case losses may diverge when the input is high dimensional. We then plot what k/d needs to be in order for the model to obtain a certain error rate. We find that as the input dimension grows, the ratio k/d needed quickly decreases. We can use the CLT 1 to analytically estimate the accuracy for the quadratic network in terms of the α i. The following proposition estimates the error rate on the inner sphere:Proposition 4.1 Consider the decision boundary of the quadratic network of the form DISPLAYFORM0 Let Z = N. We let z ∼ S 0 to denote that the vector z is uniformly distributed on the inner DISPLAYFORM1 2. Then the error rate on the inner sphere can be estimated as DISPLAYFORM2 Proposition 4.1 implies that there are many settings of α i which obtain very low error rates. As long as E[α i] ≈ (1 + R −2)/2) and their variance is not too high, the model will be extremely accurate. The histogram in Figure 3 illustrates this; i.e. the learned model has an error rate of 10 −11 but 80% of the α i are incorrect. For a typical sample, the model sums incorrect numbers together and obtains the correct answer. Flexibility in choosing α i while maintaining good accuracy increases dramatically with the input dimension. To illustrate this further consider a special case of the quadratic network where the decision boundary is of the form DISPLAYFORM3 This simplified model has two parameters, k the number of dimensions the model looks at and b a threshold separating the two classes. How large does k need to be in order for the model to obtain a desired error rate? (Assuming b is chosen optimally based on k). We answer this question using the CLT approximation in Proposition 4.1. In FIG2 we plot the fraction of input dimensions needed to obtain a desired accuracy using this simplified model. For example, if d = 3000 then the model can obtain an estimated accuracy of 10 −14 while only looking at 50% of the input. As demonstrated in section 3, neural networks trained on the sphere dataset exhibit a similar phenomenon to that of image datasets: most random samples from the data distribution are both correctly classified and close to a nearby misclassified point. In this work, we do not attempt to compare the geometry of the natural image manifold to that of the sphere, but we can explain why this property occurs on the sphere dataset. Let S 0 be in the sphere of radius 1 in d dimensions and fix E ⊆ S 0 (we interpret E to be the set of points on the inner sphere which are misclassified by some model). For x ∈ S 0 let d(x, E) denote the L 2 distance between x and the nearest point in the set E. Let d(E) = E x∼S0 d(x, E) denote the average distance from a uniformly sampled point on the sphere to the set E. Finally, let µ(E) denote the measure of E as a fraction of the sphere (so µ(S 0) = 1). We prove the following theorem in the Appendix:Theorem 5.1 Consider any model trained on the sphere dataset. Let p ∈ [0, 1.0] denote the accuracy of the model on the inner sphere, and let E denote the points on the inner sphere the model misclassifies (so in measure DISPLAYFORM0 This theorem directly links the probability of an error on the test set to the average distance to the nearest error independently of the model. Any model which misclassifies a small constant fraction of the sphere must have errors close to most randomly sampled data points, no matter how the model errors are distributed on the sphere. At a high level it follows as direct corollary of an isoperimetric inequality of BID8 . The error set E of fixed measure µ(E) which maximizes the average distance to the nearest error d(E) is a "cap", which is a set of the form E = {x ∈ S 0 : x i > α} (or more generally the sphere intersected with a half space). When d is large we can estimate d(E) by using the fact that for x chosen randomly on S 0 the distribution of a single coordinate DISPLAYFORM1 This illustrates the counter-intuitive property of high dimensional spheres, for large d a set of measure say.01% concentrated near a pole will extend all the way to within O(1/ √ d) of the equator, and a randomly chosen x ∼ p(x) will with high probability lie close to the equator. Theorem 5.1 gives an optimal trade off between the amount of generalization error and the average distance to nearest error. We can compare how the error sets of actual trained neural networks compare with this optimal bound. We do this comparison in Figure 5. We train three different architectures on the sphere dataset when d = 500, the first is a "small" ReLU network with 1000 hidden units and 2 hidden layers (ReLU-h1000-d2), the second is the quadratic network with 1000 hidden units (Quad-h1000), and the third is a "large" ReLU network with 2000 hidden units and depth 8 (Relu-h2000-d8). We train these networks with varying number of samples from the data distribution, in particular N ∈ {1000, 5000, 10000, 100000, ∞}. We then sample the performance of the networks several times during training, computing both the error rate of the model and the average distance to nearest error. The error rate is estimated from 100000 random samples from the data distribution and the average distance is estimated by 100 random points and running PGD for 1000 steps with step size of.001 (searching only on the data manifold for errors). Each point on the plot is a network at a certain snapshot during training (when N >= 10000 the networks later in training become so accurate that the error rate cannot be estimated statistically from 100000 samples, these points do not appear in the graph). Surprisingly we see that the trade off between the average distance to nearest error and the amount of error is close to what is optimal if all the errors were concentrated near a "cap" (as represented by a black line). Note that there is some noise in estimating the error rates and average distances (for example PGD is not guaranteed to find the closest error) as a some networks when sampled appear slightly better than optimal. This plot suggests that the decision boundaries of these networks are all well behaved given the amount of test error observed. For the quadratic network, the error set on the inner sphere is of the form E = {x ∈ R n : ||x|| 2 = 1, DISPLAYFORM2 Geometrically this is the area of the sphere which is outside the ellipsoid decision boundary. If α i > 1 for 2 or more i, then E is a connected region. This gives some intuition as to why the quadratic network might approach the optimal tradeoff between d(E) and µ(E). Perhaps more interesting is that both the small and large ReLU networks have similar ReLU-h1000-d2 ReLU-h2000-d8 Quad-h1000 Optimal Figure 5: We compare the average distance to nearest error with error rate for 3 networks trained on the sphere dataset. All errors are reported for the inner sphere. The 3 networks are trained with 5 different training set sizes, and their performance are measured at various points during training (the networks eventually become too accurate to appear on this graph, as the error rate will be too small to estimate statistically). Amazingly, we observe that the trade off between the amount of error and the average distance to nearest error closely tracks what is optimal, as would be observed if all the errors were concentrated near a pole of the sphere.tradeoffs between d(E) and µ(E), even though they have much more complicated architectures. The large ReLU network, for example, has over 29 million parameters. Despite the complexity, the error region of this large network demonstrates a similar tradeoff as the quadratic network. In this work we attempted to gain insight into the existence of adversarial examples for image models by studying a simpler synthetic dataset. After training different neural network architectures on this dataset we observe a similar phenomenon to that of image models -most random points in the data distribution are both correctly classified and are close to a misclassified point. We then explained this phenomenon for this particular dataset by proving a theoretical tradeoff between the error rate of a model and the average distance to nearest error independently of the model. We also observed that several different neural network architectures closely match this theoretical bound. Theorem 5.1 is significant because it reduces the question of why models are vulnerable to adversarial examples to the question of why is there a small amount of classification error. It is unclear if anything like theorem 5.1 would hold for an image manifold, and future work should investigate if a similar principal applies. Our work suggests that even a small amount of classification error may sometimes logically force the existence of many adversarial examples. This could explain why fixing the adversarial example problem has been so difficult despite substantial research interest. For example, one recent work uses adversarial training to increase robustness in the L ∞ metric BID18. Although this did increase the size,, of the perturbation needed to reliably produce an error, local errors still remain for larger than those adversarially trained for BID21.Several defenses against adversarial examples have been proposed recently which are motivated by the assumption that adversarial examples are off the data manifold BID2 a; BID16. Our challenge whether or not this assumption holds in general. As shown in section 3 there are local errors both on and off the data manifold. Our raise many questions as to whether or not it is possible to completely solve the adversarial example problem without reducing test error to 0. The test error rate of state of the art image models is non-zero, this implies that a constant fraction of the data manifold is misclassified and is the unbiased estimate of µ(E). Perhaps this alone is an indication that local adversarial errors exist. The concentric spheres dataset is an extremely simple problem which is unlikely to capture all of the complexities of the geometry of a natural image manifold. Thus we cannot reach the same about the nature of adversarial examples for real-world datasets. However, we hope that the insights gained from this very simple case will point the way forward to explore how complex real-world data sets leads to adversarial examples.

represent the natural language For example, Rederive a wide class of inference procedures from an global information bottleneck objective. should be similar to In classic papers, Zellner demonstrated that Bayesian inference could be derived as the solution to an information theoretic functional. Below we derive a generalized form of this functional as a variational lower bound of a predictive information bottleneck objective. This generalized functional encompasses most modern inference procedures and suggests novel ones. Consider a data generating process φ ∼ p(φ) from which we have some N draws that constitute our training set, x P = {x 1, x 2, . . ., x N} ∼ p(x|φ). We can also imagine (potentially infinitely many) future draws from this same process x F = {x N +1, . . .} ∼ p(x|φ). The predictive information I(x P ; x F) 1 gives a unique measure of the complexity of a data generating process . The goal of learning is to capture this complexity. To perform learning, we form a global representation of the dataset p(θ|x P). This can be thought of as a learning algorithm, that, given a set of observations, produces a summary statistic of the dataset that we hope is useful for predicting future draws from the same process. This algorithm could be deterministic or more generally, stochastic. For example, imagine training a neural network on some data with stochastic gradient descent. Here the training data would be x P, the test data x F and the neural network parameters would be θ. Our training procedure implicitly samples from the distribution p(θ|x P). How do we judge the utility of this learned global representation? The mutual information I(θ; x F) quantifies the amount of information our representation captures about future draws. 2 To maximize learning we therefore aim to maximize this quantity. 1. We use I(x; y) for the mutual information between two random variables: 2. It is interesting to note that in the limit of an infinite number of future draws, I(θ; x F) approaches I(θ; φ). Therefore, the amount of information we have about an infinite number of future draws from This is, of course, only interesting if we constrain how expressive our global representation is, for otherwise we could simply retain the full dataset. The amount of information retained about the observed data: I(θ; x P) is a direct measure of our representation's complexity. The bits a learner extracts from data provides upper bounds on generalization . Combined, these motivate the predictive information bottleneck objective, a generalized information bottleneck : We can turn this into an unconstrained optimization problem with the use of a Lagrange multiplier β: max While this objective seems wholly out of reach, we can make progress by noting that our random variables satisfy the Markov chain: x F ← φ → x P → θ, in which θ and x F are conditionally independent given x P: This implies: and the equivalent unconstrained optimization problem: 3 The first term here: I(θ; x P |x F) is the residual information between our global representation and the dataset after we condition on full knowledge of the data generating procedure. This is a direct measure of the inefficiency of our proposed representation. Simple variational bounds can be derived for this objective, just as was done for the (local) information bottleneck objective in. First, we demonstrate a variational upper bound on I(θ; the process is the same as the amount of information we have about the nature and identity of the data generating process itself. 3. A similar transformation for the (local) variational information bottleneck appeared in. 4. · is used to denote expectations, and unless denoted otherwise with respect to the full joint density Here we upper bound the residual information by using a variational approximation to p(θ|x F), the marginal of our global representation over all datasets drawn from the same data generating procedure. Any distribution q(θ) independent of x F suffices. Next we variationally lower bound I(θ; x P) with: The entropy of the training data H(x P) is a constant outside of our control that can be ignored. Here we variationally approximate the "posterior" of our global representation with a factorized "likelihood": i q(x i |θ) = q(x P |θ) ∼ p(x P |θ). Notice that while p(x P |θ) will not factorize in general, we can certainly consider a family of variational approximations that do. Combining these variational bounds, we generate the objective: We have thus derived, as a variational lower bound on the predictive information bottleneck, the objective postulates (with β = 1) is satisfied for inference procedures that optimally process information. demonstrates, this encompasses a wide array of modern inference procedures, including Generalized Bayesian Inference and a generalized Variational Inference, dubbed Gibbs VI . 5 Below we highlight some of these and other connections. If, in Equation, we identity q(θ) with a fixed prior and q(x|θ) with a fixed likelihood of a generative model, optimizing this objective for p(θ|x P) in the space of all probability densities gives the generalized Boltzmann distribution : where Z is the partition function. 6 This is a generalized form of Bayesian Inference called the power likelihood . Here the inverse temperature β acts as a Lagrange multiplier controlling the trade-off between the amount of information we retain about our observed data (I(θ; x P)) and how much predictive information we capture (I(θ; x F)). As β → ∞ (temperature goes to zero), we recover the maximum likelihood solution. At β = 1 (temperature = 1) we recover ordinary Bayesian inference. As β → 0 (temperature goes to infinity), we recover just prior predictive inference that ignores the data entirely. These limits are summarized in Table 1. 5. To incorporate the Generalized VI with divergence measures other than KL, we need only replace our mutual informations (which are KL based) with their corresponding generalizations. Table 1: Power Bayes can be recovered as a variational lower bound on the predictive information bottleneck objective (Equation). More generally, notice that in Equation the densities q(x|θ) and q(θ) are not literally the likelihood and prior of a generative model, they are variational approximations that we have complete freedom to specify. This allows us to describe other more generalized forms of Bayesian inference such as Divergence Bayes or the full Generalized Bayes provided we can interpret the chosen loss function as a conditional distribution. If we limit the domain of p(θ|x P) to a restricted family of parametric distributions, we immediately recover not only standard variational inference, but a broad generalization known as Gibbs Variational Inference (; ;). Furthermore, nothing prevents us from making q(x|θ) or q(θ) themselves parametric and simultaneously optimizing those. Optimizing the prior with a fixed likelihood, unconstrained p(θ|x P), and β = 1 the objective mirrors Empirical Bayesian approaches, including the notion of reference priors . Alternatively, optimizing a parametric likelihood with a parametric representation p(θ|x P), fixed prior, and β = 1 equates to a Neural Process . Consider next data augmentation, where we have some stochastic process that modifies our data with implicit conditional density t(x |x). If the augmentation procedure is centered about zero so that x t(x |x) = x and our chosen likelihood function is concave, then we have: which maintains our bound. For example, for an exponential family likelihood and any centered augmentation procedure (like additive mean zero noise), doing generalized Bayesian inference on an augmented dataset is also a lower bound on the predictive information bottleneck objective. We have shown that a wide range of existing inference techniques are variational lower bounds on a single predictive information bottleneck objective. This connection highlights the drawbacks of these traditional forms of inference. In all cases considered in the previous section, we made two choices that loosened our variational bounds. First, we approximated p(x P |θ), with a factorized approximation q(x P |θ) = i q(x i |θ). Second, we approximated the future conditional marginal p(θ|x F) = dx P p(θ|x P)p(x P |x F) as an unconditional "prior". Neither of these approximations is necessary. For example, consider the following tighter "prior": q(θ|x F) ∼ dx P p(θ|x P)q(x P |x F). Here we reuse a tractable global representation p(θ|x P) and instead create a variational approximation to the density of alternative datasets drawn from the same process: q(x P |x F). We believe this information-theoretic, representation-first perspective on learning has the potential to motivate new and better forms of inference. 7

We hypothesize that the vulnerability of image models to small adversarial perturbation is a naturally occurring result of the high dimensional geometry of the data manifold. We explore and theoretically prove this hypothesis for a simple synthetic dataset.

Represent the TLDR for finding the corresponding scientific article

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.


Question: Represent the article for finding a single-sentence TLDR

Question: It has been established that diverse behaviors spanning the controllable subspace of a Markov decision process can be trained by rewarding a policy for being distinguishable from other policies. However, one limitation of this formulation is the difficulty to generalize beyond the finite set of behaviors being explicitly learned, as may be needed in subsequent tasks. Successor features provide an appealing solution to this generalization problem, but require defining the reward function as linear in some grounded feature space. In this paper, we show that these two techniques can be combined, and that each method solves the other's primary limitation. To do so we introduce Variational Intrinsic Successor FeatuRes (VISR), a novel algorithm which learns controllable features that can be leveraged to provide enhanced generalization and fast task inference through the successor features framework. We empirically validate VISR on the full Atari suite, in a novel setup wherein the rewards are only exposed briefly after a long unsupervised phase. Achieving human-level performance on 12 games and beating all baselines, we believe VISR represents a step towards agents that rapidly learn from limited feedback. Unsupervised learning has played a major role in the recent progress of deep learning. Some of the earliest work of the present deep learning era posited unsupervised pre-training as a method for overcoming optimization difficulties inherent in contemporary supervised deep neural networks . Since then, modern deep neural networks have enabled a renaissance in generative models, with neural decoders allowing for the training of large scale, highly expressive families of directed models (; Van den) as well as enabling powerful amortized variational inference over latent variables . We have repeatedly seen how representations from unsupervised learning can be leveraged to dramatically improve sample efficiency in a variety of supervised learning domains . In the reinforcement learning (RL) setting, the coupling between behavior, state visitation, and the algorithmic processes that give rise to behavior complicate the development of "unsupervised" methods. The generation of behaviors by means other than seeking to maximize an extrinsic reward has long been studied under the psychological auspice of intrinsic motivation (; ;), often with the goal of improved exploration (Şimşek and ; ;). However, while exploration is classically concerned with the discovery of rewarding states, the acquisition of useful state representations and behavioral skills can also be cast as an unsupervised (i.e. extrinsically unrewarded) learning problem for agents interacting with an environment. In the traditional supervised learning setting, popular classification benchmarks have been employed (with labels removed) as unsupervised representation learning benchmarks, wherein the acquired representations are evaluated based on their usefulness for some downstream task (most commonly the original classification task with only a fraction of the labels reinstated). Analogously, we propose removing the rewards from an RL benchmark environment for unsupervised pre-training of an agent, with their subsequent reinstatement testing for dataefficient adaptation. This setup emulates scenarios where unstructured interaction with the environment, or a closely related environment, is relatively inexpensive to acquire and the agent is expected to perform one or more tasks defined in this environment in the form of rewards. The current state-of-the-art for RL with unsupervised pre-training comes from a class of algorithms which, independent of reward, maximize the mutual information between latent variable policies and their behavior in terms of state visitation, an objective which we refer to as behavioral mutual information (; ; ;). These objectives yield policies which exhibit a great deal of diversity in behavior, with variational intrinsic control (, VIC) and diversity is all you need (, DIAYN) even providing a natural formalism for adapting to the downstream RL problem. However, both methods suffer from poor generalization and a slow inference process when the reward signal is introduced. The fundamental problem faced by these methods is the requirement to effectively interpolate between points in the latent behavior space, as the most task-appropriate latent skill likely lies "between" those learnt during the unsupervised period. The construction of conditional policies which efficiently and effectively generalize to latent codes not encountered during training is an open problem for such methods. Our main contribution is to address this generalization and slow inference problem by making use of another recent advance in RL, successor features . Successor features (SF) enable fast transfer learning between tasks that differ only in their reward function, which is assumed to be linear in some features. Prior to this work, the automatic construction of these reward function features was an open research problem. We show that, despite being previously cast as learning a policy space, behavioral mutual information (BMI) maximization provides a compelling solution to this feature learning problem. Specifically, we show that the BMI objective can be adapted to learn precisely the features required by SF. Together, these methods give rise to an algorithm, Variational Intrinsic Successor FeatuRes (VISR), which significantly improves performance in the RL with unsupervised pre-training scenario. In order to illustrate the efficacy of the proposed method, we augment the popular 57-game Atari suite with such an unsupervised phase. The use of this well-understood collection of tasks allows us to position our contribution more clearly against the current literature. VISR achieves human-level performance on 12 games and outperforms all baselines, which includes algorithms that operate in three regimes: strictly unsupervised, supervised with limited data, and both. As usual, we assume that the interaction between agent and environment can be modeled as a Markov decision process . An MDP is defined as a tuple M ≡ (S, A, p, r, γ) where S and A are the state and action spaces, p(·|s, a) gives the nextstate distribution upon taking action a in state s, and γ ∈ is a discount factor that gives smaller weights to future rewards. The function r: S × A × S → R specifies the reward received at transition s a − → s; more generally, we call any signal defined as c: S × A × S → R a cumulant . As previously noted, we consider the scenario where the interaction of the agent with the environment can be split into two stages: an initial unsupervised phase in which the agent does not observe any rewards, and the usual reinforcement learning phase in which rewards are observable. During the reinforcement learning phase the goal of the agent is to find a policy π: S → A that maximizes the expected return G t = ∞ i=0 γ i R t+i, where R t = r(S t, A t, S t+1). A principled way to address this problem is to use methods derived from dynamic programming, which heavily rely on the concept of a value function . The action-value function of a policy π is defined as expected value when following policy π. Based on Q π we can compute a greedy policy π is guaranteed to do at least as well as π, that is: and π are called policy evaluation and policy improvement, respectively; under certain conditions their successive application leads to the optimal value function Q *, from which one can derive an optimal policy using. The alternation between policy evaluation and policy improvement is at the core of many RL algorithms, which usually carry out these steps only approximately . Clearly, if we replace the reward r(s, a, s) with an arbitrary cumulant c(s, a, s) all the above still holds. In this case we will use Q π c to refer to the value of π under cumulant c and the associated optimal policies will be referred to as π c, where π c (s) is the greedy policy on Q * c (s, a). Usually it is assumed, either explicitly or implicitly, that during learning there is a cost associated with each transition in the environment, and therefore the agent must learn a policy as quickly as possible. Here we consider that such a cost is only significant in the reinforcement learning phase, and therefore during the unsupervised phase the agent is essentially free to interact with the environment as much as desired. The goal in this stage is to collect information about the environment to speed up the reinforcement learning phase as much as possible. In what follows we will make this definition more precise. Following Barreto et al. (2017;, we assume that there exist features φ(s, a, s) ∈ R d such that the reward function which specifies a task of interest can be written as where w ∈ R d are weights that specify how desirable each feature component is, or a'task vector' for short. Note that, unless we constrain φ somehow, is not restrictive in any way: for example, by making φ i (s, a, s) = r(s, a, s) for some i we can clearly recover the rewards exactly. note that allows one to decompose the value of a policy π as where φ t = φ(S t, A t, S t+1) and ψ π (s, a) are the successor features (SFs) of π. SFs can be seen as multidimensional value functions in which φ(s, a, s) play the role of rewards, and as such they can be computed using standard RL algorithms (Szepesvári, 2010). One of the benefits provided by SFs is the possibility of quickly evaluating a policy π. Suppose that during the unsupervised learning phase we have computed ψ π; then, during the supervised phase, we can find a w ∈ R d by solving a regression problem based on and then compute Q π through. Once we have Q π, we can apply to derive a policy π that will likely outperform π. Since π was computed without access to the reward, its is not deliberately trying to maximize it. Thus, the solution π relies on a single step of policy improvement over a policy that is agnostic to the rewards. It turns out that we can do better than that by extending the strategy above to multiple policies. Let e: (S → A) → R k be a policy-encoding mapping, that is, a function that turns policies π into vectors in R k. Borsa et al.'s universal successor feature (USFs) are defined as ψ(s, a, e(π)) ≡ ψ π (s, a). Note that, using USFs, we can evaluate any policy π by simply computing Now that we can compute Q π for any π, we should be able to leverage this information to improve our previous solution based on a single policy. This is possible through generalized policy improvement (, GPI). Let ψ be USFs, let π 1, π 2,..., π n be arbitrary policies, and let It can be shown that is a strict generalization of, in the sense that Q π (s, a) ≥ Q πi (s, a) for all π i, s, and a. This can be extended to the case in which holds only approximately and ψ is replaced by a universal successor feature approximator (USFA) ψ θ ≈ ψ(s, a) . The above suggests an approach to leveraging unsupervised pre-training for more dataefficient reinforcement learning. First, during the unsupervised phase, the agent learns a USFA ψ θ. Then, the rewards observed at the early stages of the RL phase are used to find an approximate solution w for. Finally, n policies π i are generated and a policy π is derived through. If the approximations used in this process are reasonably accurate, π will be an improvement over π 1, π 2,.., π n. However, in order to actually implement the strategy above we have to answer two fundamental questions: (i) Where do the features φ in come from? (ii) How do we define the policies π i used in? It turns out that these questions allow for complementary answers, as we discuss next. Features φ should be defined in such a way that the down-stream task reward is likely to be a simple function of them (see). Since in the RL with unsupervised pre-training regime the task reward is not available during the long unsupervised phase, this amounts to utilizing a strong inductive bias that is likely to yield features relevant to the rewards of any'reasonable' task. One such bias is to only represent the subset of observation space that the agent can control . This can be accomplished by maximizing the mutual information between a policy conditioning variable and the agent's behavior. There exist many algorithms that maximize this quantity through various means and for various definitions of'behavior' . The objective F(θ) is to find policy parameters θ that maximize the mutual information (I) between some policy-conditioning variable, z, and some function f of the trajectory τ induced by the conditioned policy, where H is the entropy of some variable: While in general z will be a function of the state , it is common to assume that z is drawn from a fixed (or at least state-independent) distribution for the purposes of stability . This simplifies the objective to minimizing the conditional entropy of the conditioning variable given the trajectory. When the trajectory is sufficiently long, this corresponds to sampling from the steady state distribution induced by the policy. Commonly f is assumed to return the final state, but for simplicity we will consider that f samples a single state s uniformly over τ π θ. This intractable conditional distribution can be lower-bounded by a variational approximation (q) which produces the loss function used in practice (see Section 8.1 for a derivation based on) The variational parameters can be optimized by minimizing the negative log likelihood of samples from the true conditional distribution, i.e., q is a discriminator trying to predict the correct z from behavior. However, it is not obvious how to optimize the policy parameters θ, as they only affect the loss through the non-differentiable environment. The appropriate intrinsic reward function can be derived (see Section 8.2 for details) through application of the REINFORCE trick, which in log q(z|s) serving this role. Figure 1: VISR model diagram. In practice w t is also fed into ψ as an input, which also allows for GPI to be used (see Algorithm 1 in Appendix). For the random feature baseline, the discriminator q is frozen after initialization, but the same objective is used to train ψ. Traditionally, the desired product of this optimization was the conditional policy (π). While the discriminator q could be used for imitating demonstrated behaviors (i.e. by inferring the most likely z for a given τ), for down-stream RL it was typically discarded in favor of explicit search over all possible z . In the next section we discuss an alternative approach to leverage the behaviors learned during the unsupervised phase. The primary motivation behind our proposed approach is to combine the rapid task inference mechanism provided by SFs with the ability of BMI methods to learn many diverse behaviors in an unsupervised way. We begin by observing that both approaches use vectors to parameterize tasks. In the SF formulation tasks correspond to linear weightings w of features φ(s). The reward for a task given by w is r SF (s; w) = φ(s) T w. BMI objectives, on the other hand, define tasks using conditioning vectors z, with the reward for task z given by r BM I (s; z) = log q(z|s). We propose restricting conditioning vectors z to correspond to task-vectors w of the SFs formulation. The restriction that z ≡ w, in turn, requires that r SF (s; w) = r BM I (s; w), which implies that the BMI discriminator q must have the form log q(w|s) = φ(s) T w. One way to satisfy this requirement is by restricting the task vectors w and features φ(s) to be unit length and paremeterizing the discriminator q as the Von Mises-Fisher distribution with a scale parameter of 1. Note that this form of discriminator differs from the standard choice of parameterizing q as a multivariate Gaussian, which does not satisfy equation 10. With this variational family for the discriminator, all that is left to complete the base algorithm is to factorize the conditional policy into the policy-conditional successor features (ψ) and the task vector (w). This is straightforward as any conditional policy can be represented by a UVFA , and any UVFA can be represented by a USFA given an appropriate feature basis, such as the one we have just derived. Figure 1 shows the ing model. Training proceeds as in other algorithms maximizing BMI: by randomly sampling a task vector w and then trying to infer it from the state produced by the conditioned policy (in our case w is sampled from a uniform distribution over the unit circle). The key difference is that in VISR the structure of the conditional policy (equation 5) enforces the task/dynamics factorization as in SF (equations 2 and 4), which in turn reduces task inference to a regression problem derived from equation 2. Now that SFs have been given a feature-learning mechanism, we can return to the second question raised at the end of Section 3: how can we obtain a diverse set of policies over which to apply GPI? Recall that we are training a USFA ψ(s, a, e(π)) whose encoding function is e(π) = w (that is, π is the policy that tries to maximize the reward in for a particular value of w). So, the question of which policies to use with GPI comes down to the selection of a set of vectors w. One natural w candidate is the solution for a regression problem derived from. Let us call this solution w base, that is, φ(s, a, s) w base ≈ r(s, a, s). But what should the other task vectors w's be? Given that task vectors are sampled from a uniform distribution over the unit circle during training, there is no single subset that has any privileged status. So, following, we sample additional w's on the basis of similarity to w base. Since the discriminator q enforces similarity on the basis of probability under a Von Mises-Fisher distribution, these additional w's are sampled from such a distribution centered on w base, with the concentration parameter κ acting as a hyper-parameter specifying how diverse the additional w's should be. Calculating the improved policy is thus done as follows: Our experiments are divided in four groups corresponding to Sections 6.1 to 6.4. First, we assess how well VISR does in the RL setup with an unsupervised pre-training phase described in Section 2. Since this setup is unique in the literature on the Atari Suite, for the full two-phase process we only compare to ablations on the full VISR model and a variant of DIAYN adapted for these tasks (Table 1, bottom section). In order to frame performance relative to prior work, in Section 6.2 we also compare to for algorithms that operate in a purely unsupervised manner (Table 1, top section). Next, in Section 6.3, we contrast VISR's performance to that of standard RL algorithms in a low data regime (Table 1, middle section). Finally, we assess how well the proposed approach of inferring the task through the solution of a regression derived from does as compared to random search. To evaluate VISR, we impose a two-phase setup on the full suite of 57 Atari games . Agents are allowed a long unsupervised training phase (250M steps) without access to rewards, followed by a short test phase with rewards (100k steps). The full VISR algorithm includes features learned through the BMI objective and GPI to improve the execution of policies during both the training and test phases (see Algorithm 1 in the Appendix). The main baseline model, RF VISR, removes the BMI objective, instead learning SFs over features given by a random convolutional network (the same architecture as the φ network in the full model). The remaining ablations remove GPI from each of these models. The ablation shown in Table 1 (bottom) confirm that these components of VISR play complementary roles in the overall functioning of our model (also see Figure 2a). In addition, DIAYN has been adapted for the Atari domain, using the same training and testing conditions, base RL algorithm, and network architecture as VISR . With the standard 50-dimensional categorical z, performance was worse than random. While decreasing the dimensionality to 5 (matching that of VISR) improved this, it was still significantly weaker than even the ablated versions of VISR. Table 1: Atari Suite comparisons. @N represents the amount of RL interaction utilized. M dn is median, M is mean, > 0 is the number of games with better than random performance, and > H is the number of games with human-level performance as defined in. Top: unsupervised learning only (Sec. 6.2). Mid: data-limited RL (Sec. 6.3). Bottom: RL with unsupervised pre-training (Sec. 6.1). Standard deviations given in Table 2 (Appendix). Comparing against fully unsupervised approaches, our main external baseline is the Intrinsic Curiosity Module . This uses forward model prediction error in some feature-space to produce an intrinsic reward signal. Two variants have been evaluated on a 47 game subset of the Atari suite . One uses random features as the basis of their forward model (RF Curiosity), and the other uses features learned via an inverse-dynamics model (IDF Curiosity). It is important to note that, in addition to the extrinsic rewards, these methods did not use the terminal signals provided by the environment, whereas all other methods reported here do use them. The reason for not using the terminal signal was to avoid the possibility of the intrinsic reward reducing to a simple "do not die" signal. To rule this out, an explicit "do not die" baseline was run (Pos Reward NSQ), wherein the terminal signal remains and a small constant reward is given at every time-step. Finally, the full VISR model was run purely unsupervised. In practice this means not performing the fast-adaptation step (i.e. reward regression), instead switching between random w vectors every 40 time-steps (as is done during the training phase). Results shown in Table 1 (top and bottom) make it clear that while VISR is not a particularly outstanding in the unsupervised regime, when allowed 100k steps of RL it can vastly outperform these existing unsupervised methods on all criteria. Comparisons to reinforcement learning algorithms in the low-data regime are largely based on similar analysis by on the 26 easiest games in the Atari suite (as judged by above random performance for their algorithm). In that work the authors introduce a model-based agent (SimPLe) and show that it compares favorably to standard RL algorithms when data is limited. Three canonical RL algorithms are compared against: proximal policy optimization (PPO) , Rainbow , and DQN. For each, the from the lowest data regime reported in the literature are used. In addition, we also compare to a version of N-step Q-learning (NSQ) that uses the same codebase and base network architecture as VISR. Results shown in Table 1 (middle) indicate that VISR is highly competitive with the other RL methods. Note that, while these methods are actually solving the full RL problem, VISR's performance is based exclusively on the solution of a linear regression problem (equation 2). Obviously, this solution can be used to "warm start" an agent which can then refine its policy using any RL algorithm. We expect this version of VISR to have even better performance. In the previous , it was assumed that solving the linear reward-regression problem is the best way to infer the appropriate task vector. suggest a simpler approach: exhaustive search. As there are no guarantees that extrinsic rewards will be linear in the learned features (φ), it is not obvious which approach is best in practice. We hypothesize that exploiting the reward-regression task inference mechanism provided by VISR should yield more efficient inference than random search. To show this, 50 episodes (or 100k steps, whichever comes first) are rolled out using a trained VISR, each conditioned on a task vector chosen uniformly on a 5-dimensional sphere. From these initial episodes, one can either pick the task vector corresponding to the trajectory with the highest return (random search), or combine the data across all episodes and solve the linear regression problem. In each condition the VISR policy given by the inferred task vector is executed for 30 episodes and the average returns compared. As shown in Figure 2b, linear regression substantially improves performance despite using data generated specifically to aid in random search. The mean performance across all 57 games was 109.16 for reward-regression, compared to random search at 63.57. Even more dramatically, the median score for reward-regression was 8.99 compared to random search at 3.45. Overall, VISR outperformed the random search alternative on 41 of the 57 games, with one tie, using the exact same data for task inference. This corroborates the main hypothesis of this paper, namely, that endowing features derived from BMI with the fast task-inference provided by SFs gives rise to a powerful method able to quickly learn competent policies when exposed to a reward signal. Our suggest that VISR is the first algorithm to achieve notable performance on the full Atari task suite in a setting of few-step RL with unsupervised pre-training, outperforming all baselines and buying performance equivalent to hundreds of millions of interaction steps compared to DQN on some games (Figure 2c). As a suggestion for future investigations, the somewhat underwhelming for the fully unsupervised version of VISR suggest that there is much room for improvement. While curiosity-based methods are transient (i.e., asymptotically their intrinsic reward vanishes) and lack a fast adaptation mechanism, they do seem to encourage exploratory behavior slightly more than VISR. A possible direction for future work would be to use a curiosity-based intrinsic reward inside of VISR, to encourage it to better explore the space of controllable policies. Another interesting avenue for future investigation would be to combine the approach recently proposed by to enforce the policies computed by VISR to be not only distinguishable but also far apart in a given metric space. By using SFs on features that maximize BMI, we proposed an approach, VISR, that solves two open questions in the literature: how to compute features for the former and how to infer tasks in the latter. Beyond the concrete method proposed here, we believe bridging the gap between BMI and SFs is an insightful contribution that may inspire other useful methods. For convenience, we can refer to maximizing F(θ) as minimizing the loss function for parameters θ = (θ π, θ q), where θ π and θ q refer to the parameters of the policy π and variational approximation q, respectively. We can minimize L θ with respect to θ q, the parameters of q, using back-propagation. However, properly adjusting the parameters of π, θ π, is more difficult, as we lack a differentiable model of the environment. We now show that we can still derive an appropriate score function estimator using the log-likelihood or REINFORCE trick . Since in this section we will be talking about θ π only (that is, we will not discuss θ q), we will drop the subscript and refer to the parameters of π as simply θ. Let τ be a length T trajectory sampled under policy π, and let p θ be the probability of the trajectory τ under the combination of the policy and environment transition probabilities. We can compute the gradient of p θ with respect to θ as: This means that we can adjust p θ to make τ more likely under it. If we interpret p θ as the distribution induced by the policy π, then minimizing corresponds to maximizing the following value function: We can then use the policy gradient theorem to calculate the gradient of our loss function with respect to the parameters of the policy, θ, for trajectories τ beginning in state s, Since standard policy gradient (with rewards r t = log q(z|s t)) can be expressed as: Figure 3: VISR features φ learned by a variational distribution q(w|s) in a 10-by-10 gridworld. we can conclude that log q(z | s) serves the role of the reward function and treat it as such for arbitrary reinforcement learning algorithms (n-step Q-learning is used throughout this paper). The complexity and black-box nature of the Atari task suite make any significant analisis of the representations learned by VISR difficult (apart from their indirect effect on fastinference). Thus, in order to analyze the representation learned by VISR we have conducted a much smaller-scale experiment on a standard 10-by-10 grid-world. Here VISR still uses the full 5-sphere for its space of tasks, but it is trained with a much smaller network architecture for both the successor features ψ and variational approximation φ (both consist of 2 fullyconnected layers of 100 units with ReLU non-linearities, the latter L2-normalized so as to make mean predictions on the 5-sphere). We train this model for longer than necessary (960,000 trajectories of length 40 for 38,400,000 total steps) so as to best capture what representations might look like at convergence. Figure 3 shows each of the 5 dimension of φ across all states of the grid-world. It should be noted that, since these states were observed as one-hot vectors, all of the structure present is the of the mutual information training objective rather than any correlations in the input space. Figure 4 shows 49 randomly sampled reward functions, generated by sampling a w vector uniformly on the 5-sphere and taking the inner product with φ. This demonstrates that the space of φ contains many different partitionings of the state-space, which lends credence to our claim that externally defined reward functions are likely to be not far outside of this space, and thus fast-inference can yield substantial benefits. Figure 5 shows the 49 value functions corresponding to the reward function sampled in Figure 4. These value functions were computed via generalized policy improvement over the policies from 10 uniformly sampled w's. The clear correspondance between these value functions and their respective reward functions demonstrate that even though VISR is tasked with learning an infinite space of value functions, it does not significantly suffer from underfitting. These value functions can be thought of as the desired cumulative state-occupancies, and appear to represent distinct regions of the state space. A distributed reinforcement learning setup was utilized to accelerate experimentation as per. This involved having 100 separate actors, each running on its own instance of the environment. After every roll-out of 40 steps, the experiences are added to a queue. This queue is used by the centralized learner to calculate all of the losses and change the weights of the network, which are then passed back to the actors. The roll-out length implicitly determines other hyper-parameters out of convenience, namely the amount of backpropagation through time is done before truncation , as the sequential structure of the data is lost outside of the roll-out window. The task vector W is also resampled every 40 steps for similar reasons. Under review as a conference paper at ICLR 2020 The approximations to the optimal value functions for the reward functions in Figure 4, computed by VISR through GPI on 10 randomly sampled policies. In all (modulo some reported from other papers) are the average of 3 random seeds per game per condition. Due to the high computational cost of the controlled fast-inference experiments, for the experiments comparing the effect of training steps on fast-inference performance (e.g. Figure 6), an online evaluation scheme was utilized. Rather than actually performing no-reward reinforcement learning as 2 distinct phases, reward information 2 was exposed to 5 of the 100 actors which used the task vector ing from solving the reward regression via OLS. This regression was continuously solved using the most recent 100, 000 experiences from these actors.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce Variational Intrinsic Successor FeatuRes (VISR), a novel algorithm which learns controllable features that can be leveraged to provide fast task inference through the successor features framework.

We isolate one factor of RL generalization by analyzing the case when the agent only overfits to the observations. We show that architectural implicit regularizations occur in this regime.