Question: Represent the scientific article to find a short TLDR.

Question: We propose Bayesian hypernetworks: a framework for approximate Bayesian inference in neural networks. A Bayesian hypernetwork, h, is a neural network which learns to transform a simple noise distribution, p(e) = N(0,I), to a distribution q(t):= q(h(e)) over the parameters t of another neural network (the ``primary network). We train q with variational inference, using an invertible h to enable efficient estimation of the variational lower bound on the posterior p(t | D) via sampling. In contrast to most methods for Bayesian deep learning, Bayesian hypernets can represent a complex multimodal approximate posterior with correlations between parameters, while enabling cheap iid sampling of q(t). In practice, Bayesian hypernets provide a better defense against adversarial examples than dropout, and also exhibit competitive performance on a suite of tasks which evaluate model uncertainty, including regularization, active learning, and anomaly detection. Simple and powerful techniques for Bayesian inference of deep neural networks' (DNNs) parameters have the potential to dramatically increase the scope of applications for deep learning techniques. In real-world applications, unanticipated mistakes may be costly and dangerous, whereas anticipating mistakes allows an agent to seek human guidance (as in active learning), engage safe default behavior (such as shutting down), or use a "reject option" in a classification context. DNNs are typically trained to find the single most likely value of the parameters (the "MAP estimate"), but this approach neglects uncertainty about which parameters are the best ("parameter uncertainty"), which may translate into higher predictive uncertainty when likely parameter values yield highly confident but contradictory predictions. Conversely, Bayesian DNNs model the full posterior distribution of a model's parameters given the data, and thus provides better calibrated confidence estimates, with corresponding safety benefits BID9 BID0. 1 Maintaining a distribution over parameters is also one of the most effective defenses against adversarial attacks BID4.Techniques for Bayesian DNNs are an active research topic. The most popular approach is variational inference BID2 BID8, which typically restricts the variational posterior to a simple family of distributions, for instance a factorial Gaussian BID2 BID16. Unfortunately, from a safety perspective, variational approximations tend to underestimate uncertainty, by heavily penalizing approximate distributions which place mass in regions where the true posterior has low density. This problem can be exacerbated by using a restricted family of posterior distribution; for instance a unimodal approximate posterior will generally only capture a single mode of the true posterior. With this in mind, we propose learning an extremely flexible and powerful posterior, parametrized by a DNN h, which we refer to as a Bayesian hypernetwork in reference to BID17.A Bayesian hypernetwork (BHN) takes random noise ∼ N (0, I) as input and outputs a sample from the approximate posterior q(θ) for another DNN of interest (the "primary network"). The key insight for building such a model is the use of an invertible hypernet, which enables Monte Carlo estimation of the entropy term − logq(θ) in the variational inference training objective. We begin the paper by reviewing previous work on Bayesian DNNs, and explaining the necessary components of our approach (Section 2). Then we explain how to compose these techniques to yield Bayesian hypernets, as well as design choices which make training BHNs efficient, stable and robust (Section 3). Finally, we present experiments which validate the expressivity of BHNs, and demonstrate their competitive performance across several tasks (Section 4). We begin with an overview of prior work on Bayesian neural networks in Section 2.1 before discussing the specific components of our technique in Sections 2.2 and 2.3. Bayesian DNNs have been studied since the 1990s BID30 BID29. For a thorough review, see BID8. Broadly speaking, existing methods either 1) use Markov chain Monte Carlo BID39 BID30, or 2) directly learn an approximate posterior distribution using (stochastic) variational inference BID16 BID9 BID2, expectation propagation BID19 BID36, or α-divergences BID27. We focus here on the most popular approach: variational inference. Notable recent work in this area includes BID9, who interprets the popular dropout BID37 algorithm as a variational inference method ("MC dropout"). This has the advantages of being simple to implement and allowing cheap samples from q(θ). emulates Gaussian dropout, but yields a unimodal approximate posterior, and does not allow arbitrary dependencies between the parameters. The other important points of reference for our work are Bayes by Backprop (BbB) BID2, and multiplicative normalizing flows BID28. Bayes by Backprop can be viewed as a special instance of a Bayesian hypernet, where the hypernetwork only performs an element-wise scale and shift of the input noise (yielding a factorial Gaussian distribution).More similar is the work of BID28, who propose and dismiss BHNs due to the issues of scaling BHNs to large primary networks, which we address in Section 3.3.2 Instead, in their work, they use a hypernet to generate scaling factors, z on the means µ of a factorial Gaussian distribution. Because z follows a complicated distribution, this forms a highly flexible approximate posterior: q(θ) = q(θ|z)q(z)dz. However, this approach also requires them to introduce an auxiliary inference network to approximate q(z|θ) in order to estimate the entropy term of the variational lower bound, ing in lower bound on the variational lower bound. Finally, the variational autoencoder (VAE) BID21 BID23 family of generative models is likely the best known application of variational inference in DNNs, but note that the VAE is not a Bayesian DNN in our sense. VAEs approximate the posterior over latent variables, given a datapoint; Bayesian DNNs approximate the posterior over model parameters, given a dataset. A hypernetwork BID17 BID3 BID1 ) is a neural net that outputs parameters of another neural net (the "primary network"). 3 The hypernet and primary net together form a single model which is trained by backpropagation. The number of parameters of a DNN scales quadratically in the number of units per layer, meaning naively parametrizing a large primary net requires an impractically large hypernet. One method of addressing this challenge is Conditional Batch Norm (CBN) BID7, and the closely related Conditional Instance Normalization (CIN) BID20 BID38, and Feature-wise Linear Modulation (FiLM) BID31 BID26, which can be viewed as specific forms of a hypernet. In these works, the weights of the primary net are parametrized directly, and the hypernet only outputs scale (γ) and shift (β) parameters for every neuron; this can be viewed as selecting which features are significant (scaling) or present (shifting). In our work, we employ the related technique of weight normalization, which normalizes the input weights for every neuron and introduces a separate parameter g for their scale. Our proposed Bayesian hypernetworks employ a differentiable directed generator network (DDGN) BID15 ) as a generative model of the primary net parameters. DDGNs use a neural net to transform simple noise (most commonly isotropic Gaussian) into samples from a complex distribution, and are a common component of modern deep generative models such as variational autoencoders (VAEs) BID23 BID21 and generative adversarial networks (GANs) BID13 BID12.We take advantage of techniques for invertible DDGNs developed in several recent works on generative modeling BID5 and variational inference of latent variables BID32. Training these models uses the change of variables formula, which involves computing the log-determinant of the inverse Jacobian of the generator network. This computation involves a potentially costly matrix determinant, and these works propose innovative architectures which reduce the cost of this operation but can still express complicated deformations, which are referred to as "normalizing flows". We now describe how variational inference is applied to Bayesian deep nets (Section 3.1), and how we compose the methods described in Sections 2.2 and 2.3 to produce Bayesian hypernets (Section 3.2). In variational inference, the goal is to maximize a lower bound on the marginal log-likelihood of the data, log p(D) under some statistical model. This involves both estimating parameters of the model, and approximating the posterior distribution over unobserved random variables (which may themselves also be parameters, e.g., as in the case of Bayesian DNNs). Let θ ∈ R D be parameters given the Bayesian treatment as random variables, D a training set of observed data, and q(θ) a learned approximation to the true posterior p(θ|D). Since the KL divergence is always non-negative, we have, for any q(θ): DISPLAYFORM0 The right hand side of FORMULA0 is the evidence lower bound, or "ELBO".The above derivation applies to any statistical model and any dataset. In our experiments, we focus on modeling conditional likelihoods p(D) = p(Y|X). Using the conditional independence assumption, we further decompose log p(D|θ):= log p(Y|X, θ) as n i=1 log p(y i |x i, θ), and apply stochastic gradient methods for optimization. Computing the expectation in is generally intractable for deep nets, but can be estimated by Monte Carlo sampling. For a given value of θ, log p(D|θ) and log p(θ) can be computed and differentiated exactly as in a non-Bayesian DNN, allowing training by backpropagation. The entropy term E q [− logq(θ)] is also straightforward to evaluate for simple families of approximate posteriors such as Gaussians. Similarly, the likelihood of a test data-point under the predictive posterior using S samples can be estimated using Monte Carlo: DISPLAYFORM0 Bayesian hypernets (BHNs) express a flexible q(θ) by using a DDGN (section 2.3), h ∈ R D → R D, to transform random noise ∼ N (0, I D) into independent samples from q(θ). This makes it cheap to compute Monte Carlo estimations of expectations with respect to q; these include the ELBO, and its derivatives, which can be backpropagated to train the hypernet h. Since BHNs are both trained and evaluated via samples of q(θ), expressing q(θ) as a generative model is a natural strategy. However, while DDGNs are convenient to sample from, computing the entropy term (E q [− logq(θ)]) of the ELBO additionally requires evaluating the likelihood of generated samples, and most popular DDGNs (such as VAEs and GANs) do not provide a convenient way of doing so. 5 In general, these models can be many-to-one mappings, and computing the likelihood of a given parameter value requires integrating over the latent noise variables ∈ R D: DISPLAYFORM0 To avoid this issue, we use an invertible h, allowing us to compute q(θ) simply by using the change of variables formula: DISPLAYFORM1 where q is the distribution of and θ = h.As discussed in Section 2.3, a number of techniques have been developed for efficiently training such invertible DDGNs. In this work, we employ both RealNVP (RNVP) BID6 and Inverse Autoregressive Flows (IAF). Note that the latter can be efficiently applied, since we only require the ability to evaluate likelihood of generated samples (not arbitrary points in the range of h, as in generative modeling applications, e.g., BID6); and this also means that we can use a lower-dimensional to generate samples along a submanifold of the entire parameter space, as detailed below. In order to scale BHNs to large primary networks, we use the weight normalization reparametrization 6: DISPLAYFORM0 where θ j are the input weights associated with a single unit j in the primary network. We only output the scaling factors g from the hypernet, and learn a maximum likelihood estimate of v. 7 This allows us to overcome the computational limitations of naively-parametrized BHNs noted by BID28, since computation now scales linearly, instead of quadratically, in the number of BID2. In the second subplot, we place a prior on the scaling factor g and infer the posterior distribution using a BHN, while in the third subplot the hypernet is used to generate the whole weight matrices of the primary net. Each shaded region represents half a standard deviation in the posterior on the predictive mean. The red crosses are 50 examples from the training dataset.primary net units. Using this parametrization restricts the family of approximate posteriors, but still allows for a high degree of multimodality and dependence between the parameters. We also employ weight normalization within the hypernet, and found this stabilizes training dramatically. Initialization plays an important role as well; we recommend initializing the hypernet weights to small values to limit the impact of noise at the beginning of training. We also find clipping the outputs of the softmax to be within (0.001, 0.999) critical for numerical stability. We perform experiments on MNIST, CIFAR10, and a 1D regression task. There is no single metric for how well a model captures uncertainty; to evaluate our model, we perform experiments on regularization (Section 4.2), active learning (Section 4.3), anomaly detection (Section 4.4), and detection of adversarial examples (Section 4.5). Active learning and anomaly detection problems make natural use of uncertainty estimates: In anomaly detection, higher uncertainty indicates a likely anomaly. In active learning, higher uncertainty indicates a greater opportunity for learning. Parameter uncertainty also has regularization benefits: integrating over the posterior creates an implicit ensemble. Intuitively, when the most likely hypothesis predicts "A", but the posterior places more total mass on hypotheses predicting "B", we prefer predicting "B". By improving our estimate of the posterior, we more accurately weigh the evidence for different hypotheses. Adversarial examples are an especially difficult kind of anomaly designed to fool a classifier, and finding effective defenses against adversarial attacks remains an open challenge in deep learning. For the hypernet architecture, we try both RealNVP BID6 and IAF with MADE BID11, with 1-layer ReLU-MLP coupling functions with 200 hidden units (each). In general, we find that IAF performs better. We use an isotropic standard normal prior on the scaling factors (g) of the weights of the network. We also use Adam with default hyper-parameter settings BID22 and gradient clipping in all of our experiments. Our mini-batch size is 128, and to reduce computation, we use the same noise-sample (and thus the same primary net parameters) for all examples in a mini-batch. We experimented with independent noise, but did not notice any benefit. Our baselines for comparison are Bayes by Backprop (BbB) BID2, MC dropout (MCdropout) BID9, and non-Bayesian DNN baselines (with and without dropout). We first demonstrate the behavior of the network on the toy 1D-regression problem from BID2 in FIG0. As expected, the uncertainty of the network increases away from the observed data. We also use this experiment to evaluate the effects of our proposal for scaling BHNs via the weight norm parametrization (Section 3.3) by comparing with a model which generates the full set of parameters, and find that the two models produce very similar , suggesting that our proposed method strikes a good balance between scalability and expressiveness. Next, we demonstrate the distinctive ability of Bayesian hypernets to learn multi-modal, dependent distributions. FIG4 (appendix) shows that BHNs do learn approximate posteriors with dependence between different parameters, as measured by the Pearson correlation coefficient. Meanwhile, FIG1 shows that BHNs are capable of learning multimodal posteriors. For this experiment, we trained an over-parametrized linear (primary) network:ŷ = a · b · x on a dataset generated as y = x +, and the BHN learns capture both the modes of a = b = 1 and a = b = −1. We now show that BHNs act as a regularizer, outperforming dropout and traditional mean field (BbB). Results are presented in TAB0. In our experiments, we find that BHNs perform on par with dropout on full datasets of MNIST and CIFAR10; furthermore, increasing the flexibility of the posterior by adding more coupling layers improves performance, especially compared with models with 0 coupling layers, which cannot model dependencies between the parameters. We also evaluate on a subset of MNIST (the first 5,000 examples); are presented in the last two columns of TAB0. Replicating these experiments (with 8 coupling layers) for 10 trials yields Figure 3. In these MNIST experiments, we use MLPs with 2 hidden layers of 800 or 1200 hidden units each. For CIFAR10, we train a convolutional neural net (CNN) with 4 hidden layers of channels, 2 × 2 max pooling after the second and the fourth layers, filter size of 3, and a single fully connected layer of 512 units. Figure 4: Active learning: Bayesian hypernets outperform other approaches after sufficient acquisitions when warm-starting (left), for both random acquisition function (top) and BALD acquisition function (bottom). Warm-starting improves stability for all methods, but hurts performance for other approaches, compared with randomly re-initializing parameters as in (right). We also note that the baseline model (no dropout) is competitive with MCdropout, and outperforms the Dropout baseline used by. 9 These curves are the average of three experiments. We now turn to active learning, where we compare to the MNIST experiments of, replicating their architecture and training procedure. Briefly, they use an initial dataset of 20 examples (2 from each class), and acquire 10 new examples at a time, training for 50 epochs between each acquisition. While re-initialize the network after every acquisition, we found that "warm-starting" from the current learned parameters was essential for good performance with BHNs, although it is likely that longer training or better initialization schemes could perform the same role. Overall, warm-started BHNs suffered at the beginning of training, but outperformed all other methods for moderate to large numbers of acquisitions. For anomaly detection, we take BID18 as a starting point, and perform the same suite of MNIST experiments, evaluating the ability of networks to determine whether an input came from their training distribution ("Out of distribution detection"). BID18 found that the confidence expressed in the softmax probabilities of a (non-Bayesian) DNN trained on a single dataset provides a good signal for both of these detection problems. We demonstrate that Bayesian DNNs outperform their non-Bayesian counterparts. Just as in active learning, in anomaly detection, we use MC to estimate the predictive posterior, and use this to score datapoints. For active learning, we would generally like to acquire points where there is higher uncertainty. In a well-calibrated model, these points are also likely to be challenging or anomalous examples, and thus acquisition functions from the active learning literature are good candidates for scoring anomalies. We consider all of the acquisition functions listed in as possible scores for the Area Under the Curve (AUC) of Precision-Recall (PR) and Receiver Operating Characteristic (ROC) metrics, but found that the maximum confidence of the softmax probabilities (i.e., "variation ratio") acquisition function used by BID18 gave the best performance. Both BHN and MCdropout achieve significant performance gains over the non-Bayesian baseline, and MCdropout performs significantly better than BHN in this task. Results are presented in TAB1.Second, we follow the same experimental setup, using all the acquisition functions, and exclude one class in the training set of MNIST at a time. We take the excluded class of the training data as out-of-distribution samples. The is presented in TAB2 (Appendix). This experiment shows the benefit of using scores that reflect dispersion in the posterior samples (such as mean standard deviation and BALD value) in Bayesian DNNs. Finally, we consider this same anomaly detection procedure as a novel tool for detecting adversarial examples. Our setup is similar to BID27 and BID28, where it is shown that when more perturbation is added to the data, model uncertainty increases and then drops. We use the Fast Gradient Sign method (FGS) BID14 for adversarial attack, and use one sample of our model to estimate the gradient. 10 We find that, compared with dropout, BHNs are less confident on data points which are far from the data manifold. In particular, BHNs constructed with IAF consistently outperform RealNVP-BHNs and dropout in detecting adversarial examples and errors. Results are shown in FIG3. We introduce Bayesian hypernets (BHNs), a new method for variational Bayesian deep learning which uses an invertible hypernetwork as a generative model of parameters. BHNs feature efficient we found the BALD values our implementation computes provide a better-than-random acquisition function (compare the blue line in the top and bottom plots). 10 and BID28 used 10 and 1 model samples, respectively, to estimate gradient. We report the with 1 sample; with more samples are given in the appendix. when more perturbation is added to the data (left), uncertainty measures also increase (first row). In particular, the BALD and Mean STD scores, which measure epistemic uncertainty, are strongly increasing for BHNs, but not for dropout. The second row and third row plots show for adversary detection and error detection (respectively) in terms of the AUC of ROC (y-axis) with increasing perturbation along the x-axis. Gradient direction is estimated with one Monte Carlo sample of the weights/dropout mask.training and sampling, and can express complicated multimodal distributions, thereby addressing issues of overconfidence present in simpler variational approximations. We present a method of parametrizing BHNs which allows them to scale successfully to real world tasks, and show that BHNs can offer significant benefits over simpler methods for Bayesian deep learning. Future work could explore other methods of parametrizing BHNs, for instance using the same hypernet to output different subsets of the primary net parameters. A We replicate the experiments of anomaly detection with unseen classes of MNIST. Here we use 32 samples to estimate the gradient direction with respect to the input. A better estimate of gradient amounts to a stronger attack, so accuracy drops lower for a given step size while an adversarial example can be more easily detected with a more informative uncertainty measure. In this paper, we employ weight normalization in the primary network, treating (only) the scaling factors g as random variables. We choose an isotropic Gaussian prior for g: p(g) = N (g; 0, λI), which in an L 2 weight-decay penalty on g, or, equivalently, w = g ≥ E ∼q ,g=h φ [log p(D|g; v, b) + log p(g) − log q + log det DISPLAYFORM0 where v and b are the direction and bias parameters of the primary net, and φ is the parameters of the hypernetwork. We optimize this bound with respect to {v, b, φ} during training.

We propose Bayesian hypernetworks: a framework for approximate Bayesian inference in neural networks.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Evolutionary-based optimization approaches have recently shown promising in domains such as Atari and robot locomotion but less so in solving 3D tasks directly from pixels. This paper presents a method called Deep Innovation Protection (DIP) that allows training complex world models end-to-end for such 3D environments. The main idea behind the approach is to employ multiobjective optimization to temporally reduce the selection pressure on specific components in a world model, allowing other components to adapt. We investigate the emergent representations of these evolved networks, which learn a model of the world without the need for a specific forward-prediction loss. The ability of the brain to model the world arose from the process of evolution. It evolved because it helped organisms to survive and strive in their particular environments and not because such forward prediction was explicitly optimized for. In contrast to the emergent neural representations in nature, current world model approaches are often directly rewarded for their ability to predict future states of the environment (; ; ;). While it is undoubtedly useful to be able to explicitly encourage a model to predict what will happen next, in this paper we are interested in what type of representations can emerge from the less directed process of artificial evolution and what ingredients might be necessary to encourage the emergence of such predictive abilities. In particular, we are building on the recently introduced world model architecture introduced by. This agent model contains three different components: a visual module, mapping high-dimensional inputs to a lower-dimensional representative code, an LSTM-based memory component, and a controller component that takes input from the visual and memory module to determine the agent's next action. In the original approach, each component of the world model was trained separately and to perform a different and specialised function, such as predicting the future. demonstrated that these models can also be trained end-to-end through a population-based genetic algorithm (GA) that exclusively optimizes for final performance, the approach was only applied to the simpler 2D car racing domain and it is an open question how such an approach will scale to the more complex 3D VizDoom task that first validated the effectiveness of the world model approach. Here we show that a simple genetic algorithm fails to find a solution to solving the VizDoom task and ask the question what are the missing ingredients necessary to encourage the evolution of more powerful world models? The main insight in this paper is that we can view the optimization of a heterogeneous neural network (such as world models) as a co-evolving system of multiple different sub-systems. The other important insight is that representational innovations discovered in one subsystem (e.g. the visual system learns to track moving objects) require the other sub-systems to adapt. In fact, if the other systems are not given time to adapt, such innovation will likely initially have an adversarial effect on overall performance! In order to optimize such co-evolving heterogeneous neural systems, we propose to reduce the selection pressure on individuals whose visual or memory system was recently changed, given the controller component time to readapt. This Deep Innovation Protection (DIP) approach is inspired by the recently introduced morphological innovation protection method of , which allows for the scalable co-optimization of controllers and robot body plans. Our approach is able to find a solution to the VizDoom: Take Cover task, which was first solved by the original world model approach . More interestingly, the emergent world models learned to predict events important for the survival of the agent, even though they were not explicitly trained to predict the future. Additionally, our investigates into the training process show that DIP allows evolution to carefully orchestrate the training of the components in these heterogeneous architectures. We hope this work inspires more research that focuses on investigating representations emerging from approaches that do not necessarily only rely on gradient-based optimization. The hypothesis in this paper is that to optimize complex world models end-to-end for more complex tasks requires each of its components to be carefully tuned to work well together. For example, an innovation in the visual or memory component could adversely impact the controller component, leading to reduced performance. However, in the long run such innovation could allow an individual to outperform its predecessors. The agent model is based on the world model approach introduced by. The network includes a sensory component, implemented as a variational autoencoder (VAE) that compresses the high-dimensional sensory information into a smaller 32-dimensional representative code (Fig. 1). This code is fed into a memory component based on a recurrent LSTM , which should predict future representative codes based on previous information. Both the output from the sensory component and the memory component are then fed into a controller that decides on the action the agent should take at each time step. , we train these world models end-to-end with a genetic algorithm, in which mutations add Gaussian noise to the parameter vectors of the networks: θ = θ + σ, where ∼ N (0, I). The approach introduced in this paper aims to train heterogeneous neural systems end-to-end by temporally reducing the selection pressure on individuals with recently changed modules, allowing other components to adapt. For example, in case of the world model, in which a mutation can either affect the VAE, MDN-RNN or controller, selection pressure should be reduced if a mutation affects the VAE or MDN-RNN, giving the controller time to readapt to the changes in the learned representation. Inspired by the multi-objective morphological innovation protection introduced by , we employ the well-known multiobjective optimization approach NSGA-II , in which a second "age" objective keeps track of when a mutation changes either the VAE or the MDN-RNN. Every generation an individual's age is increased by 1, however, if a mutation changes the VAE or MDN-RNN, this age objective is set to zero (lower is better). Therefore, if two neural networks reach the same performance (i.e. the same final reward), the one that had less time to adapt (i.e. whose age is lower) would have a higher chance of being selected for the next generation. The second objective is the accumulated reward received during an episode. Pseudocode of the approach applied to world models is shown in Algorithm 1. It is important to note that this approach is different to the traditional usage of "age" in multiobjective optimization, in which age is used to increase diversity and keeps track of how long individuals have been in the population . In the approach in this paper, age counts how many generations the controller component of an individual had time to adapt to an unchanged visual and memory system. In the original world model approach the visual and memory component were trained separately and through unsupervised learning based on data from random rollouts. In this paper they are optimized through a genetic algorithm without evaluating each component individually. In other words, the VAE is not directly optimized to reconstruct the original input data and neither is the memory component optimized to predict the next time step; the whole network is trained in an endto-end fashion. Here we are interested in what type of neural representations emerge by themselves that allow the agent to solve the given task. Following the original world model approach , in the experiments presented here an agent is trained to solve the car racing tasks, and the more challenging VizDoom task from 64×64 RGB pixel inputs (Fig. 2). In the continuous control task Assign ranks based on Pareto fronts 9: Generate set of non-dominated solutions 10: Add solutions, starting from first front, until number solution = N 11: Generate child population through binary tournament selection and mutations 12: Reset age to 0 for all individuals whose VAE or MDN-RNN was mutated 13: end for A visual component (the encoder of a variational autoencoder) produces a latent code z t at each time step t, which is concatenated with the hidden state h t of the LSTM-based memory component that takes z t and previously performed action a t−1 as input. The combined vector (z t, h t) is input into the controller component to determine the next action of the agent. In this paper, the agent model is trained end-to-end with a multiobjective genetic algorithm. CarRacing-v0 the agent is presented with a new procedurally generated track every episode, receiving a reward of -0.1 every frame and a reward of +100/N for each visited track tile, where N is the total number of tiles in the track. The network controlling the agent (Fig. 1) has three outputs to control left/right steering, acceleration and braking. Further details on the network model, which is the same for both domains, can be found in the Appendix. In the VizDoom:Take Cover task the agent has to try to stay alive for 2,100 timesteps, while avoiding fireballs shot at it by strafing to the left or the right. The agent receives a +1 reward for every frame it is alive. The network controlling the agent has one output a to control left (a < −0.3) and right strafing (a > 0.3), or otherwise standing still. In this domain, a solution is defined as surviving for over 750 timesteps, averaged across 100 random rollouts . Following the NSGA-II approach, individuals for the next generation are determined stochastically through 2-way tournament selection from the 50% highest ranked individuals in the population (Algorithm 1). No crossover operation was employed. The population size was 200. Because of the randomness in this domain, we evaluate the top three individuals of each generation one additional time to get a better estimate of the true elite. We compare a total of four different approaches: The age objective is reset to zero when either the VAE or MDN-RNN is changed. The idea behind this approach is that the controller should get time to readapt if one of the components that precede it in the network change. Here the age objective is set to zero if the controller changes. This setting tests if protecting components upstream can be effective in optimizing heterogeneous neural models. Figure 2: In the CarRacing-v0 task the agent has to learn to drive across many procedurally generated tracks as fast as possible from 64 ×64 RGB color images. In the VizDoom: Take Cover domain the agent has to learn to avoid fireballs and to stay alive as long as possible. This setup is the same as the controller protection approach but we additionally reset age if the MDN-RNN changes. On average, this treatment will reset the age objective as often as DIP. 4. Random age objective: In this setup the age objective is assigned a random number between at each evaluation. This treatment tests if better performance can be reached just through introducing more diversity in the population. 5. Standard GA -no innovation protection: In this non-multi-objective setup, which is the same one as introduced in 1, only the accumulated reward is taken into account when evaluating individuals. For all treatments, a mutation has an equal probability to either mutate the visual, memory, or controller component of the network. Interestingly, while reported that this approach performs similarly well to an approach that always mutates all components, we noticed that it performs significantly worse in the more complicated VizDoom domain. This suggests that the more complex the tasks, the more important it is to be able to fine-tune individual components in the overall world model architecture. All are averaged over ten independent evolutionary runs. In the car racing domain we find that there is no noticeable difference between an approach with and without innovation protection Figure 4: Still frames of a learned policy. The evolved champion learned to primarily pay attention to the walls and fireballs, while ignoring the floor and ceiling. Interestingly the agent also seems to pay attention to the health and ammo indicator. and both can solve the domain. However, in the more complex VizDoom task, the DIP approach that protects innovations in both VAE and MDN-RNN, outperforms all other approaches (Fig. 3a). The approach is able to find a solution to the task, effectively avoiding fireballs and reaching an average score of 824.33 (sd=491.59). To better understand the decision-making process of the agent, we calculate perturbation-based saliency maps (see Appendix for details) to determine the parts of the environment the agent is paying most attention to (Fig. 4). The idea behind perturbation-based saliency maps is to measure to what extent the output of the model changes if parts of the input image are altered . Not surprisingly, the agent learned to pay particular attention to the walls, fireballs, and the position of the monsters. The better performance of the random age objective compared to no innovation protection suggests that increasing diversity in the population improves performance but less effectively than selectivity resetting the age objective as in DIP. Interestingly, the controller and the MDN-RNN&Controller protection approach perform less well, confirming our hypothesis that it is important to protect innovations upstream in the network for downstream components. We further investigate what type of world model can emerge from an evolutionary process that does not directly optimize for forward prediction or reconstruction loss. To gain insights into the learned representations we employ the t-SNE dimensionality reduction technique , which has proven valuable for visualizing the inner workings of deep neural networks ). We are particularly interested in the information contained in the compressed 32-dimensional vector of the VAE and the information stored in the hidden states of the MDN-RNN (which are both fed into the controller that decides on the agent's action). Different combinations of sequences of these latent vectors collected during one rollout are visualized in two dimensions in Fig. 5. Interestingly, while the 32-dimensional z vector from the VAE does not contain enough information to infer the correct action, either the hidden state alone or in combination with z in grouping the states into two distinct classes (one for moving left and one for moving right). The temporal dimension captured by the recurrent network proves invaluable in deciding what action is best. For example, not getting stuck in a position that makes avoiding incoming fireballs impossible, seems to require a level of forward prediction by the agent. To gain a deeper understanding of this issue we look more closely into the learned temporal representation next. In order to analyze the learned temporal dynamics of the forward model, we are taking a closer look at the average activation x t of all 256 hidden nodes at time step t and how much they differ from the overall average across all time stepsX = 1 N N 1x t. The variance ofx t is thus calculated as σ t = (X −x t) 2, and normalized to the range before plotting. The hypothesis is that activation levels far from the mean might indicate a higher importance and should have a greater impact on the agent's controller component. In other words, they likely indicate critical situations in which the agent needs to pay particular attention to the predictions of the MDN-RNN. Fig. 6 depicts frames from the learned policies in two different situations, which shows that the magnitude of LSTM activations are closely tied to specific situations. The forward model does not seem to react to fireballs by themselves but instead depends on the agent being in the line of impact of an approaching fireball, which is critical information for the agent to stay alive. Evolutionary Innovations In addition to analyzing the learned representations of the final networks, it is interesting to study the different stepping stones evolution discovered to solve the VizDoom task. We show one particular evolutionary run in Fig. 7, with other ones following similar progressions. In the first 30 generations the agent starts to learn to pay attention to fireballs but only tries avoiding them by either standing still or moving to the right. A jump in performance happens around generation 34 when the agent starts to discover moving to either the left or right; however, the learned representation between moving left or right is not well defined yet. This changes around generation 56, leading to another jump in fitness and some generations of quick fine-tuning later the agent is able to differentiate well between situations requiring different actions, managing to survive for the whole length of the episode. Motivated by the approach of to analyse the gradient descent-based training of neural networks, we investigate the weight distances of the world model components of the best-performing networks found during training to the final solution representation (Fig. 3b). The VAE is the component with the steepest decrease in distance with a noticeable jump around generation 60 due to another lineage taking over. The MDN-RNN is optimized slowest, which is likely due to the fact that the correct forward model dynamics are more complicated to discover than the visual component. These suggest that DIP is able to orchestrate the training of these heterogeneous world model architectures in an automated way. Generation 34 Generation 56 Generation 145 Figure 7: Development of the evolved representation over generations. Shown are t-SNE mappings of the 288-dimensional vectors (32-dimensional latent vectors + 256-dimensional hidden state vector) together with saliency maps of specific game situations. Early on in evolution the agent starts paying attention to the fireballs (generation 24) but only moves to the right (blue) or stands still (black). Starting around generation 34 the agent starts to move to the left and right, with the saliency maps becoming more pronounced. From generation 56 on the compressed learned representation (latent vector+hidden state vector) allows the agent to infer the correct action almost all the time. The champion discovered in generation 145 discovered a visual encoder and LSTM mapping that shows a clear division for left and right strafing actions. A variety of different RL algorithms have recently been shown to work well on a diverse set of problems when combined with the representative power of deep neural networks (; ; . While most approaches are based on variations of or policy gradient methods (;, recently evolutionary-based methods have emerged as a promising alternative for some domains). showed that a type of evolution strategy (ES) can reach competitive performance in the Atari benchmark and at controlling robots in MuJoCo. Additionally, Such et al. demonstrated that a simple genetic algorithm is in fact able to reach similar performance to deep RL methods such as DQN or A3C. Earlier approaches that evolved neural networks for RL tasks worked well in complex RL tasks with lower-dimensional input spaces (; ;). However, when trained end-to-end these networks are often still orders of magnitude simpler than networks employed for supervised learning problems or depend on additional losses that are responsible for training certain parts of the network . For complex agent models, different network components can be trained separately (Wahlström et al., 2015;). For example, in the world model approach , the authors first train a variational autoencoder (VAE) on 10,000 rollouts from a random policy to compress the high-dimensional sensory data and then train a recurrent network to predict the next latent code. Only after this process is a smaller controller network trained to perform the actual task, taking information from both the VAE and recurrent network as input to determine the action the agent should perform. Evolutionary approaches solving 3D tasks directly from pixels has so far proven difficult although a few notable approaches exist. Koutník et al. evolved an indirectly encoded and recurrent controller for car driving in TORCS, which learned to drive based on a raw 64×64 pixel image. The approach was based on an indirect encoding of the network's weights analogous to the JPEG compression in images. To scale to 3D FPS tasks, first trained an autoencoder in an unsupervised way and then evolved the controller giving the compressed representation as input. In another approach, trained an object recognizer in a supervised way and then in a separate step evolved a controller module. More recently, introduced an approach called safe mutations, in which the magnitude of mutations to weight connections is scaled based on the sensitivity of the network's output to that weight. The safe mutations approach allowed the evolution of large-scale deep networks for a simple 3D maze task and is a complementary approach that could be combined with DIP in the future. The approach introduced in this paper can be viewed as a form of diversity maintenance, in which selection pressure on certain mutated neural networks is reduced. Many other methods for encouraging diversity were invented by the evolutionary computation community, such as novelty search , quality diversity , or speciation . Using the concept of age to maintain diversity has a long history in evolutionary algorithms. first introduced the idea of using age in a genetic algorithm in which individuals are removed from the population if they reach a specific age limit. In the Age-Layered Population Structure approach introduced by , the population is segregated into different age layers and newly generated individuals are introduced into the youngest age layer to increase diversity. combine the idea of age with a multi-objective approach, in which individuals are rewarded for performance and low age. Inspired by the morphological innovation approach and in contrast to previous approaches , DIP does not introduce new random individuals into the generation but instead resets the age of individuals whose sensory or memory system have been mutated. That is, it is not a measure of how long the individual has been in the population, as in the traditional usage of age in multi-objective optimization. Approaches to learning dynamical models have mainly focused on gradient descent-based methods, with early work on RNNs in the 1990s . More recent work includes PILCO , which is a probabilistic model-based policy search method and Black-DROPS , which employs CMA-ES for data-efficient optimization of complex control problems. Additionally, interest has increased in learning dynamical models directly from high-dimensional pixel images for robotic tasks and also video games . Work on evolving forward models has mainly focused on neural networks that contain orders of magnitude fewer connections and lower-dimensional feature vectors than the models in this paper. The paper demonstrated that a world model representation for a 3D task can emerge under the right circumstances without being explicitly rewarded for it. To encourage this emergence, we introduced deep innovation protection, an approach that can dynamically reduce the selection pressure for different components in a heterogeneous neural architecture. The main insight is that when components upstream in the neural network change, such as the visual or memory system in a world model, components downstream need time to adapt to changes in those learned representations. The neural model learned to represent situations that require similar actions with similar latent and hidden codes (Fig. 5 and 7). Additionally, without a specific forward-prediction loss, the agent learned to predict "useful" events that are necessary for its survival (e.g. predicting when the agent is in the line-of-fire of a fireball). In the future it will be interesting to compare the differences and similarities of emergent representations and learning dynamics ing from evolutionary and gradient descent-based optimization approaches . Interestingly, without the need for a variety of specialized learning methods employed in the original world model paper, a simple genetic algorithm augmented with DIP can not only solve the simpler 2D car racing domain , but also more complex 3D domains such as VizDoom. That the average score across 100 random rollouts is lower when compared to the one reported in the original world model paper (824 compared to 1092) is maybe not surprising; if random rollouts are available, training each component separately can in a higher performance. However, in more complicated domains, in which random rollouts might not be able to provide all relevant experiences (e.g. a random policy might never reach a certain level), the proposed DIP approach could become increasingly relevant. An exciting future direction is to combine the end-to-end training regimen of DIP with the ability of training inside the world model itself . However, because the evolved representation is not directly optimized to predict the next time step and only learns to predict future events that are useful for the agent's survival, it is an interesting open question how such a different version of a hallucinate environment could be used for training. A natural extension to this work is to evolve the neural architectures in addition to the weights of the network. Searching for neural architectures in RL has previously only been applied to smaller networks (; ; ; ;) but could potentially now be scaled to more complex tasks. While our innovation protection approach is based on evolution, ideas presented here could also be incorporated in gradient descent-based approaches that optimize neural systems with multiple interacting components end-to-end. The size of each population was 200 and evolutionary runs had a termination criterion of 200 generations. The genetic algorithm σ was determined empirically and set to 0.03 for the experiments in this paper. The code for the DIP approach can be found at: [Removed for anonymous review]. An overview of the agent model is shown in Fig. 1, which employs the same architecture as the original world model approach. The sensory model is implemented as a variational autoencoder that compresses the high-dimensional input to a latent vector z. The VAE takes as input an RGB image of size 64 × 64 × 3, which is passed through four convolutional layers, all with stride 2. Details on the encoder are depicted in the visual component shown in Fig. 1, where layer details are shown as: activation type (e.g. ReLU), number of output channels × filter size. The decoder, which is in effect only used to analyze the evolved visual representation, takes as input a tensor of size 1 × 1 × 104 and processes it through four deconvolutional layers each with stride 2 and sizes of 128 × 5, 64 × 5, 32 × 6, and 32 × 6. The network's weights are set using the default The memory model combines a recurrent LSTM network with a mixture density Gaussian model as network outputs, known as a MDN-RNN (; b). The network has 256 hidden nodes and models P (z t+1 |a t, z t, h t), where a t is the action taken by the agent at time t and h t is the hidden state of the recurrent network. Similar models have previously been used for generating sequences of sketches and handwriting (a We performed an analysis of the cumulative reward per age and the number of individuals with a certain age averaged across all ten runs and all generations (Fig. 8). While the average reward increases with age, there are fewer and fewer individuals at higher age levels. This suggest that the two objectives (minimising age and increasing cumulative reward) are in competition with each other, motivating the choice for a multi-objective optimization approach. These indicate that the multi-objective optimization is working as intended; staying alive for longer becomes increasingly difficult and a high age needs to be compensated for by a high task reward. Similarly to the approach by , we calculate perturbation-based saliency maps by applying a Gaussian blur of 5 × 5 pixels to the coordinates (i, j) of an image I from the game. The Gaussian blur can be interpreted as adding uncertainty to a particular location of the screen. For example, if a fireball is at location (i, j) then adding noise to that location makes the agent less certain about the fireball's location. The saliency map intensity S(i, j) is calculated as the difference between the policy output π given the original image I and modified image I with added Gaussian blur at location (i, j): S(i, j) = |π(I) − π(I)|.

Represent the short summary for finding the corresponding scientific article (long)

Deep Innovation Protection allows evolving complex world models end-to-end for 3D tasks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.


Question: Represent the article for finding a single-sentence TLDR:

Question: Adversarial training is one of the most popular ways to learn robust models but is usually attack-dependent and time costly. In this paper, we propose the MACER algorithm, which learns robust models without using adversarial training but performs better than all existing provable l2-defenses. Recent work shows that randomized smoothing can be used to provide certified l2 radius to smoothed classifiers, and our algorithm trains provably robust smoothed classifiers via MAximizing the CErtified Radius (MACER). The attack-free characteristic makes MACER faster to train and easier to optimize. In our experiments, we show that our method can be applied to modern deep neural networks on a wide range of datasets, including Cifar-10, ImageNet, MNIST, and SVHN. For all tasks, MACER spends less training time than state-of-the-art adversarial training algorithms, and the learned models achieve larger average certified radius. Modern neural network classifiers are able to achieve very high accuracy on image classification tasks but are sensitive to small, adversarially chosen perturbations to the inputs . Given an image x that is correctly classified by a neural network, a malicious attacker may find a small adversarial perturbation δ such that the perturbed image x + δ, though visually indistinguishable from the original image, is assigned to a wrong class with high confidence by the network. Such vulnerability creates security concerns in many real-world applications. Researchers have proposed a variety of defense methods to improve the robustness of neural networks. Most of the existing defenses are based on adversarial training (; ; ; ;). During training, these methods first learn on-the-fly adversarial examples of the inputs with multiple attack iterations and then update model parameters using these perturbed samples together with the original labels. However, such approaches depend on a particular (class of) attack method. It cannot be formally guaranteed whether the ing model is also robust against other attacks. Moreover, attack iterations are usually quite expensive. As a , adversarial training runs very slowly. Another line of algorithms trains robust models by maximizing the certified radius provided by robust certification methods (; ; c). Using linear or convex relaxations of fully connected ReLU networks, a robust certification method computes a "safe radius" r for a classifier at a given input such that at any point within the neighboring radius-r ball of the input, the classifier is guaranteed to have unchanged predictions. However, the certification methods are usually computationally expensive and can only handle shallow neural networks with ReLU activations, so these training algorithms have troubles in scaling to modern networks. In this work, we propose an attack-free and scalable method to train robust deep neural networks. We mainly leverage the recent randomized smoothing technique . A randomized smoothed classifier g for an arbitrary classifier f is defined as g(x) = E η f (x + η), in which η ∼ N (0, σ 2 I). derived how to analytically compute the certified radius of the randomly smoothed classifier g, they did not show how to maximize that radius to make the classifier g robust. proposed SmoothAdv to improve the robustness of g, but it still relies on the expensive attack iterations. Instead of adversarial training, we propose to learn robust models by directly taking the certified radius into the objective. We outline a few challenging desiderata any practical instantiation of this idea would however have to satisfy, and provide approaches to address each of these in turn. A discussion of these desiderata, as well as a detailed implementation of our approach is provided in Section 4. And as we show both theoretically and empirically, our method is numerically stable and accounts for both classification accuracy and robustness. Our contributions are summarized as follows: • We propose an attack-free and scalable robust training algorithm by MAximizing the CErtified Radius (MACER). MACER has the following advantages compared to previous works: -Different from adversarial training, we train robust models by directly maximizing the certified radius without specifying any attack strategies, and the learned model can achieve provable robustness against any possible attack in the certified region. Additionally, by avoiding time-consuming attack iterations, our proposed algorithm runs much faster than adversarial training. -Different from other methods that maximize the certified radius but are not scalable to deep neural networks, our method can be applied to architectures of any size. This makes our algorithm more practical in real scenarios. • We empirically evaluate our proposed method through extensive experiments on Cifar-10, ImageNet, MNIST, and SVHN. On all tasks, MACER achieves better performance than state-of-the-art algorithms. MACER is also exceptionally fast. For example, on ImageNet, MACER uses 39% less training time than adversarial training but still performs better. Neural networks trained by standard SGD are not robust -a small and human imperceptible perturbation can easily change the prediction of a network. In the white-box setting, methods have been proposed to construct adversarial examples with small ∞ or 2 perturbations (; ; ;). Furthermore, even in the black-box setting where the adversary does not have access to the model structure and parameters, adversarial examples can be found by either transfer attack or optimization-based approaches (; ;). It is thus important to study how to improve the robustness of neural networks against adversarial examples. Adversarial training So far, adversarial training has been the most successful robust training method according to many recent studies. Adversarial training was first proposed in and , where they showed that adding adversarial examples to the training set can improve the robustness against such attacks. More recently, showed that adversarial training can be formulated as a min-max optimization problem and demonstrated that adversarial training with PGD attack can lead to very robust models empirically. Zhang et al. (2019b) further proposed to decompose robust error as the sum of natural error and boundary error to achieve better performance. Although models obtained by adversarial training empirically achieve good performance, they do not have certified error guarantees. Despite the popularity of PGD-based adversarial training, one major issue is that its speed is too slow. Some recent papers propose methods to accelerate adversarial training. For example, Freem replays an adversarial example several times in one iteration, YOPO-m-n (a) restricts back propagation in PGD within the first layer, and estimates the adversary with local linearization. Robustness certification and provable defense Many defense algorithms proposed in the past few years were claimed to be effective, but showed that most of them are based on "gradient masking" and can be bypassed by more carefully designed attacks. It is thus important to study how to measure the provable robustness of a network. A robustness certification algorithm takes a classifier f and an input point x as inputs, and outputs a "safe radius" r such that for any δ subject to δ ≤ r, f (x) = f (x + δ). Several algorithms have been proposed recently, including the convex polytope technique, abstract interpretation methods (; and the recursive propagation algrithms (; . These methods can provide attack-agnostic robust error lower bounds. Moreover, to achieve networks with nontrivial certified robust error, one can train a network by minimizing the certified robust error computed by the above-mentioned methods, and several algorithms have been proposed in the past year ; c; . Unfortunately, they can only be applied to shallow networks with limited activation and run very slowly. More recently, researchers found a new class of certification methods called randomized smoothing. The idea of randomization has been used for defense in several previous works but without any certification. Later on, first showed that if a Gaussian random noise is added to the input or any intermediate layer. A certified guarantee on small 2 perturbation can be computed via differential privacy. and then provided improved ways to compute the 2 certified robust error for Gaussian smoothed models. In this paper, we propose a new algorithm to train on these 2 certified error bounds to significantly reduce the certified error and achieve better provable adversarial robustness. It is thus quite natural to improve model robustness via maximizing the robust radius. Unfortunately, computing the robust radius of a classifier induced by a deep neural network is very difficult. showed that computing the l 1 robust radius of a deep neural network is NP-hard. Although there is no for the l 2 radius yet, it is very likely that computing the l 2 robust radius is also NP-hard. Certified radius Many previous works proposed certification methods that seek to derive a tight lower bound of R(f θ ; x, y) for neural networks (see Section 2 for related work). We call this lower bound certified radius and denote it by CR(f θ ; x, y). The certified radius satisfies 0 ≤ CR(f θ ; x, y) ≤ R(f θ ; x, y) for any f θ, x, y. The certified radius leads to a guaranteed upper bound of the 0/1 robust classification error, which is called 0/1 certified robust error. The 0/1 certified robust error of classifier f θ on sample (x, y) is defined as l e. a sample is counted as correct only if the certified radius reaches. The expectation of certified robust error over (x, y) ∼ p data serves as a performance metric of the provable robustness: Recall that CR(f θ ; x, y) is a lower bound of the true robust radius, which immediately implies that L 0/1 Therefore, a small 0/1 certified robust error leads to a small 0/1 robust classification error. Randomized smoothing In this work, we use the recent randomized smoothing technique , which is scalable to any architectures, to obtain the certified radius of smoothed deep neural networks. The key part of randomized smoothing is to use the smoothed version of f θ, which is denoted by g θ, to make predictions. The formulation of g θ is defined as follows. Definition 1. For an arbitrary classifier f θ ∈ F and σ > 0, the smoothed classifier g θ of f θ is defined as g θ (x) = arg max In short, the smoothed classifier g θ (x) returns the label most likely to be returned by f θ when its input is sampled from a Gaussian distribution N (x, σ 2 I) centered at x. proves the following theorem, which provides an analytic form of certified radius: Theorem 1. Let f θ ∈ F, and η ∼ N (0, σ 2 I). Let the smoothed classifier g θ be defined as in. Let the ground truth of an input x be y. If g θ classifies x correctly, i.e. Then g θ is provably robust at x, with the certified radius given by where Φ is the c.d.f. of the standard Gaussian distribution. As we can see from Theorem 1, the value of the certified radius can be estimated by repeatedly sampling Gaussian noises. More importantly, it can be computed for any deep neural networks. This motivates us to design a training method to maximize the certified radius and learn robust models. To minimize the 0/1 robust classification error in or the 0/1 certified robust error in, many previous works (b;) proposed to first decompose the error. Note that a classifier g θ has a positive 0/1 certified robust error on sample (x, y) if and only if exactly one of the following two cases happens: • g θ (x) = y, i.e. the classifier misclassifies x. • g θ (x) = y, but CR(g θ ; x, y) <, i.e. the classifier is correct but not robust enough. Thus, the 0/1 certified robust error can be decomposed as the sum of two error terms: a 0/1 classification error and a 0/1 robustness error: Minimizing the 0-1 error directly is intractable. A classic method is to minimize a surrogate loss instead. The surrogate loss for the 0/1 classification error is called classification loss and denoted by l C (g θ ; x, y). The surrogate loss for the 0/1 robustness error is called robustness loss and denoted by l R (g θ ; x, y). Our final objective function is We would like our loss functions l C (g θ ; x, y) and l R (g θ ; x, y) to satisfy some favorable conditions. These conditions are summarized below as (C1) -(C3): • (C1) (Surrogate condition): Surrogate loss should be an upper bound of the original error function, i.e. l C (g θ ; x, y) and l R (g θ ; x, y) should be upper bounds of 1 {g θ (x) =y} and 1 {g θ (x)=y,CR(g θ ;x,y)< }, respectively. • (C2) (Differentiablity): l C (g θ ; x, y) and l R (g θ ; x, y) should be (sub-)differentiable with respect to θ. • (C3) (Numerical stability): The computation of l C (g θ ; x, y) and l R (g θ ; x, y) and their (sub-)gradients with respect to θ should be numerically stable. The surrogate condition (C1) ensures that l(g θ ; x, y) itself meets the surrogate condition, i.e. Conditions (C2) and (C3) ensure that can be stably minimized with first order methods. We next discuss choices of the surrogate losses that ensure we satisfy condition (C1). The classification surrogate loss is relatively easy to design. There are many widely used loss functions from which we can choose, and in this work we choose the cross-entropy loss as the classification loss: For the robustness surrogate loss, we choose the hinge loss on the certified radius: where˜ > 0 and λ ≥ 1. We use the hinge loss because not only does it satisfy the surrogate condition, but also it is numerically stable, which we will discuss in Section 4.4. The classification surrogate loss in is differentiable with respect to θ, but the differentiability of the robustness surrogate loss in requires differentiability of CR(g θ ; x, y). In this section we will show that the randomized smoothing certified radius in does not meet condition (C2), and accordingly, we will introduce soft randomized smoothing to solve this problem. Whether the certified radius is sub-differentiable with respect to θ boils down to the differentiablity of E η 1 {f θ (x+η)=y}. Theoretically, the expectation is indeed differentiable. However, from a practical point of view, the expectation needs to be estimated by Monte Carlo sampling Gaussian noise and k is the number of samples. This estimation, which is a sum of indicator functions, is not differentiable. Hence, condition (C2) is still not met from the algorithmic perspective. To tackle this problem, we leverage soft randomized smoothing (Soft-RS). In contrast to the original version of randomized smoothing (Hard-RS), Soft-RS is applied to a neural network z θ (x) whose last layer is softmax. The soft smoothed classifierg θ is defined as follows. Definition 2. For a neural network z θ: X → P(K) whose last layer is softmax and σ > 0, the soft smoothed classifierg θ of z θ is defined as Using Lemma 2 in , we prove the following theorem in Appendix A: Theorem 2. Let the ground truth of an input x be y. Ifg θ classifies x correctly, i.e. Theng θ is provably robust at x, with the certified radius given by where Φ is the c.d.f. of the standard Gaussian distribution. We notice that in (see its Appendix B), a similar technique was introduced to overcome the non-differentiability in creating adversarial examples to a smoothed classifier. Different from their work, our method uses Soft-RS to obtain a certified radius that is differentiable in practice. The certified radius given by soft randomized smoothing meets condition (C2) in the algorithmic design. Even if we use Monte Carlo sampling to estimate the expectation, is still sub-differentiable with respect to θ as long as z θ is sub-differentiable with respect to θ. Connection between Soft-RS and Hard-RS We highlight two main properties of Soft-RS. Firstly, it is a differentiable approximation of the original Hard-RS. To see this, note that when a.e. − − → 1 {y=arg max c u c θ (x)}, sog θ converges to g θ almost everywhere. Consequently, the Soft-RS certified radius converges to the Hard-RS certified radius almost everywhere as β goes to infinity. Secondly, Soft-RS itself provides an alternative way to get a provable robustness guarantee. In Appendix A, we will provide Soft-RS certification procedures that certifỹ g θ with the Hoeffding bound or the empirical Bernstein bound. In this section, we will address the numerical stability condition (C3). While Soft-RS does provide us with a differentiable certified radius which we could maximize with first-order optimization methods, directly optimizing suffers from exploding gradients. The problem stems from the inverse cumulative density function Φ −1 (x), whose derivative is huge when x is close to 0 or 1. Fortunately, by minimizing the robustness loss instead, we can maximize the robust radius free from exploding gradients. The hinge loss restricts that samples with non-zero robustness loss must satisfy 0 < CR(g θ ; x, y) < +˜, which is equivalent to 0 < ξ θ (x, y) < γ where. Under this restriction, the derivative of Φ −1 is always bounded as shown in the following proposition. The proof can be found in Appendix B., γ} with respect to p 1 and p 2 is bounded. We are now ready to present the complete MACER algorithm. Expectations over Gaussian samples are approximated with Monte Carlo sampling. Let ). During training we minimize E (x,y)∼p data l(g θ ; x, y). Detailed implementation is described in Algorithm 1. To simplify the implementation, we choose γ to be a hyperparameter instead of˜. The inverse temperature of softmax β is also a hyperparameter. For each Compute the empirical expectations: Update θ with one step of any first-order optimization method to minimize 10: end for Compare to adversarial training Adversarial training defines the problem as a mini-max game and solves it by optimizing the inner loop (attack generation) and the outer loop (model update) iteratively. In our method, we only have a single loop (model update). As a , our proposed algorithm can run much faster than adversarial training because it does not require additional back propagations to generate adversarial examples. Compare to previous work The overall objective function of our method, a linear combination of a classification loss and a robustness loss, is similar to those of adversarial logit pairing (ALP) and TRADES (b). In MACER, the λ in the objective function can also be viewed as a trade-off factor between accuracy and robustness. However, the robustness term of MACER does not depend on a particular adversarial example x, which makes it substantially different from ALP and TRADES. In this section, we empirically evaluate our proposed MACER algorithm on a wide range of tasks. We also study the influence of different hyperparameters in MACER on the final model performance. To fairly compare with previous works, we follow and to use LeNet for MNIST, ResNet-110 for Cifar-10 and SVHN, and ResNet-50 for ImageNet. MACER Training For Cifar-10, MNIST and SVHN, we train the models for 440 epochs using our proposed algorithm. The learning rate is initialized to be 0.01, and is decayed by 0.1 at the 200 th /400 th epoch. For all the models, we use k = 16, γ = 8.0 and β = 16.0. The value of λ trades off the accuracy and robustness and we find that different λ leads to different robust accuracy when the model is injected by different levels (σ) of noise. We find setting λ = 12.0 for σ = 0.25 and λ = 4.0 for σ = 0.50 works best. For ImageNet, we train the models for 120 epochs. The initial learning rate is set to be 0.1 and is decayed by 0.1 at the 30 th /60 th /90 th epoch. For all models on ImageNet, we use k = 2, γ = 8.0 and β = 16.0. More details can be found in Appendix C. Baselines We compare the performance of MACER with two previous works. The first work trains smoothed networks by simply minimizing cross-entropy loss. The second one uses adversarial training on smoothed networks to improve the robustness. For both baselines, we use checkpoints provided by the authors and report their original numbers whenever available. In addition, we run's method on all tasks as it is a speical case of MACER by setting k = 1 and λ = 0. Certification Following previous works, we report the approximated certified test set accuracy, which is the fraction of the test set that can be certified to be robust at radius r. However, the approximated certified test set accuracy is a function of the radius r. It is hard to compare two models unless one is uniformly better than the other for all r. Hence, we also use the average certified radius (ACR) as a metric: for each test data (x, y) and model g, we can estimate the certified radius CR(g; x, y). The average certified radius is defined as 1 |Stest| (x,y)∈Stest CR(g; x, y) where S test is the test set. To estimate the certified radius for data points, we use the source code provided by. We report the on Cifar-10 and ImageNet in the main body of the paper. Results on MNIST and SVHN can be found in Appendix C.2. Performance The performance of different models on Cifar-10 are reported in Table 1, and in Figure 1 we display the radius-accuracy curves. Note that the area under a radius-accuracy curve is equal to the ACR of the model. First, the plots show that our proposed method consistently achieves significantly higher approximated certified test set accuracy than. This shows that robust training via maximizing the certified radius is more effective than simply minimizing the cross entropy classification loss. Second, the performance of our model is different from that of Salman et al. The gain of our model is relatively smaller when σ = 1.0. This is because σ = 1.0 is a very large noise level and both models perform poorly. The ImageNet are displayed in Table 2 and Figure 2, and the observation is similar. All experimental show that our proposed algorithm is more effective than previous ones. Training speed Since MACER does not require adversarial attack during training, it runs much faster to learn a robust model. Empirically, we compare MACER with on the average training time per epoch and the total training hours, and list the statistics in Table 3. For a fair comparison, we use the codes 34 provided by the original authors and run all algorithms on the same machine. For Cifar-10 we use one NVIDIA P100 GPU and for ImageNet we use four NVIDIA P100 GPUs. According to our experiments, on ImageNet, MACER achieves ACR=0.544 in 117.90 hours. On the contrary, only achieves ACR=0.528 but uses 193.10 hours, which clearly shows that our method is much more efficient. One might question whether the higher performance of MACER comes from the fact that we train for more epochs than previous methods. In Section C.3 we also run MACER for 150 epochs and compare it with the models in Table 3. The show that when run for only 150 epochs, MACER still achieves a performance comparable with SmoothAdv, and is 4 times faster at the same time. In this section, we carefully examine the effect of different hyperparameters in MACER. All experiments are run on Cifar-10 with σ = 0.25 or 0.50. The for σ = 0.25 are shown in Figure 3. All details can be found in Appendix C.4. Effect of k We sample k Gaussian samples for each input to estimate the expectation in. We can see from Figure 3 (a) that using more Gaussian samples usually leads to better performance. For example, the radius-accuracy curve of k = 16 is uniformly above that of k = 1. Effect of λ The radius-accuracy curves in Figure 3 (b) demonstrate the trade-off effect of λ. From the figure, we can see that as λ increases, the clean accuracy drops while the certified accuracy at large radii increases. Effect of γ γ is defined as the hyperparameter in the hinge loss. From Figure 3 (c) we can see that when γ is small, the approximated certified test set accuracy at large radii is small since γ "truncates" the large radii. As γ increases, the robust accuracy improves. It appears that γ also acts as a trade-off between accuracy and robustness, but the effect is not as significant as the effect of λ. Effect of β Similar to's finding (see its Appendix B), we also observe that using a larger β produces better . pointed out that a large β may make training unstable, we find that if we only apply a large β to the robustness loss, we can maintain training stability and achieve a larger average certified radius as well. In this work we propose MACER, an attack-free and scalable robust training method via directly maximizing the certified radius of a smoothed classifier. We discuss the desiderata such an algorithm would have to satisfy, and provide an approach to each of them. According to our extensive experiments, MACER performs better than previous provable l 2 -defenses and trains faster. Our strong empirical suggest that adversarial training is not a must for robust training, and defense based on certification is a promising direction for future research. Moreover, several recent papers (; ; suggest that using unlabeled data helps improve adversarially robust generalization. We will also extend MACER to the semisupervised setting. In this section we provide theoretical analysis and certification procedures for Soft-RS. Our proof is based on the following lemma: Proof of Theorem 2. Let y Because z c θ : Meanwhile, z B ≤ 1 − z A, so we can take z B = 1 − z A, and It reduces to find a confidence lower bound of z A . Here we provide two bounds: Hoeffding Bound The random variable z . By Hoeffding's inequality we have Hence, a 1 − α confidence lower bound z A of z A is where S 2 is the sample variance of X 1, · · ·, X k, i.e. Consequently, a 1 − α confidence lower bound z A of z A is The full certification procedure with the above two bounds is described in Algorithm 2. Algorithm 2 Soft randomized smoothing certification 1: # Certify the robustness ofg around an input x with Hoeffding bound 2: function CERTIFYHOEFFDING(z, σ 2, x, n 0, n, α) if z A > A ← SAMPLEUNDERNOISE(z, x, n, σ 2) 23: for j = 1 to num do 24: Sample noise η j ∼ N (0, σ 2 I) Compute: z j = z(x + η j) Figure 4. The show that Hard-RS consistently gives a larger lower bound of robust radius than Soft-RS. We also observe that there is a gap between Soft-RS and Hard-RS when β → ∞, which implies that the empirical Bernstein bound, though tighter than the Hoeffding bound, is still looser than the Clopper-Pearson bound. Proof of Proposition 1. We only need to consider the case when is a strictly increasing function of p, p * is unique, and Since p 1 is the largest value and p 1 + p 2 +... is continuous in any closed interval of, the derivative of Φ −1 (p 1) − Φ −1 (p 2) with respect to p 1 is bounded. Similarly, p 2 is the largest among p 2,..., and the derivative of Φ −1 (p 1) − Φ −1 (p 2) with respect to p 2 is bounded. In this section we list all compared models in the main body of this paper. Cifar-10 models are listed in Table 4, and ImageNet models are listed in Table 5. 1.00 Salman-1.00 2-sample 10-step SmoothAdv P GD with = 2.00 MACER-1.00 MACER with k = 16, dynamic λ 5, β = 16.0 and γ = 8.0 The are reported in Table 6. For all σ, we use k = 16, λ = 16.0, γ = 8.0 and β = 16.0. Table 7. We use k = 16, λ = 12.0, γ = 8.0 and β = 16.0. Table 8 we report the performance and training time of MACER on Cifar-10 when it is only run for 150 epochs, and compare with SmoothAdv and MACER (440 epochs). The learning rate is decayed by 0.1 at epochs 60 and 120. All other hyperparameters are kept the same as in Table 4. Table 9 for detailed experimental settings. Results are reported in Tables 10-13.

Represent the TLDR for finding the corresponding scientific article

We propose MACER: a provable defense algorithm that trains robust models by maximizing the certified radius. It does not use adversarial training but performs better than all existing provable l2-defenses.

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.