Question: Represent the article for finding a single-sentence TLDR

Question: Learning an efficient update rule from data that promotes rapid learning of new tasks from the same distribution remains an open problem in meta-learning. Typically, previous works have approached this issue either by attempting to train a neural network that directly produces updates or by attempting to learn better initialisations or scaling factors for a gradient-based update rule. Both these approaches pose challenges. On one hand, directly producing an update forgoes a useful inductive bias and can easily lead to non-converging behaviour. On the other hand, approaches that try to control a gradient-based update rule typically resort to computing gradients through the learning process to obtain their meta-gradients, leading to methods that can not scale beyond few-shot task adaptation. In this work we propose Warped Gradient Descent (WarpGrad), a method that intersects these approaches to mitigate their limitations. WarpGrad meta-learns an efficiently parameterised preconditioning matrix that facilitates gradient descent across the task distribution. Preconditioning arises by interleaving non-linear layers, referred to as warp-layers, between the layers of a task-learner. Warp-layers are meta-learned without backpropagating through the task training process in a manner similar to methods that learn to directly produce updates. WarpGrad is computationally efficient, easy to implement, and can scale to arbitrarily large meta-learning problems. We provide a geometrical interpretation of the approach and evaluate its effectiveness in a variety of settings, including few-shot, standard supervised, continual and reinforcement learning. to learn implies inferring a learning strategy from some set of past experiences via a meta-learner that a task-learner can leverage when learning a new task. One approach is to directly parameterise an update rule via the memory of a recurrent neural network (; ; ;). Such memory-based methods can, in principle, represent any learning rule by virtue of being universal function approximators (; ; Schäfer &). They can also scale to long learning processes by using truncated backpropagation through time, but they lack an inductive bias as to what constitutes a reasonable learning rule. This renders them hard to train and brittle to generalisation as their parameter updates have no guarantees of convergence. An alternative family of approaches defines a gradient-based update rule and meta-learns a shared initialisation that facilitates task adaptation across a distribution of tasks (; ;). Such methods are imbued with a strong inductive biasgradient descent-but restrict knowledge transfer to the initialisation. Recent work has shown that it is beneficial to more directly control gradient descent by meta-learning an approximation of a parameterised matrix (; ;) that preconditions gradients during task training, similarly to second-order and Natural Gradient Descent methods . To meta-learn preconditioning, these methods backpropagate through the gradient descent process, limiting them to few-shot learning. In this paper, we propose a novel framework called Warped Gradient Descent (WarpGrad), that relies on the inductive bias of gradient-based meta-learners by defining an update rule that preconditions gradients, but that is meta-learned using insights from memory-based methods. In particular, we Meta-Learning Task-learners Shared Warp Figure 1: Schematics of WarpGrad. WarpGrad preconditioning is embedded in task-learners f by interleaving warp-layers (ω, ω ) between each task-learner's layers (h, h ). WarpGrad achieve preconditioning by modulating layer activations in the forward pass and gradients in the backward pass by backpropagating through warp-layers (Dω), which implicitly preconditions gradients by some matrix (P). Warp-parameters (φ) are meta-learned over the joint search space induced by task adaptation (E θ [J(φ)]) to form a geometry that facilitates task learning. leverage that gradient preconditioning is defined point-wise in parameter space and can be seen as a recurrent operator of order 1. We use this insight to define a trajectory agnostic meta-objective over a joint parameter search space where knowledge transfer is encoded in gradient preconditioning. To achieve a scalable and flexible form of preconditioning, we take inspiration from works that embed preconditioning in task-learners , but we relax the assumption that task-learners are feed-forward and replace their linear projection with a generic neural network ω, referred to as a warp layer. By introducing non-linearity, preconditioning is rendered data-dependent. This allows WarpGrad to model preconditioning beyond the block-diagonal structure of prior works and enables it to meta-learn over arbitrary adaptation processes. We empirically validate WarpGrad and show it surpasses baseline gradient-based meta-learners on standard few-shot learning tasks (miniImageNet, tieredImageNet; ; ;), while scaling beyond few-shot learning to standard supervised settings on the "multi"-shot Omniglot benchmark and a multi-shot version of tieredImageNet. We further find that WarpGrad outperforms competing methods in a reinforcement learning (RL) setting where previous gradient-based meta-learners fail (maze navigation with recurrent neural networks ) and can be used to meta-learn an optimiser that prevents catastrophic forgetting in a continual learning setting. WarpGrad belongs to the family of optimisation-based meta-learners that parameterise an update rule θ ← U (θ; ξ) with some meta-parameters ξ. Specifically gradient-based meta-learners define an update rule by relying on the gradient descent, U (θ; ξ):= θ − α∇ L(θ) for some objective L and learning rate α. A task is defined by a training set D τ train and a test set D τ test, which defines learning objectives L D τ (θ):= E (x,y)∼D τ [(f (x, θ), y)] over the task-learner f for some loss. MAML meta-learns a shared initialisation θ 0 by backpropagating through K steps of gradient descent across a given task distribution p(τ), θ 0 θ 0 Figure 2: Gradient-based meta-learning. Colours denote different tasks (τ), dashed lines denote backpropagation through the adaptation process, and solid black lines denote optimiser parameter (φ) gradients w.r.t. one step of task parameter (θ) adaptation. Left: A meta-learned initialisation compresses trajectory information into a single initial point (θ 0). Middle: MAML-based optimisers interact with adaptation trajectories at every step and backpropagate through each interaction. Right: WarpGrad is trajectory agnostic. Task adaptation defines an empirical distribution p(τ, θ) over which WarpGrad learns a geometry for adaptation by optimising for steepest descent directions. Subsequent work on gradient-based meta-learning primarily differ in the parameterisation of U. Meta-SGD (MSGD;) learns a vector of learning rates, whereas Meta-Curvature (MC;) defines a block-diagonal preconditioning matrix B, and T-Nets embed block-diagonal preconditioning in feed-forward learners via linear projections, T-Nets. These methods optimise for meta-parameters ξ = {θ 0, φ} by backpropagating through the gradient descent process, as per Eq. 1. This limits them to few-shot learning as they become computationally expensive, susceptible to exploding/vanishing gradients, and susceptible to a credit assignment problem; ). Our goal is to develop a meta-learner that overcomes all three limitations. To do so, we depart from the paradigm of backpropagating to the initialisation and exploit the fact that learning to precondition gradients can be seen as a Markov Process of order 1, a special form of a recurrent update rule . To develop this notion, we first establish a general-purpose form of preconditioning (Section 2.2). Based on this, we obtain a canonical meta-objective from a geometrical point of view (Section 2.3), from which we derive a trajectory-agnostic meta-objective (Section 2.4). A preconditioned gradient descent rule, U (θ; φ):= θ − αP (θ; φ)∇ L(θ), defines a geometry via P. To disentangle the expressive capacity of this geometry from the expressive capacity of the tasklearner f, we take inspiration from T-Nets that embed linear projections T in feed-forward layers, h = σ(T W x+b). This in itself is not sufficient to achieve disentanglement since the parameterisation of T is directly linked to that of W, but it can be achieved under non-linear preconditioning. To this end, we relax the assumption that the task-learner is feed-forward and consider an arbitrary. We insert warp-layers that are universal function approximators parameterised by neural networks into the task-learner without restricting their form or how they interact with f. In the simplest case, we interleave warp-layers between layers of the task-learner to, but other forms of interaction can be beneficial (see Ap- Figure 3: Left: synthetic experiment illustrating how WarpGrad warps gradients (see Appendix D for full details). Each task f ∼ p(f) defines a distinct loss surface (W, bottom row). Gradient descent (black) on these surfaces struggles to find a minimum. WarpGrad meta-learns a warp ω to produce better update directions (magenta; Section 2.4). In doing so, WarpGrad learns a meta-geometry P where standard gradient descent is well behaved (top row). Right: gradient descent in P is equivalent to first-order Riemannian descent in W under a meta-learned Riemann metric (Section 2.3). pendix A for practical guidelines). Backpropagation automatically induces gradient preconditioning, as in T-Nets, but in our case via the Jacobians of the warp-layers: where D x and D θ denote the Jacobian with respect to input and parameters, respectively. In the special case where f is feed-forward and each ω a linear projection, we obtain an instance of WarpGrad that is akin to T-Nets since preconditioning is given by D x ω = T. Conversely, by making warp-layers non-linear, we can induce interdependence between warp-layers, allowing WarpGrad to model preconditioning beyond the block-diagonal structure imposed by prior works. Further, this enables a form of task-conditioning by making Jacobians of warp-layers data dependent. As we have made no assumptions on the form of the task-learner or warp-layers, WarpGrad methods can act on any neural network through any form of warping, including recurrence. We show that increasing the capacity of the meta-learner by defining warp-layers as Residual Networks improves performance on classification tasks (Section 4.1). We also introduce recurrent warp-layers for agents in a gradient-based meta-learner that is the first, to the best of our knowledge, to outperform memory-based meta-learners on a maze navigation task that requires memory (Section 4.3). Warp-layers imbue WarpGrad with three powerful properties. First, thanks to the preconditioned gradients, WarpGrad inherits gradient descent properties, importantly guarantees of convergence. Second, warp-layers form a distributed representation of preconditioning that disentangles the expressiveness of the geometry it encodes from the expressive capacity of the task-learner. Third, warp-layers are meta-learned across tasks and trajectories and can therefore capture properties of the task-distribution beyond local information. Figure 3 illustrates these properties in a synthetic scenario, where we construct a family of tasks f: R 2 → R (see Appendix D for details) and meta-learn across the task distribution. WarpGrad learns to produce warped loss surfaces (illustrated on two tasks τ and τ) that are smoother and more well-behaved than their respective native loss-surfaces. If the preconditioning matrix P is invertible, it defines a valid Riemann metric and therefore enjoys similar convergence guarantees to gradient descent. Thus, if warp-layers represent a valid (meta-learned) Riemann metric, WarpGrad is well-behaved. For T-Nets, it is sufficient to require T to be full rank, since T explicitly defines P as a block-diagonal matrix with block entries T T T. In contrast, non-linearity in warp-layers precludes such an explicit identification. Instead, we must consider the geometry that warp-layers represent. For this, we need a metric tensor, G, which is a positive-definite, smoothly varying matrix that measures curvature on a manifold W. The metric tensor defines the steepest direction of descent by −G −1 ∇ L , hence our goal is to establish that warp-layers approximate some G −1. Let Ω represent the effect of warp-layers by a reparameterisation h (i) (x; Ω(θ; φ) ∀x, i that maps from a space P onto the manifold W with γ = Ω(θ; φ). We induce a metric G on W by push-forward (Figure 2): where Provided Ω is not degenerate (G is non-singular), G −1 is positivedefinite, hence a valid Riemann metric. While this is the metric induced on W by warp-layers, it is not the metric used to precondition gradients since we take gradient steps in P which introduces an error term (Figure 2). We can bound the error by first-order Taylor series expansion to establish first-order equivalence between the WarpGrad update in P (Eq. 7) and the ideal update in W (Eq. 8), Consequently, gradient descent under warp-layers (in P-space) is first-order equivalent to warping the native loss surface under a metric G to facilitate task adaptation. Warp parameters φ control the geometry induced by warping, and therefore what task-learners converge to. By meta-learning φ we can accumulate information that is conducive to task adaptation but that may not be available during that process. This suggest that an ideal geometry (in W-space) should yield preconditioning that points in the direction of steepest descent, accounting for global information across tasks, In contrast to MAML-based approaches (Eq. 1), this objective avoids backpropagation through learning processes. Instead, it defines task learning abstractly by introducing a joint distribution over objectives and parameterisations, opening up for general-purpose meta-learning at scale. The canonical objective in Eq. 10 describes a meta-objective for learning a geometry on first principles that we can render into a trajectory-agnostic update rule for warp-layers. To do so, we define a by a task-learnerf that is embedded with a shared WarpGrad optimiser, a meta-training objective L τ meta, and a task adaptation objective L τ task. We use L τ task to adapt task parameters θ and L τ meta to adapt warp parameters φ. Note that we allow meta and task objectives to differ in arbitrary ways, but both are expectations over some data, as above. In the simplest case they differ in terms of validation versus training data, but they may differ in terms of learning paradigm as well, as we demonstrate in continual learning experiment (Section 4.3). To obtain our meta-objective, we recast the canonical objective (Eq. 10) in terms of θ using first-order equivalence of gradient steps (Eq. 9). Next, we factorise p(τ, θ) into p(θ | τ)p(τ). Since p(τ) is given, it remains to consider a sampling strategy for p(θ | τ). For meta-learning of warp-layers, we assume this distribution is given. We later show how to incorporate meta-learning of a prior p(θ 0 | τ). While any sampling strategy is valid, in this paper we exploit that task learning under stochastic gradient descent can be seen as sampling from an empirical prior p(θ | τ) ; in particular, each iterate θ K and sampling such chains defines an empirical distribution p(θ | τ) around some prior p(θ 0 | τ), which we will discuss in Section 2.5. The joint distribution p(τ, θ) defines a joint search space across tasks. Meta-learning therefore learns a geometry over this space with the steepest expected direction of descent. This direction is however not with respect to the objective that produced the gradient, L τ task, but with respect to L τ meta, Decoupling the task gradient operator ∇L τ task from the geometry learned by L τ meta lets us infuse global knowledge in warp-layers, a promising avenue for future research . As an example of this, in Section 4.3 we meta-learn an update-rule that mitigates catastrophic forgetting by defining L τ meta over current and previous tasks seen at any given point of adaptation. In contrast to other gradient-based meta-learners, the WarpGrad meta-objective is defined as a one-step recurrent operator that is meta-learned across a joint search space. This meta-objective is trajectory agnostic and thus compatible with arbitrary task learning processes. It does not suffer from vanishing/exploding gradients nor from the credit assignment problem, but it does rely on second-order gradients, a requirement we can relax by detaching task parameter gradients (∇L where sg is the stop-gradient operator. In contrast to the first-order approximation of MAML , which ignores the entire trajectory except for the final gradient, this approximation retains all gradient terms and only discards local second-order effects, which are typically dominated by first-order effect in long parameter trajectories . Empirically, we find that our approximation only incurs a minor loss of performance in an ablation study on Omniglot (Appendix F). Interestingly, this approximation is a form of multitask learning with respect to φ (; ;) that marginalises over task parameters θ τ. WarpGrad is a method for learning warp layer parameters φ over a joint search space defined by p(τ, θ). Because WarpGrad takes this distribution as given, we can integrate WarpGrad with methods that define or learn some form of "prior" p(θ 0 | τ) over θ τ 0. For instance, (a) Multi-task solution: in online learning, we can alternate between updating a multi-task solution and tuning warp parameters. We use this approach in our Reinforcement Learning experiment (Section 4.3); (b) Meta-learned point-estimate: when task adaptation occurs in batch mode, we can meta-learn a shared initialisation θ 0. Our few-shot and supervised learning experiments take this approach (Section 4.1); (c) Metalearned prior: WarpGrad can be combined with Bayesian methods that define a full prior (; ; ;). We incorporate such methods by some objective C (potentially vacuous) over θ 0 that we optimise jointly with WarpGrad, where L can be substituted byL and λ ∈ [0, ∞) is a hyper-parameter. We train the WarpGrad optimiser via stochastic gradient descent. We solve Eq. 13 by alternating between sampling task parameters from p(τ, θ) given the current parameter values for φ and taking meta-gradient steps over these samples to update φ. As such, our method can also be seen as a generalised form of gradient descent in the form of Mirror Descent with a meta-learned dual space . The details of the sampling procedure may vary depending on the specifics of the tasks (static, sequential), the design of the task-learner (feed-forward, recurrent), and the learning objective (supervised, self-supervised, reinforcement learning). In Algorithm 1 we illustrate a simple online algorithm with constant memory and linear complexity in K, assuming the same holds for C. A drawback of this approach is that it is relatively data inefficient; in Appendix B we detail a more complex offline training algorithm that stores task parameters in a replay buffer for mini-batched training of φ. The gains of the offline variant can be dramatic: in our Omniglot experiment (Section 4.1), offline meta-training allows us to update warp parameters 2000 times with each meta-batch, improving final test accuracy from 76.3% to 84.3% (Appendix F). Learning to learn, or meta-learning, has previously been explored in a variety of settings. Early work focused on evolutionary approaches (; ;). introduced gradient descent methods to meta-learning, specifically for recurrent meta-learning algorithms, extended to RL by and. A similar approach was taken by and to meta-learn a parameterised update rule in the form of a Recurrent Neural Network (RNN). A related idea is to separate parameters into "slow" and "fast" weights, where the latter captures meta-information and the former encapsulates rapid adaptation (; ;). This can be implemented by embedding a neural network that dynamically adapt the parameters of a main architecture . WarpGrad can be seen as learning slow warp-parameters that precondition adaptation of fast task parameters. Recent meta-learning has focused almost exclusively on the special case of few-shot learning, where tasks are characterised by severe data scarcity. In this settings, tasks must be relatively similar, such that a single or handful of examples are sufficient to learn the task at hand (; ; ;). Several meta-learners have been proposed that directly predict the parameters of the task-learner (; ; ;). To scale, such methods typically pretrain a feature extractor and predict a small subset of the parameters. Closely related to our work are gradient-based few-shot learning methods that extend MAML by sharing some subset of parameters between task-learners that is fixed during task training but meta-learner across tasks, which may reduce overfitting (; ;) or induce more robust convergence . It can also be used to model latent variables for concept or task inference, which implicitly induce gradient modulation (; ; ;). Our work is also related to gradient-based meta-learning of a shared initialisation that scales beyond few-shot learning . Require: p(τ): distribution over tasks Require: α, β, λ: hyper-parameters 1: initialise φ and p(θ 0 | τ) 2: while not done do sample mini-batch of task T from p(τ) 4: for all τ ∈ T do 6: for all k in 0,..., K τ −1 do 8: 10: end for sample mini-batch of task T from p(τ) for all τ ∈ T do 6: end for 12: end for for all (τ, k) ∈ B do 15: 16: end for 24: end while Meta-learned preconditioning is closely related to parallel work on second-order optimisation methods for high dimensional non-convex loss surfaces (; ; ;). In this setting, second-order optimisers typically struggle to improve upon first-order baselines . As second-order curvature is typically intractable to compute, such methods resort to low-rank approximations (; ;) and suffer from instability . In particular, Natural Gradient Descent is a method that uses the Fisher Information Matrix as curvature metric . Several proposed methods for amortising the cost of estimating the metric (; ;). As noted by , expressing preconditioning through interleaved projections can be seen as a form of Mirror Descent . WarpGrad offers a new perspective on gradient preconditioning by introducing a generic form of model-embedded preconditioning that exploits global information beyond the task at hand. Leap 73.9 ± 2.2 75.5 ± 2.6 Warp-Leap 80.4 ± 1.6 83.6 ± 1.9 We evaluate WarpGrad in a set of experiments designed to answer three questions: do WarpGrad methods retain the inductive bias of MAML-based fewshot learners? Can WarpGrad methods scale to problems beyond the reach of such methods? Can WarpGrad generalise to complex meta-learning problems? For few-shot learning, we test whether WarpGrad retains the inductive bias of gradient-based meta-learners while avoiding backpropagation through the gradient descent process. To isolate the effect of the WarpGrad objective, we use linear warp-layers that we train using online meta-training (Algorithm 1) to make WarpGrad as close to T-Nets as possible. For a fair comparison, we meta-learn the initialisation using All task-learners use a convolutional architecture that stacks 4 blocks made up of a 3 × 3 convolution, max-pooling, batch-norm, and ReLU activation. We define Warp-MAML by inserting warp-layers in the form of 3 × 3 convolutions after each block in the baseline task-learner. All baselines are tuned with identical and independent hyper-parameter searches (including filter sizes -full experimental settings in Appendix H), and we report best from our experiments or the literature. Warp-MAML outperforms all baselines (Table 1), improving 1-and 5-shot accuracy by 3.6 and 5.5 percentage points on miniImageNet and by 5.2 and 3.8 percentage points on tieredImageNet , which indicates that WarpGrad retains the inductive bias of MAML-based meta-learners. Next, we evaluate whether WarpGrad can scale beyond few-shot adaptation on similar supervised problems. We propose a new protocol for tieredImageNet that increases the number of adaptation Omniglot test accuracies on held-out tasks after meta-training on a varying number of tasks. Shading represents standard deviation across 10 independent runs. We compare WarpLeap, Leap, and Reptile, multi-headed finetuning, as well as SGD and KFAC which used random initialisation but with 10x larger batch size and learning rate. Right: On a RL maze navigation task, mean cumulative return is shown. Shading represents inter-quartile ranges across 10 independent runs. † Simple modulation and ‡ retroactive modulation are used . steps to 640 and use 6 convolutional blocks in task-learners, which are otherwise defined as above. Since MAML-based approaches cannot backpropagate through 640 adaptation steps for models of this size, we evaluate WarpGrad against two gradient-based meta-learners that meta-learn an initialisation without such backpropagation, Reptile and Leap , and we define a Warp-Leap meta-learner by Leap is an attractive complement as it minimises the expected gradient descent trajectory length across tasks. Under WarpGrad, this becomes a joint search for a geometry in which task adaptation defines geodesics (shortest paths, see Appendix C for details). While Reptile outperforms Leap by 2.6 percentage points on this benchmark, Warp-Leap surpasses both, with a margin of 3.88 to Reptile (Table 1). We further evaluate Warp-Leap on the multi-shot Omniglot protocol proposed by , where each of the 50 alphabets is a 20-way classification task. Task adaptation involves 100 gradient steps on random samples that are preprocessed by random affine transformations. We report for Warp-Leap under offline meta-training (Algorithm 2), which updates warp parameters 2000 times per meta step (see Appendix E for experimental details). WarpLeap enjoys similar performance on this task as well, improving over Leap and Reptile by 8.1 and 12.8 points respectively (Table 1). We also perform an extensive ablation study varying the number of tasks in the meta-training set. Except for the case of a single task, Warp-Leap substantially outperforms all baselines (Figure 4), achieving a higher rate of convergence and reducing the final test error from~30% to~15%. Non-linear warps, which go beyond block-diagonal preconditioning, reach~11% test error (refer to Appendix F and Table 2 for the full ). Finally, we find that WarpGrad methods behave distinctly different from Natural Gradient Descent methods in an ablation study (Appendix G). It reduce final test error from~42% to~19%, controlling for initialisation, while its preconditioning matrices differ from what the literature suggests . (c.1) Reinforcement Learning To illustrate how WarpGrad may be used both with recurrent neural networks and in meta-reinforcement learning, we evaluate it in a maze navigation task proposed by. The environment is a fixed maze and a task is defined by randomly choosing a goal location. The agent's objective is to find the location as many times as possible, being teleported to a random location each time it finds it. We use advantage actor-critic with a basic recurrent neural network as the task-learner, and we design a Warp-RNN as a HyperNetwork that uses an LSTM that is fixed during task training. This LSTM modulates the weights of the task-learning RNN (defined in Appendix I), which in turn is trained on mini-batches of 30 episodes for 200 000 steps. We accumulate the gradient of fixed warp-parameters continually (Algorithm 3, Appendix B) at each task parameter update. Warp parameters are updated on every 30 th step on task parameters (we control for meta-LSTM capacity in Appendix I). We compare against Learning to Reinforcement Learn and Hebbian meta-learning (; 2019); see Appendix I for details. Notably, linear warps (T-Nets) do worse than the baseline RNN on this task while the Warp-RNN converges to a mean cumulative reward of~160 in 60 000 Figure 5: Continual learning regression experiment. Average log-loss over 100 randomly sampled tasks. Each task contains 5 sub-tasks learned (a) sequentially as seen during meta-training or (b) in random order [sub-task 1, 3, 4, 2, 0]. We train on each sub-task for 20 steps, for a total of K = 100 task adaptation steps. episodes, compared to baselines that reach at most a mean cumulative reward of~125 after 100 000 episodes (Figure 4), reaching~150 after 200 000 episodes (I). (c.2) Continual Learning We test if WarpGrad can prevent catastrophic forgetting in a continual learning scenario. To this end, we design a continual learning version of the sine regression meta-learning experiment in by splitting the input interval [−5, 5] ⊂ R into 5 consecutive sub-tasks (an alternative protocol was recently proposed independently by). Each sub-task is a regression problem with the target being a mixture of two random sine waves; for each task, we train a 4-layer feed-forward task-learner with interleaved warp-layers incrementally on one sub-task at a time (see Appendix J for details). To isolate the behaviour of WarpGrad parameters, we use a fixed random initialisation for each task sequence. Warp parameters are meta-learned to prevent catastrophic forgetting by defining L τ meta to be the average task loss over current and previous sub-tasks, for each sub-task in a task sequence. This forces warp-parameters to disentangle the adaptation process of current and previous sub-tasks. Evaluating our WarpGrad optimiser on 100 random tasks, we find that it learns new sub-tasks well, with mean losses on a 10 −3 magnitude. When switching tasks, performance immediately deteriorates to an order of magnitude 10 −2, where it remains for the remainder of training (Figure 5). These indicate that WarpGrad can be an effective mechanism against catastrophic forgetting, a promising avenue for further research. For detailed , see Appendix J. We propose WarpGrad, a novel meta-learner that combines the expressive capacity and flexibility of memory-based meta-learners with the inductive bias of gradient-based meta-learners. WarpGrad meta-learns to precondition gradients during task adaptation without backpropagating through the adaptation process and we find empirically that it retains the inductive bias of MAML-based few-shot learners while being able to scale to complex problems and architectures. Further, by expressing preconditioning through warp-layers that are universal function approximators, WarpGrad is able to express geometries beyond the block-diagonal structure of prior works. WarpGrad provides a principled framework for general-purpose meta-learning that integrates learning paradigms, such as continual learning, an exciting avenue for future research. We introduce novel means for preconditioning, for instance with residual and recurrent warp-layers. Understanding how WarpGrad manifolds relate to second-order optimisation methods will further our understanding of gradient-based meta-learning and aid us in designing warp-layers with stronger inductive bias. In their current form, WarpGrad methods share some of the limitations of many popular metalearning approaches. While WarpGrad is designed to avoid backpropagating through the task training process, as in Warp-Leap, the WarpGrad objective samples from parameter trajectories and has therefore linear computational complexity in the number of adaptation steps, currently an unresolved limitation of gradient-based meta-learning. Our offline algorithm (Algorithm 2) hints at exciting possibilities for overcoming this limitation. WarpGrad is a model-embedded meta-learned optimiser that allows for a number of implementation strategies. Indeed, there is a number of ways warp-layers can be embedded in an architecture of choice. To embed warp-layers given a task-learner architecture, we may either insert new warp-layers in the given architecture or designate some layers as warp-layers and some as task layers. We found that WarpGrad can both be used in a high-capacity mode, where task-learners are relatively weak to avoid overfitting, as well as in a low-capacity mode where task-learners are powerful and warp-layers are relatively weak. The best approach depends on the problem at hand. We highlight three approaches to designing WarpGrad optimisers, starting from a given architecture: (a) Model partitioning. Given a desired architecture, designate some operations as task-adaptable and the rest as warp-layers. Task layers do not have to interleave exactly with warp-layers as gradient warping arises both through the forward pass and through backpropagation. This was how we approached the tieredImageNet and miniImageNet experiments. (b) Model augmentation. Given a model, designate all layers as task-adaptable and interleave warplayers. Warp-layers can be relatively weak as backpropagation through non-linear activations ensures expressive gradient warping. This was our approach to the Omniglot experiment; our main architecture interleaves linear warp-layers in a standard architecture. (c) Information compression. Given a model, designate all layers as warp and interleave weak task layers. In this scenario, task-learners are prone to overfitting. Pushing capacity into the warp allows it to encode general information the task-learner can draw on during task adaptation. This approach is similar to approaches in transfer and meta-learning that restrict the number of free parameters during task training (; ; Figure 6 illustrates this process. In this section, we provide the variants of WarpGrad training algorithms used in this paper. Algorithm 1 describes a simple online algorithm, which accumulates meta-gradients online during task adaptation. This algorithm has constant memory and scales linearly in the length of task trajectories. In Algorithm 2, we describe an offline meta-training algorithm. This algorithm is similar to Algorithm 1 in many respects, but differs in that we do not compute meta-gradients online during task adaptation. Instead, we accumulate them into a replay buffer of sampled task parameterisations. This buffer is a Monte-Carlo sample of the expectation in the meta objective (Eq. 13) that can be thought of as a dataset in its own right. Hence, we can apply standard mini-batching with respect to the buffer and perform mini-batch gradient descent on warp parameters. This allows us to update warp parameters several times for a given sample of task parameter trajectories, which can greatly improve data efficiency. In our Omniglot experiment, we found offline meta-training to converge faster: in fact, a mini-batch size of 1 (i.e. η = 1 in Algorithm 2 converges rapidly without any instability. Finally, in Algorithm 3, we present a continual meta-training process where meta-training occurs throughout a stream of learning experiences. Here, C represents a multi-task objective, such as the average task loss, task . Meta-learning arises by collecting experiences continuously (across different tasks) and using these to accumulate the meta-gradient online. Warp parameters are updated intermittently with the accumulated meta-gradient. We use this algorithm in our maze navigation experiment, where task adaptation is internalised within the RNN tasklearner. In this section, we detail WarpGrad methods used in our experiments. Warp-MAML We use this model for few-shot learning (Section 4.1). We use the full warpobjective in Eq. 11 together with the MAML objective (Eq. 1), where C MAML = L MAML under the constraint P = I. In our experiments, we trained Warp-MAML using the online training algorithm (Algorithm 1). Warp-Leap We use this model for multi-shot meta-learning. It is defined by applying Leap to θ 0 (Eq. 16), where the Leap objective is defined by minimising the expected cumulative chordal distance, Note that the Leap meta-gradient makes a first-order approximation to avoid backpropagating through the adaptation process. It is given by where In our experiments, we train Warp-Leap using Algorithm 1 in the multi-shot tieredImageNet experiment and Algorithm 2 in the Omniglot experiment. We perform an ablation study for training algorithms, comparing exact (Eq. 11) versus approximate (Eq. 12) meta-objectives, and several implementations of the warp-layers on Omniglot in Appendix F. Warp-RNN For our Reinforcement Learning experiment, we define a WarpGrad optimiser by meta-learning an LSTM that modulates the weights of the task-learner (see Appendix I for details). For this algorithm, we face a continuous stream of experiences (episodes) that we meta-learn over using our continual meta-training algorithm (Algorithm 3). In our experiment, both L τ task and L τ meta are the advantage actor-critic objective ; C is computed on one batch of 30 episodes, whereas L is accumulated over η = 30 such batches, for a total of 900 episodes. As each episode involves 300 steps in the environment, we cannot apply the exact meta objective, but use the approximate meta objective (Eq. 12). Specifically, let E τ = {s 0, a 1, r 1, s 1, . . ., s T, a T, r T, s T +1} denote an episode on task τ, where s denotes state, a action, and r instantaneous reward. Denote a mini-batch of randomly sampled task episodes by E = {E τ} τ ∼p(τ) and an ordered set of k consecutive mini-batches by Sample mini-batch of task B from p(τ) g φ, g θ0 ← 0 for all τ ∈ B do 6: for all k in 0,..., K τ −1 do 8: g φ ← g φ + ∇L(φ; θ Sample mini-batch of task B from p(τ) for all τ ∈ B do 6: g θ ← g θ + ∇C(θ; φ) Sample mini-batch of task B from p(τ) T ← {τ : for all τ ∈ B do 6: for all k in 0, . . ., K τ −1 do 8: T 16: φ ← φ − βg φ 20: i, g φ, g θ0 ← 0 end if end while 24: end while and C multi (θ;, where n and n are normalising constants. The Warp-RNN objective is defined by WarpGrad for Continual Learning For this experiment, we focus on meta-learning warpparameters. Hence, the initialisation for each task sequence is a fixed random initialisation, (i.e. λC(θ 0) = 0). For the warp meta-objective, we take expectations over N task sequences, where each task sequence is a sequence of T = 5 sub-tasks that the task-learner observes one at a time; thus while the task loss is defined over the current sub-task, the meta-loss averages of the current and all prior sub-tasks, for each sub-task in the sequence. See Appendix J for detailed definitions. Importantly, because WarpGrad defines task adaptation abstractly by a probability distribution, we can readily implement a continual learning objective by modifying the joint task parameter distribution p(τ, θ) that we use in the meta-objective (Eq. 11). A task defines a sequence of sub-tasks over which we generate parameter trajectories θ τ. Thus, the only difference from multi-task meta-learning is that parameter trajectories are not generated under a fixed task, but arise as a function of the continual learning algorithm used for adaptation. We define the conditional distribution p(θ | τ) as before by sampling sub-task parameters θ τt from a mini-batch of such task trajectories, keeping track of which sub-task t it belongs to and which sub-tasks came before it in the given task sequence τ. The meta-objective is constructed, for any sub-task parameterisation θ τt, as, where D j is data from sub-task j (Appendix J). The meta-objective is an expectation over task parameterisations, To build intuition for what it means to warp space, we construct a simple 2-D problem over loss surfaces. A learner is faced with the task of minimising an objective function of the form where each task f τ is defined by scale and rotation functions g τ that are randomly sampled from a predefined distribution. Specifically, each task is defined by the objective function The task is to minimise the given objective from a randomly sampled initialisation, x {i=1,2} ∼ U (−3, 3). During meta-training, we train on a task for 100 steps using a learning rate of 0.1. Each task has a unique loss-surface that the learner traverses from the randomly sampled initialisation. While each loss-surface is unique, they share an underlying structure. Thus, by metalearning a warp over trajectories on randomly sampled loss surfaces, we expect WarpGrad to learn a warp that is close to invariant to spurious descent directions. In particular, WarpGrad should produce a smooth warped space that is quasi-convex for any given task to ensure that the task-learner finds a minimum as fast as possible regardless of initialisation. To visualise the geometry, we use an explicit warp Ω defined by a 2-layer feed-forward network with a hidden-state size of 30 and tanh non-linearities. We train warp parameters for 100 meta-training steps; Figure 7: Example trajectories on three task loss surfaces. We start Gradient Descent (black) and WarpGrad (magenta) from the same initialisation; while SGD struggles with the curvature, the WarpGrad optimiser has learned a warp such that gradient descent in the representation space (top) leads to rapid convergence in model parameter space (bottom). in each meta-step we sample a new task surface and a mini-batch of 10 random initialisations that we train separately. We train to convergence and accumulate the warp meta-gradient online (Algorithm 1). We evaluate against gradient descent on randomly sampled loss surfaces (Figure 7). Both optimisers start from the same initialisation, chosen such that standard gradient descent struggles; we expect the WarpGrad optimisers to learn a geometry that is robust to the initialisation (top row). This is indeed what we find; the geometry learned by WarpGrad smoothly warps the native loss surface into a well-behaved space where gradient descent converges to a local minimum. We follow the protocol of , including choice of hyper-parameters. In this setup, each of the 50 alphabets that comprise the dataset constitutes a distinct task. Each task is treated as a 20-way classification problem. Four alphabets have fewer than 20 characters in the alphabet and are discarded, leaving us with 46 alphabets in total. 10 alphabets are held-out for final meta-testing; which alphabets are held out depend on the seed to account for variations across alphabets; we train and evaluate all baselines on 10 seeds. For each character in an alphabet, there are 20 raw samples. Of these, 5 are held out for final evaluation on the task while the remainder is used to construct a training set. Raw samples are pre-processed by random affine transformations in the form of (a) scaling between [0.8, 1.2], (b) rotation, and (c) cropping height and width by a factor of [−0.2, 0.2] in each dimension. This ensures tasks are too hard for few-shot learning. During task adaptation, mini-batches are sampled at random without ensuring class-balance (in contrast to few-shot classification protocols ). Note that benchmarks under this protocol are not compatible with few-shot learning benchmarks. We use the same convolutional neural network architecture and hyper-parameters as in. This learner stacks a convolutional block comprised of a 3 × 3 convolution with 64 filters, followed by 2 × 2 max-pooling, batch-normalisation, and ReLU activation, four times. All images are down-sampled to 28 × 28, ing in a 1 × 1 × 64 feature map that is passed on to a final linear layer. We create a Warp Leap meta-learner that inserts warp-layers between each convolutional block, where each h is defined as above. In our main experiment, each ω i is simply a 3 × 3 convolutional layer with zero padding; in Appendix F we consider both simpler and more sophisticated versions. We find that relatively simple warp-layers do quite well. However, adding capacity does improve generalisation performance. We meta-learn the initialisation of task parameters using the Leap objective (Eq. 16), detailed in Appendix C. Both L τ meta and L τ task are defined as the negative log-likelihood loss; importantly, we evaluate them on different batches of task data to ensure warp-layers encourage generalisation. We found no additional benefit in this experiment from using held-out data to evaluate L τ meta. We use the offline meta-training algorithm (Appendix B, Algorithm 2); in particular, during meta-training, we sample mini-batches Top: test accuracies on held-out tasks after meta-training on a varying number of tasks. Bottom: AUC under accuracy curve on held-out tasks after meta-training on a varying number of tasks. Shading represents standard deviation across 10 independent runs. We compare between Warp-Leap, Leap, and Reptile, multi-headed finetuning, as well as SGD and KFAC which used random initialisation but with 10x larger batch size and learning rate. of 20 tasks and train task-learners for 100 steps to collect 2000 task parameterisations into a replay buffer. Task-learners share a common initialisation and warp parameters that are held fixed during task adaptation. Once collected, we iterate over the buffer by randomly sampling mini-batches of task parameterisations without replacement. Unless otherwise noted, we used a batch size of η = 1. For each mini-batch, we update φ by applying gradient descent under the canonical meta-objective (Eq. 11), where we evaluate L τ meta on a randomly sampled mini-batch of data from the corresponding task. Consequently, for each meta-batch, we take (up to) 2000 meta-gradient steps on warp parameters φ. We find that this form of mini-batching causes the meta-training loop to converge much faster and induces no discernible instability. We compare Warp-Leap against no meta-learning with standard gradient descent (SGD) or KFAC . We also benchmark against baselines provided in; Leap, Reptile , MAML, and multi-headed fine-tuning. All learners benefit substantially from large batch sizes as this enables higher learning rates. To render no-pretraining a competitive option within a fair computational budget, we allow SGD and KFAC to use 10x larger batch sizes, enabling 10x larger learning rates. This renders them computationally costly, taking 2x and 4x longer to train on a given task during meta-test time than Warp-Leap, respectively. No. Meta-training tasks 1 49.5 ± 7.8 37.6 ± 4.8 40.4 ± 4.0 53.8 ± 5.0 40.0 ± 2.6 56.0 51.0 3 68.8 ± 2.8 53.4 ± 3.1 53.1 ± 4.2 64.6 ± 3.3 48.6 ± 2.5 56.0 51.0 5 75.0 ± 3.6 59.5 ± 3.7 58.3 ± 3.3 67.7 ± 2.8 51.6 ± 3.8 56.0 51.0 10 81.2 ± 2.4 67.4 ± 2.4 65.0 ± 2.1 71.3 ± 2.0 54.1 ± 2.8 56.0 51.0 15 82.7 ± 3.3 70.0 ± 2.4 66.6 ± 2.9 73.5 ± 2.4 54.8 ± 3.4 56.0 51.0 20 82.0 ± 2.6 73.3 ± 2.3 69.4 ± 3.4 75.4 ± 3.2 56.6 ± 2.0 56.0 51.0 25 83.8 ± 1.9 74.8 ± 2.7 70.8 ± 1.9 76.4 ± 2.2 56.7 ± 2.1 56.0 51.0 F ABLATION STUDY: WARP LAYERS, META-OBJECTIVE, AND META-TRAINING WarpGrad provides a principled approach for model-informed meta-learning and offers several degrees of freedom. To evaluate these design choices, we conduct an ablation study on Warp-Leap where we vary the design of warp-layers as well as meta-training approach. For the ablation study, we fixed the number of pretraining tasks to 25 and report final test accuracy over 4 independent runs. All ablations use the same hyper-parameters, except for online meta-training which uses a learning rate of 0.001. First, we vary the meta-training protocol by (a) using the approximate objective (Eq. 12), (b) using online meta-training (Algorithm 1), and (c) whether meta-learning the learning rate used for task adaptation is beneficial in this experiment. We meta-learn a single scalar learning rate (as warp parameters can learn layer-wise scaling). Meta-gradients for the learning rate are clipped at 0.001 and we use a learning rate of 0.001. Note that when using offline meta-training, we store both task parameterisations and the momentum buffer in that phase and use them in the update rule when computing the canonical objective (Eq. 11). Further, we vary the architecture used for warp-layers. We study simpler versions that use channelwise scaling and more complex versions that use non-linearities and residual connections. We also evaluate a version where each warp-layer has two stacked convolutions, where the first warp convolution outputs 128 filters and the second warp convolution outputs 64 filters. Finally, in the two-layer warp-architecture, we evaluate a version that inserts a FiLM layer between the two warp convolutions. These are adapted during task training from a 0 initialisation; they amount to task embeddings that condition gradient warping on task statistics. Full are reported in Table 3. G ABLATION STUDY: WARPGRAD AND NATURAL GRADIENT DESCENT and Natural Neural Nets . First, we isolate the effect of warping task loss surfaces by fixing a random initialisation and only meta-learning warp parameters. That is, in this experiment, we set λC(θ 0) = 0. We compare against two baselines, stochastic gradient descent (SGD) and KFAC, both trained from a random initialisation. We use task mini-batch sizes of 200 and task learning rates of 1.0, otherwise we use the same hyper-parameters as in the main experiment. For WarpGrad, we meta-train with these hyper-parameters as well. We evaluate two WarpGrad architectures, in one, we use linear warp-layers, which gives a block-diagonal preconditioning, as in KFAC. In the other, we use our most expressive warp configuration from the ablation experiment in appendix F, where warp-layers are two-layer convolutional block with residual connections, batch normalisation, and ReLU activation. We find that warped geometries facilitate task adaptation on held-out tasks to a greater degree than either SGD or KFAC by a significant margin (table 4). We further find that going beyond block-diagonal preconditioning yields a significant improvement in performance. Second, we explore whether the geometry that we meta-learn under in the full Warp-Leap algorithm is approximately Fisher. In this experiment we use the main Warp-Leap architecture. We use a meta-learner trained on 25 tasks and that we evaluate on 10 held-out tasks. Because warp-layers are linear in this configuration, if the learned geometry is approximately Fisher, post-warp activations should be zero-centred and the layer-wise covariance matrix should satisfy where I is the identify matrix . If true, Warp-Leap would learn a block-diagonal approximation to the Inverse Fisher Matrix, as Natural Neural Nets. To test this, during task adaptation on held-out tasks, we compute the mean activation in each convolutional layer pre-and post-warping. We also compute the Shatten-1 norm of the difference between layer activation covariance and the identity matrix pre-and post-warping, as described above. We average statistics over task and adaptation step (we found no significant variation in these dimensions). Figure 9 summarise our . We find that, in general, WarpGrad-Leap has zero-centered post-warp activations. That pre-warp activations are positive is an artefact of the ReLU activation function. However, we find that the correlation structure is significantly different from what we would expect if Warp-Leap were to represent the Fisher matrix; post-warp covariances are significantly dissimilar from the identity matrix and varies across layers. These indicate that WarpGrad methods behave distinctly different from Natural Gradient Descent methods. One possibility is that WarpGrad methods do approximate the Fisher Information Matrix, but with higher accuracy than other methods. A more likely explanation is that WarpGrad methods encode a different geometry since they can learn to leverage global information beyond the task at hand, which enables them to express geometries that standard Natural Gradient Descent cannot. H miniIMAGENET AND tieredIMAGENET miniImageNet This dataset is a subset of 100 classes sampled randomly from the 1000 base classes in the ILSVRC-12 training set, with 600 images for each class. Following , classes are split into non-overlapping meta-training, meta-validation and meta-tests sets with 64, 16, and 20 classes in each respectively. tieredImageNet As described in , this dataset is a subset of ILSVRC-12 that stratifies 608 classes into 34 higher-level categories in the ImageNet human-curated hierarchy . In order to increase the separation between meta-train and meta-evaluation splits, 20 of these categories are used for meta-training, while 6 and 8 are used for meta-validation and metatesting respectively. Slicing the class hierarchy closer to the root creates more similarity within each split, and correspondingly more diversity between splits, rendering the meta-learning problem more challenging. High-level categories are further divided into 351 classes used for meta-training, 97 for meta-validation and 160 for meta-testing, for a total of 608 base categories. All the training images in ILSVRC-12 for these base classes are used to generate problem instances for tieredImageNet, of which there are a minimum of 732 and a maximum of 1300 images per class. For all experiments, N -way K-shot classification problem instances were sampled following the standard image classification methodology for meta-learning proposed in. A subset of N classes was sampled at random from the corresponding split. For each class, K arbitrary images were chosen without replacement to form the training dataset of that problem instance. As usual, a disjoint set of L images per class were selected for the validation set. Few-shot classification In these experiments we used the established experimental protocol for evaluation in meta-validation and meta-testing: 600 task instances were selected, all using N = 5, K = 1 or K = 5, as specified, and L = 15. During meta-training we used N = 5, K = 5 or K = 15 respectively, and L = 15. Task-learners used 4 convolutional blocks defined by with 128 filters (or less, chosen by hyperparameter tuning), 3 × 3 kernels and strides set to 1, followed by batch normalisation with learned scales and offsets, a ReLU non-linearity and 2 × 2 max-pooling. The output of the convolutional stack (5 × 5 × 128) was flattened and mapped, using a linear layer, to the 5 output units. The last 3 convolutional layers were followed by warp-layers with 128 filters each. Only the final 3 task-layer parameters and their corresponding scale and offset batch-norm parameters were adapted during task-training, with the corresponding warp-layers and the initial convolutional layer kept fixed and meta-learned using the WarpGrad objective. Note that, with the exception of CAVIA, other baselines do worse with 128 filters as they overfit; MAML and T-Nets achieve 46% and 49 % 5-way-1-shot test accuracy with 128 filters, compared to their best reported (48.7% and 51.7%, respectively). Hyper-parameters were tuned independently for each condition using random grid search for highest test accuracy on meta-validation left-out tasks. Grid sizes were 50 for all experiments. We choose the optimal hyper-parameters (using early stopping at the meta-level) in terms of meta-validation test set accuracy for each condition and we report test accuracy on the meta-test set of tasks. 60000 metatraining steps were performed using meta-gradients over a single randomly selected task instances and their entire trajectories of 5 adaptation steps. Task-specific adaptation was done using stochastic gradient descent without momentum. We use Adam for meta-updates. Multi-shot classification For these experiments we used N = 10, K = 640 and L = 50. Tasklearners are defined similarly, but stacking 6 convolutional blocks defined by 3 × 3 kernels and strides set to 1, followed by batch normalisation with learned scales and offsets, a ReLU non-linearity and 2 × 2 max-pooling (first 5 layers). The sizes of convolutional layers were chosen by hyper-parameter tuning to {64, 64, 160, 160, 256, 256}. The output of the convolutional stack (2 × 2 × 256) was flattened and mapped, using a linear layer, to the 10 output units. Hyper-parameters were tuned independently for each algorithm, version and baseline using random grid search for highest test accuracy on meta-validation left-out tasks. Grid sizes were 200 for all multi-shot experiments. We choose the optimal hyper-parameters in terms of mean meta-validation test set accuracy AUC (using early stopping at the meta-level) for each condition and we report test accuracy on the meta-test set of tasks. 2000 meta-training steps were performed using averaged meta-gradients over 5 random task instances and their entire trajectories of 100 adaptation steps with batch size 64, or inner-loops. Task-specific adaptation was done using stochastic gradient descent with momentum (0.9). Meta-gradients were passed to Adam in the outer loop. We test WarpGrad against Leap, Reptile, and training from scratch with large batches and tuned momentum. We tune all meta-learners for optimal performance on the validation set. WarpGrad outperforms all baselines both in terms of rate of convergence and final test performance (Figure 10). To illustrate both how WarpGrad may be used with Recurrent Neural Networks in an online metalearning setting, as well as in a Reinforcement Learning environment, we evaluate it in a maze navigation task proposed by. The environment is a fixed maze and a task is defined by randomly choosing a goal location in the maze. During a task episode of length 200, the goal location is fixed but the agent gets teleported once it finds it. Thus, during an episode the agent must first locate the goal, then return to it as many times as possible, each time being randomly teleported to a new starting location. We use an identical setup as , except our grid is of size 11 × 11 as opposed to 9 × 9. We compare our Warp-RNN to a Learning to Reinforcement Learn and Hebbian meta-learners (; 2019). The task-learner in all cases is an advantage actor-critic , where the actor and critic share an underlying basic RNN, whose hidden state is projected into a policy and value function by two separate linear layers. The RNN has a hidden state size of 100 and tanh non-linearities. Following , for all benchmarks, we train the task-learner using Adam with a learning rate of 1e − 3 for 200 000 steps using batches of 30 episodes, each of length 200. Metalearning arises in this setting as each episode encodes a different task, as the goal location moves, and by learning across episodes the RNN is encoding meta-information in its parameters that it can leverage during task adaptation (via its hidden state ). for further details. We design a Warp-RNN by introducing a warp-layer in the form of an LSTM that is frozen for most of the training process. , we use this meta-LSTM to modulate the task RNN. Given an episode with input vector x t, the task RNN is defined by where W, V, b are task-adaptable parameters; each U (i) j,t is a diagonal warp matrix produced by projecting from the hidden state of the meta-LSTM, U Figure 6, Appendix A). Because the meta-LSTM is frozen for most of the training process, task adaptable parameters correspond to those of the baseline RNN. To control for the capacity of the meta-LSTM, we also train a HyperRNN where the LSTM is updated with every task adaptation; we find this model does worse than the WarpGrad-RNN. We also compare the non-linear preconditioning that we obtain in our Warp-RNN to linear forms of preconditioning defined in prior works. We implement a T-Nets-RNN meta-learner, defined by embedding linear projections T h, T x and T b that are meta-learned in the task RNN, h t = tanh(T h V h t + T x W x t + b). Note that we cannot backpropagate to these meta-parameters as per the T-Nets (MAML) framework. Instead, we train T h, T x, T b with the meta-objective and meta-training algorithm we use for the Warp-RNN. The T-Nets-RNN does worse than the baseline RNN and generally fails to learn. We meta-train the Warp-RNN using the continual meta-training algorithm (Algorithm 3, see Appendix B for details), which accumulates meta-gradients continuously during training. Because task training is a continuous stream of batches of episodes, we accumulating the meta-gradient using the approximate objective (Eq. 12, where L τ task and L τ meta are both the same advantage actor-critic objective) and update warp-parameters on every 30th task parameter update. We detail the meta-objective in Appendix C (see Eq. 18). Our implementation of a Warp-RNN can be seen as meta-learning "slow" weights to facilitate learning of "fast" weights . Implementing Warp-RNN requires four lines of code on top of the standard training script. The task-learner is the same in all experiments with the same number of learnable parameters and hidden state size. Compared to all baselines, we find that the Warp-RNN converges faster and achieves a higher cumulative reward (Figure 4 and Figure 11). J META-LEARNING FOR CONTINUAL LEARNING Online SGD and related optimisation methods tend to adapt neural network models to the data distribution encountered last during training, usually leading to what has been termed "catastrophic meta-learned). Each non-linearity is followed by a residual warping block consisting of 2-layer feedforward networks with 100 hidden units and tanh non-linearities, with meta-learned parameters φ which are fixed during the task adaptation process. Continual learning as task adaptation The task target function g τ is partition into 5 sets of subtask. The task-learner sees one partition at a time and is given n = 20 gradient steps to adapt, for a total of K = 100 steps of online gradient descent updates for the full task sequence; recall that every such sequence starts from a fixed random initialisation θ 0. The adaptation is completely online since at step k = 1,..., K we sample a new mini-batch D k task of 5 samples from a single sub-task (sub-interval). The data distribution changes after each n = 20 steps with inputs x coming from the next sub-interval and targets form the same function g τ (x). During meta-training we always present tasks in the same order, presenting intervals from left to right. The online (sub-)task loss is defined on the current mini-batch D k task at step k: 2. Adaptation to each sub-task uses sub-task data only to form task parameter updates θ We used a constant learning rate α = 0.001. Warp-parameters φ are fixed across the full task sequence during adaptation and are meta-learned across random samples of task sequences, which we describe next. Meta-learning an optimiser for continual learning To investigate the ability of WarpGrad to learn an optimiser for continual learning that mitigates catastrophic forgetting, we fix a random initialisation prior to meta-training that is not meta-learned; every task-learner is initialised with these parameters. To meta-learn an optimiser for continual learning, we need a meta-objective that encourages such behaviour. Here, we take a first step towards a framework for meta-learned continual learning. We define the meta-objective L τ meta as an incremental multitask objective that, for each sub-task!τ t in a given task sequence τ, averages the validation sub-task losses (Eq. 22) for the current and every preceding loss in the task sequence. The task meta-objective is defined by summing over all sub-tasks in the task sequence. For some sub-task parameterisation θ τt, we have As before, the full meta-objective is an expectation over the joint task parameter distribution (Eq. 11); for further details on the meta-objective, see Appendix C, Eq. 19. This meta-objective gives equal weight to all the tasks in the sequence by averaging the regression step loss over all sub-tasks where a prior sub-task should be learned or remembered. For example, losses from the first sub-task, defined using the interval [−5, −3], will appear nT times in the meta-objective. Conversely, the last sub-task in a sequence, defined on the interval, is learned only in the last n = 20 steps of task adaptation, and hence appers n times in the meta-objective. Normalising on number of appearances corrects for this bias. We trained warp-parameters using Adam and a meta-learning rate of 0.001, sampling 5 random tasks to form a meta-batch and repeating the process for 20 000 steps of meta-training. Results Figure 12 shows a breakdown of the validation loss across the 5 sequentially learned tasks over the 100 steps of online learning during task adaptation. Results are averaged over 100 random regression problem instances. The meta-learned WarpGrad optimiser reduces the loss of the task currently being learned in each interval while also largely retaining performance on previous tasks. There is an immediate relatively minor loss of performance, after which performance on previous tasks is retained. We hypothesise that this is because the meta-objectives averages over the full learning curve, as opposed to only the performance once a task has been adapted to. As such, the WarpGrad optimiser may allow for some degree of performance loss. Intriguingly, in all cases, after an initial spike in previous sub-task losses when switching to a new task, the spike starts to revert back some way towards optimal performance, suggesting that the WarpGrad optimiser facilitates positive backward transfer, without this being explicitly enforced in the meta-objective. Deriving a principled meta-objective for continual learning is an exciting area for future research.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We propose a novel framework for meta-learning a gradient-based update rule that scales to beyond few-shot learning and is applicable to any form of learning, including continual learning.

Represent the short summary for finding the corresponding scientific article (long)

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: represent the article for finding a single-sentence tldr.

Question: In recent years, deep neural network approaches have been widely adopted for machine learning tasks, including classification. However, they were shown to be vulnerable to adversarial perturbations: carefully crafted small perturbations can cause misclassification of legitimate images. We propose Defense-GAN, a new framework leveraging the expressive capability of generative models to defend deep neural networks against such attacks. Defense-GAN is trained to model the distribution of unperturbed images. At inference time, it finds a close output to a given image which does not contain the adversarial changes. This output is then fed to the classifier. Our proposed method can be used with any classification model and does not modify the classifier structure or training procedure. It can also be used as a defense against any attack as it does not assume knowledge of the process for generating the adversarial examples. We empirically show that Defense-GAN is consistently effective against different attack methods and improves on existing defense strategies. Despite their outstanding performance on several machine learning tasks, deep neural networks have been shown to be susceptible to adversarial attacks BID20 BID4. These attacks come in the form of adversarial examples: carefully crafted perturbations added to a legitimate input sample. In the context of classification, these perturbations cause the legitimate sample to be misclassified at inference time BID20 BID4 BID16 BID11. Such perturbations are often small in magnitude and do not affect human recognition but can drastically change the output of the classifier. Recent literature has considered two types of threat models: black-box and white-box attacks. Under the black-box attack model, the attacker does not have access to the classification model parameters; whereas in the white-box attack model, the attacker has complete access to the model architecture and parameters, including potential defense mechanisms BID21 BID2.Various defenses have been proposed to mitigate the effect of adversarial attacks. These defenses can be grouped under three different approaches: modifying the training data to make the classifier more robust against attacks, e.g., adversarial training which augments the training data of the classifier with adversarial examples BID20 BID4, modifying the training procedure of the classifier to reduce the magnitude of gradients, e.g., defensive distillation BID18, and attempting to remove the adversarial noise from the input samples BID6 BID13. All of these approaches have limitations in the sense that they are effective against either white-box attacks or black-box attacks, but not both BID21 BID13. Furthermore, some of these defenses are devised with specific attack models in mind and are not effective against new attacks. In this paper, we propose a novel defense mechanism which is effective against both white-box and black-box attacks. We propose to leverage the representative power of Generative Adversarial Networks (GAN) to diminish the effect of the adversarial perturbation, by "projecting" input images onto the range of the GAN's generator prior to feeding them to the classifier. In the GAN framework, two models are trained simultaneously in an adversarial setting: a generative model that emulates the data distribution, and a discriminative model that predicts whether a certain input came from real data or was artificially created. The generative model learns a mapping G from a low-dimensional vector z ∈ R k to the high-dimensional input sample space R n. During training of the GAN, G is encouraged to generate samples which resemble the training data. It is, therefore, expected that legitimate samples will be close to some point in the range of G, whereas adversarial samples will be further away from the range of G. Furthermore, "projecting" the adversarial examples onto the range of the generator G can have the desirable effect of reducing the adversarial perturbation. The projected output, computed using Gradient Descent (GD), is fed into the classifier instead of the original (potentially adversarially modified) image. We empirically demonstrate that this is an effective defense against both black-box and white-box attacks on two benchmark image datasets. The rest of the paper is organized as follows. We introduce the necessary regarding known attack models, defense mechanisms, and GANs in Section 2. Our defense mechanism, which we call Defense-GAN, is formally motivated and introduced in Section 3. Finally, experimental , under different threat models, as well as comparisons to other defenses are presented in Section 4. In this work, we propose to use GANs for the purpose of defending against adversarial attacks in classification problems. Before detailing our approach in the next section, we explain related work in three parts. First, we discuss different attack models employed in the literature. We, then, go over related defense mechanisms against these attacks and discuss their strengths and shortcomings. Lastly, we explain necessary information regarding GANs. Various attack models and algorithms have been used to target classifiers. All attack models we consider aim to find a perturbation δ to be added to a (legitimate) input x ∈ R n, ing in the adversarial examplex = x + δ. The ∞ -norm of the perturbation is denoted by BID4 and is chosen to be small enough so as to remain undetectable. We consider two threat levels: black-and white-box attacks. White-box models assume that the attacker has complete knowledge of all the classifier parameters, i.e., network architecture and weights, as well as the details of any defense mechanism. Given an input image x and its associated ground-truth label y, the attacker thus has access to the loss function J(x, y) used to train the network, and uses it to compute the adversarial perturbation δ. Attacks can be targeted, in that they attempt to cause the perturbed image to be misclassified to a specific target class, or untargeted when no target class is specified. In this work, we focus on untargeted white-box attacks computed using the Fast Gradient Sign Method (FGSM) BID4, the Randomized Fast Gradient Sign Method (RAND+FGSM) BID21, and the Carlini-Wagner (CW) attack BID2. Although other attack models exist, such as the Iterative FGSM, the Jacobian-based Saliency Map Attack (JSMA) BID16, and Deepfool , we focus on these three models as they cover a good breadth of attack algorthims. FGSM is a very simple and fast attack algorithm which makes it extremely amenable to real-time attack deployment. On the other hand, RAND+FGSM, an equally simple attack, increases the power of FGSM for white-box attacks BID21, and finally, the CW attack is one of the most powerful white-box attacks to-date BID2.Fast Gradient Sign Method (FGSM) Given an image x and its corresponding true label y, the FGSM attack sets the perturbation δ to: DISPLAYFORM0FGSM BID4 was designed to be extremely fast rather than optimal. It simply uses the sign of the gradient at every pixel to determine the direction with which to change the corresponding pixel value. Randomized Fast Gradient Sign Method (RAND+FGSM) The RAND+FGSM BID21 attack is a simple yet effective method to increase the power of FGSM against models which were adversarially trained. The idea is to first apply a small random perturbation before using FGSM. More explicitly, for α <, random noise is first added to the legitimate image x: DISPLAYFORM1 Then, the FGSM attack is computed on x, ing iñ DISPLAYFORM2 The Carlini-Wagner (CW) attack The CW attack is an effective optimization-based attack model BID2. In many cases, it can reduce the classifier accuracy to almost 0% BID2 BID13. The perturbation δ is found by solving an optimization problem of the form: DISPLAYFORM3 where f is an objective function that drives the example x to be misclassified, and c > 0 is a suitably chosen constant. The 2, 0, and ∞ norms are considered. We refer the reader to BID2 for details regarding the approach to solving and setting the constant c. For black-box attacks we consider untargeted FGSM attacks computed on a substitute model. As previously mentioned, black-box adversaries have no access to the classifier or defense parameters. It is further assumed that they do not have access to a large training dataset but can query the targeted DNN as a black-box, i.e., access labels produced by the classifier for specific query images. The adversary trains a model, called substitute, which has a (potentially) different architecture than the targeted classifier, using a very small dataset augmented by synthetic images labeled by querying the classifier. Adversarial examples are then found by applying any attack method on the substitute network. It was found that such examples designed to fool the substitute often end up being misclassified by the targeted classifier BID20. In other words, black-box attacks are easily transferrable from one model to the other. Various defense mechanisms have been employed to combat the threat from adversarial attacks. In what follows, we describe one representative defense strategy from each of the three general groups of defenses. A popular approach to defend against adversarial noise is to augment the training dataset with adversarial examples BID20 BID4 BID14. Adversarial examples are generated using one or more chosen attack models and added to the training set. This often in increased robustness when the attack model used to generate the augmented training set is the same as that used by the attacker. However, adversarial training does not perform as well when a different attack strategy is used by the attacker. Additionally, it tends to make the model more robust to white-box attacks than to black-box attacks due to gradient masking BID17 BID21. Defensive distillation BID18 trains the classifier in two rounds using a variant of the distillation BID7 method. This has the desirable effect of learning a smoother network and reducing the amplitude of gradients around input points, making it difficult for attackers to generate adversarial examples BID18. It was, however, shown that, while defensive distillation is effective against white-box attacks, it fails to adequately protect against black-box attacks transferred from other networks BID2. Recently, BID13 introduced MagNet as an effective defense strategy. It trains a reformer network (which is an auto-encoder or a collection of auto-encoders) to move adversarial examples closer to the manifold of legitimate, or natural, examples. When using a collection of auto-encoders, one reformer network is chosen at random at test time, thus strengthening the defense. It was shown to be an effective defense against gray-box attacks where the attacker knows everything about the network and defense, except the parameters. MagNet is the closest defense to our approach, as it attempts to reform an adversarial sample using a learnt auto-encoder. The main differences between MagNet and our approach are: we use GANs instead of auto-encoders, and, most importantly, we use GD minimization to find latent codes as opposed to a feedforward encoder network. This makes Defense-GAN more robust, especially against white-box attacks. GANs, originally introduced by, consist of two neural networks, G and D. G: R k → R n maps a low-dimensional latent space to the high dimensional sample space of x. D is a binary neural network classifier. In the training phase, G and D are typically learned in an adversarial fashion using actual input data samples x and random vectors z. An isotropic Gaussian prior is usually assumed on z. While G learns to generate outputs G(z) that have a distribution similar to that of x, D learns to discriminate between "real" samples x and "fake" samples G(z). D and G are trained in an alternating fashion to minimize the following min-max loss: DISPLAYFORM0 It was shown that the optimal GAN is obtained when the ing generator distribution p g = p data.However, GANs turned out to be difficult to train in practice BID5, and alternative formulations have been proposed. introduced Wasserstein GANs (WGANs) which are a variant of GANs that use the Wasserstein distance, ing in a loss function with more desirable properties: DISPLAYFORM1 In this work, we use WGANs as our generative model due to the stability of their training methods, especially using the approach in BID5.3 PROPOSED DEFENSE-GAN Figure 1: Overview of the Defense-GAN algorithm. As mentioned in Section 2.3, the GAN min-max loss in admits a global optimum when p g = p data. It can be similarly shown that WGAN admits an optimum to its own minmax loss in, when the set {x | p g (x) = p data (x)} has zero Lebesgue-measure. Formally, Lemma 1 A generator distribution p g is a global optimum for the WGAN min-max game defined in, if and only if p g (x) = p data (x) for all x ∈ R n, potentially except on a set of zero Lebesguemeasure. A sketch of the proof can be found in Appendix A.Additionally, it was shown that, if G and D have enough capacity to represent the data, and if the training algorithm is such that p g converges to p data, then DISPLAYFORM0 where G t is the generator of a GAN or WGAN 1 after t steps of its training algorithm BID8. This serves to show that, under ideal conditions, the addition of the GAN reconstruction loss minimization step should not affect the performance of the classifier on natural, legitimate samples, as such samples should be almost exactly recovered. Furthermore, we hypothesize that this step will help reduce the adversarial noise which follows a different distribution than that of the GAN training examples. Defense-GAN is a defense strategy to combat both white-box and black-box adversarial attacks against classification networks. At inference time, given a trained GAN generator G and an image x to be classified, z * is first found so as to minimize DISPLAYFORM0 G(z *) is then given as the input to the classifier. The algorithm is illustrated in Figure 1. As FORMULA7 is a highly non-convex minimization problem, we approximate it by doing a fixed number L of GD steps using R different random initializations of z (which we call random restarts), as shown in Figures 1 and 2.The GAN is trained on the available classifier training dataset in an unsupervised manner. The classifier can be trained on the original training images, their reconstructions using the generator G, or a combination of the two. As was discussed in Section 3.1, as long as the GAN is appropriately trained and has enough capacity to represent the data, original clean images and their reconstructions should not defer much. Therefore, these two classifier training strategies should, at least theoretically, not differ in performance. Compared to existing defense mechanisms, our approach is different in the following aspects: 1. Defense-GAN can be used in conjunction with any classifier and does not modify the classifier structure itself. It can be seen as an add-on or pre-processing step prior to classification.2. If the GAN is representative enough, re-training the classifier should not be necessary and any drop in performance due to the addition of Defense-GAN should not be significant.3. Defense-GAN can be used as a defense to any attack: it does not assume an attack model, but simply leverages the generative power of GANs to reconstruct adversarial examples.4. Defense-GAN is highly non-linear and white-box gradient-based attacks will be difficult to perform due to the GD loop. A detailed discussion about this can be found in Appendix B. We assume three different attack threat levels:1. Black-box attacks: the attacker does not have access to the details of the classifier and defense strategy. It therefore trains a substitute network to find adversarial examples.2. White-box attacks: the attacker knows all the details of the classifier and defense strategy. It can compute gradients on the classifier and defense networks in order to find adversarial examples.3. White-box attacks, revisited: in addition to the details of the architectures and parameters of the classifier and defense, the attacker has access to the random seed and random number generator. In the case of Defense-GAN, this means that the attacker knows all the random initializations {z DISPLAYFORM0 We compare our method to adversarial training BID4 and MagNet BID13 under the FGSM, RAND+FGSM, and CW (with 2 norm) white-box attacks, as well as the FGSM black-box attack. Details of all network architectures used in this paper can be found in Appendix C. When the classifier is trained using the reconstructed images (G(z *)), we refer to our method as Defense-GAN-Rec, and we use Defense-GAN-Orig when the original images (x) are used to train the classifier. Our GAN follows the WGAN training procedure in BID5, and details of the generator and discriminator network architectures are given in TAB4. The reformer network (encoder) for the MagNet baseline is provided in TAB5. Our implementation is based on TensorFlow BID0 and builds on open-source software: CleverHans by BID15 and improved WGAN training by BID5. We use machines equipped with NVIDIA GeForce GTX TITAN X GPUs. In our experiments, we use two different image datasets: the MNIST handwritten digits dataset BID10 and the Fashion-MNIST (F-MNIST) clothing articles dataset BID22. Both datasets consist of 60, 000 training images and 10, 000 testing images. We split the training images into a training set of 50, 000 images and hold-out a validation set containing 10, 000 images. For white-box attacks, the testing set is kept the same (10, 000 samples). For black-box attacks, the testing set is divided into a small hold-out set of 150 samples reserved for adversary substitute training, as was done in, and the remaining 9, 850 samples are used for testing the different methods. In this section, we present experimental on FGSM black-box attacks. As previously mentioned, the attacker trains a substitute model, which could differ in architecture from the targeted model, using a limited dataset consisting of 150 legitimate images augmented with synthetic images labeled using the target classifier. The classifier and substitute model architectures used and referred to throughout this section are described in TAB3 in the Appendix. In TAB0, we present our classification accuracy and compare to other defense methods. As can be seen, FGSM black-box attacks were successful at reducing the classifier accuracy by up to 70%. All considered defense mechanisms are relatively successful at diminishing the effect of the attacks. We note that, as expected, the performance of Defense-GAN-Rec and that of Defense-GAN-Orig are very close. In addition, they both perform consistently well across different classifier and substitute model combinations. MagNet also performs in a consistent manner, but achieves lower accuracy than Defense-GAN. Two adversarial training defenses are presented: the first one obtains the adversarial examples assuming the same attack = 0.3, and the second assumes a different = 0.15. With incorrect knowledge of, the performance of adversarial training generally decreases. In addition, the classification performance of this defense method has very large variance across the different architectures. It is worth noting that adversarial training defense is only fit against FGSM attacks, because the adversarially augmented data, even with a different, is generated using the same method as the black-box attack (FGSM). In contrast, Defense-GAN and MagNet are general defense mechanisms which do not assume a specific attack model. The performances of defenses on the F-MNIST dataset, shown in TAB1, are noticeably lower than on MNIST. This is due to the large = 0.3 in the FGSM attack. Please see Appendix D for qualitative examples showing that = 0.3 represents very high noise, which makes F-MNIST images difficult to classify, even by a human. In addition, the Defense-GAN parameters used in this experiment were kept the same for both Tables, in order to study the effect of dataset complexity, and can be further optimized as investigated in the next section. FIG1 shows the effect of varying the number of GD iterations L as well as the random restarts R used to compute the GAN reconstructions of input images. Across different L and R values, Defense-GAN-Rec and Defense-GAN-Orig have comparable performance. Increasing L has the expected effect of improving performance when no attack is present. Interestingly, with an FGSM attack, the classification performance decreases after a certain L value. With too many GD iterations on the mean squared error (MSE) ||G(z) − (x + δ)|| 2 2, some of the adversarial noise components are retained. In the right Figure, the effect of varying R is shown to be extremely pronounced. This is due to the non-convex nature of the MSE, and increasing R enables us to sample different local minima. We now investigate the effect of changing the attack in Table 3. As expected, with higher, the FGSM attack is more successful, especially on the F-MNIST dataset where the noise norm seems to have a more pronounced effect with nearly 37% drop in performance between = 0.1 and 0.3. Figure 7 in Appendix D shows adversarial samples as well as their reconstructions with Defense-GAN at different values of. We can see that for large, the class is difficult to discern, even for the human eye. Even though it seems that increasing is a desirable strategy for the attacker, this increases the likelihood that the adversarial noise is discernible and therefore the attack is detected. It is trivial for the attacker to provide adversarial images at very high, and a good measure of an attack's strength is its ability to affect performance at low. In fact, in the next section, we discuss how Defense-GAN can be used to not only diminish the effect of attacks, but to also detect them. We intuitively expect that clean, unperturbed images will lie closer to the range of the Defense-GAN generator G than adversarial examples. This is due to the fact that G was trained to produce images which resemble the legitimate data. In light of this observation, we propose to use the MSE of an image with it is reconstruction from as a "metric" to decide whether or not the image was Table 3: Classification accuracy of Model F using Defense-GAN (L = 400, R = 10), under FGSM black-box attacks for various noise norms and substitute Model E. adversarially manipulated. In order words, for a given threshold θ > 0, the hypothesis test is: We compute the reconstruction MSEs for every image from the test dataset, and its adversarially manipulated version using FGSM. We show the Receiver Operating Characteristic (ROC) curves as well as the Area Under the Curve (AUC) metric for different Defense-GAN parameters and values in FIG2. The show that this attack detection strategy is effective especially when the number of GD iterations L and random restarts R are large. From the left and middle Figures, we can conclude that the number of random restarts plays a very important role in the detection false positive and true positive rates as was discussed in Section 4.1.1. Furthermore, when is very small, it becomes difficult to detect attacks at low false positive rates. We now present on white-box attacks using three different strategies: FGSM, RAND+FGSM, and CW. We perform the CW attack for 100 iterations of projected GD, with learning rate 10.0, and use c = 100 in equation FORMULA3. Table 4 shows the classification performance of different classifier models across different attack and defense strategies. We note that Defense-GAN significantly outperforms the two other baseline defenses. We even give the adversarial attacker access to the random initializations of z. However, we noticed that the performance does not change much when the attacker does not know the initialization. Adversarial training was done using FGSM to generate the adversarial samples. It is interesting to mention that when CW attack is used, adversarial training performs extremely poorly. As previously discussed, adversarial training does not generalize well against different attack methods. Due to the loop of L steps of GD, Defense-GAN is resilient to GD-based white-box attacks, since the attacker needs to "un-roll" the GD loop and propagate the gradient of the loss all the way across L steps. In fact, from Table 4 L = 25, the performance of the same network drops to 0.947 (more than 5% drop). This shows that using a larger L significantly increases the robustness of Defense-GAN against GD-based whitebox attacks. This comes at the expense of increased inference time complexity. We present a more detailed discussion about the difficulty of GD-based white-box attacks in Appendix B and time complexity in Appendix G. Additional white-box experimental on higher-dimensional images are reported in Appendix F. Table 4: Classification accuracies of different classifier models using various defense strategies on the MNIST (top) and F-MNIST (bottom) datasets, under FGSM, RAND+FGSM, and CW white-box attacks. Defense-GAN has L = 200 and R = 10. Classifier In this paper, we proposed Defense-GAN, a novel defense strategy utilizing GANs to enhance the robustness of classification models against black-box and white-box adversarial attacks. Our method does not assume a particular attack model and was shown to be effective against most commonly considered attack strategies. We empirically show that Defense-GAN consistently provides adequate defense on two benchmark computer vision datasets, whereas other methods had many shortcomings on at least one type of attack. It is worth mentioning that, although Defense-GAN was shown to be a feasible defense mechanism against adversarial attacks, one might come across practical difficulties while implementing and deploying this method. The success of Defense-GAN relies on the expressiveness and generative power of the GAN. However, training GANs is still a challenging task and an active area of research, and if the GAN is not properly trained and tuned, the performance of Defense-GAN will suffer on both original and adversarial examples. Moreover, the choice of hyper-parameters L and R is also critical to the effectiveness of the defense and it may be challenging to tune them without knowledge of the attack. A OPTIMALITY OF p g = p DATA FOR WGANS Sketch of proof of Lemma 1: The WGAN min-max loss is given by: DISPLAYFORM0 DISPLAYFORM1 For a fixed G, the optimal discriminator D which maximizes V W (D, G) is such that: DISPLAYFORM2 Plugging D * G back into, we get: DISPLAYFORM3 }. Clearly, to minimize, we need to set p data (x) = p g (x) for x ∈ X. Then, since both pdfs should integrate to 1, DISPLAYFORM4 However, this is a contradiction since p g (x) < p data (x) for x ∈ X c, unless µ(X c) = 0 where µ is the Lebesgue measure. This concludes the proof. In order to perform a GD-based white-box attack on models using Defense-GAN, an attacker needs to compute the gradient of the output of the classifier with respect to the input. From Figure 1, the generator and the classifier can be seen as one, combined, feedforward network, through which it is easy to propagate gradients. The difficulty lies in the orange box of the GD optimization detailed in FIG0.For the sake of simplicity, let's assume that R = 1. Define L(x, z) = ||G(z) − x|| 2 2. Then z * = z L, which is computed recursively as follows: DISPLAYFORM0 and so on. Therefore, computing the gradient of z * with respect to x involves a large number (L) of recursive chain rules and high-dimensional Jacobian tensors. This computation gets increasingly prohibitive for large L. We describe the neural network architectures used throughout the paper. The detail of models A through F used for classifier and substitute networks can be found in TAB3. In TAB4, the GAN architectures are described, and in TAB5, the encoder architecture for the MagNet baseline is given. In what follows:• Conv(m, k × k, s) refers to a convolutional layer with m feature maps, filter size k × k, and stride s• ConvT(m, k × k) refers to the transpose (gradient) of Conv (sometimes referred to as "deconvolution") with m feature maps, filter size k × k, and stride s• FC(m) refers to a fully-connected layer with m outputs• Dropout(p) refers to a dropout layer with probability p• ReLU refers to the Rectified Linear Unit activation• LeakyReLU(α) is the leaky version of the Rectified Linear Unit with parameter α Generator Discriminator DISPLAYFORM0 We report on white-box attacks on the CelebFaces Attributes dataset (CelebA) BID12 in TAB0. The CelebA dataset is a large-scale face dataset consisting of more than 200, 000 face images, split into training, validation, and testing sets. The RGB images were center-cropped and resized to 64 × 64. We performed the task of gender classification on this dataset. The GAN architecture is the same as that in TAB4, except for an additional ConvT(128, 5 × 5, 1) layer in the generator network. G TIME COMPLEXITYThe computational complexity of reconstructing an image using Defense-GAN is on the order of the number of GD iterations performed to estimate z *, multiplied by the time to compute gradients. The number of random restarts R has less effect on the running time, since random restarts are independent and can run in parallel if enough resources are available. TAB0 shows the average running time, in seconds, to find the reconstructions of MNIST and F-MNIST images on one NVIDIA GeForce GTX TITAN X GPU. For most applications, these running times are not prohibitive. We can see a tradeoff between running time and defense robustness as well as accuracy.

Defense-GAN uses a Generative Adversarial Network to defend against white-box and black-box attacks in classification models.

Represent the short summary for finding the corresponding scientific article (long)!

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.