Question: represent the scientific article to find a short tldr

Question: Over the past four years, neural networks have been proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions. We show that adversarial vulnerability increases with the gradients of the training objective when viewed as a function of the inputs. For most current network architectures, we prove that the L1-norm of these gradients grows as the square root of the input size. These nets therefore become increasingly vulnerable with growing image size. Our proofs rely on the network’s weight distribution at initialization, but extensive experiments confirm that our still hold after usual training. Following the work of BID7, Convolutional Neural Networks (CNNs) have been found vulnerable to adversarial examples: an adversary can drive the performance of state-of-the art CNNs down to chance level with imperceptible changes of the inputs. A number of studies have tried to address this issue, but only few have stressed that, because adversarial examples are essentially small input changes that create large output variations, they are inherently caused by large gradients of the neural network with respect to its inputs. Of course, this view, which we will focus on here, assumes that the network and loss are differentiable. It has the advantage to yield a large body of specific mathematical tools, but might not be easily extendable to masked gradients, non-smooth models or the 0-1-loss. Nevertheless, our might even hold for non-smooth models, given that the latter can often be viewed as smooth at a coarser level. Contributions. More specifically, we provide theoretical and empirical arguments supporting the existence of a monotonic relationship between the gradient norm of the training objective (of a differentiable classifier) and its adversarial vulnerability. Evaluating this norm based on the weight statistics at initialization, we show that CNNs and most feed-forward networks, by design, exhibit increasingly large gradients with input dimension d, almost independently of their architecture. That leaves them increasingly vulnerable to adversarial noise. We corroborate our theoretical by extensive experiments. Although some of those experiments involve adversarial regularization schemes, our goal is not to advocate a new adversarial defense (these schemes are already known), but to show how their effect can be explained by our first order analysis. We do not claim to explain all aspects of adversarial vulnerability, but we claim that our first order argument suffices to explain a significant part of the empirical findings on adversarial vulnerability. This calls for researching the design of neural network architectures with inherently smaller gradients and provides useful guidelines to practitioners and network designers. Suppose that a given classifier ϕ classifies an image x as being in category ϕ(x). An adversarial image is a small modification of x, barely noticeable to the human eye, that suffices to fool the classifier into predicting a class different from ϕ(x). It is a small perturbation of the inputs, that creates a large variation of outputs. Adversarial examples thus seem inherently related to large gradients of the network. A connection, that we will now clarify. Note that visible adversarial examples sometimes appear in the literature, but we deliberately focus on imperceptible ones. Adversarial vulnerability and adversarial damage. In practice, an adversarial image is constructed by adding a perturbation δ to the original image x such that δ ≤ for some (small) number and a given norm · over the input space. We call the perturbed input x + δ an -sized · -attack and say that the attack was successful when ϕ(x + δ) = ϕ(x). This motivates Definition 1. Given a distribution P over the input-space, we call adversarial vulnerability of a classifier ϕ to an -sized · -attack the probability that there exists a perturbation δ of x such that δ ≤ and ϕ(x) = ϕ(x + δ).We call the average increase-after-attack E x∼P [∆L] of a loss L the (L-) adversarial damage (of the classifier ϕ to an -sized · -attack).When L is the 0-1-loss L 0/1, adversarial damage is the accuracy-drop after attack. The 0-1-loss damage is always smaller than adversarial vulnerability, because vulnerability counts all class-changes of ϕ(x), whereas some of them may be neutral to adversarial damage (e.g. a change between two wrong classes). The L 0/1 -adversarial damage thus lower bounds adversarial vulnerability. Both are even equal when the classifier is perfect (before attack), because then every change of label introduces an error. It is hence tempting to evaluate adversarial vulnerability with L 0/1 -adversarial damage. From ∆L 0/1 to ∆L and to ∂ x L. In practice however, we do not train our classifiers with the non-differentiable 0-1-loss but use a smoother loss L, such as the cross-entropy loss. For similar reasons, we will now investigate the adversarial damage E x [∆L(x, c)] with loss L rather than L 0/1. Like for BID7; BID13; and many others, a classifier ϕ will hence be robust if, on average over x, a small adversarial perturbation δ of x creates only a small variation δL of the loss. Now, if δ ≤, then a first order Taylor expansion in shows that DISPLAYFORM0 where ∂ x L denotes the gradient of L with respect to x, and where the last equality stems from the definition of the dual norm |||·||| of ·. Now two remarks. First: the dual norm only kicks in because we let the input noise δ optimally adjust to the coordinates of ∂ x L within its -constraint. This is the brand mark of adversarial noise: the different coordinates add up, instead of statistically canceling each other out as they would with random noise. For example, if we impose that δ 2 ≤, then δ will strictly align with ∂ x L. If instead δ ∞ ≤, then δ will align with the sign of the coordinates of ∂ x L. Second remark: while the Taylor expansion in becomes exact for infinitesimal perturbations, for finite ones it may actually be dominated by higher-order terms. Our experiments FIG1 however strongly suggest that in practice the first order term dominates the others. Now, remembering that the dual norm of an p -norm is the corresponding q -norm, and summarizing, we have proven Lemma 2. At first order approximation in, an -sized adversarial attack generated with norm · increases the loss L at point x by |||∂ x L|||, where |||·||| is the dual norm of ·. In particular, an -sized p -attack increases the loss by ∂ x L q where 1 ≤ p ≤ ∞ and 1 p + 1 q = 1. Consequently, the adversarial damage of a classifier with loss L to -sized attacks generated with norm · is E x |||∂ x L|||. This is valid only at first order, but it proves that at least this kind of first-order vulnerability is present. We will see that the first-order predictions closely match the experiments, and that this insight helps protecting even against iterative (non-first-order) attack methods (Figure 1).Calibrating the threshold to the attack-norm ·. Lemma 2 shows that adversarial vulnerability depends on three main factors: (i) ·, the norm chosen for the attack (ii), the size of the attack, and (iii) E x |||∂ x L|||, the expected dual norm of ∂ x L. We could see Point (i) as a measure of our sensibility to image perturbations, (ii) as our sensibility threshold, and (iii) as the classifier's expected marginal sensibility to a unit perturbation. E x |||∂ x L||| hence intuitively captures the discrepancy between our perception (as modeled by ·) and the classifier's perception for an input-perturbation of small size. Of course, this viewpoint supposes that we actually found a norm · (or more generally a metric) that faithfully reflects human perception -a project in its own right, far beyond the scope of this paper. However, it is clear that the threshold that we choose should depend on the norm · and hence on the input-dimension d. In particular, for a given pixel-wise order of magnitude of the perturbations δ, the p -norm of the perturbation will scale like d. This suggests to write the threshold p used with p -attacks as: DISPLAYFORM0 where ∞ denotes a dimension-independent constant. In Appendix D we show that this scaling also preserves the average signal-to-noise ratio x 2 / δ 2, both across norms and dimensions, so that p could correspond to a constant human perception-threshold. With this in mind, the impatient reader may already jump to Section 3, which contains our main contributions: the estimation of E x ∂ x L q for standard feed-forward nets. Meanwhile, the rest of this section shortly discusses two straightforward defenses that we will use later and that further illustrate the role of gradients. A new old regularizer. Lemma 2 shows that the loss of the network after an 2 -sized · -attack is DISPLAYFORM1 It is thus natural to take this loss-after-attack as a new training objective. Here we introduced a factor 2 for reasons that will become clear in a moment. Incidentally, for · = · 2, this new loss reduces to an old regularization-scheme proposed by BID4 called double-backpropagation. At the time, the authors argued that slightly decreasing a function's or a classifier's sensitivity to input perturbations should improve generalization. In a sense, this is exactly our motivation when defending against adversarial examples. It is thus not surprising to end up with the same regularization term. Note that our reasoning only shows that training with one specific norm |||·||| in helps to protect against adversarial examples generated from ·. A priori, we do not know what will happen for attacks generated with other norms; but our experiments suggest that training with one norm also protects against other attacks (see FIG1 and Section 4.1).Link to adversarially-augmented training. In, designates an attack-size threshold, while in, it is a regularization-strength. Rather than a notation conflict, this reflects an intrinsic duality between two complementary interpretations of, which we now investigate further. Suppose that, instead of using the loss-after-attack, we augment our training set with -sized · -attacks x + δ, where for each training point x, the perturbation δ is generated on the fly to locally maximize the loss-increase. Then we are effectively training with DISPLAYFORM2 where by construction δ satisfies. We will refer to this technique as adversarially augmented training. It was first introduced by BID7 with · = · ∞ under the name of FGSM 1 -augmented training. Using the first order Taylor expansion in of, this'old-plus-postattack' loss of simply reduces to our loss-after-attack, which proves Proposition 3. Up to first-order approximations in,L, · = L,|||·|||. Said differently, for small enough, adversarially-augmented training with -sized · -attacks amounts to penalizing the dual norm |||·||| of ∂ x L with weight /2. In particular, double-backpropagation corresponds to training with 2 -attacks, while FGSM-augmented training corresponds to an 1 -penalty on ∂ x L.This correspondence between training with perturbations and using a regularizer can be compared to Tikhonov regularization: Tikhonov regularization amounts to training with random noise BID2, while training with adversarial noise amounts to penalizing ∂ x L. Section 4.1 verifies the correspondence between adversarial augmentation and gradient regularization empirically, which also strongly suggests the empirical validity of the first-order Taylor expansion in.3 ESTIMATING ∂ x L q TO EVALUATE ADVERSARIAL VULNERABILITY In this section, we evaluate the size of ∂ x L q for standard neural network architectures. We start with fully-connected networks, and finish with a much more general theorem that, not only encompasses CNNs (with or without strided convolutions), but also shows that the gradient-norms are essentially independent of the network topology. We start our analysis by showing how changing q affects the size of ∂ x L q. Suppose for a moment that the coordinates of ∂ x L have typical magnitude DISPLAYFORM3 This equation carries two important messages. First, we see how ∂ x L q depends on d and q. The dependence seems highest for q = 1. But once we account for the varying perceptibility threshold DISPLAYFORM4, we see that adversarial vulnerability scales like d · |∂ x L|, whatever p -norm we use. Second, shows that to be robust against any type of p -attack at any input-dimension d, the average absolute value of the coefficients of ∂ x L must grow slower than 1/d. Now, here is the catch, which brings us to our core insight. In order to preserve the activation variance of the neurons from layer to layer, the neural weights are usually initialized with a variance that is inversely proportional to the number of inputs per neuron. Imagine for a moment that the network consisted only of one output neuron o linearly connected to all input pixels. For the purpose of this example, we assimilate o and L. Because we initialize the weights with a variance of 1/d, their average absolute value |∂ x o| ≡ |∂ x L| grows like 1/ √ d, rather than the required 1/d. By, the adversarial vulnerability DISPLAYFORM0 This toy example shows that the standard initialization scheme, which preserves the variance from layer to layer, causes the average coordinate-size |∂ x L| to grow like 1/ √ d instead of 1/d. When an ∞ -attack tweaks its -sized input-perturbations to align with the coordinate-signs of ∂ x L, all coordinates of ∂ x L add up in absolute value, ing in an output-perturbation that scales like √ d and leaves the network increasingly vulnerable with growing input-dimension. Our next theorems generalize the previous toy example to a very wide class of feedforward nets with ReLU activation functions. For illustration purposes, we start with fully connected nets and only then proceed to the broader class, which includes any succession of (possibly strided) convolutional layers. In essence, the proofs iterate our insight on one layer over a sequence of layers. They all rely on the following set (H) of hypotheses: H1 Non-input neurons are followed by a ReLU killing half of its inputs, independently of the weights. H2 Neurons are partitioned into layers, meaning groups that each path traverses at most once. H3 All weights have 0 expectation and variance 2/(in-degree) ('He-initialization'). H4 The weights from different layers are independent. H5 Two distinct weights w, w from a same node satisfy E [w w] = 0.If we follow common practice and initialize our nets as proposed by BID8, then H3-H5 are satisfied at initialization by design, while H1 is usually a very good approximation BID1. Note that such i.i.d. weight assumptions have been widely used to analyze neural nets and are at the heart of very influential and successful prior work (e.g., equivalence between neural nets and Gaussian processes as pioneered by Neal 1996). Nevertheless, they do not hold after training. That is why all our statements in this section are to be understood as orders of magnitudes that are very well satisfied at initialization in theory and in practice, and that we will confirm experimentally after training in Section 4. Said differently, while our theorems rely on the statistics of neural nets at initialization, our experiments confirm their after training. Theorem 4 (Vulnerability of Fully Connected Nets). Consider a succession of fully connected layers with ReLU activations which takes inputs x of dimension d, satisfies assumptions (H), and outputs logits f k (x) that get fed to a final cross-entropy-loss layer L. Then the coordinates of DISPLAYFORM0 These networks are thus increasingly vulnerable to p -attacks with growing input-dimension. Theorem 4 is a special case of the next theorem, which will show that the previous are essentially independent of the network-topology. We will use the following symmetry assumption on the neural connections. For a given path p, let the path-degree d p be the multiset of encountered in-degrees along path p. For a fully connected network, this is the unordered sequence of layer-sizes preceding the last path-node, including the input-layer. Now consider the multiset {d p} p∈P(x,o) of all path-degrees when p varies among all paths from input x to output o. The symmetry assumption (relatively to o) is (S) All input nodes x have the same multiset {d p} p∈P(x,o) of path-degrees from x to o. Intuitively, this means that the statistics of degrees encountered along paths to the output are the same for all input nodes. This symmetry assumption is exactly satisfied by fully connected nets, almost satisfied by CNNs (up to boundary effects, which can be alleviated via periodic or mirror padding) and exactly satisfied by strided layers, if the layer-size is a multiple of the stride. Theorem 5 (Vulnerability of Feedforward Nets). Consider any feed-forward network with linear connections and ReLU activation functions. Assume the net satisfies assumptions (H) and outputs logits f k (x) that get fed to the cross-entropy-loss L. DISPLAYFORM1 Moreover, if the net satisfies the symmetry assumption (S), then DISPLAYFORM2 Theorems 4 and 5 are proven in Appendix B. The main proof idea is that in the gradient norm computation, the He-initialization exactly compensates the combinatorics of the number of paths in the network, so that this norm becomes independent of the network topology. In particular, we getCorollary 6 (Vulnerability of CNNs). In any succession of convolution and dense layers, strided or not, with ReLU activations, that satisfies assumptions (H) and outputs logits that get fed to the cross-entropy-loss L, the gradient of the logit-coordinates scale like 1/ √ d and FORMULA8 is satisfied. It is hence increasingly vulnerable with growing input-resolution to attacks generated with any p -norm. Appendix A shows that the network gradient are dampened when replacing strided layers by average poolings, essentially because average-pooling weights do not follow the He-init assumption H3. In Section 4.1, we empirically verify the validity of the first-order Taylor approximation made in (Fig.1), for example by checking the correspondence between loss-gradient regularization and adversarially-augmented training FIG1 ). Section 4.2 then empirically verifies that both the average 1 -norm of ∂ x L and the adversarial vulnerability grow like √ d as predicted by Corollary 6. For all experiments, we approximate adversarial vulnerability using various attacks of the Foolboxpackage . We use an ∞ attack-threshold of size ∞ = 0.005 (and later 0.002) which, for pixel-values ranging from 0 to 1, is completely imperceptible but suffices to fool the classifiers on a significant proportion of examples. This ∞ -threshold should not be confused with the regularization-strengths appearing in and, which will be varied in some experiments. Figure 1: Adversarial vulnerability approximated by different attack-types for 10 trained networks as a function of (a) the 1 gradient regularization-strength used to train the nets and (b) the average gradient-norm. These curves confirm that the first-order expansion term in is a crucial component of adversarial vulnerability. DISPLAYFORM0 suggests that protecting against a given attack-norm also protects against others. (f): Merging 2band 2c shows that all adversarial augmentation and gradient-regularization methods achieve similar accuracy-vulnerability trade-offs. We train several CNNs with same architecture to classify CIFAR-10 images BID12.For each net, we use a specific training method with a specific regularization value. The training methods used were 1 -and 2 -penalization of ∂ x L (Eq. 4), adversarial augmentation with ∞ -andValidity of first order expansion. The following observations support the validity of the first order Taylor expansion in and suggest that it is a crucial component of adversarial vulnerability: (i) the efficiency of the first-order defense against iterative (non-first-order) attacks (Fig.1a); (ii) the striking similarity between the PGD curves (adversarial augmentation with iterative attacks) and the other adversarial training training curves (one-step attacks/defenses); (iii) the functional-like dependence between any approximation of adversarial vulnerability and E x ∂ x L 1 (Fig.1b), and its independence on the training method FIG1. (iv) the excellent correspondence between the gradient-regularization and adversarial training curves (see next paragraph). Said differently, adversarial examples seem indeed to be primarily caused by large gradients of the classifier as captured via the induced loss.. The excellent match between the adversarial augmentation curve with p = ∞ (p = 2) and its gradient-regularization dual counterpart with q = 1 (resp. q = 2) illustrates the duality between as a threshold for adversarially-augmented training and as a regularization constant in the regularized loss (Proposition 3). It also supports the validity of the first-order Taylor expansion in.Confirmation of. Still on the upper row, the curves for p = ∞, q = 1 have no reason to match those for p = q = 2 when plotted against, because -threshold is relative to a specific attack-norm. However, suggested that the rescaled thresholds d 1/p may approximately correspond to a same'threshold-unit' across p -norms and across dimension. This is well confirmed by the upper row plots: by rescaling the x-axis, the p = q = 2 and q = 1, p = ∞ curves get almost super-imposed. Accuracy-vs-Vulnerability Trade-Off. FIG1 by taking out, FIG1 shows that all gradient regularization and adversarial training methods yield equivalent accuracyvulnerability trade-offs. Incidentally, for higher penalization values, these trade-offs appear to be much better than those given by cross Lipschitz regularization. The penalty-norm does not matter. We were surprised to see that on Figures 2d and 2f, the L,q curves are almost identical for q = 1 and 2. This indicates that both norms can be used interchangeably in (modulo proper rescaling of via FORMULA2), and suggests that protecting against a specific attacknorm also protects against others. may provide an explanation: if the coordinates of ∂ x L behave like centered, uncorrelated variables with equal variance -which follows from assumptions (H) -, then the 1 -and 2 -norms of FIG1 confirms this explanation. The slope is independent of the training method. Therefore, penalizing ∂ x L(x) 1 during training will not only decrease E x ∂ x L 1 (as shown in FIG1), but also drive down E x ∂ x L 2 and vice-versa. DISPLAYFORM1 Theorems 4-5 and Corollary 6 predict a linear growth of the average 1 -norm of ∂ x L with the square root of the input dimension d, and therefore also of adversarial vulnerability (Lemma 2). To test these predictions, we upsampled the CIFAR-10 images (of size 3 x 32 x 32) by copying pixels so as to get 4 datasets with, respectively, 32, 64, 128 and 256 pixels per edge. We then trained a CNN on each dataset All networks had exactly the same amount of parameters and very similar structure across the various input-resolutions. The CNNs were a succession of 8'convolution → batchnorm → ReLU' layers with 64 output channels, followed by a final full-connection to the 12 logit-outputs. We used 2 × 2-max-poolings after the convolutions of layers 2,4, 6 and 8, and a final max-pooling after layer 8 that fed only 1 neuron per channel to the fully-connected layer. To ensure that the convolution-kernels cover similar ranges of the images across each of the 32, 64, 128 and 256 input-resolutions, we respectively dilated all convolutions ('à trous') by a factor 1, 2, 4 and 8. There is a clear discrepancy: on the training set, the gradient norms decrease (after an initialization phase) and are dimension-independent; on the test set, they increase and scale like √ d. This suggests that, outside the training points, the nets tend to recover their prior gradient-properties (i.e. naturally large gradients). Our theoretical show that the priors of classical neural networks yield vulnerable functions because of naturally high gradients. And our experiments FIG3 suggest that usual training does not escape these prior properties. But how may these insights help understanding the vulnerability of robustly trained networks? Clearly, to be successful, robust training algorithms must escape ill-behaved priors, which explains why most methods (e.g. FGSM, PGD) are essentially gradient penalization techniques. But, MNIST aside, even state-of-the-art methods largely fail at protecting current network architectures BID14, and understanding why is motivation to this and many other papers. Interestingly, BID14 recently noticed that those methods actually do protect the nets on training examples, but fail to generalize to the test set. They hence conclude that state-of-the-art robustification algorithms work, but need more data. Alternatively however, when generalization fails, one can also reduce the model's complexity. Large fully connected nets for example typically fail to generalize to out-of-sample examples: getting similar accuracies than CNNs would need prohibitively many training points. Similarly, Schmidt et al.'s observations may suggest that, outside the training points, networks tend to recover their prior properties, i.e. naturally large gradients. FIG4 corroborates this hypothesis. It plots the evolution over training epochs of the 1 -gradient-norms of the CNNs from Section 4.2 FIG3 on the training and test sets respectively. The discrepancy is unmistakable: after a brief initialization phase, the norms decrease on the training set, but increase on the test set. They are moreover almost input-dimension independent on the training set, but scale as √ d on the test set (as seen in FIG3 up to respectively 2, 4, 8 and 16 times the training set values. These observations suggest that, with the current amount of data, tackling adversarial vulnerability may require new architectures with inherently smaller gradients. Searching these architectures among those with well-behaved prior-gradients seems a reasonable start, where our theoretical may prove very useful. On network vulnerability. BID7 already stressed that adversarial vulnerability increases with growing dimension d. But their argument only relied on a linear 'one-output-to-manyinputs'-model with dimension-independent weights. They therefore concluded on a linear growth of adversarial vulnerability with d. In contrast, our theory applies to almost any standard feed-forward architecture (not just linear), and shows that, once we adjust for the weight's dimension-dependence, adversarial vulnerability increases like √ d (not d), almost independently of the architecture. Nevertheless, our experiments confirm Goodfellow et al.'s idea that our networks are "too linear-like", in the sense that a first-order Taylor expansion is indeed sufficient to explain the adversarial vulnerability of neural networks. As suggested by the one-output-to-many-inputs model, the culprit is that growing 3 Appendix A investigates such a preliminary direction by introducing average poolings, which have a weight-size 1 /in−channels rather than the typical 1 / √ in−channels of the other He-initialized weights. dimensionality gives the adversary more and more room to'wriggle around' with the noise and adjust to the gradient of the output neuron. This wriggling, we show, is still possible when the output is connected to all inputs only indirectly, even when no neuron is directly connected to all inputs, like in CNNs. This explanation of adversarial vulnerability is independent of the intrinsic dimensionality or geometry of the data (compare to BID0 BID6 . Finally, let us mention that show a close link between the vulnerability to small worst-case perturbation (as studied here) and larger average perturbations. Our findings on the adversarial vulnerability NNs to small perturbation could thus be translated accordingly. On robustification algorithms. Incidentally, BID7 also already relate adversarial vulnerability to large gradients of the loss L, an insight at the very heart of their FGSM-algorithm. They however do not propose any explicit penalizer on the gradient of L other than indirectly through adversarially-augmented training. propose the old double-backpropagation to robustify networks but make no connection to FGSM and adversarial augmentation. BID13 discuss and use the connection between gradient-penalties and adversarial augmentation, but never actually compare both in experiments. This comparison however is essential to test the validity of the first-order Taylor expansion in, as confirmed by the similarity between the gradient-regularization and adversarial-augmentation curves in FIG1. BID9 derived yet another gradient-based penalty -the cross-Lipschitz-penalty-by considering (and proving) formal guarantees on adversarial vulnerability itself, rather than adversarial damage. While both penalties are similar in spirit, focusing on the adversarial damage rather than vulnerability has two main advantages. First, it achieves better accuracy-to-vulnerability ratios, both in theory and practice, because it ignores class-switches between misclassified examples and penalizes only those that reduce the accuracy. Second, it allows to deal with one number only, ∆L, whereas Hein & Andriushchenko's cross-Lipschitz regularizer and theoretical guarantees explicitly involve all K logit-functions (and their gradients). See Appendix C. Penalizing network-gradients is also at the heart of contractive auto-encoders as proposed by , where it is used to regularize the encoder-features. Seeing adversarial training as a generalization method, let us also mention BID10, who propose to enhance generalization by searching for parameters in a "flat minimum region" of the loss. This leads to a penalty involving the gradient of the loss, but taken with respect to the weights, rather than the inputs. In the same vein, a gradientregularization of the loss of generative models also appears in Proposition 6 of , where it stems from a code-length bound on the data (minimum description length). More generally, the gradient regularized objective is essentially the first-order approximation of the robust training objective max δ ≤ L(x + δ, c) which has a long history in math , machine learning and now adversarial vulnerability . Finally, BID3 propose new network-architectures that have small gradients by design, rather than by special training: an approach that makes all the more sense, considering the of Theorems 4 and 5. For further details and references on adversarial attacks and defenses, we refer to. For differentiable classifiers and losses, we showed that adversarial vulnerability increases with the gradients ∂ x L of the loss, which is confirmed by the near-perfect functional relationship between gradient norms and vulnerability FIG1 We then evaluated the size of ∂ x L q and showed that, at initialization, usual feed-forward nets (convolutional or fully connected) are increasingly vulnerable to p -attacks with growing input dimension d (the image-size), almost independently of their architecture. Our experiments show that, on the tested architectures, usual training escapes those prior gradient (and vulnerability) properties on the training, but not on the test set. BID14 suggest that alleviating this generalization gap requires more data. But a natural (complementary) alternative would be to search for architectures with naturally smaller gradients, and in particular, with well-behaved priors. Despite all their limitations (being only first-order, assuming a prior weight-distribution and a differentiable loss and architecture), our theoretical insights may thereby still prove to be precious future allies. It is common practice in CNNs to use average-pooling layers or strided convolutions to progressively decrease the number of pixels per channel. Corollary 6 shows that using strided convolutions does not protect against adversarial examples. However, what if we replace strided convolutions by convolutions with stride 1 plus an average-pooling layer? Theorem 5 considers only randomly initialized weights with typical size 1/ √ in-degree. Average-poolings however introduce deterministic weights of size 1/(in-degree). These are smaller and may therefore dampen the input-to-output gradients and protect against adversarial examples. We confirm this in our next theorem, which uses a slightly modified version (H) of (H) to allow average pooling layers. (H) is (H), but where the He-init H3 applies to all weights except the (deterministic) average pooling weights, and where H1 places a ReLU on every non-input and non-average-pooling neuron. Theorem 7 (Effect of Average-Poolings). Consider a succession of convolution layers, dense layers and n average-pooling layers, in any order, that satisfies (H) and outputs logits f k (x). Assume the n average pooling layers have a stride equal to their mask size and perform averages over a 1,..., a n nodes respectively. Then ∂ x f k 2 and |∂ x f k | scale like 1/ √ a 1 · · · a n and 1/ √ d a 1 · · · a n respectively. Proof in Appendix B.4. Theorem 7 suggest to try and replace any strided convolution by its non-strided counterpart, followed by an average-pooling layer. It also shows that if we systematically reduce the number of pixels per channel down to 1 by using only non-strided convolutions and average-pooling layers (i.e. d = n i=1 a i), then all input-to-output gradients should become independent of d, thereby making the network completely robust to adversarial examples. Our following experiments (FIG6 show that after training, the networks get indeed robustified to adversarial examples, but remain more vulnerable than suggested by Theorem 7.Experimental setup. Theorem 7 shows that, contrary to strided layers, average-poolings should decrease adversarial vulnerability. We tested this hypothesis on CNNs trained on CIFAR-10, with 6 blocks of 'convolution → BatchNorm →ReLU' with 64 output-channels, followed by a final average pooling feeding one neuron per channel to the last fully-connected linear layer. Additionally, after every second convolution, we placed a pooling layer with stride and mask-size (thus acting on 2 × 2 neurons at a time, without overlap). We tested average-pooling, strided and max-pooling layers and trained 20 networks per architecture. Results are shown in FIG6. All accuracies are very close, but, as predicted, the networks with average pooling layers are more robust to adversarial images than the others. However, they remain more vulnerable than what would follow from Theorem 7. We also noticed that, contrary to the strided architectures, their gradients after training are an order of magnitude higher than at initialization and than predicted. This suggests that assumptions (H) get more violated when using average-poolings instead of strided layers. Understanding why will need further investigations. Proof. Let δ be an adversarial perturbation with δ = 1 that locally maximizes the loss increase at point x, meaning that δ = arg max δ ≤1 ∂ x L · δ. Then, by definition of the dual norm of ∂ x L we have: DISPLAYFORM0 Proof. Let x designate a generic coordinate of x. To evaluate the size of ∂ x L q, we will evaluate the size of the coordinates ∂ x L of ∂ x L by decomposing them into DISPLAYFORM0 where f k (x) denotes the logit-probability of x belonging to class k. We now investigate the statistical properties of the logit gradients ∂ x f k, and then see how they shape ∂ x L.Step 1: Statistical properties of ∂ x f k. Let P(x, k) be the set of paths p from input neuron x to output-logit k. Let p − 1 and p be two successive neurons on path p, andp be the same path p but without its input neuron. Let w p designate the weight from p − 1 to p and ω p be the path-product ω p:= p∈p w p. Finally, let σ p (resp. σ p) be equal to 1 if the ReLU of node p (resp. if path p) is active for input x, and 0 otherwise. As previously noticed by BID1 using the chain rule, we see that ∂ x f k is the sum of all ω p whose path is active, i.e. ∂ x f k (x) = p∈P(x,k) ω p σ p. Consequently: DISPLAYFORM1 The first equality uses H1 to decouple the expectations over weights and ReLUs, and then applies Lemma 10 of Appendix B.3, which uses H3-H5 to kill all cross-terms and take the expectation over weights inside the product. The second equality uses H3 and the fact that the ing product is the same for all active paths. The third equality counts the number of paths from x to k and we conclude by noting that all terms cancel out, except d p−1 from the input layer which is d. Equation 8 shows DISPLAYFORM2 Step 2: Statistical properties of DISPLAYFORM3 f h (x) (the probability of image x belonging to class k according to the network), we have, by definition of the cross-entropy loss, L(x, c):= − log q c (x), where c is the label of the target class. Thus: DISPLAYFORM4 otherwise, and DISPLAYFORM5 Using again Lemma 10, we see that the ∂ x f k (x) are K centered and uncorrelated variables. So DISPLAYFORM6 is approximately the sum of K uncorrelated variables with zero-mean, and its total variance is given by DISPLAYFORM7. concludes. Remark 1. Equation 9 can be rewritten as DISPLAYFORM8 As the term k = c disappears, the norm of the gradients ∂ x L(x) appears to be controlled by the total error probability. This suggests that, even without regularization, trying to decrease the ordinary classification error is still a valid strategy against adversarial examples. It reflects the fact that when increasing the classification margin, larger gradients of the classifier's logits are needed to push images from one side of the classification boundary to the other. This is confirmed by Theorem 2.1 of BID9. See also in Appendix C. The proof of Theorem 5 is very similar to the one of Theorem 4, but we will need to first generalize the equalities appearing in. To do so, we identify the computational graph of a neural network to an abstract Directed Acyclic Graph (DAG) which we use to prove the needed algebraic equalities. We then concentrate on the statistical weight-interactions implied by assumption (H), and finally throw these together to prove the theorem. In all the proof, o will designate one of the output-logits f k (x).Lemma 8. Let x be the vector of inputs to a given DAG, o be any leaf-node of the DAG, x a generic coordinate of x. Let p be a path from the set of paths P(x, o) from x to o,p the same path without node x, p a generic node inp, and d p be its input-degree. Then: DISPLAYFORM0 Proof. We will reason on a random walk starting at o and going up the DAG by choosing any incoming node with equal probability. The DAG being finite, this walk will end up at an input-node x with probability 1. Each path p is taken with probability p∈p 1 dp. And the probability to end up at an input-node is the sum of all these probabilities, i.e. DISPLAYFORM1 The sum over all inputs x in being 1, on average it is 1/d for each x, where d is the total number of inputs (i.e. the length of x). It becomes an equality under assumption (S):Lemma 9. Under the symmetry assumption (S), and with the previous notations, for any input x ∈ x: p. By using and the fact that, by (S), the multiset D(x, o) is independent of x, we hence conclude DISPLAYFORM2 DISPLAYFORM3 Now, let us relate these considerations on graphs to gradients and use assumptions (H). We remind that path-product ω p is the product p∈p w p.Lemma 10. Under assumptions (H), the path-products ω p, ω p of two distinct paths p and p starting from a same input node x, satisfy: DISPLAYFORM4 Furthermore, if there is at least one non-average-pooling weight on path p, then E W [ω p] = 0.Proof. Hypothesis H4 yields DISPLAYFORM5 Now, take two different paths p and p that start at a same node x. Starting from x, consider the first node after which p and p part and call p and p the next nodes on p and p respectively. Then the weights w p and w p are two weights of a same node. Applying H4 and H5 hence gives DISPLAYFORM6 Finally, if p has at least one non-average-pooling node p, then successively applying H4 and H3 yields: DISPLAYFORM7 We now have all elements to prove Theorem 5.Proof. (of Theorem 5) For a given neuron p inp, let p − 1 designate the previous node in p of p. Let σ p (resp. σ p) be a variable equal to 0 if neuron p gets killed by its ReLU (resp. path p is inactive), and 1 otherwise. Then: DISPLAYFORM8 Consequently: DISPLAYFORM9 where the firs line uses the independence between the ReLU killings and the weights (H1), the second uses Lemma 10 and the last uses Lemma 9. The gradient ∂ x o thus has coordinates whose squared expectations scale like 1/d. Thus each coordinate scales like 1/ √ d and ∂ x o q like d DISPLAYFORM10 Step 2 of the proof of Theorem 4.Finally, note that, even without the symmetry assumption (S), using Lemma 8 shows that DISPLAYFORM11 Thus, with or without (S), ∂ x o 2 is independent of the input-dimension d. To prove Theorem 7, we will actually prove the following more general theorem, which generalizes Theorem 5. Theorem 7 is a straightforward corollary of it. Theorem 11. Consider any feed-forward network with linear connections and ReLU activation functions that outputs logits f k (x) and satisfies assumptions (H). Suppose that there is a fixed multiset of integers {a 1, . . ., a n} such that each path from input to output traverses exactly n average pooling nodes with degrees {a 1, . . ., a n}. Then: DISPLAYFORM0 Furthermore, if the net satisfies the symmetry assumption (S), then: DISPLAYFORM1 Two remarks. First, in all this proof, "weight" encompasses both the standard random weights, and the constant (deterministic) weights equal to 1/(in-degree) of the average-poolings. Second, assumption H5 implies that the average-pooling nodes have disjoint input nodes: otherwise, there would be two non-zero deterministic weights w, w from a same neuron that would hence satisfy: DISPLAYFORM2 Proof. As previously, let o designate any fixed output-logit f k (x). For any path p, let a be the set of average-pooling nodes of p and let q be the set of remaining nodes. Each path-product ω p satisfies: ω p = ω q ω a, where ω a is a same fixed constant. For two distinct paths p, p, Lemma 10 therefore yields: DISPLAYFORM3 Combining this with Lemma 9 and under assumption (S), we get similarly to: DISPLAYFORM4 Again, note that, even without assumption (S), using and Lemma 8 shows that DISPLAYFORM5 which proves. In their Theorem 2.1, BID9 show that the minimal = δ p perturbation to fool the classifier must be bigger than: DISPLAYFORM0 They argue that the training procedure typically already tries to maximize f c (x) − f k (x), thus one only needs to additionally ensure that ∂ x f c (x) − ∂ x f k (x) q is small. They then introduce what they call a Cross-Lipschitz Regularization, which corresponds to the case p = 2 and involves the gradient differences between all classes: DISPLAYFORM1 In contrast, using, (the square of) our proposed regularizer ∂ x L q from can be rewritten, for p = q = 2 as: DISPLAYFORM2 Although both FORMULA0 and FORMULA0 cross-interaction between the K classes, the big difference is that while in all classes play exactly the same role, in the summands all refer to the target class c in at least two different ways. First, all gradient differences are always taken with respect to ∂ x f c. Second, each summand is weighted by the probabilities q k (x) and q h (x) of the two involved classes, meaning that only the classes with a non-negligible probability get their gradient regularized. This reflects the idea that only points near the margin need a gradient regularization, which incidentally will make the margin sharper. To keep the average pixel-wise variation constant across dimensions d, we saw in that the threshold p of an p -attack should scale like d 1/p. We will now see another justification for this scaling. Contrary to the rest of this work, where we use a fixed p for all images x, here we will let p depend on the 2 -norm of x. If, as usual, the dataset is normalized such that the pixels have on average variance 1, both approaches are almost equivalent. Suppose that given an p -attack norm, we want to choose p such that the signal-to-noise ratio (SNR) x 2 / δ 2 of a perturbation δ with p -norm ≤ p is never greater than a given SNR threshold 1/. For p = 2 this imposes 2 = x 2. More generally, studying the inclusion of p -balls in 2 -balls yields DISPLAYFORM0 Note that this gives again DISPLAYFORM1. This explains how to adjust the threshold with varying p -attack norm. Now, let us see how to adjust the threshold of a given p -norm when the dimension d varies. Suppose that x is a natural image and that decreasing its dimension means either decreasing its resolution or cropping it. Because the statistics of natural images are approximately resolution and scale invariant BID11, in either case the average squared value of the image pixels remains unchanged, which implies that x 2 scales like √ d. Pasting this back into, we again get: DISPLAYFORM2 In particular, ∞ ∝ is a dimension-free number, exactly like in of the main part. Now, why did we choose the SNR as our invariant reference quantity and not anything else? One reason is that it corresponds to a physical power ratio between the image and the perturbation, which we think the human eye is sensible to. Of course, the eye's sensitivity also depends on the spectral frequency of the signals involved, but we are only interested in orders of magnitude here. Another point: any image x yields an adversarial perturbation δ x, where by constraint x 2 / δ x ≤ 1/. For 2 -attacks, this inequality is actually an equality. But what about other p -attacks: (on average over x,) how far is the signal-to-noise ratio from its imposed upper bound 1/? For p ∈ {1, 2, ∞}, the answer unfortunately depends on the pixel-statistics of the images. But when p is 1 or ∞, then the situation is locally the same as for p = 2. Specifically:Lemma 12. Let x be a given input and > 0. Let p be the greatest threshold such that for any δ with δ p ≤ p, the SNR DISPLAYFORM3 Moreover, for p ∈ {1, 2, ∞}, if δ x is the p -sized p -attack that locally maximizes the loss-increase i.e. δ x = arg max δ p ≤ p |∂ x L · δ|, then: DISPLAYFORM4 and E x [SNR(x)] = 1.Proof. The first paragraph follows from the fact that the greatest p -ball included in an 2 -ball of radius x 2 has radius x 2 d DISPLAYFORM5 Under review as a conference paper at ICLR 2019The second paragraph is clear for p = 2. For p = ∞, it follows from the fact that δ x = ∞ sign ∂ x L which satisfies: DISPLAYFORM6 Intuitively, this means that for p ∈ {1, 2, ∞}, the SNR of p -sized p -attacks on any input x will be exactly equal to its fixed upper limit 1/. And in particular, the mean SNR over samples x is the same (1/) in all three cases. We also ran a similar experiment as in Section 4.2, but instead of using upsampled CIFAR-10 images, we created a 12-class dataset of approximately 80, 000 3 × 256 × 256-sized RGBimages by merging similar ImageNet-classes, resizing the smallest image-edge to 256 pixels and quantiles. The are identical to Section 4.2: after usual training, the vulnerability and gradient-norms still increase like √ d. Note that, as the gradients get much larger at higher dimensions, the first order approximation in becomes less and less valid, which explains the little inflection of the adversarial vulnerability curve. For smaller -thresholds, we verified that the inflection disappears. Here we plot the same curves as in the main part, but using an 2 -attack threshold of size 2 = 0.005 √ d instead of the ∞ -threshold and deep-fool attacks instead of iterative ∞ -ones in Figs. 8 and 9. Note that contrary to ∞ -thresholds, 2 -thresholds must be rescaled by √ d to stay consistent across dimensions (see Eq.3 and Appendix D). All curves look essentially the same as their counterparts in the main text. In usual adversarially-augmented training, the adversarial image x + δ is generated on the fly, but is nevertheless treated as a fixed input of the neural net, which means that the gradient does not get backpropagated through δ. This need not be. As δ is itself a function of x, the gradients could actually also be backpropagated through δ. As it was only a one-line change of our code, we used this opportunity to test this variant of adversarial training (FGSM-variant in FIG1) and thank Martín Arjovsky for suggesting it. But except for an increased computation time, we found no significant difference compared to usual augmented training.

Represent the TLDR for finding the corresponding scientific article

Neural nets have large gradients by design; that makes them adversarially vulnerable.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the article for finding a single-sentence TLDR

Question: Effective performance of neural networks depends critically on effective tuning of optimization hyperparameters, especially learning rates (and schedules thereof). We present Amortized Proximal Optimization (APO), which takes the perspective that each optimization step should approximately minimize a proximal objective (similar to the ones used to motivate natural gradient and trust region policy optimization). Optimization hyperparameters are adapted to best minimize the proximal objective after one weight update. We show that an idealized version of APO (where an oracle minimizes the proximal objective exactly) achieves global convergence to stationary point and locally second-order convergence to global optimum for neural networks. APO incurs minimal computational overhead. We experiment with using APO to adapt a variety of optimization hyperparameters online during training, including (possibly layer-specific) learning rates, damping coefficients, and gradient variance exponents. For a variety of network architectures and optimization algorithms (including SGD, RMSprop, and K-FAC), we show that with minimal tuning, APO performs competitively with carefully tuned optimizers. Tuning optimization hyperparameters can be crucial for effective performance of a deep learning system. Most famously, carefully selected learning rate schedules have been instrumental in achieving state-of-the-art performance on challenging datasets such as ImageNet BID6 and WMT BID36. Even algorithms such as RMSprop BID34 and Adam , which are often interpreted in terms of coordinatewise adaptive learning rates, still have a global learning rate parameter which is important to tune. A wide variety of learning rate schedules have been proposed BID24 BID14 BID2. Seemingly unrelated phenomena have been explained in terms of effective learning rate schedules BID35. Besides learning rates, other hyperparameters have been identified as important, such as the momentum decay factor BID31, the batch size BID28, and the damping coefficient in second-order methods BID20 BID19.There have been many attempts to adapt optimization hyperparameters to minimize the training error after a small number of updates BID24 BID1 BID2. This approach faces two fundamental obstacles: first, learning rates and batch sizes have been shown to affect generalization performance because stochastic updates have a regularizing effect BID5 BID18 BID27 BID35. Second, minimizing the short-horizon expected loss encourages taking very small steps to reduce fluctuations at the expense of long-term progress BID37. While these effects are specific to learning rates, they present fundamental obstacles to tuning any optimization hyperparameter, since basically any optimization hyperparameter somehow influences the size of the updates. In this paper, we take the perspective that the optimizer's job in each iteration is to approximately minimize a proximal objective which trades off the loss on the current batch with the average change in the predictions. Specifically, we consider proximal objectives of the form J(φ) = h(f (g(θ, φ))) + λD(f (θ), f (g(θ, φ))), where f is a model with parameters θ, h is an approximation to the objective function, g is the base optimizer update with hyperparameters φ, and D is a distance metric. Indeed, approximately solving such a proximal objective motivated the natural gradient algorithm BID0, as well as proximal reinforcement learning algorithms BID26. We introduce Amortized Proximal Optimization (APO), an approach which adapts optimization hyperparameters to minimize the proximal objective in each iteration. We use APO to tune hyperparameters of SGD, RMSprop, and K-FAC; the hyperparameters we consider include (possibly layer-specific) learning rates, damping coefficients, and the power applied to the gradient covariances. Notice that APO has a hyperparameter λ which controls the aggressiveness of the updates. We believe such a hyperparameter is necessary until the aforementioned issues surrounding stochastic regularization and short-horizon bias are better understood. However, in practice we find that by performing a simple grid search over λ, we can obtain automatically-tuned learning rate schedules that are competitive with manual learning rate decay schedules. Furthermore, APO can automatically adapt several optimization hyperparameters with only a single hand-tuned hyperparameter. We provide theoretical justification for APO by proving strong convergence for an oracle which solves the proximal objective exactly in each iteration. In particular, we show global linear convergence and locally quadratic convergence under mild assumptions. These motivate the proximal objective as a useful target for meta-optimization. We evaluate APO on real-world tasks including image classification on MNIST, CIFAR-10, CIFAR-100, and SVHN. We show that adapting learning rates online via APO yields faster training convergence than the best fixed learning rates for each task, and is competitive with manual learning rate decay schedules. Although we focus on fast optimization of the training objective, we also find that the solutions found by APO generalize at least as well as those found by fixed hyperparameters or fixed schedules. We view a neural network as a parameterized function z = f (x, θ), where x is the input, θ are the weights and biases of the network, and z can be interpreted as the output of a regression model or the un-normalized log-probabilities of a classification model. Let the training dataset be {( DISPLAYFORM0, where input x i is associated with target t i . Our goal is to minimize the loss function: DISPLAYFORM1 where Z is the matrix of network outputs on all training examples x 1, . . ., x N, and T is the vector of labels. We design an iterative optimization algorithm to minimize Eq. 1 under the following framework: in the kth iteration, one aims to update θ to minimize the following proximal objective: DISPLAYFORM2 where x is the data used in the current iteration, P is the distribution of data, θ k is the parameters of the neural network at the current iteration, h(·) is some approximation of the loss function, and D(·, ·) represents the distance between network outputs under some metric (for notational convenience, we use mini-batch size of 1 to describe the algorithm). We first provide the motivation for this proximal objective in Section 2.1; then in Section 2.2, we propose an algorithm to optimize it in an online manner. In this section, we show that by approximately minimizing simple instances of Eq. 2 in each iteration (similar to BID25), one can recover the classic Gauss-Newton algorithm and Natural Gradient Descent BID0. In general, updating θ so as to minimize the proximal objective is impractical due to the complicated nonlinear relationship between θ and z. However, one can find an approximate solution by linearizing the network function: DISPLAYFORM0 where J = ∇ θ f (x, θ) is the Jacobian matrix. We consider the following instance of Eq. 2: DISPLAYFORM1 where ∆z f (x, θ) − f (x, θ k) is the change of network output, t is the label of current data x. Here h(·) is defined as the first-order Taylor approximation of the loss function. Using the linear approximation (Eq. 3), and a local second-order approximation of D, this proximal objective can be written as: DISPLAYFORM2 DISPLAYFORM3 is the Hessian matrix of the dissimilarity measured atz = f (x, θ k).Solving Eq. 5 yields: DISPLAYFORM4 where G Ex ∼P J ∇ 2DJ is the pre-conditioning matrix. Different settings for the dissimilarity DISPLAYFORM5 is defined as the squared Euclidean distance, Eq. 6 recovers the classic Gauss-Newton algorithm. When DISPLAYFORM6 is defined as the Bregman divergence, Eq. 6 yields the Generalized Gauss-Newton (GGN) method. When the output of neural network parameterizes an exponential-family distribution, the dissimilarity term can be defined as Kullback-Leibler divergence: DISPLAYFORM7 in which case Eq. 6 yields Natural Gradient Descent BID0. Since different versions of our proximal objective lead to various efficient optimization algorithms, we believe it is a useful target for meta-optimization. Although optimizers including the Gauss-Newton algorithm and Natural Gradient Descent can be seen as ways to approximately solve Eq. 2, they rely on a local linearization of the neural network and usually require more memory and more careful tuning in practice. We propose to instead directly minimize Eq. 2 in an online manner. Finding good hyperparameters (e.g., the learning rate for SGD) is a challenging problem in practice. We propose to adapt these hyperparameters online in order to best optimize the proximal objective. Consider any optimization algorithm (base-optimizer) of the following form: θ ← g(x, t, θ, ξ, φ). Here, θ is the set of model parameters, x is the data used in this iteration, t is the corresponding label, ξ is a vector of statistics computed online during optimization, and φ is a vector of optimization hyperparameters to be tuned. For example, ξ contains the exponential moving averages of the squared gradients of the parameters in RMSprop. φ usually contains the learning rate (global or layer-specific), and possibly other hyperparameters dependent on the algorithm. For each step, we formulate the meta-objective from Eq. 2 as follows (for notational convenience we omit variables other than θ and φ of g): DISPLAYFORM0 Here,x is a random mini-batch sampled from the data distribution P. We compute the approximation to the loss, h, using the same mini-batch as the gradient of the base optimizer, to avoid the short horizon bias problem BID37; we measure D on a different mini-batch to avoid instability that would if we took a large step in a direction that is unimportant for the current batch, but important for other batches. The hyperparameters φ are optimized using a stochastic gradient-based algorithm (the meta-optimizer) using the gradient ∇ φ J(φ) (similar in spirit to BID24 BID17). We refer to our framework as Amortized Proximal Optimization (APO). The simplest version of APO, which uses SGD as the meta-optimizer, is shown in Algorithm 1. One can choose any meta-optimizer; we found that RMSprop was the most stable and best-performing meta-optimizer in practice, and we used it for all our experiments. DISPLAYFORM1 When considering optimization meta-objectives, it is useful to analyze idealized versions where the meta-objective is optimized exactly (even when doing so is prohibitively expensive in practice). For instance, BID37 analyzed an idealized SMD algorithm, showing that even the idealized version suffered from short-horizon bias. In this section, we analyze two idealized versions of APO where an oracle is assumed to minimize the proximal objective exactly in each iteration. In both cases, we obtain strong convergence , suggesting that our proximal objective is a useful target for meta-optimization. We view the problem in output space (i.e., explicitly designing an update schedule for z i). Consider the space of outputs on all training examples; when we train a neural network, we are optimizing over a manifold in this space: DISPLAYFORM0 We assume that f is continuous, so that M is a continuous manifold. Given an oracle that for each iteration exactly minimizes the expectation of proximal objective Eq. 2 over the dataset, we can write one iteration of APO in output space as: DISPLAYFORM1 where z i is ith column of Z, corresponding to the network output on data x i after update, z k,i is the current network output on data x i. We first define the proximal objective as Eq. 4, using the Euclidean distance as the dissimilarity measure, which corresponds to Gauss-Newton algorithm under the linearization of network. With an oracle, this proximal objective leads to projected gradient descent: DISPLAYFORM0 Consider a loss function on one data point (z): R d → R, where d is the dimension of neural network's output. 1 We say the gradient is L-Lipschitz if: DISPLAYFORM1 When the manifold M is smooth, a curve in M is called geodesic if it is the shortest curve connecting the starting and the ending point. We say M have a C-bounded curvature if for each trajectory DISPLAYFORM2 going along some geodesic and v(t) 2 = 1, there is v(t) ≤ C with spectral norm. For each point Z ∈ M, consider the tangent space at point Z as T Z M. We call the projection of ∇ L(Z) onto the hyperplane T Z M as the effective gradient of L at Z ∈ M. It is worth noting that zero effective gradient corresponds to stationary point of the neural network. We have the following theorem stating the global convergence of Eq. 14 to stationary point: Theorem 1. Assume the loss satisfies A1. Furthermore, assume L is lower bounded by L * and has gradient norm upper bound G. Let g * T be the effective gradient in the first T iterations with minimal norm. When the manifold is smooth with C-bounded curvature, with λ ≥ {CG, L 4}, the norm of g * T converges with rate O 1 T as: DISPLAYFORM3 This convergence differs from usual neural network convergence , because here the Lipschitz constants are defined for the output space, so they are known and generally nice. For instance, L = 1 when we use a quadratic loss. In contrast, the gradient is in general not Lipschitz continuous in weight space for deep networks. We further replace the dissimilarity term with: DISPLAYFORM0 which is the second-order approximation of Eq. 8. With a proximal oracle, this variant of APO turns out to be Proximal Newton Method in the output space, if we set λ = 1 2: DISPLAYFORM1 where DISPLAYFORM2 H is the norm with local Hessian as metric. In general, Newton's method can't be applied directly to neural nets in weight space, because it is nonconvex BID4. However, Proximal Newton Method in output space can be efficient given a strongly convex loss function. Consider a loss (z) with µ-strongly convex: DISPLAYFORM3 where z * is the unique minimizer and µ is some positive real number, and L H -smooth Hessian: for any vector v ∈ R d such that v = 1, there is: DISPLAYFORM4 The following theorem suggests the locally fast convergence rate of iteration Eq. 17: Theorem 2. Under assumptions A2 and A3, if the unique minimum Z * ∈ M, then whenever iteration converges to Z *, it converges locally quadratically 2: DISPLAYFORM5 Hence, the proximal oracle achieves second-order convergence for neural network training under fairly reasonable assumptions. Of course, we don't expect practical implementations of APO (or any other practical optimization method for neural nets) to achieve the second-order convergence rates, but we believe the second-order convergence still motivates our proximal objective as a useful target for meta-optimization. Finding good optimization hyperparameters is a longstanding problem BID3. Classic methods for hyperparameter optimization, such as grid search, random search, and Bayesian optimization BID29 BID33, are expensive, as they require performing many complete training runs, and can only find fixed hyperparameter values (e.g., a constant learning rate). Hyperband can reduce the cost by terminating poorly-performing runs early, but is still limited to finding fixed hyperparameters. Population Based Training (PBT) BID10 ) trains a population of networks simultaneously, and throughout training it terminates poorly-performing networks, replaces their weights by a copy of the weights of a better-performing network, perturbs the hyperparameters, and continues training from that point. PBT can find a coarse-grained learning rate schedule, but because it relies on random search, it is far less efficient than gradient-based meta-optimization. There have been a number of approaches to gradient-based adaptation of learning rates. Gradientbased optimization algorithms can be unrolled as computation graphs, allowing the gradients of hyperparameters such as learning rates to be computed via automatic differentiation. BID17 propagate gradients through the full unrolled training procedure to find optimal learning rate schedules offline. Stochastic meta-descent (SMD) adapts hyperparameters online. Hypergradient descent (HD) BID2 takes the gradient of the learning rate with respect to the optimizer update in each iteration, to minimize the expected loss in the next iteration. In particular, HD suffers from short horizon bias BID37, while in Appendix F we show that APO does not. Some authors have proposed learning entire optimization algorithms BID14 BID35 BID1. BID14 view this problem from a reinforcement learning perspective, where the state consists of the objective function L and the sequence of prior iterates {θ t} and gradients {∇ θ L(θ t)}, and the action is the step ∆θ. In this setting, the update rule φ is a policy, which can be found via policy gradient methods BID32. Approaches that learn optimizers must be trained on a set of objective functions {f 1, . . ., f n} drawn from a distribution F; this setup can be restrictive if we only have one instance of an objective function. In addition, the initial phase of training the optimizer on a distribution of functions can be expensive. APO requires only the objective function of interest and finds learning rate schedules in a single training run. In principle, APO could be used to learn a full optimization algorithm; however, learning such an algorithm would be just as hard as the original optimization problem, so one would not expect an out-of-the-box meta-optimizer (such as RMSprop with learning rate 0.001) to work as well as it does for adapting few hyperparameters. In this section, we evaluate APO empirically on a variety of learning tasks; Table 1 gives an overview of the datasets, model architectures, and base optimizers we consider. In our proximal objective, DISPLAYFORM0, h can be any approximation to the loss function (e.g., a linearization); in our experiments, we directly used the loss value h =, as we found this to work well in many settings. As the dissimilarity term D, we used the squared Euclidean norm. We used APO to tune the optimization hyperparameters of four base-optimizers: SGD, SGD with Nesterov momentum (denoted SGDm), RMSprop, and K-FAC. For SGD, the only hyperparameter is the learning rate; we consider both a single, global learning rate, as well as per-layer learning rates. For SGDm, the update rule is given by: DISPLAYFORM1 where g = ∇. Since adapting µ requires considering long-term performance BID31, it is not appropriate to adapt it with a one-step objective like APO. Instead, we just adapt the learning rate with APO as if there's no momentum, but then apply momentum with µ = 0.9 on top of the updates. For RMSprop, the optimizer step is given by: DISPLAYFORM2 We note that, in addition to the learning rate η, we can also consider adapting and the power to which s is raised in the denominator of Eq. 21-we denote this parameter ρ, where in standard RMSprop we have ρ = 1 2. Both and ρ can be interpreted as having a damping effect on the update. K-FAC is an approximate natural gradient method based on preconditioning the gradient by an approximation to the Fisher matrix, θ ← θ − F −1 ∇. For K-FAC, we tune the global learning rate and the damping factor. Meta-Optimization Setup. Throughout this section, we use the following setup for metaoptimization: we use RMSprop as the meta-optimizer, with learning rate 0.1, and perform 1 metaoptimization update for every 10 steps of the base optimization. We show in Appendix E that with this default configuration, APO is robust to the initial learning rate of the base optimizer. Each meta-optimization step takes approximately the same amount of computation as a base optimization step; by performing meta-updates once per 10 base optimization steps, the computational overhead of using APO is just a small fraction more than the original training procedure. Rosenbrock. We first validated APO on the two-dimensional Rosenbrock function, f (x, y) = (1 − x) 2 + 100(y − x 2) 2, with initialization (x, y) = (1, −1.5). We used APO to tune the learning rate of RMSprop, and compared to standard RMSprop with several fixed learning rates. Because this problem is deterministic, we set λ = 0 for APO. FIG0 shows that RMSprop-APO was able to achieve a substantially lower objective value than the baseline RMSprop. The learning rates for each method are shown in FIG0; we found that APO first increases the learning rate to make rapid progress at the start of optimization, and then gradually decreases it as it approaches the local optimum. In Appendix D we show that APO converges quickly from many different locations on the Rosenbrock surface, and in Appendix E we show that APO is robust to the initial learning rate of the base optimizer. Badly-Conditioned Regression. Next, we evaluated APO on a badly-conditioned regression problem BID22, which is intended to be a difficult test problem for optimization algorithms. In this problem, we consider a dataset of input/output pairs {(x, y)}, where the outputs are given by y = Ax, where A is an ill-conditioned matrix with κ(A) = 10 10. The task is to fit a two-layer linear model f (x) = W 2 W 1 x to this data; the loss to be minimized is FIG0 (c) compares the performance of RMSprop with a hand-tuned fixed learning rate to the performance of RMSprop-APO, with learning rates shown in FIG0. Again, the adaptive learning rate enabled RMSprop-APO to achieve a loss value orders of magnitude smaller than that achieved by RMSprop with a fixed learning rate. DISPLAYFORM0 For each of the real-world datasets we consider-MNIST, CIFAR-10, CIFAR-100, SVHN, and FashionMNIST-we chose the learning rates for the baseline optimizers via grid searches: for SGD and SGDm, we performed a grid search over learning rates {0.1, 0.01, 0.001}, while for RMSprop, we performed a grid search over learning rates {0.01, 0.001, 0.0001}. For SGD-APO and SGDm-APO, we set the initial learning rate to 0.1, while for RMSprop-APO, we set the initial learning rate to 0.0001. These initial learning rates are used for convenience; we show in Appendix E that APO is robust to the choice of initial learning rate. The only hyperparameter we consider for APO is the value of λ: for SGD-APO and SGDm-APO, we select the best λ from a grid search over {0.1, 0.01, 1e-3}; for RMSprop, we choose λ from a grid search over {0.1, 0.01, 1e-3, 1e-4, 1e-5, 0}. Note that because each value of λ yields a learning rate schedule, performing a search over λ is much more effective than searching over fixed learning rates. In particular, we show that the adaptive learning rate schedules discovered by APO are competitive with manual learning rate schedules. First, we compare SGD and RMSprop with their APO-tuned variants on MNIST, and show that APO outperforms fixed learning rates. As the classification network for MNIST, we used a twolayer MLP with 1000 hidden units per layer and ReLU nonlinearities. We trained on mini-batches of size 100 for 100 epochs. SGD with APO. We used APO to tune the global learning rate of SGD and SGD with Nesterov momentum (denoted SGDm) on MNIST, where the momentum is fixed to 0.9. For baseline SGDm, we used learning rate 0.01, while for baseline SGD, we used both learning rates 0.1 and 0.01. The training curve of SGD with learning rate 0.1 almost coincides with that of SGDm with learning rate 0.01. For SGD-APO, the best λ was 1e-3, while for SGDm-APO, the best λ was 0.1. A comparison of the algorithms is shown in FIG1. APO substantially improved the training loss for both SGD and SGDm. We compare K-FAC with a fixed learning rate and a manual learning rate schedule to APO, used to tune 1) the learning rate; and 2) both the learning rate and damping coefficient. RMSprop with APO. Next, we used APO to tune the global learning rate of RMSprop. For baseline RMSprop, the best fixed learning rate was 1e-4, while for RMSprop-APO, the best λ was 1e-5. FIG1 (b) compares RMSprop and its APO-tuned variant on MNIST. RMSprop-APO achieved a training loss about three orders of magnitude smaller than the baseline. We trained a 34-layer residual network (ResNet34) BID7 on CIFAR-10 , using mini-batches of size 128, for 200 epochs. We used batch normalization and standard data augmentation (horizontal flipping and cropping). For each optimizer, we compare APO to 1) fixed learning rates; and 2) manual learning rate decay schedules. SGD with APO. For SGD, we used both learning rates 0.1 and 0.01 since both work well. For SGD with momentum, we used learning rate 0.01. We also consider a manual schedule for both SGD and SGDm: starting from learning rate 0.1, and we decay it by a factor of 5 every 60 epochs. For the APO variants, we found that λ=1e-3 was best for SGD, while λ = 0.1 was best for SGDm. As shown in FIG1 (c), APO not only accelerates training, but also achieves higher accuracy on the test set at the end of training. RMSprop with APO. For RMSprop, we use fixed learning rates 1e-3 and 1e-4, and we consider a manual learning rate schedule in which we initialize the learning rate to 1e-3 and decay by a factor of 5 every 60 epochs. For RMSprop-APO, we used λ = 1e-3. The training curves, test accuracies, and learning rates for RMSprop and RMSprop-APO on CIFAR-10 are shown in FIG1 (d). We found that APO achieved substantially lower training loss than fixed learning rates, and was competitive with the manual decay schedule. In particular, both the final training loss and final test accuracy achieved by APO are similar to those achieved by the manual schedule. K-FAC with APO. We also used APO to tune the learning rate and damping coefficient of K-FAC. Similarly to the previous experiments, we use K-FAC to optimize a ResNet34 on CIFAR-10. We used mini-batches of size 128 and trained for 100 epochs. For the baseline, we used a fixed learning rate of 1e-3 as well as a decay schedule with initial learning rate 1e-3, decayed by a factor of 10 at epochs 40 and 80. For APO, we used λ = 1e-2. In experiments where the damping is not tuned, it is fixed at 1e-3. The are shown in FIG2. We see that K-FAC-APO performs competitively with the manual schedule when tuning just the global learning rate, and that both training loss and test accuracy improve when we tune both the learning rate and damping coefficient simultaneously. Next, we evaluated APO on the CIFAR-100 dataset. Similarly to our experiments on CIFAR-10, we used a ResNet34 network with batch-normalization and data augmentation, and we trained on minibatches of size 128, for 200 epochs. We compared SGD-APO/SGDm-APO to standard SGD/SGDm using a fixed learning rate found by grid search; a custom learning rate schedule in which the learning rate is decayed by a factor of 5 at epochs 60, 120, and 180. We set λ = 1e-3 for SGD-APO and λ = 0.1 for SGDm-APO. Figure 4 shows the training loss, test accuracy, and the tuned learning rate. It can be seen that APO generally achieves smaller training loss and higher test accuracy. We also used APO to train an 18-layer residual network (ResNet18) with batch normalization on the SVHN dataset BID21. Here, we used the standard train and test sets, without additional training data. We used mini-batches of size 128 and trained our networks for 160 epochs. We compared APO to 1) fixed learning rates, and 2) a manual schedule in which we initialize the learning rate to 1e-3 and decay by a factor of 10 at epochs 80 and 120. We show the training loss, test accuracy, and learning rates for each method in Figure 5. Here, RMSprop-APO achieves similar training loss to the manual schedule, and obtains higher test accuracy than the schedule. We also see that the learning rate adapted by APO spans two orders of magnitude, similar to the manual schedule. Figure 6: SGD with weight decay compared to SGD-APO without weight decay, on CIFAR-10. BID9 is a widely used technique to speed up neural net training. Networks with BN are commonly trained with weight decay. It was shown that the effectiveness of weight decay for networks with BN is not due to the regularization, but due to the fact that weight decay affects the scale of the network weights, which changes the effective learning rate (; BID8 BID35 . In particular, weight decay decreases the scale of the weights, which increases the effective learning rate; if one uses BN without regularizing the norm of the weights, then the weights can grow without bound, pushing the effective learning rate to 0. Here, we show that using APO to tune learning rates allows for effective training of BN networks without using weight decay. In particular, we compared SGD-APO without weight decay and SGD with weight decay 5e-4. Figure 6 shows that SGD-APO behaved better than SGD with a fixed learning rate, and achieved comparable performance as SGD with a manual schedule. We introduced amortized proximal optimization (APO), a method for online adaptation of optimization hyperparameters, including global and per-layer learning rates, and damping parameters for approximate second-order methods. We evaluated our approach on real-world neural network optimization tasks-training MLP and CNN models-and showed that it converges faster and generalizes better than optimal fixed learning rates. Empirically, we showed that our method overcomes short horizon bias and performs well with sensible default values for the meta-optimization parameters. Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281, 2018.A PROOF OF THEOREM 1We first introduce the following lemma:Lemma 1. Assume the manifold is smooth with C-bounded curvature, the gradient norm of loss function L is upper bounded by G. If the effective gradient at point Z k ∈ M is g k, then for any DISPLAYFORM0 Proof. We construct the Z satisfying the above inequality. Consider the following point in R d: DISPLAYFORM1 We show that Z is a point satisfying the inequality in the lemma. Firstly, we notice that DISPLAYFORM2 This is because when we introduce the extra curveṽ DISPLAYFORM3 Here we use the fact thatv = 0 and v ≤ C. Therefore we have DISPLAYFORM4 Here the first equality is by introducing the extra Y, the first inequality is by triangle inequality, the second equality is by the definition of g k being ∇ Z L(Z k) projecting onto a plane, the second inequality is due to the above bound of Y − Z, the last inequality is due to DISPLAYFORM5, there is therefore DISPLAYFORM6 which completes the proof. Proof. For the ease of notation, we denote the effective gradient at iteration k as g k. For one iteration, there is DISPLAYFORM0 Here the first inequality is due to the Lipschitz continuity and the fact that total loss equals to the sum of all loss functions, and the second inequality is due to λ ≥ L 4, the third inequality is due to Lemma 1 with γ = So we have DISPLAYFORM1 Telescoping, there is DISPLAYFORM2 Proof. For notational convenience, we think of Z as a vector rather than a matrix in this proof. The Hessian ∇ 2 L(Z) is therefore a block diagonal matrix, where each block is the Hessian of loss on a single data. First, we notice the following equation: DISPLAYFORM0 is the norm of vector v defined by the positive definite matrix A. DISPLAYFORM1, therefore also positive definite. As a of the above equivalence, one step of Proximal Newton Method can be written as: DISPLAYFORM2.Since Z * ∈ M by assumption, there is: DISPLAYFORM3 Now we have the following inequality for one iteration: DISPLAYFORM4 Here the first inequality is because of triangle inequality, the second inequality is due to the previous , the equality is because ∇ L(Z *) = 0, the last inequality is because of the strong convexity. By the Lipschitz continuity of the Hessian, we have: DISPLAYFORM5 Therefore, we have: Here we highlight the ability of APO to tune several optimization hyperparameters simultaneously. We used APO to adapt all of the RMSprop hyperparameters {η, ρ,}. As shown in FIG6 (a), tuning ρ and in addition to the learning rate η can stabilize training. We also used APO to adapt per-layer learning rates. FIG6 (b) shows the per-layer learning rates tuned by APO, when using SGD on MNIST. FIG6 (c) uses APO to tune per-layer learning rate of RMSprop on MNIST. FIG7 shows the adaptation of the additional ρ and hyperparameters of RMSprop, for training an MLP on MNIST. Tuning per-layer learning rates is a difficult optimization problem, and we found that it was useful to use a smaller meta learning rate of 0.001 and perform meta-updates more frequently. DISPLAYFORM6 We also used APO to train a convolutional network on the FashionMNIST dataset BID38. The network we use consists of two convolutional layers with 16 and 32 filters respectively, both with kernel size 5, followed by a fully-connected layer. The are shown in FIG6 (d), where we also compare K-FAC to hand-tuned RMSprop and RMSprop-APO on the same problem. We find that K-FAC with a fixed learning rate outperforms RMSprop-APO, while K-FAC-APO substantially outperforms K-FAC. The are shown in FIG6 (d). We also show the adaptation of both the learning rate and damping coefficient for K-FAC-APO in FIG6 (d). In this section, we present additional experiments on the Rosenbrock problem. We show in FIG9 that APO converges quickly from different starting points on the Rosenbrock surface. In this section we show that APO is robust to the choice of initial learning rate of the base optimizer. With a suitable meta learning rate, APO quickly adapts many different initial learning rates to the same range, after which the learning rate adaptation follows a similar trajectory. Thus, APO helps to alleviate the difficulty involved in selecting an initial learning rate. First, we used RMSprop-APO to optimize Rosenbrock, starting with a wide range of initial learning rates; we see in FIG0 that the training losses and learning rates are nearly identical between all these experiments. Next, we trained an MLP on MNIST and ResNet34 on CIFAR-10 using RMSprop-APO, with the learning rate of the base optimizer initialized to 1e-2, 1e-3, 1e-4, 1e-5, 1e-6, and 1e-7. We used the default meta learning rate 0.1. As shown in FIG0, the training loss, test accuracy, and learning rate adaptation are nearly identical using these initial learning rates, which span 5 orders of magnitude. In this section we apply APO to the noisy quadratic problem investigated in BID37 BID23, and demonstrate that APO overcomes the short horizon bias problem. We optimize a quadratic function f (x) = x T Hx, where x ∈ R 1000, H is a diagonal matrix H = diag{h 1, h 2, · · ·, h 1000}, with eigenvalues h i evenly distributed in interval [0.01, 1]. Initially, we set x with each dimension being 100. For each iteration, we can access the noisy version of the function, i.e., the gradient and function value of functioñ DISPLAYFORM0 Here c is the vector of noise: each dimension of c is independently randomly sampled from a normal distribution at each iteration, and the variance of dimension i is set to be 1 hi. For SGD, we consider the following four learning rate schedules: optimal schedule, exponential schedule, linear schedule and a fixed learning rate. For SGD with APO, we directly use functionf as the loss approximation h, use Euclidean distance norm square as the dissimilarity term D, and consider the following schedules for λ: optimal schedule(with λ ≥ 0), exponential schedule, linear schedule and a fixed λ. We calculate the optimal parameter for each schedule of both algorithms so as to achieve a minimal function value at the end of 300 iterations. We optimize the schedules with 10000 steps of Adam and learning rate 0.001 after unrolling the entire 300 iterations. The function values at the end of 300 iterations with each schedule are shown in Table 2. FIG0 plots the training loss and learning rate of SGD during the 300 iterations under optimal schedule, figure 13 plots the training loss and λ under optimal schedule for SGD with APO. It can be seen that SGD with APO achieves almost the same training loss as optimal SGD for noisy quadratics task. This indicates that APO doesn't suffer from the short-horizon bias mentioned in BID37. Adam BID11 is an adaptive optimization algorithm widely used for training neural networks, which can be seen as RMSProp with momentum. The update rule is given by: DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 where g t = ∇ θ t (θ t−1).Similar to SGD with Nesterov momentum, we fixed the β 1 and β 2 for Adam, and used APO to tune the global learning rate η. We tested Adam-APO with a ResNet34 network on CIFAR-10 dataset, and compared it with both Adam with fixed learning rate and Adam with a learning rate schedule where the learning rate is initialized to 1e-3 and is decayed by a factor of 5 every 60 epochs. Similarly to SGD with momentum, we found that Adam generally benefits from larger values of λ. Thus, we recommend performing a grid search over λ values from 1e-2 to 1. As shown in FIG0, APO improved both the training loss and test accuracy compared to the fixed learning rate, and achieved comparable performance as the manual learning rate schedule. Population-based training (PBT) BID10 is an approach to hyperparameter optimization that trains a population of N neural networks simultaneously: each network periodically evaluates its performance on a target measure (e.g., the training loss); poorly-performing networks can exploit better-performing members of the population by cloning the weights of the better network, copying and perturbing the hyperparameters used by the better network, and resuming training. In this way, a single model can essentially experience multiple hyperparameter settings during training; in particular, we are interested in evaluating the learning rate schedule found using PBT.Here, we used PBT to tune the learning rate for RMSprop, to optimize a ResNet34 model on CIFAR-10. For PBT, we used a population of size 4 (which we found to perform better than a population of size 10), and used a perturbation strategy that consists of randomly multiplying the learning rate by either 0.8 or 1.2. In PBT, one can specify the probability with which to re-sample a hyperparameter value from an underlying distribution. We found that it was critical to set this to 0; otherwise, the learning rate could jump from small to large values and cause instability in training. FIG0 compares PBT with APO; we show the best training loss achieved by any of the models in the PBT population, as a function of wall-clock time. For a fair comparison between these methods, we ran both PBT and APO using 1 GPU. We see that APO outperforms PBT, achieving a training loss an order of magnitude smaller than PBT, and achieves the same test accuracy, much more quickly.

Represent the TLDR for finding the corresponding scientific article:

We introduce amortized proximal optimization (APO), a method to adapt a variety of optimization hyperparameters online during training, including learning rates, damping coefficients, and gradient variance exponents.

Represent this text to find a much much longer article about it with say hundreds-thousands of words