Question: represent the scientific article to find a short tldr

Question: In adversarial attacks to machine-learning classifiers, small perturbations are added to input that is correctly classified. The perturbations yield adversarial examples, which are virtually indistinguishable from the unperturbed input, and yet are misclassified. In standard neural networks used for deep learning, attackers can craft adversarial examples from most input to cause a misclassification of their choice. We introduce a new type of network units, called RBFI units, whose non-linear structure makes them inherently resistant to adversarial attacks. On permutation-invariant MNIST, in absence of adversarial attacks, networks using RBFI units match the performance of networks using sigmoid units, and are slightly below the accuracy of networks with ReLU units. When subjected to adversarial attacks based on projected gradient descent or fast gradient-sign methods, networks with RBFI units retain accuracies above 75%, while ReLU or Sigmoid see their accuracies reduced to below 1%. Further, RBFI networks trained on regular input either exceed or closely match the accuracy of sigmoid and ReLU network trained with the help of adversarial examples. The non-linear structure of RBFI units makes them difficult to train using standard gradient descent. We show that RBFI networks of RBFI units can be efficiently trained to high accuracies using pseudogradients, computed using functions especially crafted to facilitate learning instead of their true derivatives. Machine learning via deep neural networks has been remarkably successful in a wide range of applications, from speech recognition to image classification and language processing. While very successful, deep neural networks are affected by adversarial examples: small, especially crafter modifications of correctly classified input that are misclassified BID20 ). The trouble with adversarial examples is twofold. The modifications to regular input are so small as to be difficult or impossible to detect for a human: this has been shown both in the case of images BID20; BID14 ) and sounds BID9; BID5 ). Further, the adversarial examples are in some measure transferable from one neural network to another BID7; BID14; BID16; BID22 ), so they can be crafted even without precise knowledge of the weights of the target neural network. At a fundamental level, it is hard to provide guarantees about the behavior of a deep neural network, when every correctly classified input is tightly encircled by very similar, yet misclassified, inputs. Thus far, the approach for obtaining neural networks that are more resistant to adversarial attacks has been to feed to the networks, as training data, an appropriate mix of the original training data, and adversarial examples BID7; BID12 ). In training neural networks using adversarial examples, if the examples are generated via efficient heuristics such as the fast gradient sign method, the networks learn to associate the specific adversarial examples to the original input from which they were derived, in a phenomenon known as label leaking BID10; BID12; BID21 ). This does not in increased resistance to general adversarial attacks BID12; BID4 ). If the adversarial examples used in training are generated via more general optimization techniques, as in BID12 ), networks with markedly increased resistance to adversarial attacks can be obtained, at the price of a more complex and computationally expensive training regime, and an increase in required network capacity. We pursue here a different approach, proposing the use of neural network types that are, due to their structure, inherently impervious to adversarial attacks, even when trained on standard input only. In BID7 ), the authors connect the presence of adversarial examples to the (local) linearity of neural networks. In a purely linear form n i=1 x i w i, we can perturb each x i by, taking x i + if w i > 0, and x i − if w i < 0. This causes an output perturbation of magnitude n i=1 |w i |, or nw forw the average modulus of w i. When the number of inputs n is large, as is typical of deep neural networks, a small input perturbation can cause a large output change. Of course, deep neural networks are not globally linear, but the insight of BID7 ) is that they may be sufficiently locally linear to allow adversarial attacks. Following this insight, we develop networks composed of units that are highly non-linear. The networks on which we settled after much experimentation are a variant of the well known radial basis functions (RBFs) BID0; BID6 BID15 ); we call our variant RBFI units. RBFI units are similar to classical Gaussian RBFs, except for two differences that are crucial in obtaining both high network accuracy, and high resistance to attacks. First, rather than being radially symmetrical, RBFIs can scale each input component individually; in particular, they can be highly sensitive to some inputs while ignoring others. This gives an individual RBFI unit the ability to cover more of the input space than its symmetrical variants. Further, the distance of an input from the center of the Gaussian is measured not in the Euclidean, or 2, norm, but in the infinity norm ∞, which is equal to the maximum of the differences of the individual components. This eliminates all multi-input linearity from the local behavior of a RBFI: at any point, the output depends on one input only; the n in the above discussion is always 1 for RBFIs, so to say. The "I" in RBFI stands for the infinity norm. Using deeply nonlinear models is hardly a new idea, but the challenge has been that such models are typically difficult to train. Indeed, we show that networks with RBFI units cannot be satisfactorily trained using gradient descent. To get around this, we show that the networks can be trained efficiently, and to high accuracy, using pseudogradients. A pseudogradient is computed just as an ordinary gradient, except that we artificially pretend that some functions have a derivative that is different from the true derivative, and especially crafted to facilitate training. In particular, we use pseudoderivatives for the exponential function, and for the maximum operator, that enter the definition of Gaussian RBFI units. Gaussians have very low derivative away from their center, which makes training difficult; our pseudoderivative artificially widens the region of detectable gradient around the Gaussian center. The maximum operator appearing in the infinity norm has non-zero derivative only for one of its inputs at a time; we adopt a pseudogradient that propagates back the gradient to all of its inputs, according to their proximity in value to the maximum input. Tampering with the gradient may seem unorthodox, but methods such as AdaDelta BID23 ), and even gradient descent with momentum, cause training to take a trajectory that does not follow pure gradient descent. We simply go one step further, devising a scheme that operates at the granularity of the individual unit. We show that with these two changes, RBFIs can be easily trained with standard random (pseudo)gradient descent methods, yielding networks that are both accurate, and resistant to attacks. To conduct our experiments, we have implemented RBFI networks on top of the PyTorch framework BID18 ). The code will be made available in a final version of the paper. We consider permutation invariant MNIST, which is a version of MNIST in which the 28 × 28 pixel images are flattened into a one-dimensional vector of 784 values and fed as a feature vector to neural networks BID7 ). On this test set, we show that for nets of 512,512,512,10 units, RBFI networks match the classification accuracy of networks of sigmoid units ((96.96 ± 0.14)% for RBFI vs. (96.88 ± 0.15)% for sigmoid), and are close to the performance of network with ReLU units ((98.62 ± 0.08)%). When trained over standard training sets, RBFI networks retain accuracies over 75% for adversarial attacks that reduce the accuracy of ReLU and sigmoid networks to below 2% (worse than random). We show that RBFI networks trained on normal input are superior to ReLU and sigmoid networks trained even with adversarial examples. Our experimental can be summarized as follows:• In absence of adversarial attacks, RBFI networks match the accuracy of sigmoid networks, and are slightly lower in accuracy than ReLU networks.• When networks are trained with regular input only, RBFI networks are markedly more resistant to adversarial attacks than sigmoid or ReLU networks.• In presence of adversarial attacks, RBFI networks trained on regualar input provide higher accuracy than sigmoid or ReLU networks, even when the latter are trained also on adversarial examples, and even when the adversarial examples are obtained via general projected gradient descent BID12 ).• RBFI networks can be successfully trained with pseudogradients; the training via standard gradient descent yields instead markedly inferior .• Appropriate regularization helps RBFI networks gain increased resistance to adversarial attacks. Much work remains to be done, including experimenting with convolutional networks using RBFI units for images. However, the seem promising, in that RBFI seem to offer a viable alternative to current adversarial training regimes in achieving robustness to adversarial attacks. Adversarial examples were first noticed in BID20, where they were generated via the solution of general optimization problems. In BID7, a connection was established between linearity and adversarial attacks. A fully linear form n i=1 x i w i can be perturbed by using x i + sign(w i), generating an output change of magnitude · n i=1 |w i |. In analogy, BID7 introduced the fast gradient sign method (FGSM) method of creating adversarial perturbations, by taking DISPLAYFORM0, where ∇ i L is the loss gradient with respect to input i. The work also showed how adversarial examples are often transferable across networks, and it asked the question of whether it would be possible to construct non-linear structures, perhaps inspired by RBFs, that are less linear and are more robust to adversarial attacks. This entire paper is essentially a long answer to the conjectures and suggestions expressed in BID7.It was later discovered that training on adversarial examples generated via FGSM does not confer strong resistance to attacks, as the network learns to associate the specific examples generated by FGSM to the original training examples in a phenomenon known as label leaking BID10; BID12; BID21. The FGSM method for generating adversarial examples was extended to an iterative method, I-FGSM, in BID9. In BID21, it is shown that using small random perturbations before applying FSGM enhances the robustness of the ing network. The network trained in BID21 using I-FSGM and ensemble method won the first round of the NIPS 2017 competition on defenses with respect to adversarial attacks. Carlini and Wagner in a series of papers show that training regimes based on generating adversarial examples via simple heuristics, or combinations of these, in general fail to convey true resistance to attacks BID3 b). They further advocate measuring the resistance to attacks with respect to attacks found via more general optimization processes. In particular, FGSM and I-FGSM rely on the local gradient, and training techniques that break the association between the local gradient and the location of adversarial examples makes networks harder to attack via FGSM and I-FGSM, without making the networks harder to attack via general optimization techniques. In this paper, we follow this suggestion by using a general optimization method, projected gradient descent (PGD), to generate adversarial attacks and evaluate network robustness. BID2 also shows that the technique of defensive distillation, which consists in appropriately training a neural network on the output of another BID17, protects the networks from FGSM and I-FGSM attacks, but does not improve network resistance in the face of general adversarial attacks. In BID12 it is shown that by training neural networks on adversarial examples generated via PGD, it is possible to obtain networks that are genuinely more resistant to adversarial examples. The price to pay is a more computationally intensive training, and an increase in the network capacity required. We provide an alternative way of reaching such resistance, one that does not rely on a new training regime. In BID7, the adversarial attacks are linked to the linearity of the models. Following this insight, we seek to use units that do not exhibit a marked linear behavior, and specifically, units which yield small output variations for small variations of their inputs measured in infinity norm. A linear form g(x) = i x i w i represents the norm-2 distance of the input vector x to a hyperplane perpendicular to vector w, scaled by |w| and its orientation. It is not advantageous to simply replace this norm-2 distance with an infinity-norm distance, as the infinity-norm distance between a point and a plane is not a very useful concept. It is preferable to consider the infinity-norm distance between points. Hence, we define our units as variants of the classical Gaussian radial basis functions BID1 BID15 ). We call our variant RBFI, to underline the fact that they are built using infinity norm. An RBFI unit U(u, w) for an input in IR n is parameterized by two vectors of weights u = u 1,..., u n and w = w 1,..., w n Given an input x ∈ IR n, the unit produces output DISPLAYFORM0 where is the Hadamard, or element-wise, product. In, the vector w is a point from which the distance to x is measured in infinity norm, and the vector u provides scaling factors for each coordinate. Without loss of expressiveness, we require the scaling factors to be non-negative, that is, u i ≥ 0 for all 1 ≤ i ≤ n. The scaling factors provide the flexibility of disregarding some inputs x i, by having u i ≈ 0, while emphasizing the influence of other inputs. Writing out explicitly, we have: DISPLAYFORM1 The output of a RBFI unit is close to 1 only when x is close to w in the coordinates that have large scaling factors. Thus, the unit is reminiscent of an And gate, with normal or complemented inputs, which outputs 1 only for one value of its inputs. Logic circuits are composed both of And and of Or gates. Thus, we introduce an Or RBFI unit by U OR (u, w) = 1 − U(u, w). We construct neural networks out of RBFI units using layers consisting of And units, layers consisting of Or units, and mixed layers, in which the unit type is chosen at random at network initialization. To form an intuitive idea of why networks with RBFI units might resist adversarial attacks, it is useful to compute the sensitivity of individual units to such attacks. For x ∈ IR n and > 0, let B (x) = {x | x − x ∞ ≤} be the set of inputs within distance from x in infinity norm. Given a function f: IR n → IR, we call its sensitivity to adversarial attacks the quantity: DISPLAYFORM2 The sensitivity represents the maximum change in output we can obtain via an input change within in infinity norm, as a multiple of itself. For a single ReLU unit with weight vector w, the sensitivity is given by s = n i=1 |w i | = w 1. This formula can be understood by noting that the worst case for a ReLU unit corresponds to considering an input x for which the output is positive, and taking x i = x i + if w i > 0, and BID7 ). Similarly, for a single sigmoid unit with weight vector w, we have s = 1 4 w 1, where the factor of 1/4 corresponds to the maximum derivative of the sigmoid. For a RBFI unit U(u, w), on the other hand, from we have: DISPLAYFORM3 DISPLAYFORM4 ∞. Thus, the sensitivity of ReLU and Sigmoid units increases linearly with input size, whereas the sensitivity of RBFI units is essentially constant with respect to input size. These formulas can be extended to bounds for whole networks. For a ReLU network with K 0 inputs and layers of DISPLAYFORM5 where w (k) ij is the weight for input i of unit j of layer k, for 1 ≤ k ≤ K M. We can compute an upper boundŝ for the sensitivity of the network via: DISPLAYFORM6 The formula for Sigmoid networks is identical except for the 1/4 factors. Using similar notation, for RBFI networks we have: DISPLAYFORM7 By connecting in a simple way the sensitivity to attacks to the network weights, these formulas suggest the possibility of using weight regularization to achieve robustness: by adding cŝ to the loss function for c > 0, we might be able to train networks that are both accurate and robust to attacks. We will show in Section 6.5 that such a regularization helps train more robust RBFI networks, but it does not help train more robust ReLU networks. The non-linearities in make neural networks containing RBFI units difficult to train using standard gradient descent, as we will show experimentally. The problem lies in the shape of Gaussian functions. Far from its peak for x = w, a function of the form is rather flat, and its derivative may not be large enough to cause the vector of weights w to move towards useful places in the input space during training. To obtain networks that are easy to train, we replace the derivatives for exp and max with alternate functions, which we call pseudoderivatives. These pseudoderivatives are then used in the chain-rule computation of the loss gradient in lieu of the true derivatives, yielding a pseudogradient. Exponential function. In computing the partial derivatives of via the chain rule, the first step consists in computing d dz e −z, which is of course equal to −e −z. The problem is that −e −z is very close to 0 when z is large, and z in is u (x − w) 2 ∞, which can be large. Hence, in the chain-rule computation of the gradient, we replace −e −z with the alternate "pseudoderivative" −1/ √ 1 + z, which has a much longer tail. Max. The gradient of y = max 1≤i≤n z i, of course, is given by ∂y ∂zi = 1 if z i = y, and ∂y ∂zi = 0 if z i < y. The problem is that this transmits feedback only to the largest input(s). This slows down training and can create instabilities. We use as pseudoderivative e zi−y, so that some of the feedback is transmitted to inputs z i that approach y. One may be concerned that by using the loss pseudogradient as the basis of optimization, rather than the true loss gradient, we may converge to solutions where the pseudogradient is null, and yet, we are not at a minimum of the loss function. This can indeed happen. We experimented with switching to training with true gradients once the pseudogradients failed to yield improvements; this increased the accuracy on the training set, but barely improved it on the testing set. It is conceivable that more sophisticated ways of mixing training with regular and pseudo-gradients would allow training RBFI networks to higher accuracy on the testing set. Given a correctly classified input x for a network, and a perturbation size > 0, an input x is an adversarial example for if x is misclassified, and x − x ∞ ≤ η. Consider a network trained with cost function J(θ, x, y), where θ is the set of network parameters, x is the input, and y is the output. Indicate with ∇ x J(θ, x, y) the gradient of J wrt its input x computed at values x of the inputs, parameters θ, and output y. For each input x belonging to the testing set, given a perturbation amount > 0, we produce adversarial examplesx with x −x ∞ ≤ using the following techniques. Fast Gradient Sign Method (FGSM) BID7 ). If the cost were linear around x, the optimal -max-norm perturbation of the input would be given by sign(∇ x J(θ, x, y) ). This suggests taking as adversarial example: BID9 ). Instead of computing a single perturbation of size using the sign of the gradient, we apply M perturbations of size /M, each computed from the endpoint of the previous one. Precisely, the attack computes a sequencẽ x 0,x 1,...,x M, wherex 0 = x, and where eachx i+1 is obtained, for 0 ≤ i < M, by: DISPLAYFORM0 We then takex =x M as our adversarial example. This attack is more powerful than its singlestep version, as the direction of the perturbation can better adapt to non-linear cost gradients in the neighborhood of x BID9 ).Projected Gradient Descent (PGD) BID12 ). For an input x ∈ IR n and a given maximum perturbation size > 0, we consider the set B (x) ∩ n of valid inputs around x, and we perform projected gradient descent (PGD) in B (x) ∩ n of the negative loss with which the network has been trained (or, equivalently, projected gradient ascent wrt. the loss). By following the gradient in the direction of increasing loss, we aim at finding mis-classified inputs in B (x)∩ n. As the gradient is non-linear, to check for the existence of adversarial attacks we perform the descent multiple times, each time starting from a point of B (x) ∩ n chosen uniformly at random. Noise. In addition to the above adversarial examples, we will study the robustness of our networks by feeding them inputs affected by noise. For a testing input x and a noise amount ∈, we produce an -noisy versionx viax = (1 −)x + χ, where χ is a random element of the input space, which for MNIST is n. We have implemented FGSM, I-FGSM, and PGD attacks for RBFI both relying on standard gradients, and relying on pseudogradients. In the , we denote pseudogradient-based via RBFI [psd]. The idea is that if pseudogradients are useful in training, they are likely to be useful also in attacking the networks, and an adversary may well rely on them. BID4 show that many networks that resist FGSM and I-FGSM attacks can still be attacked by using general optimization-based methods. Thus, they and argue that the evaluation of attack resistance should include general optimization methods; the PGD attacks we consider are an example of such methods. 6.1 EXPERIMENTAL SETUP Implementation. We implemented RBFI networks in the PyTorch framework BID18 ). In order to extend PyTorch with a new function f, it is necessary to specify the function behavior f (x), and the function gradient ∇ x f. To implement RBFI, we extend PyTorch with two new functions: a LargeAttractorExp function, with forward behavior e −x and backward gradient propagation according to −1/ √ 1 + x, and SharedFeedbackMax, with forward behavior y = max n i=1 x i and backward gradient propagation according to e xi−y. These two functions are used in the definition of RBFI units, as per FORMULA2, with the AutoGrad mechanism of PyTorch providing backward (pseudo)gradient propagation for the complete networks. Dataset. We use the MNIST dataset BID11 ) for our experiments, following the standard setup of 60,000 training examples and 10,000 testing examples. Each digit image was flattened to a one-dimensional feature vector of length 28 × 28 = 784, and fed to a fully-connected neural network; this is the so-called permutation-invariant MNIST.Neural networks. We compared the accuracy of the following fully-connected network structures.• ReLU networks BID13 ) whose output is fed into a softmax, and the network is trained via cross-entropy loss. Table 1: Performance of 512-512-512-10 networks for MNIST testing input, and for input corrupted by adversarial attacks and noise computed with perturbation size = 0.3.• Sigmoid networks trained with square-error loss.• RBFI networks, trained using square-error loss. For a RBFI network with m layers, we denote its type as RBFI(K 1, . . ., K m | t 1, . . ., t m), where K 1,..., K m are the numbers of units in each layer, and where the units in layer i are And units if t i = ∧, Or units if t i = ∨, and are a random mix of And and Or units if t m = *.Square-error loss worked as well or better than other loss functions for Sigmoid and RBFI networks. Unless otherwise noted, we use networks with layers 512, 512, 512, and 10 units, and in case of RBFI networks, we used geometry RBFI(512, 512, 512, 10 | ∧, ∨, ∧, ∨). For RBFI networks we use a bound of [0.01, 3] for the components of the u-vectors, and of for the w-vectors, the latter corresponding to the value range of MNIST pixels. We experimented with RBFI networks with various geometries, and we found the performance differences to be rather small, for reasons we do not yet fully understand. We trained all networks with the AdaDelta optimizer BID23 ), which yielded good for all networks considered. Attacks. We applied FGSM, I-FGSM, and noise attacks to the whole test set. In I-FGSM attacks, we performed 10 iterations of. As PGD attacks are considerably more computationally intensive, we apply them to one run only, and we compute the performance under PGD attacks for the first 5,000 examples in the test set. For each input x in the test set, we perform 20 searches, or restarts. In each search, we start from a random point in B (x) and we perform 100 steps of projected gradient descent using the AdaDelta algorithm to tune step size; if at any step a misclassified example is generated, the attack is considered successful. In Table 1 we summarize the on the accuracy and resistance to adversarial examples for networks trained on the standard MNIST training set. The are computed from 10 training runs for ReLU and Sigmoid networks, and from 5 runs for RBFI and RBFI [psd]. In each run we used different seeds for the random generator used for weight initialization; each run consisted of 30 training epochs. In a of the form a ± e, a is the percentage accuracy, and e is the standard deviation in the accuracy of the individual runs. In absence of perturbations, RBFI networks lose (1.66 ± 0.21)% performance compared to ReLU networks (from (98.62±0.07)% to (96.96±0.14)%), and perform comparably to sigmoid networks (the difference is below the standard deviation of the ). When perturbations are present, in the form of adversarial attacks or noise, the performance of RBFI networks is superior. We note that the FGSM and I-FGSM attacks performed using regular gradients are not effective against RBFI networks. This phenomenon is called gradient masking: the gradient in proximity of valid inputs offers little information about the possible location of adversarial examples BID4. Pseudogradients do avoid gradient masking, and indeed the most effective attack against RBFI networks is I-FGSM performed using pseudogradients, which lowers the accuracy to (78.92 ± 1.91)% for = 0.3. Including adversarial examples in the training set is the most common method used to make neural networks more resistant to adversarial attacks BID7; BID12 ). We explored whether ReLU and Sigmoid networks trained via a mix of normal and adversarial exam- ples offer a resistance to adversarial attacks compared to that offered by RBFI networks trained on standard examples only. For brevity, we omit the for Sigmoid networks, as they were consistently inferior to those for ReLU networks. We compared the performance of a RBFI network with that of ReLU network trained normally (indicated simply by ReLU), and with ReLU networks trained as follows:• ReLU(FGSM) and ReLU(I-FSGM): for each (x, t) in the training set, we construct an adversarial examplex via or FORMULA10, and we feed both (x, t) and (x, t) to the network for training.• ReLU(PGD): for each (x, t) in the training set, we perform 100 steps of projected gradient descent from a point chosen at random in B (x) ∩ n; denoting by x the ending point of the projected gradient descent, we feed both (x, t) and (x, t) to the network for training. We generated adversarial examples for training for = 0.3, which is consistent with BID12. Due to the high computational cost of adversarial training (and in particular, PGD adversarial training), we performed one run, and performed the training of ReLU networks for 10 epochs, which seemed sufficient for their accuracy to plateau. The are given in FIG1. Overall, the best networks may be the simple RBFI networks, trained without the use of adversarial examples: for each class of attack, they exhibit either the best performance, or they are very close in performance to the best performer; this is true for no other network type. For PGD attacks, the best performance is obtained by ReLU(PGD) networks trained on PGD attacks, but this may be simply due to gradient masking: note that ReLU(PGD) networks do not perform well with respect to I-FGSM attacks. We note that ReLU(FGSM) networks seem to learn that = 0.3 FGSM attacks are likely, but they have not usefully generalized the lesson, for instance, to attacks of size 0.1. The S-shaped performance curve of ReLU(FGSM) with respect to FGSM or noise is known as label leaking: the network learns to recognize the original input given its perturbed version BID10 ). We compared the performance achieved by training RBFI networks with standard gradients, and with pseudogradients. After 30 epochs of training RBFI(512, 512, 512, 10 | *, *, *, ∨) networks, pseudogradients yielded (96.79 ± 0.17)% accuracy, while regular gradients only (86.35 ± 0.75)%. On smaller networks, that should be easier to train, the gap even widened: for RBFI(128, 128, 10 | *, *, ∨) networks, it went from (95.00 ± 0.29)% for pseudogradients to (82.40 ± 3.72)% for regular gradients. In Section 3, we developed upper bounds for the sensitivity of ReLU and RBFI networks to adversarial attacks on the basis of network weights. It is reasonable to ask whether, using those upper bounds as weight regularizations, we might achieve robustness to adversarial attacks. For ReLU networks, the answer is substantially negative. We experimented adding to the loss used to train the network a term cŝ, for c ≥ 0 andŝ as in. We experimented systematically for many values of c. Large values prevented the network from learning. Smaller values ed in little additional robustness: for = 0.3, simple FGSM attacks lowered the network accuracy to below 10%.For RBFI networks, regularization did help. The choice of upper bound for the components of the u-vector influences the resistance of the trained networks to adversarial examples, as can be seen from. In the experiments reported thus far, we used an upper bound of 3. One may ask: would RBFI networks perform as well, if a higher bound were used? The answer is yes, provided weight regularization is used in place of a tighter bound. If we raise the bound to 10, and use no regularization, the accuracy under PGD attacks with = 0.3 drops from 93.32% to 83.62%. By adding to the loss the regularization cŝ, for c = 0.0001 andŝ as in FORMULA8, we can recover most of the lost accuracy, obtaining accuracy 89.38% at = 0.3. In this paper, we have shown that non-linear structures such as RBFI can be efficiently trained using artificial, "pseudo" gradients, and can attain both high accuracy and high resistance to adversarial attacks.

Represent the short summary for finding the corresponding scientific article (long)

We introduce a type of neural network that is structurally resistant to adversarial attacks, even when trained on unaugmented training sets.  The resistance is due to the stability of network units wrt input perturbations.

represent the short summary for finding the corresponding scientific article (long)

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question:
represent the article for finding a single-sentence tldr

------

For example, 'Semantic parsing which maps a natural language sentence into a formal machine-readable representation of its meaning, is highly constrained by the limited annotated training data. Inspired by the idea of coarse-to-fine, we propose a general-to-detailed neural network(GDNN) by incorporating cross-domain sketch(CDS) among utterances and their logic forms. For utterances in different domains, the General Network will extract CDS using an encoder-decoder model in a multi-task learning setup. Then for some utterances in a specific domain, the Detailed Network will generate the detailed target parts using sequence-to-sequence architecture with advanced attention to both utterance and generated CDS. Our experiments show that compared to direct multi-task learning, CDS has improved the performance in semantic parsing task which converts users' requests into meaning representation language(MRL). We also use experiments to illustrate that CDS works by adding some constraints to the target decoding process, which further proves the effectiveness and rationality of CDS. Recently many natural language processing (NLP) tasks based on the neural network have shown promising and gained much attention because these studies are purely data-driven without linguistic prior knowledge. Semantic parsing task which maps a natural language sentence into a machine-readable representation BID6 ), as a particular translation task, can be treated as a sequence-to-sequence problem BID3 ). Lately, a compositional graph-based semantic meaning representation language (MRL) has been introduced BID14 ), which converts utterance into logic form (action-object-attribute), increasing the ability to represent complex requests. This work is based on MRL format for semantic parsing task. Semantic parsing highly depends on the amount of annotated data and it is hard to annotate the data in logic forms such as Alexa MRL. Several researchers have focused on the area of multi-task learning and transfer learning BID10, BID6, BID15 ) with the observation that while these tasks differ in their domains and forms, the structure of language composition repeats across domains BID12 ). Compared to the model trained on a single domain only, a multi-task model that shares information across domains can improve both performance and generalization. However, there is still a lack of interpretations why the multi-task learning setting works BID26 ) and what the tasks have shared. Some NLP studies around language modeling BID18, BID29, BID2 ) indicate that implicit commonalities of the sentences including syntax and morphology exist and can share among domains, but these commonalities have not been fully discussed and quantified. To address this problem, in this work, compared to multi-task learning mentioned above which directly use neural networks to learn shared features in an implicit way, we try to define these cross-domain commonalities explicitly as cross-domain sketch (CDS). E.g., Search weather in 10 days in domain Weather and Find schedule for films at night in domain ScreeningEvent both have action SearchAction and Attribute time, so that they share a same MRL structure like SearchAction(Type(time@?)), where Type indicates domain and? indicates attribute value which is copying from the original utterance. We extract this domain general MRL structure as CDS. Inspired by the research of coarse-to-fine BID4 ), we construct a two-level encoder-decoder by using CDS as a middle coarse layer. We firstly use General Network to get the CDS for every utterance in all domains. Then for a single specific domain, based on both utterance and extracted CDS, we decode the final target with advanced attention while CDS can be seen as adding some constraints to this process. The first utterance-CDS process can be regarded as a multi-task learning setup since it is suitable for all utterances across the domains. This work mainly introducing CDS using multi-task learning has some contributions listed below: 1) We make an assumption that there exist cross-domain commonalities including syntactic and phrasal similarity for utterances and extract these commonalities as cross-domain sketch (CDS) which for our knowledge is the first time. We then define CDS on two different levels (action-level and attribute-level) trying to seek the most appropriate definition of CDS.2) We propose a general-to-detailed neural network by incorporating CDS as a middle coarse layer. CDS is not only a high-level extraction of commonalities across all the domains, but also a prior information for fine process helping the final decoding.3) Since CDS is cross-domain, our first-level network General Network which encodes the utterance and decodes CDS can be seen as a multi-task learning setup, capturing the commonalities among utterances expressions from different domains which is exactly the goal of multi-task learning. Traditional spoken language understanding (SLU) factors language understanding into domain classification, intent prediction, and slot filling, which proves to be effective in some domains BID9 ). Representations of SLU use pre-defined fixed and flat structures, which limit its expression skills like that it is hard to capture the similarity among utterances when the utterances are from different domains BID15 ). Due to SLU's limited representation skills, meaning representation language (MRL) has been introduced which is a compositional graph-based semantic representation, increasing the ability to represent more complex requests BID14 ). There are several different logic forms including lambda-calculus expression BID17 ), SQL BID33 ), Alexa MRL BID14 ). Compared to fixed and flat SLU representations, MRL BID14 ) based on a large-scale ontology, is much stronger in expression in several aspects like cross-domain and complex utterances. Mapping a natural language utterance into machine interpreted logic form (such as MRL) can be regarded as a special translation task, which is treated as a sequence-to-sequence problem BID28 ). Then BID1 and BID22 advance the sequence-tosequence network with attention mechanism learning the alignments between target and input words, making great progress in the performance. BID23 explore the attention mechanism with some improvements by replacing attention function with attention sparsity. Besides, to deal with the rare words, BID8 incorporate the copy mechanism into the encoder-decoder model by directly copying words from inputs. Lately, many researchers have been around improving sequence-to-sequence model itself, in interpreting the sentence syntax information. BID5 encode the input sentence recursively in a bottom-up fashion. BID30 generate the target sequence and syntax tree through actions simultaneously. Another aspect which has caught much attention is constrained decoding. BID16 and BID25 add some constraints into decoding process, making it more controllable. BID3 use the recurrent network as encoder which proves effective in sequence representation, and respectively use the recurrent network as decoder and tree-decoder. BID16 employ the grammar to constrain the decoding process. BID4, believe utterance understanding is from high-level to low-level and by employing sketch, improve the performance. For semantic parsing task especially in MRL format, it is expensive and time-consuming to annotate the data, and it is challenging to train semantic parsing neural models. Multi-task learning aims to use other related tasks to improve target task performance. BID20 deal with traditional SLU piper-line network by jointly detecting intent and doing slot filling. BID27 share parameters among various tasks, according to the low-level and high-level difference. BID11 divide the representation network into task-specific and general which is shared during multi-task learning. BID6 and BID12 directly share the encoder or decoder neural layers (model params) through different semantic parsing tasks. In BID15, multi-task learning also mainly acts sharing the params of the network. For human language expressions, especially task-oriented requests, there exist commonalities across sentences including linguistic syntax and phrase similarity. They can be seen with general sentence templates. TAB1 Since sentence templates are too many, we try to leverage these common regularities more abstractly. We extract these invariant commonalities which are implicit across domains, and call them as crossdomain sketch (CDS) in a canonical way. We define CDS in meaning representation language (MRL) format (action-object-attribute) and on two levels (action-level and attribute-level). Action-level CDS means to acquire the same action for utterances from different domains while the attribute-level CDS means to extract more detailed information. See examples in TAB1. Instead of extracting CDS from utterance directly, we try converting from logic form into CDS reversely, because it is easier to deal with structural logic form than utterance in natural language form. We analyze the dataset Snips and use a rule-based method to obtain CDS. We strip logic forms from domain-specific components and preserve domainindependent parts including general actions and attributes. We do some statistics on the dataset Snips BID7 ) used in this paper. We convert attributes [object type, movie type, restaurant type] into {object type}, [object name, restaurant name, movie name] into {object name}, [year, timeRange] into {time}, [location name, current location] into {object location}, [country, city] into {place}. All those attributes account for 55% of all attributes which indicate the existence and feasibility of CDS. Figure 1 shows our overall network, which contains a two-level encoder-decoder. The General Network encodes utterance and decodes cross-domain sketch (CDS). Since this process is domaingeneral, it can be done to all domains, which is a multi-task setup. The Detailed Network firstly encodes the CDS and the utterance, then it decodes the target based on both utterance and CDS. This process is domain-dependent, so that it is a fine-tuning process in a specific domain. For an input utterance u = u 1, u 2,...u |u|, and its middle cross-domain sketch (CDS) c = c 1, c 2,...c |c|, and its final logic form y = y 1, y 2,...y |y|, the conditional probability is: Figure 1: Overall Network. General Network (red dashed box below) encodes the utterance with bi-directional LSTM and decodes cross-domain sketch (CDS) using unidirectional LSTM with attention to utterance in all domains. For identical encoding, general utterance encoding and specific utterance encoding share the same encoder while for separate encoding, they are not (see Section 3.2.2). Then Detailed Network, in one specific domain, encodes CDS and utterance using bi-directional LSTM, decodes the final target with advanced attention to both utterance and CDS. DISPLAYFORM0 where y <t = y 1, y 2,...y |t−1|, and c <t = c 1, c 2,...c |t−1|. The neural encoder of our model is similar to neural machine translation (NMT) model, which uses a bi-directional recurrent neural network. Firstly each word of utterance is mapped into a vector u t ∈ R d via embedding layer and we get a word sequence u = (u 1, ..., u |u|). Then we use a bi-directional recurrent neural network with long short-term memory units (LSTM) BID13 ) to learn the representation of word sequence. We generate forward hidden state DISPLAYFORM0 The t-th word will be h DISPLAYFORM1 We construct two kinds of utterance encoders, general utterance encoder for General Network and specific utterance encoder for Detailed Network (see in Figure 1), so as to extract different information for different purposes. The general utterance encoder, meant to pay more attention to cross-domain commonalities of utterances, is used by all utterances from all domains. The specific utterance encoder, which is domain-dependent, belongs to one specific domain and is more sensitive to details. We call encoder outputs h ug t from general utterance encoder and h us t from specific utterance encoder. When the two encoders share the same parameters that is h ug t = h us t, we call it identical encoding and when they are not, we call it separate encoding, inspired by BID27; BID16; BID21; BID0 ) which explore the sharing mechanisms of multi-task learning and propose some improvements. The General Network is meant to obtain cross-domain sketch (CDS) c conditioned on utterance u, using an encoder-decoder network. After encoding utterance by general utterance encoder for all domains, we obtain h ug t (see Section 3.2.2). Then we start to decode CDS.The decoder is based on a unidirectional recurrent neural network, and the output vector is used to predict the word. The cd represents CDS decoder. DISPLAYFORM0 where c t is the previously predicted word embedding. The LuongAttention BID22 ) between decoding hidden state d t and encoding sequence e i (i = 1, 2, ...|e|) at time step t is computed as: DISPLAYFORM1 DISPLAYFORM2 Based on equations FORMULA4 and FORMULA5, we compute the attention a The t-th predicted output token will be: DISPLAYFORM0 DISPLAYFORM1 where W, b are parameters. After decoding CDS words c = (c 1, ..., c |c|), we use an encoder to represent its meaning and due to words' relation with forward and backward contexts, we choose to use a bi-directional LSTM. We generate forward hidden state − → h Through specific utterance encoder and cross-domain sketch (CDS) encoder, we acquired t-th word representation h us t and h ce t. Finally with advanced attention to both encoded utterance u and CDS c, we decode the final target y. The decoder is based on a unidirectional recurrent neural network, and the output vector is used to predict the word. The y represents target decoder. DISPLAYFORM0 where y t is the previously predicted word embedding. During target decoding process and at time step t, we not only compute the attention to utterance encoding outputs h us but also compute the attention to CDS encoding outputs h ce. The attention between target hidden state and utterance is a 1, 2,...|c|) in the same way. Then the t-th predicted output token will be based on the advanced two-aspect attention: DISPLAYFORM1 For training process, the objective is: DISPLAYFORM0 T is the training corpus. For inference process, we firstly obtain cross-domain sketch (CDS) via c = argmax p(c|u) then we get the final target logic form via y = argmax p(y|u, c). For both decoding processes, we use greedy search to generate words one by one. Existed semantic parsing datasets, e.g., GEO BID32 ), ATIS BID31 ), collect data only from one domain and have a very limited amount, which can not fully interpret the effectiveness of cross-domain sketch (CDS) since it needs large dataset among different domains. In this case, we mainly consider the semantic parsing task Snips (Goo et al. FORMULA0) based on MRL format (action-object-attribute). Snips collects users' requests from a personal voice assistant. The original dataset is annotated in spoken language understanding (SLU) format (intent-slot). It has 7 intent types and 72 slot labels, and more statistics are shown in TAB4. Based on the format (intent-slot), we pre-process this dataset into MRL format by some pre-defined rules, then we regard the intent as domain/task and share CDS among them. The details are shown in Target SearchAction (ScreeningEventType ( object type @ 2, movie type @ 4, timeRange @ 6, location name @ 8 9 10) ) TAB3: Several examples of Snips. Utterance is the user's request which is a natural language expression. Intent and slots are in formats from original dataset. Cross-domain sketch (CDS) has two levels (action-level and attribute-level). Target is the final logic form with numbers indicating copying words from utterance (index starting from 0). We use Tensorflow in all our experiments, with LuongAttention BID22 ) and copy mechanism. The embedding dimension is set to 100 and initialized with GloVe embeddings (Pennington et al. FORMULA0). The encoder and decoder both use one-layer LSTM with hidden size 50. We apply the dropout selected in {0.3,0.5}. Learning rate is initialized with 0.001 and is decaying during training. Early stopping is applied. The mini-batch size is set to 16. We use the logic form accuracy as the evaluation metric. Firstly, in order to prove the role of the cross-domain sketch (CDS) in helping to guide decoding process with multi-tasking learning setup, we do several experiments, and the are shown in Table 4. For joint learning, we apply several multi-task architectures from BID6 ), including one-to-one, one-to-many and one-to-shareMany. One-to-one architecture applies a single sequence-to-sequence model across all the tasks. One-to-many only shares the encoder across all the tasks while the decoder including the attention parameters is not shared. In one-to-shareMany model, tasks share encoder and decoder (including attention) params, but the output layer of decoder is task-independent. From the Table 4, in general, joint learning performs better than single task learning. In joint learning, one-to-one is the best and performs way better than one-to-many and one-to-shareMany, probably limited by the dataset's size and similarity among tasks. By incorporating CDS, our GDNN (general-to-detailed neural network) models have all improved the performance to different degrees. The CDS is defined on two levels (action-level and attribute-level, see examples in TAB3) and attribute-level CDS improves greater than action-level CDS, which is in our expectation since it offers more information for tasks to share. We also experiment on different utterance encoding setups with identical encoding and separate encoding (see Section 3.2.2). The separate encoding setup performs better than sharing the same encoder for utterance, which integrates the fact that different encoders pay different attention to the utterances due to different purposes which means one is more general and the other is more specific detailed. Method Snips Accuracy Single Seq2Seq BID28 62.3 Joint Seq2Seq (one-to-many) BID6 62.0 Joint Seq2Seq (one-to-shareMany) BID6 64.2 Joint Seq2Seq (one-to-one) BID6 71.4 GDNN with Action-level CDS (identical encoding) 74.9 GDNN with Action-level CDS (separate encoding) 75.1 GDNN with Attribute-level CDS (identical encoding) 76.7 GDNN with Attribute-level CDS (separate encoding) 78.1 Table 4: Multi-task Results. Single Seq2Seq means each task has a sequenece-to-sequence model. Joint Seq2Seq show with three multi-task mechanisms. Our include GDNN (generalto-detailed neural network) models with different levels of CDS (action-level/attribute level) and different utterance encoding mechanisms (identical encoding/separate encoding).We also list the full of GDNN in TAB6 Moreover, we compare our experiments with traditional models which regard the task as intent classification and slot filling (IC SF). The are shown in TAB7 below. From TAB7, we can see compared to IC SF models (based on sequence labeling format), Seq2Seq perform worse (71.4% compared to 73.2%) due to its fewer assumptions and larger decode size as well as its difficulty of training, which is usual in comparing IC SF models and sequence-tosequence models. Through using CDS, the performance has significantly improved. On the one Method Snips Accuracy Joint Seq. BID10 73.2 Atten.-Based BID19 74.1 Slot.-Gated (Intent Atten.) BID7 74.6 Slot.-Gated (Full Atten.) BID7 75.5 Joint Seq2Seq BID6 71.4 GDNN with Action-level CDS 73.2 GDNN with Attribute-level CDS 74.6 hand, CDS extract the cross-domain commonalities among tasks helping to make the multi-task learning more specific, which can be seen as an advance to multi-task learning. On the other hand, CDS can be seen adding some constraints to the final target decoding process which has offered more information for the decoding process, compared to direct joint Seq2Seq. To better prove and explain this idea, we do some experiments according to constraint decoding aspect. We try to compare the sub-process of converting utterance to CDS through different models, e.g., IC SF, Seq2Seq. From the Table 7, we can see that Seq2Seq achieve the comparable (87.7%) to IC SF model (84.9%) for generating CDS from utterance, which further explains that, the fact joint seq2seq performs worse (71.4%, see TAB7) than IC SF model (73.2%) is owing to the lack of guidance and constraints during the follow-up decoding process. By incorporating CDS, we add some constraints to this decoding process thus obtaining better performance. Table 7: Results of CDS generation in dataset Snips by two methods. IC SF is using intent classification and slot filling with evaluation metric (intent accuracy, slot labelling accuracy and final accuracy). Seq2Seq generates CDS based on utterance using an encoder-decoder. In this paper, we propose the concept of cross-domain sketch (CDS) which extracts some shared information across domains, trying to fully utilize the cross-domain commonalities such as syntactic and phrasal similarity in human expressions. We try to define CDS on two levels and give some examples to illustrate our idea. We also present a general-to-detailed neural network (GDNN) for converting an utterance into a logic form based on meaning representation language (MRL) form. The general network, which is meant to extract cross-domain commonalities, uses an encoderdecoder model to obtain CDS in a multi-task setup. Then the detailed network generates the final domain-specific target by exploiting utterance and CDS simultaneously via attention mechanism. Our experiments demonstrate the effectiveness of CDS and multi-task learning. CDS is able to generalize over a wide range of tasks since it is an extraction to language expressions. Therefore, in the future, we would like to perfect the CDS definition and extend its' ontology to other domains and tasks. Besides, in this paper, we use attention mechanism to make use of CDS which is still a indirect way. We would like to explore more effective ways such as constraint decoding to further enhance the role of CDS.' should have a representation like 'General-to-detailed neural network(GDNN)  with Multi-Task Learning by incorporating cross-domain sketch(CDS) for semantic parsing' but very far from 'We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.'.

Question: We show that there exists an inherent tension between the goal of adversarial robustness and that of standard generalization. Specifically, training robust models may not only be more resource-consuming, but also lead to a reduction of standard accuracy. We demonstrate that this trade-off between the standard accuracy of a model and its robustness to adversarial perturbations provably exists even in a fairly simple and natural setting. These findings also corroborate a similar phenomenon observed in practice. Further, we argue that this phenomenon is a consequence of robust classifiers learning fundamentally different feature representations than standard classifiers. These differences, in particular, seem to in unexpected benefits: the features learned by robust models tend to align better with salient data characteristics and human perception. Deep learning models have achieved impressive performance on a number of challenging benchmarks in computer vision, speech recognition and competitive game playing (; BID24 ; ; BID25 . However, it turns out that these models are actually quite brittle. In particular, one can often synthesize small, imperceptible perturbations of the input data and cause the model to make highly-confident but erroneous predictions BID9 BID5).This problem of so-called adversarial examples has garnered significant attention recently and ed in a number of approaches both to finding these perturbations, and to training models that are robust to them BID23; BID16 BID7; a; BID14 BID1. However, building such adversarially robust models has proved to be quite challenging. In particular, many of the proposed robust training methods were subsequently shown to be ineffective BID8 BID2 ). Only recently, has there been progress towards models that achieve robustness that can be demonstrated empirically and, in some cases, even formally verified BID13;;;; BID11 b).The vulnerability of models trained using standard methods to adversarial perturbations makes it clear that the paradigm of adversarially robust learning is different from the classic learning setting. In particular, we already know that robustness comes at a cost. This cost takes the form of computationally expensive training methods (more training time), but also, as shown recently in , the potential need for more training data. It is natural then to wonder: Are these the only costs of adversarial robustness? And, if so, once we choose to pay these costs, would it always be preferable to have a robust model instead of a standard one? The goal of this work is to explore these questions and thus, in turn, to bring us closer to understanding the phenomenon of adversarial robustness. Our contributions It might be natural to expect that training models to be adversarially robust, albeit more resource-consuming, can only improve performance in the standard classification setting. In this work, we show, however, that the picture here is much more nuanced: these two goals might be fundamentally at odds. Specifically, even though applying adversarial training, the leading method for training robust models, can be beneficial in some regimes of training data size, in general, there is a trade-off between the standard accuracy and adversarially robust accuracy of a model. In fact, we show that this trade-off provably exists even in a fairly simple and natural setting. At the root of this trade-off is the fact that features learned by the optimal standard and optimal robust classifiers are fundamentally different and, interestingly, this phenomenon persists even in the limit of infinite data. This thus also goes against the natural expectation that given sufficient data, classic machine learning tools would be sufficient to learn robust models and emphasizes the need for techniques specifically tailored to training robust models. Our exploration also uncovers certain unexpected benefit of adversarially robust models. In particular, adversarially robust learning tends to equip the ing models with invariances that we would expect to be also present in human vision. This, in turn, leads to features that align better with human perception, and could also pave the way towards building models that are easier to understand. Consequently, the feature embeddings learnt by robust models yield also clean inter-class interpolations, similar to those found by generative adversarial networks (GANs) BID23 and other generative models. This hints at the existence of a stronger connection between GANs and adversarial robustness. Recall that in the canonical classification setting, the primary focus is on maximizing standard accuracy, i.e. the performance on (yet) unseen samples from the underlying distribution. Specifically, the goal is to train models that have low expected loss (also known as population risk): DISPLAYFORM0 [L(x, y; θ)].Adversarial robustness The existence of adversarial examples largely changed this picture. In particular, there has been a lot of interest in developing models that are resistant to them, or, in other words, models that are adversarially robust. In this context, the goal is to train models that have low expected adversarial loss: DISPLAYFORM1 Here, ∆ represents the set of perturbations that the adversary can apply to induce misclassification. In this work, we focus on the case when ∆ is the set of p -bounded perturbations, i.e. ∆ = {δ ∈ R d | δ p ≤ ε}. This choice is the most common one in the context of adversarial examples and serves as a standard benchmark. It is worth noting though that several other notions of adversarial perturbations have been studied. These include rotations and translations BID15, and smooth spatial deformations (a). In general, determining the "right" ∆ to use is a domain specific question. Adversarial training The most successful approach to building adversarially robust models so far BID13;; ) was socalled adversarial training BID23. Adversarial training is motivated by viewing as a statistical learning question, for which we need to solve the corresponding (adversarial) empirical risk minimization problem: DISPLAYFORM2 The ing saddle point problem can be hard to solve in general. However, it turns out to be often tractable in practice, at least in the context of p -bounded perturbations BID13. Specifically, adversarial training corresponds to a natural robust optimization approach to solving this problem BID4. In this approach, we repeatedly find the worst-case input perturbations δ (solving the inner maximization problem), and then update the model parameters to reduce the loss on these perturbed inputs. Though adversarial training is effective, this success comes with certain drawbacks. The most obvious one is an increase in the training time (we need to compute new perturbations each parameter update step). Another one is the potential need for more training data as shown recently in . These costs make training more demanding, but is that the whole price of being adversarially robust? In particular, if we are willing to pay these costs: Are robust classifiers better than standard ones in every other aspect? This is the key question that motivates our work. Comparison of the standard accuracy of models trained against an 2 -bounded adversary as a function of size of the training dataset. We observe that when training with few samples, adversarial training has a positive effect on model generalization (especially on MNIST). However, as training data increase, the standard accuracy of robust models drops below that of the standard model (ε train = 0). Similar for ∞ trained networks are shown in FIG7 of Appendix G.Adversarial Training as a Form of Data Augmentation Our point of start is a popular view of adversarial training as the "ultimate" form of data augmentation. According to this view, the adversarial perturbation set ∆ is seen as the set of invariants that a good model should satisfy (regardless of the adversarial robustness considerations). Thus, finding the worst-case δ corresponds to augmenting the training data in the "most confusing" and thus also "most helpful" manner. A key implication of this view is that adversarial training should be beneficial for the standard accuracy of a model (; 2014; BID23).Indeed, in FIG0, we see this effect, when classifiers are trained with relatively few samples (particularly on MNIST). In this setting, the amount of training data available is insufficient to learn a good standard classifier and the set of adversarial perturbations used is "compatible" with the learning task. (That is, good standard models for this task need to be also somewhat invariant to these perturbations.) In such regime, robust training does indeed act as data augmentation, regularizing the model and leading to a better solution (from standard accuracy point of view). (Note that this effect seems less pronounced for CIFAR-10, possibly because p -invariance is not as important for a good standard CIFAR-10 classifier.) Surprisingly however, in FIG7 we see that as we include more samples in the training set, this positive effect becomes less significant. In fact, after some point adversarial training actually decreases the standard accuracy. In Figure 7 in Appendix G we study the behaviour of models trained using adversarial training with different p -bounded adversaries. We observe a steady decline in standard accuracy as the strength of the adversary increases. (Note that this still holds if we train on batches that also contain natural examples, as in Kurakin et al. (2016a). See Appendix B.) Similar effects were also observed in prior work (b; BID13 BID12 ; b; ; BID3 .The goal of this work is to illustrate and explain the roots of this phenomenon. In particular, we would like to understand:Why does there seem to be a trade-off between standard and adversarially robust accuracy?As we will show, this effect is not an artifact of our adversarial training methods but in fact is inevitable consequence of different goals of adversarial robustness and standard generalization. As we discussed above, we often observe that employing adversarial training leads to a decrease in a model's standard accuracy. In what follows, we show that this phenomenon is a manifestation of an inherent tension between standard accuracy and adversarially robust accuracy. In particular, we present a theoretical model that demonstrates it. In fact, this phenomenon can be illustrated in a fairly simple setting which suggests that it is quite prevalent. Our binary classification task Our data model consists of input-label pairs (x, y) sampled from a distribution D as follows: DISPLAYFORM0 where N (µ, σ 2) is a normal distribution with mean µ and variance σ 2, and p ≥ 0.5. We chose η to be large enough so that a simple classifier attains high standard accuracy (>99%) -e.g. η = Θ(1/ √ d) will suffice. The parameter p quantifies how correlated the feature x 1 is with the label. For the sake of example, we can think of p as being 0.95. This choice is fairly arbitrary; the trade-off between standard and robust accuracy will be qualitatively similar for any p < 1.Standard classification is easy Note that samples from D consist of a single feature that is moderately correlated with the label and d other features that are only very weakly correlated with it. Despite the fact that each one of the latter type of features individually is hardly predictive of the correct label, this distribution turns out to be fairly simple to classify from a standard accuracy perspective. Specifically, a natural (linear) classifier DISPLAYFORM1 achieves standard accuracy arbitrarily close to 100%, for d large enough. Indeed, observe that DISPLAYFORM2 Adversarially robust classification Note that in our discussion so far, we effectively viewed the average of x 2,..., x d+1 as a single "meta-feature" that is highly correlated with the correct label. For a standard classifier, any feature that is even slightly correlated with the label is useful. As a , a standard classifier will take advantage (and thus rely on) the weakly correlated features x 2,..., x d+1 (by implicitly pooling information) to achieve almost perfect standard accuracy. However, this analogy breaks completely in the adversarial setting. In particular, an ∞ -bounded adversary that is only allowed to perturb each feature by a moderate ε can effectively override the effect of the aforementioned meta-feature. For instance, if ε = 2η, an adversary can shift each weaklycorrelated feature towards −y. The classifier would now see a perturbed input x such that each of the features x 2,..., x d+1 are sampled i.i.d. from N (−ηy, 1) (i.e., now becoming anti-correlated with the correct label). Thus, when ε ≥ 2η, the adversary can essentially simulate the distribution of the weakly-correlated features as if belonging to the wrong class. Formally, the probability of the meta-feature correctly predicting y in this setting is DISPLAYFORM3 As a , the simple classifier in that relies solely on these features cannot get adversarial accuracy better than 1%.Intriguingly, this discussion draws a distinction between robust features (x 1) and non-robust features (x 2, . . ., x d+1) that arises in the adversarial setting. While the meta-feature is far more predictive of the true label, it is extremely unreliable in the presence of an adversary. Hence, a tension between standard and adversarial accuracy arises. Any classifier that aims for high accuracy (say > 99%) will have to heavily rely on non-robust features (the robust feature provides only, say, 95% accuracy). However, since the non-robust features can be arbitrarily manipulated, this classifier will inevitably have low adversarial accuracy. We make this formal in the following theorem proved in Appendix C. Theorem 2.1 (Robustness-accuracy trade-off). Any classifier that attains at least 1 − δ standard accuracy on D has robust accuracy at most p 1−p δ against an ∞ -bounded adversary with ε ≥ 2η. This bound implies that if p < 1, as standard accuracy approaches 100% (δ → 0), adversarial accuracy falls to 0%. As a concrete example, consider p = 0.95, then any classifier with standard accuracy more than 1 − δ will have robust accuracy at most 19δ 1. Also it is worth noting that the theorem is tight. If δ = 1 − p, both the standard and adversarial accuracies are bounded by p which is attained by the classifier that relies solely on the first feature. Additionally, note that compared to the scale of the features ±1, the value of ε required to manipulate the standard classifier is very small DISPLAYFORM4 On the (non-)existence of an accurate and robust classifier It might be natural to expect that in the regime of infinite data, the standard classifier itself acts as a robust classifier. Note however, that this is not true for the setting we analyze above. Here, the trade-off between standard and adversarial accuracy is an inherent trait of the data distribution itself and not due to having insufficient samples. In this particular classification task, we (implicitly) assumed that there does not exist a classifier that is both robust and very accurate (i.e. > 99% standard and robust accuracy). Thus, for this task, any classifier that is very accurate (including the Bayes classifier -the classifier minimizing classification error having full-information about the distribution) will necessarily be non-robust. This seemingly goes against the common assumption in adversarial ML that humans are such perfect robust and accurate classifiers for standard datasets. However, note that there is no concrete evidence supporting this assumption. In fact, humans often have far from perfect performance in vision benchmarks (; 2014;) and are outperformed by ML models in certain tasks (b; BID19 . It is plausible that standard ML models are able to outperform humans in these tasks by relying on brittle features that humans are naturally invariant to and the observed decrease in performance might be the manifestation of that. As we have seen in the distributional model D, a classifier that achieves very high standard accuracy will inevitably have near-zero adversarial accuracy. This is true even when a classifier with reasonable standard and robust accuracy exists. Hence, in an adversarial setting, where the goal is to achieve high adversarial accuracy, the training procedure needs to be modified. We now make this phenomenon concrete for linear classifiers trained using the soft-margin SVM loss. Specifically, in Appendix D we prove the following theorem. For η ≥ 4/ √ d and p ≤ 0.975 (the first feature is not perfect), a soft-margin SVM classifier of unit weight norm minimizing the distributional loss achieves a standard accuracy of > 99% and adversarial accuracy of < 1% against an ∞ -bounded adversary of ε ≥ 2η. Minimizing the distributional adversarial loss instead leads to a robust classifier that has standard and adversarial accuracy of p against any ε < 1.This theorem shows that if our focus is on robust models, adversarial training is crucial to achieve non-trivial adversarial accuracy in this setting. Simply optimizing the standard accuracy of the model (i.e. standard training) leads to poor robust accuracy. Soft-margin SVM classifiers and the constant 0.975 are chosen for mathematical convenience. Our proofs do not depend on them in a crucial way and can be adapted, in a straightforward manner, to other natural settings, e.g. logistic regression. Transferability An interesting implication of our analysis is that standard training produces classifiers that rely on features that are weakly correlated with the correct label. This will be true for any classifier trained on the same distribution. Hence, the adversarial examples that are created by perturbing each feature in the direction of −y will transfer across classifiers trained on independent Figure 2: Visualization of the loss gradient with respect to input pixels. Recall that these gradients highlight the input features which affect the loss most strongly, and thus are important for the classifier's prediction. We observe that the gradients are significantly more interpretable for adversarially trained networks -they align well with perceptually relevant features. In contrast, for standard networks they appear very noisy. We observe that gradients of ∞ -trained models tend to be sparser than those of 2 -trained models. (For MNIST, blue and red pixels denote positive and negative gradient regions respectively. For CIFAR-10 and ImageNet, we clip gradients to within ±3σ and rescale them to lie in the range.) Additional visualizations are in FIG0 of Appendix G.samples from the distribution. This constitutes an interesting manifestation of the generally observed phenomenon of transferability and might hint at its origin. Empirical examination In Section 2.1, we showed that the trade-off between standard accuracy and robustness might be inevitable. To examine how representative our theoretical model is of real-world datasets, we also experimentally investigate this issue on MNIST as it is amenable to linear classifiers. Interestingly, we observe a qualitatively similar behavior. For instance, in FIG6 (b) in Appendix E, we see that the standard classifier assigns weight to even weakly-correlated features. (Note that in settings with finite training data, such brittle features could arise even from noise -see Appendix E.) The robust classifier on the other hand does not assign any weight beyond a certain threshold. Further, we find that it is possible to obtain a robust classifier by directly training a standard model using only features that are relatively well-correlated with the label (without adversarial training). As expected, as more features are incorporated into the training, the standard accuracy is improved at the cost of robustness (see Appendix E FIG6 (c)). In Section 2, we established that robust and standard models might depend on very different sets of features. We demonstrated how this can lead to a decrease in standard accuracy for robust models. In this section, we will argue that the features learned by robust models can also be beneficial. At a high level, robustness to adversarial perturbations can be viewed as an invariance property of a model. A model that achieves small loss for all perturbations in the set ∆, will necessarily have learned features that are invariant to such perturbations. Thus, robust training can be viewed as a method to embed certain invariances in a model. Since we also expect humans to be invariant to these perturbations (e.g. small p -bounded changes of the pixels), robust models will be more aligned with human vision than standard models. In this section, we present evidence supporting the view. Loss gradients in the input space align well with human perception As a starting point, we want to investigate which features of the input most strongly affect the prediction of the classifier both for standard and robust models. To this end, we visualize the gradients of the loss with respect to individual features (pixels) of the input in Figure 2. We observe that gradients for adversarially: Visualizing large-ε adversarial examples for standard and robust (2 / ∞ -adversarial training) models. We construct these examples by iteratively following the (negative) loss gradient while staying with 2 -distance of ε from the original image. We observe that the images produced for robust models effectively capture salient data characteristics and appear similar to examples of a different class. (The value of ε is equal for all models and much larger than the one used for training.) Additional examples are visualized in Figure 8 and 9 of Appendix G.trained networks align well with perceptually relevant features (such as edges) of the input image. In contrast, for standard networks, these gradients have no coherent patterns and appear very noisy to humans. We want to emphasize that no preprocessing was applied to the gradients (other than scaling and clipping for visualization). On the other hand, extraction of interpretable information from the gradients of standard networks has so far only been possible using additional sophisticated techniques (; ;).This observation effectively outlines an approach to train models that align better with human perception by design. By encoding the correct prior into the set of perturbations ∆, adversarial training alone might be sufficient to yield interpretable gradients. We believe that this phenomenon warrants an in-depth investigation and we view our experiments as only exploratory. Adversarial examples exhibit salient data characteristics Given how the gradients of standard and robust models are concentrated on qualitatively different input features, we want to investigate how the adversarial examples of these models appear visually. To find adversarial examples, we start from a given test image and apply Projected Gradient Descent (PGD; a standard first-order optimization method) to find the image of highest loss within an p -ball of radius ε around the original image 2. This procedure will change the pixels that are most influential for a particular model's predictions and thus hint towards how the model is making its predictions. The ing visualizations are presented in FIG2 (details in Appendix A). Surprisingly, we can observe that adversarial perturbations for robust models tend to produce salient characteristics of another class. In fact, the corresponding adversarial examples for robust models can often be perceived as samples from that class. This behavior is in stark contrast to standard models, for which adversarial examples appear as noisy variants of the input image. These findings provide additional evidence that adversarial training does not necessarily lead to gradient obfuscation BID2. Following the gradient changes the image in a meaningful way and (eventually) leads to images of different classes. Hence, the robustness of these models does not stem from having gradients that are ill-suited for first-order methods. Smooth cross-class interpolations via gradient descent By linearly interpolating between the original image and the image produced by PGD we can produce a smooth, "perceptually plausible" interpolation between classes FIG3. Such interpolation have thus far been restricted to generative models such as GANs BID22 and VAEs , involved manipulation of learned representations , and hand-designed methods . In fact, we conjecture that the similarity of these inter-class trajectories to GAN interpolations is not a coincidence. We postulate that the saddle point problem that is key in both these approaches may be at the root of this effect. We hope that future research will investigate this connection further and explore how to utilize the loss landscape of robust models as an alternative method to smoothly interpolate between classes. Due to the large body of related work, we will only focus on the most relevant studies here and defer the full discussion to Appendix F. BID18 prove upper bounds on the robust of classifiers and exhibit a standard vs. robust accuracy trade-off for a specific classifier families on a synthetic task. Their setting also (implicitly) utilizes the notion of robust and non-robust features, however these features have small magnitude rather than weak correlation. propose regularizing the gradient of the classifier with respect to its input. They find that the ing classifiers have more interpretable gradients and targeted adversarial examples resemble the target class for digit and character recognition tasks. There has been recent of work proving upper bounds on classifier robustness BID20; BID17. However, this work is orthogonal to ours as in these settings there exist classifiers that are both robust and accurate. In this work, we show that the goal of adversarially robust generalization might fundamentally be at odds with that of standard generalization. Specifically, we identify an inherent trade-off between the standard accuracy and adversarial robustness of a model, that provably manifests in a concrete, simple setting. This trade-off stems from intrinsic differences between the feature learned by standard and robust models. Our analysis also explains the drop in standard accuracy observed when employing adversarial training in practice. Moreover, it emphasizes the need to develop robust training methods, since robustness is unlikely to arise as a consequence of standard training. We discover that even though adversarial robustness comes at a price, it has some unexpected benefits. Robust models learn features that align well with salient data characteristics. The root of this phenomenon is that the set of adversarial perturbations encodes some prior for human perception. Thus, classifiers that are robust to these perturbations are also necessarily invariant to input modifications that we expect humans to be invariant to. We demonstrate a striking consequence of this phenomenon: robust models yield clean feature interpolations similar to those obtained from generative models such as GANs BID23. This emphasizes the possibility of a stronger connection between GANs and adversarial robustness. Finally, our findings show that the interplay between adversarial robustness and standard classification might be more nuanced that one might expect. This motivates further work to fully undertand the relative costs and benefits of each of these notions. Kaiming we filter out all the images from the MNIST dataset other than the "5" and "7" labelled examples. For the ImageNet dataset, adversarial training is significantly harder since the classification problem is challenging by itself and standard classifiers are already computationally expensive to train. We thus restrict our focus to a smaller subset of the dataset. We group together a subset of existing, semantically similar ImageNet classes into 8 different super-classes, as shown in TAB1. We train and evaluate only on examples corresponding to these classes. "Dog" 151 to 268 "Cat" 281 to 285 "Frog" 30 to 32 "Turtle" 33 to 37 "Bird" 80 to 100 "Primate" 365 to 382 "Fish" 389 to 397 "Crab" 118 to 121 "Insect" 300 to 319A.2 MODELS• Binary MNIST (Section 2.2): We train a linear classifier with parameters w ∈ R 784, b ∈ R on the dataset described in Section A.1 (labels −1 and +1 correspond to images labelled as "5" and "7" respectively). We use the cross-entropy loss and perform 100 epochs of gradient descent in training.• MNIST: We use the simple convolution architecture from the TensorFlow tutorial 3.• CIFAR-10: We consider a standard ResNet model BID25. It has 4 groups of residual layers with filter sizes and 5 residual units each 4.• Restricted ImageNet: We use a ResNet-50 BID25 architecture using the code from the tensorpack repository . We do not modify the model architecture, and change the training procedure only by changing the number of examples per "epoch" from 1,280,000 images to 76,800 images. We perform adversarial training to train robust classifiers following BID13. Specifically, we train against a projected gradient descent (PGD) adversary, starting from a random initial perturbation of the training data. We consider adversarial perturbations in p norm where p = {2, ∞}.Unless otherwise specified, we use the values of ε provided in TAB2 to train/evaluate our models. The images we generated for FIG2 were allowed a much larger perturbation from the original sample in order to produce visible changes to the images. These values are listed in Table 3. Since Table 3 these levels of perturbations would allow to truly change the class of the image, training against such strong adversaries would be impossible. Still, we observe that smaller values of ε suffices to ensure that the models rely on the most robust (and hence interpretable) features. In order to make sure that the standard accuracy drop in Figure 7 is not an artifact of only training on adversarial examples, we experimented with including unperturbed examples in each training batch, following the recommendation of (a). We found that while this slightly improves the standard accuracy of the classifier, it decreases it's robust accuracy by a roughly proportional amount, see TAB3. DISPLAYFORM0 The main idea of the proof is that an adversary with ε = 2η is able to change the distribution of features x 2,..., x d+1 to reflect a label of −y instead of y by subtracting εy from each variable. Hence any information that is used from these features to achieve better standard accuracy can be used by the adversary to reduce adversarial accuracy. We define G + to be the distribution of x 2,..., x d+1 when y = +1 and G − to be that distribution when y = −1. We will consider the setting where ε = 2η and fix the adversary that replaces x i by x i − yε for each i ≥ 2. This adversary is able to change G + to G − in the adversarial setting and vice-versa. Consider any classifier f (x) that maps an input x to a class in {−1, +1}. Let us fix the probability that this classifier predicts class +1 for some fixed value of x 1 and distribution of x 2,..., x d+1. Concretely, we define p ij to be the probability of predicting +1 given that the first feature has sign i and the rest of the features are distributed according to G j. Formally, DISPLAYFORM1 Using these definitions, we can express the standard accuracy of the classifier as DISPLAYFORM2 Similarly, we can express the accuracy of this classifier against the adversary that replaces G + with G − (and vice-versa) as DISPLAYFORM3 For convenience we will define a = 1 − p ++ + p −− and b = 1 − p −+ + p +−. Then we can rewrite DISPLAYFORM4 We are assuming that the standard accuracy of the classifier is at least 1 − δ for some small δ. This implies that DISPLAYFORM5 Since p ij are probabilities, we can guarantee that a ≥ 0. Moreover, since p ≥ 0.5, we have p/(1 − p) ≥ 1. We use these to upper bound the adversarial accuracy by DISPLAYFORM6 We consider the problem of fitting the distribution D of by using a standard soft-margin SVM classifier. Specifically, this can be formulated as: DISPLAYFORM7 for some value of λ. We will assume that we tune λ such that the optimal solution w * has 2 -norm of 1. This is without much loss of generality since our proofs can be adapted to the general case. We will refer to the first term of as the margin term and the second term as the regularization term. First we will argue that, due to symmetry, the optimal solution will assign equal weight to all the features x i for i = 2,..., d + 1. Lemma D.1. Consider an optimal solution w * to the optimization problem. Then, DISPLAYFORM8 Proof. Assume that ∃ i, j ∈ {2, ..., d + 1} such that w * i = w * j. Since the distribution of x i and x j are identical, we can swap the value of w i and w j, to get an alternative set of parametersŵ that has the same loss function value (ŵ j = w i,ŵ i = w j,ŵ k = w k for k = i, j).Moreover, since the margin term of the loss is convex in w, using Jensen's inequality, we get that averaging w * andŵ will not increase the value of that margin term. Note, however, that w * +ŵ 2 2 < w * 2, hence the regularization loss is strictly smaller for the average point. This contradicts the optimality of w *.Since every optimal solution will assign equal weight to all x i for k ≥ 2, we can replace these features by their sum (and divide by √ d for convenience). We will define DISPLAYFORM9 which, by the properties of the normal distribution, is distributed as DISPLAYFORM10 By assigning a weight of v to that combined feature the optimal solutions can be parametrized as DISPLAYFORM11 where the regularization term of the loss is λ(w DISPLAYFORM12 Recall that our chosen value of η is 4/ √ d, which implies that the contribution of vz is distributed normally with mean 4yv and variance v 2 . By the concentration of the normal distribution, the probability of vz being larger than v is large. We will use this fact to show that the optimal classifier will assign on v at least as much weight as it assigns on w 1 . Lemma D.2. Consider the optimal solution (w * 1, v *) of the problem. Then DISPLAYFORM13 Proof. Assume for the sake of contradiction that v * < 1/ √ 2. Then, with probability at least 1 − p, the first feature predicts the wrong label and without enough weight, the remaining features cannot compensate for it. Concretely, DISPLAYFORM14 We will now show that a solution that assigns zero weight on the first feature (v = 1 and w 1 = 0), achieves a better margin loss. DISPLAYFORM15 Hence, as long as p ≤ 0.975, this solution has a smaller margin loss than the original solution. Since both solutions have the same norm, the solution that assigns weight only on v is better than the original solution (w * 1, v *), contradicting its optimality. We have established that the learned classifier will assign more weight to v than w 1. Since z will be at least y with large probability, we will show that the behavior of the classifier depends entirely on z. Lemma D.3. The standard accuracy of the soft-margin SVM learned for problem is at least 99%.Proof. By Lemma D.2, the classifier predicts the sign of w 1 x 1 + vz where vz ∼ N (4yv, v 2) and v ≥ 1/ √ 2. Hence with probability at least 99%, vzy > 1/ √ 2 ≥ w 1 and thus the predicted class is y (the correct class) independent of x 1.We can utilize the same argument to show that an adversary that changes the distribution of z has essentially full control over the classifier prediction. Lemma D.4. The adversarial accuracy of the soft-margin SVM learned for is at most 1% against an ∞ -bounded adversary of ε = 2η. Proof. Observe that the adversary can shift each feature x i towards y by 2η. This will cause z to be distributed as DISPLAYFORM16 Therefore with probability at least 99%, vyz < −y ≤ −w 1 and the predicted class will be −y (wrong class) independent of x 1.It remains to show that adversarial training for this classification task with ε > 2η will in a classifier that has relies solely on the first feature. Lemma D.5. Minimizing the adversarial variant of the loss in a classifier that assigns 0 weight to features x i for i ≥ 2.Proof. The optimization problem that adversarial training solves is DISPLAYFORM17 which is equivalent to DISPLAYFORM18 Consider any optimal solution w for which w i > 0 for some i > 2. The contribution of terms depending on w i to 1 − yw x + ε w 1 is a normally-distributed random variable with mean 2η − ε ≤ 0. Since the mean is non-positive, setting w i to zero can only decrease the margin term of the loss. At the same time, setting w i to zero strictly decreases the regularization term, contradicting the optimality of w. Clearly, such a classifier will have standard and adversarial accuracy of p against any ε < 1 since such a value of ε is not sufficient to change the sign of the first feature. This concludes the proof of the theorem. Our theoretical analysis shows that there is an inherent tension between standard accuracy and adversarial robustness. At the core of this trade-off is the concept of robust and non-robust features. The robustness of a feature is characterized by the strength of its correlation with the correct label. It is natural to wonder whether this concept of robust features is an artifact of our theoretical analysis or if it manifests more broadly. We thus investigate this issue experimentally on a dataset that is amenable to linear classifiers, MNIST (details in Appendix A).Recall the goal of standard classification for linear classifiers is to predict accurately, i.e. y = sign(w x). Hence the correlation of a feature i with the true label, computed as |E[yx i]|, quantifies how useful this feature is for classification. In the adversarial setting, against an ε ∞ -bounded adversary we need to ensure that y = sign(w x − εy w 1). In that case we expect a feature i to be DISPLAYFORM0 This calculation suggests that in the adversarial setting, there is an implicit threshold on feature correlations imposed by the threat model (the perturbation allowed to the adversary). While standard models may utilize all features with non-zero correlations, a robust model cannot rely on features with correlation below this threshold. In FIG6 (b), we visualize the correlation of each pixel (feature) in the MNIST dataset along with the learned weights of the standard and robust classifiers. As expected, we see that the standard classifier assigns weights even to weakly-correlated pixels so as to maximize prediction confidence. On the other hand, the robust classifier does not assign any weight below a certain correlation threshold which is dictated by the adversary's strength (ε) FIG6 )Interestingly, the standard model assigns non-zero weight even to very weakly correlated pixels (FIG6). In settings with finite training data, such non-robust features could arise from noise.(For instance, in N tosses of an unbiased coin, the expected imbalance between heads and tails is O( √ N) with high probability.) A standard classifier would try to take advantage of even this "hallucinated" information by assigning non-zero weights to these features. TAB4.) (a) Visualization of network weights per input feature. (b) Comparison of feature-label correlation to the weight assigned to the feature by each network. Adversarially trained networks put weights only on a small number of strongly-correlated or "robust" features. (c) Performance of a model trained using standard training only on the most robust features. Specifically, we sort features based on decreasing correlation with the label and train using only the most correlated ones. Beyond a certain threshold, we observe that as more non-robust or (weakly correlated) features are available to the model, the standard accuracy increases at the cost of robustness. The analysis above highlights an interesting trade-off between the predictive power of a feature and its vulnerability to adversarial perturbations. This brings forth the question -Could we use these insights to train robust classifiers with standard methods (i.e. without performing adversarial training)? As a first step, we train a (standard) linear classifier on MNIST utilizing input features (pixels) that lie above a given correlation threshold (see FIG6). As expected, as more non robust features are incorporated in training, the standard accuracy increases at the cost of robustness. Further, we observe that a standard classifier trained in this manner using few robust features attains better robustness than even adversarial training. This suggest a more direct (and potentially better) method of training robust networks in certain settings. F derive parameter-dependent bounds on the robustness of any fixed classifier. Our focus on the statistical setting itself and provide lower bounds for all classifiers learned in this setting. analyze the adversarial robustness of nearest neighbor classifiers. Instead we focus on lower bounds that are inherent to the statistical setting itself and apply to all classifiers. study the generalization aspect of adversarially robustness. They show that the number of samples needed to achieve adversarially robust generalization is polynomially larger in the dimension than the number of samples needed to ensure standard generalization. However, in the limit of infinite data, one can learn classifiers that are both robust and accurate. BID20 demonstrate a setting where even a small amount of standard error implies that most points provably have a misclassified point close to them. In this setting, achieving perfect standard accuracy (easily achieved by a simple classifier) is sufficient to achieve perfect adversarial robustness. In contrast, our work focuses on a setting where adversarial training (provably) matters and there exists a trade-off between standard and adversarial accuracy. explore the connection between robustness and generalization, showing that, in a certain sense, robustness can imply generalization. This direction is orthogonal to our, since we work in the limit of infinite data, optimizing the distributional loss directly. BID17 prove lower bounds on the robustness of any classifier based on certain generative assumptions. Since these bounds apply to all classifiers, independent of architecture and training procedure, they fail to capture the situation we face in practice where robust optimization can significantly improve the adversarial robustness of standard classifiers BID13;; ).A recent work BID6 turns out to (implicitly) rely on the distinction between robust and non-robust features in constructing a distribution for which adversarial robustness is hard from a different, computational point of view. BID23 observed that adversarial training in feature weights that depend on fewer input features (similar to FIG6). Additionally, it has been observed that for naturally trained RBF classifiers on MNIST, targeted adversarial attacks resemble images of the target class BID21. empirically observe a similar trade-off between the accuracy and robustness of standard models across different deep architectures on ImageNet. BID3 study an extreme multi-label problem and observe that for classes with relatively few examples, 1 -regularization (which corresponds to adversarial training for linear models) is helpful, while for classes with more samples, it is harmful to the model accuracy. Comparison of standard accuracies of models trained against an ∞ -bounded adversary as a function of the size of the training dataset. We observe that in the low-data regime, adversarial training has an effect similar to data augmentation and helps with generalization in certain cases (particularly on MNIST). However, in the limit of sufficient training data, we see that the standard accuracy of robust models is less than that of the standard model (ε train = 0), which supports the theoretical analysis in Section 2.1. Figure 7: Standard test accuracy of adversarially trained classifiers. The adversary used during training is constrained within some p -ball of radius ε train (details in Appendix A). We observe a consistent decrease in accuracy as the strength of the adversary increases. FIG0: Visualization of the gradient of the loss with respect to input features (pixels) for standard and adversarially trained networks for 10 randomly chosen samples, similar to those in Figure 2. Gradients are significantly more interpretable for adversarially trained networks -they align almost perfectly with perceptually relevant features. For MNIST, blue and red pixels denote positive and negative gradient regions respectively. For CIFAR10 and Restricted ImageNet we clip pixel to 3 standard deviations and scale to.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that adversarial robustness might come at the cost of standard classification performance, but also yields unexpected benefits.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.