Question: Represent the scientific article to find a short TLDR:

Question: Adversarial training is one of the strongest defenses against adversarial attacks, but it requires adversarial examples to be generated for every mini-batch during optimization. The expense of producing these examples during training often precludes adversarial training from use on complex image datasets. In this study, we explore the mechanisms by which adversarial training improves classifier robustness, and show that these mechanisms can be effectively mimicked using simple regularization methods, including label smoothing and logit squeezing. Remarkably, using these simple regularization methods in combination with Gaussian noise injection, we are able to achieve strong adversarial robustness -- often exceeding that of adversarial training -- using no adversarial examples. Deep Neural Networks (DNNs) have enjoyed great success in many areas of computer vision, such as classification BID7, object detection BID4, and face recognition BID11. However, the existence of adversarial examples has raised concerns about the security of computer vision systems BID16 BID1. For example, an attacker may cause a system to mistake a stop sign for another object BID3 or mistake one person for another BID14. To address security concerns for high-stakes applications, researchers are searching for ways to make models more robust to attacks. Many defenses have been proposed to combat adversarial examples. Approaches such as feature squeezing, denoising, and encoding BID19 BID13 BID15 BID10 have had some success at pre-processing images to remove adversarial perturbations. Other approaches focus on hardening neural classifiers to reduce adversarial susceptibility. This includes specialized non-linearities BID20, modified training processes BID12, and gradient obfuscation BID0.Despite all of these innovations, adversarial training BID5, one of the earliest defenses, still remains among the most effective and popular strategies. In its simplest form, adversarial training minimizes a loss function that measures performance of the model on both clean and adversarial data as follows DISPLAYFORM0 where L is a standard (cross entropy) loss function, (x i, y i) is an input image/label pair, θ contains the classifier's trainable parameters, κ is a hyper-parameter, and x i,adv is an adversarial example for image x. BID9 pose adversarial training as a game between two players that similarly requires computing adversarial examples on each iteration. A key drawback to adversarial training methods is their computational cost; after every mini-batch of training data is formed, a batch of adversarial examples must be produced. To train a network that resists strong attacks, one needs to train with the strongest adversarial examples possible. For example, networks hardened against the inexpensive Fast Gradient Sign Method can be broken by a simple two-stage attack BID17. Current state-of-theart adversarial training on MNIST and CIFAR-10 use expensive iterative adversaries BID9, such as the Projected Gradient Descent (PGD) method, or the closely related Basic Iterative Method (BIM) BID8. Adversarial training using strong attacks may be 10-100 times more time consuming than standard training methods. This prohibitive cost makes it difficult to scale adversarial training to larger datasets and higher resolutions. In this study, we show that it is possible to achieve strong robustness -comparable to or greater than the robustness of adversarial training with a strong iterative attack -using fast optimization without adversarial examples. We achieve this using standard regularization methods, such as label smoothing BID18 and the more recently proposed logit squeezing BID6. While it has been known for some time that these tricks can improve the robustness of models, we observe that an aggressive application of these inexpensive tricks, combined with random Gaussian noise, are enough to match or even surpass the performance of adversarial training on some datasets. For example, using only label smoothing and augmentation with random Gaussian noise, we produce a CIFAR-10 classifier that achieves over 73% accuracy against black-box iterative attacks, compared to 64% for a state-of-the-art adversarially trained classifier BID9. In the white-box case, classifiers trained with logit squeezing and label smoothing get ≈ 50% accuracy on iterative attacks in comparison to ≈ 47% for adversarial training. Regularized networks without adversarial training are also more robust against non-iterative attacks, and more accurate on non-adversarial examples. Our goal is not just to demonstrate these defenses, but also to dig deep into what adversarial training does, and how it compares to less expensive regularization-based defenses. We begin by dissecting adversarial training, and examining ways in which it achieves robustness. We then discuss label smoothing and logit squeezing regularizers, and how their effects compare to those of adversarial training. We then turn our attention to random Gaussian data augmentation, and explore the importance of this technique for adversarial robustness. Finally, we combine the regularization methods with random Gaussian augmentation, and experimentally compare the robustness achievable using these simple methods to that achievable using adversarial training. Adversarial training injects adversarial examples into the training data as SGD runs. During training, adversarial perturbations are applied to each training image to decrease the logit corresponding to its correct class. The network must learn to produce logit representations that preserve the correct labeling even when faced with such an attack. At first glance, it seems that adversarial training might work by producing a large "logit gap," i.e., by producing a logit for the true class that is much larger than the logit of other classes. Surprisingly, adversarial training has the opposite effect -we will see below that it decreases the logit gap. To better understand what adversarial training does, and how we can replicate it, we now break down the different strategies for achieving robustness. This section presents a simple metric for adversarial robustness that will help us understand adversarial training. Consider an image x, and its logit representation z (i.e. pre-softmax activation) produced by a neural network. Let z y denote the logit corresponding to the correct class y. If we add a small perturbation δ to x, then the corresponding change in logits is approximately δ T ∇ x z y under a linear approximation, where ∇ x z y is the gradient of z y with respect to x. Under a linearity assumption, we can calculate the step-size L needed to move an example from class y to another classȳ. A classifier is susceptible to adversarial perturbation δ if the perturbed logit of the true class is smaller than the perturbed logit of any other class: DISPLAYFORM0 Assuming a one-step ∞ attack such as FGSM, the perturbation δ can be expressed as DISPLAYFORM1 where L is the ∞ -norm of the perturbation. Using this choice of δ, Equation 2 becomes Table 1: Experimental and predicted accuracy of classifiers for MNIST and CIFAR-10. The predicted accuracy is the percentage of images for which < L. The empirical accuracy is the percent of images that survive a perturbation of size. Attacks on both the cross-entropy (X-ent) and logits as in BID2 (CW) are presented. DISPLAYFORM2 where · 1 denotes the 1 -norm of a vector. Therefore the smallest ∞ -norm of the perturbation required is the ratio of "logit gap" to "gradient gap", i.e., DISPLAYFORM3 Equation FORMULA4 measures robustness by predicting the smallest perturbation size needed to switch the class of an image. While the formula for L makes linearity assumptions, the approximation L fairly accurately predicts the robustness of classifiers of the CIFAR-10 dataset (where perturbations are small and linearity assumptions cause little distortion). It is also a good ballpark approximation on MNIST, even after adversarial training (see Table 1).Maximal robustness occurs when L is as large as possible. From equation 4, we observe 3 different strategies for hardening a classifier:• Increase the logit gap: Maximize the numerator of equation 4 by producing a classifier with relatively large z y.• Squash the adversarial gradients: Train a classifier that has small adversarial gradients ∇ x zȳ for any classȳ. In this case a large perturbation is needed to significantly change zȳ.• Maximize gradient coherence: Produce adversarial gradients ∇ x zȳ that are highly correlated with the gradient for the correct class ∇ x z y. This will shrink the denominator of equation 4, and produce robustness even if adversarial gradients are large. In this case, one cannot decrease z y without also decreasing zȳ, and so large perturbations are needed to change the class label. The most obvious strategy for achieving robustness is to increase the numerator in equation 4 while fixing the denominator. Remarkably, our experimental investigation reveals that adversarial training does not rely on this strategy at all, but rather it decreases the logit gap and gradient gap simultaneously. This can be observed in FIG0, which shows distributions of logit gaps for naturally and adversarially trained models on MNIST. Note that the cross entropy loss actually limits adversarial training from increasing logit gaps. The accuracy of the classifier goes down in the presence of adversarial examples, and so the cross entropy loss is minimized by smaller logit gaps that reflect the lower level of certainty in the adversarial training environment. Adversarial training succeeds by minimizing the denominator in Equation 4; it simultaneously squeezes the logits and crushes the adversarial gradients. FIG0 shows that the adversarial gradients shrink dramatically more than the logit gaps, and so the net effect is an increase in robustness. If we closely examine the phenomenon of shrinking the logit gaps, we find that this shrink is due in part to an overall shrink in the size of the logits themselves, (i.e., |z i | for any class i). To see this, we plot histograms of the logits when classifiers are adversarially trained with strong adversaries 1, weak adversaries 2, and with no adversarial examples. FIG1 shows that adversarial training does indeed squash the logits, although not enough to fully explain the decrease in |z y − zȳ| in FIG0. 3 We have seen that adversarial training works by squashing adversarial gradients and slightly increasing gradient coherence. But the fact that it cannot do this without decreasing the logit gap leads us to suspect that these quantities are inherently linked. This leads us to ask an important question: If we directly decrease the logit gap, or the logits themselves, using an explicit regularization term, will this have the desired effect of crushing the adversarial gradients? There are two approaches to replicating the effect on the logits produced by adversarial training. The first is to replicate the decrease in logit gap seen in FIG0. This can be achieved by label smoothing. A second approach to replicating adversarial training is to just directly crush all logit values and mimic the behavior in FIG1. This approach is known as "logit squeezing," and works by adding a regularization term to the training objective that explicitly penalizes large logits. Label smoothing Label smoothing converts "one-hot" label vectors into "one-warm" vectors that represents a low-confidence classification. Because large logit gaps produce high-confidence classifications, label-smoothed training data forces the classifier to produce small logit gaps. Label smoothing is a commonly used trick to prevent over-fitting on general classification problems, and it was first observed to boost adversarial robustness by BID18, where it was used as an inexpensive replacement for the network distillation defense BID12. A one-hot label vector y hot is smoothed using the formula DISPLAYFORM0 where α ∈ is the smoothing parameter, and N c is the number of classes. If we pick α = 0 we get a hard decision vector with no smoothing, while α = 1 creates an ambiguous decision by assigning equal certainty to all classes. Logit squeezing A second approach to replicating adversarial training is to just directly crush all logit values and mimic the behavior in FIG1. This approach is known as "logit squeezing," and works by adding a regularization term to the training objective that explicitly penalizes large logits. BID6 were the first to introduce logit-squeezing as an alternative to a "logit pairing" defense. Logit squeezing relies on the loss function DISPLAYFORM1 where β is the squeezing parameter (i.e., coefficient for the logit-squeezing term) and ||.|| F is the Frobenius norm of the logits for the mini-batch. Can such simple regularizers really replicate adversarial training? Our experimental suggest that simple regularizers can hurt adversarial robustness, which agrees with the findings in BID20. However, these strategies become highly effective when combined with a simple trick from the adversarial training literature -data augmentation with Gaussian noise. Adding Gaussian noise to images during training (i.e, Gaussian noise augmentation) can be used to improve the adversarial robustness of classifiers BID6 BID20. However, the effect of Gaussian noise is not well understood BID6. Here, we take a closer look at the behavior of Gaussian augmentation through systematic experimental investigations, and see that its effects are more complex than one might think. Label smoothing and logit squeezing become shockingly effective at hardening networks when they are combined with Gaussian noise augmentation. From the robustness plots in FIG2, we can see that training with Gaussian noise alone produces a noticeable change in robustness, which seems to be mostly attributable to a widening of the logit gap and slight decrease in the gradient gap (∇ x z y − ∇ x zȳ 1). The small increase in robustness from random Gaussian augmentation was also reported by BID6. We also see that label smoothing alone causes a very slight drop in robustness; the shrink in the gradient gap is completely offset by a collapse in the logit gap. Surprisingly, Gaussian noise and label smoothing have a powerful synergistic effect. When used together they cause a dramatic drop in the gradient gap, leading to a surge in robustness. A similar effect happens in the case of logit squeezing, and are shown in Appendix B (Figure 8). Regularization methods have the potential to squash or align the adversarial gradients, but these properties are only imposed during training on images from the manifold that the "true" data lies on. At test time, the classifier sees adversarial images that do not "look like" training data because they lie off of, but adjacent to, the image manifold. By training the classifier on images with random perturbations, we teach the classifier to enforce the desired properties for input images that lie off the manifold. The generalization property of Gaussian augmentation seems to be independent from, and sometimes conflicting with, the synergistic properties discuss above. In our experiments below, we find that smaller noise levels lead to a stronger synergistic effect, and yield larger L and better robustness to FGSM attacks. However, larger noise levels enable the regularization properties to generalize further off the manifold, ing in better robustness to iterative attacks or attacks that escape the flattened region by adding an initial random perturbation. See the on MNIST in Table 2 and the on CIFAR-10 in TAB3 for various values of σ (standard deviation of Gaussian noise). For more comprehensive experiments on the different parameters that contribute to the regularizers see Table 6 for MNIST and Tables 7 & 8 Label smoothing (i.e. reducing the variance of the logits) is helpful because it causes the gradient gap to decrease. The decreased gradient gap may be due to smaller element-wise gradient amplitudes, the alignments of the adversarial gradients, or both. To investigate the causes, we plot the 1 norm of the gradients of the logits with respect to the input image 4 and the cosine of the angle between the gradients FIG3. We see that in label smoothing (with Gaussian augmentation), both the gradient magnitude decreases and the gradients get more aligned. Larger smoothing parameters α cause the gradient to be both smaller and more aligned. When logit squeezing is used with Gaussian augmentation, the magnitudes of the gradients decrease. The distribution of the cosines between gradients widens, but does not increase like it did for label smoothing. These effects are very similar to the behavior of adversarial training in FIG0. Interestingly, in the case of logit squeezing with Gaussian noise, unlike label smoothing, the numerator of Equation 4 increases as well. This increase in the logit gap disappears once we take away Gaussian augmentation (See Appendix B Figure 8). Simultaneously increasing the numerator and decreasing the denominator of Equation 4 potentially gives a slight advantage to logit squeezing. There are multiple factors that can affect the robustness of the MNIST classifier 5. While regularizers do not yield more robustness than adversarial training for MNIST, the are promising given that these relatively high values of robustness come at a cheap cost in comparison to adversarial training. In Table 2 we notice that as we increase the number of training iterations k, we get more robust models for both logit squeezing and label smoothing 6. We get more robust models when we use larger smoothing (α) and squeezing (β) parameters, and when Gaussian augmentation is used with standard deviation σ that is greater than the desired (the maximum perturbation size). Table 2: Accuracy of different MNIST classifiers against PGD and FGSM attacks on X-ent and CW losses under the white-box and black-box threat models. Attacks have maximum ∞ perturbation = 0.3. The iterative white-box attacks have an initial random step. The naturally trained model is used for generating black-box attacks. We use CW loss for the black-box attack. We trained Wide-Resnet CIFAR-10 classifiers (depth=28 and k=10) using aggressive values for the smoothing and squeezing parameters on the CIFAR10 data set. Similar to BID9, we use the standard data-augmentation techniques and weight-decay. We compare our to those of BID9. Note that the adversarially trained model from BID9 has been trained for 80,000 iterations on adversaries which are generated using a 7-step PGD. Keeping in mind that each step requires a forward and backward pass, the running time of training for 80,000 iterations on 7-step PDG examples is equivalent to 640,000 iterations of training with label smoothing or logit squeezing. A short version of our on white-box attacks are summarized in TAB3. The of some of our black-box experiments are summarized in Table 4 7. While logit squeezing seems to outperform label smoothing in the white-box setting, label smoothing is slightly better under the black-box threat. We see that aggressive logit squeezing with squeezing parameter β = 10 and σ = 20 in a model that is more robust than the adversarially trained model from BID9 BID9 63.39%* 64.38%* 63.39%* 64.38%* 67.00% 67.25% Table 4: Black-box attacks on CIFAR-10 models. Attacks are ∞ with = 8. Similar to BID9, We build 7-step PGD attacks and FGSM attacks for a public adversarially trained model. *Values taken from the original paper by BID9. Some defenses work by obfuscating gradients and masking the gradients. BID0 suggest these models can be identified by performing "sanity checks" such as attacking them with Figure 5: The cross-entropy landscape of the first eight images of the validation set for the model trained for 160k iterations with hyper-parameters β = 10 and σ = 30. To plot the loss landscape we take two random directions r 1 and r 2 (i.e. r 1, r 2 ∼ Rademacher(0.5) ). We plot the cross-entropy (i.e. xent) loss at different points x = x i + 1 · r 1 + 2 · r 2. Where x i is the clean image and −10 ≤ 1, 2 ≤ 10.unbounded strong adversaries (i.e. unbounded with many iterations). By attacking our robust models using these unbounded attacks, we verify that the unbounded adversary can degrade the accuracy to 0.00% which implies that the adversarial example generation optimization attack (PGD) is working properly. Also, it is known that models which do not completely break the PGD attack (such as us) can possibly lead to a false sense of security by creating a loss landscape that prevents an -bounded but weak adversary from finding strong adversarial examples. This can be done by convoluting the loss landscape such that many local-optimal points exist. This false sense of security can be identified by increasing the strength of the adversary using methods such as increasing the number of steps and random restarts of the PGD attack. We run the stronger PGD attacks on a sample model from TAB3 with hyper-parameters k = 160k, β = 10, and σ = 30. We notice that performing 9 random restarts for the PGD attack on the cross-entropy loss only drops the accuracy slightly to 49.86%. Increasing the number of PGD steps to 1000 decreases the accuracy slightly more to 40.94% 8. While for such strong iterative white-box attacks our sample model is less robust than the adversarially trained model, there are other areas where this hardened model is superior to the adversarially trained model: Very high robustness in the black box setting (roughly 3% higher than that for adversarial training according to Table 4) and against white-box non-iterative (or less iterative) attacks (roughly 15%); high test accuracy on clean data (roughly 3%); and, very fast training time compared to adversarial training. To further verify that our robust model is not falsely creating a sense of robustness by "breaking" the optimization problem that generates adversarial examples by either masking the gradients or making the loss landscape convoluted, we visualize the loss landscape of our sample model from TAB3. We plot the classification (e.g., cross-entropy) loss for points surrounding the first eight validation images that belong to the subspace spanned by two random directions 9 in Figure 5. It seems that the loss landscape has not become convoluted. This observation is backed up by the fact that increasing the number of PGD attack steps does not substantially affect accuracy. To check the performance of our proposed regularizers on more complicated datasets with more number of classes, we perform aggressive logit squeezing on the CIFAR-100 dataset which contains 100 categories. We use the same architecture and settings used for training the CIFAR-10 classifiers. The white-box performance of two hardened models with logit squeezing and a PGD adversarially trained model for the same architecture are summarized in Table 5 Table 5: White-box iterative attacks on CIFAR-100 models. We use ∞ attacks with = 8. For brevity, we only report the for attacking the cross-entropy loss. We attack the models with adversaries having different strengths by varying the number of PGD steps. Similar to CIFAR-10, aggressive logit squeezing can in models as robust as those that are adversarially trained at a fraction of the cost. The logit-squeezed model that is trained for only 80k iterations achieves high classification accuracy for natural/clean examples. It is also more robust against white-box PGD attacks compared to the adversarially trained model that requires much more training time. The logit-squeezed model with k = 160k improves the robustness and clean accuracy even further and still trains faster than the adversarially trained model (28.8 hours vs 34.3 hours). We studied the robustness of adversarial training, label smoothing, and logit squeezing through a linear approximation L that relates the magnitude of adversarial perturbations to the logit gap and the difference between the adversarial directions for different labels. Using this simple model, we observe how adversarial training achieves robustness and try to imitate this robustness using label smoothing and logit squeezing. The ing methods perform well on MNIST, and can get on CIFAR-10 and CIFAR-100 that can excel over adversarial training in both robustness and accuracy on clean examples. By demonstrating the effectiveness of these simple regularization methods, we hope this work can help make robust training easier and more accessible to practitioners. Similarly to what we observed about adversarial training on MNIST, adversarial training on CIFAR-10 works by greatly shrinking the adversarial gradients and also shrinking the logit gaps. The shrink in the gradients is much more dramatic than the shrink in the logit gap, and overwhelms the decrease in the numerator of Equation 4. See FIG4. As shown empirically in Table 6, and analytically using the linear approximation in Equation 4 evaluated in Figure 8, logit squeezing worsens robustness when Gaussian augmentation is not used. However, when fused with Gaussian augmentation, logit squeezing achieves good levels of robustness. This addition of Gaussian augmentation has three observed effects: the gradients get squashed, the logit gap increases, and the gradients get slightly more aligned. The increase in the logit gaps increases the numerator in Equation 4. This gives a slight edge to logit squeezing in comparison to label smoothing, that mostly works by decreasing the denominator in Equation 4. The for all of our experiments on MNIST are summarized in Table 6. As can be seen, Gaussian random augmentation by itself (β = α = 0) is effective in increasing robustness on black-box attacks. It does not, however, significantly increase robustness on white-box attacks. Models that are only trained with either logit squeezing or label smoothing without random Gaussian augmentation (σ = 0), can be fooled by an adversary that has knowledge of the model parameters (white-box) with accuracy 100%. In the black-box setting, they are also not robust. Increasing the magnitude of the noise (σ) generally increases robustness but degrades the performance on the clean examples. Keeping the number of iterations k constant, for extremely large σ Table 6: Accuracy of different models trained on MNIST with a 40 step PGD attack on the crossentropy (X-ent) loss and the Carlini-Wagner (CW) loss under the white-box and black-box threat models. Attacks are ∞ attacks with a maximum perturbation of = 0.3. The iterative white-box attacks have an initial random step. The naturally trained model was used for generating the attacks for the black-box threat model. We use the CW loss for the FGSM attack in the blackbox case. k is the number of training iterations. Here we take a deeper look at reguarlized training for CIFAR-10. The that can be drawn in this case are parallel with those of MNIST discussed in Appendix C. It is worth noting that while the of logit squeezing outperform those from label smoothing in the white-box setting, training with large squeezing coefficient β often fails and in low accuracy on test data. This breakdown of training rarely happens for label smoothing (even for very large smoothing parameters α). BID9 100.00% 87.25% 45.84% 46.90% 56.22% 55.57% Table 7: White-box attacks on the CIFAR-10 models. All attacks are ∞ attacks with = 8. For the 20-step PGD, similar to BID9, we use an initial random perturbation. While it seems that logit squeezing, label smoothing, and adversarial training have a lot in common when we look at quantities affecting the linear approximation L, we wonder whether they still look similar with respect to other metrics. Here, we look at the sum of the activations in the logit layer for every logit FIG0 ) and the sum of activations for every neuron of the penultimate layer (FIG0). The penultimate activations are often seen as the "feature representation" that the neural network learns. By summing over the absolute value of the activations of all test examples for every neuron in the penultimate layer, we can identify how many of the neurons are effectively inactive. When we perform natural training, all neurons become active for at least some images. After adversarial training, this is no longer the case. Adversarial training is causing the effective dimensionality of the deep feature representation layer to decrease. One can interpret this as adversarial training learning to ignore features that the adversary can exploit (400 out of the 1024 neurons of the penultimate layer are deactivated). Shockingly, both label smoothing and logit squeezing do the samethey deactivate roughly 400 neurons from the deep feature representation layer. BID9 63.39%* 64.38%* 63.39%* 64.38%* 67.00% 67.25% Table 8: Black-box attacks on the CIFAR-10 models. All attacks are ∞ attacks with = 8. Similar to BID9, We build 7-step PGD attacks and FGSM attacks for the public adversarial trained model of MadryLab. We then use the built attacks for attacking the different models. *: Since we do not have the Madry model, we cannot evaluate it under the PGD attack with and without random initialization and therefore we use the same value that is reported by them for both. As a sanity check, and to verify that the robustness of our models are not due to degrading the functionality of PGD attacks, here we verify that our models indeed have zero accuracy when the adversary is allowed to make huge perturbations. In Table 9 by performing an unbounded PGD attack on a sample logit-squeezed model for the CIFAR-10 (β = 10, σ = 30, and k = 160k), we verify that our attack is not benefiting from obfuscating the gradients. Table 9: The effect of unbounded on the accuracy. The decline in the accuracy as a sanity check shows that the sample model is at least not completely breaking the PGD attack and is not obfuscating the gradients. We attack our sample hardened CIFAR-10 model (β = 10, Gaussian augmentation σ = 30 model, and k = 160k iterations), by performing many random restarts. Random restarts can potentially increase the strength of the PGD attack that has a random initialization. As shown in Table 10, increasing the number of random restarts does not significantly degrade the robustness. It should be noted that attacking with more iterations and more random restarts hurts adversarially trained models as well (see the leaderboard in Madry's Cifar10 Github Repository). Table 10: The effect of the number of random restarts while generating the adversarial examples on the accuracy of a model trained with the logit squeezing. It shows that the accuracy plateaus at 9 random restarts. Another way to increase the strength of an adversary is by increasing the number of PGD steps for the attack. We notice that increasing the number of steps does not greatly affect the robustness of our sample logit-squeezed model (See Table 11). Table 11: The effect of the number of steps of the white-box PGD attack on the CW loss (worst case based on TAB3) for the model trained in 160k steps with logit squeezing parameters β = 10 and σ = 30 on CIFAR-10 dataset. The model remains resistant against ∞ attacks with = 8 Figure 9: The cross-entropy landscape of the first eight images of the validation set for the model trained for 160k iteration and hyper-parameters β = 10 and σ = 30. To plot the loss landscape we take walks in one random direction r 1 ∼ Rademacher(0.5) and the adversarial direction a 2 = sign(∇ x xent) where xent is the cross-entropy loss. We plot the cross-entropy (i.e. xent) loss at different points x = x i + 1 · r 1 + 2 · a 2. Where x i is the clean image and −10 ≤ 1, 2 ≤ 10. As it can be seen moving along the adversarial direction changes the loss value a lot and moving along the random direction does not make any significant major changes. Similar to Figure 5, we plot the classification loss landscape surrounding the input images for the first eight images of the validation set. Unlike in Figure 5 which we changed the clean image along two random directions, in Figure 9, we wander around the clean image by moving in the adversarial direction and one random direction. From Figure 9 we observe that the true direction that the classification loss changes is along the adversarial direction which illustrates that the logit-squeezed model is not masking the gradients. 2) with random noise, and logit squeezing (β = 0.5) with random noise. In all cases, the noise is Gaussian with σ = 0.5. Interestingly, the combination of Gaussian noise and label smoothing, similar to the combination of Gaussian noise and logit squeezing, deactivates roughly 400 neurons. This is similar to adversarial training. In some sense it seems that all three methods are causing the "effective" dimensionality of the deep feature representation layer to shrink. FIG0: For MNIST, we plot the cumulative sum of activation magnitudes for all neurons in logit layer of a network produced by natural training, adversarial training, natural training with random noise, label smoothing (LS = 0.2) with random noise, and logit squeezing (β = 0.5) with random noise. In all cases, the noise is Gaussian with σ = 0.5.

Achieving strong adversarial robustness comparable to adversarial training without training on adversarial examples

represent the short summary for finding the corresponding scientific article (long) Examples:
Provided: "We introduce an alternative GAN design based on random routes in generator, which can serve as a tool for generative models interpretability." Match: "In this paper, we introduce Random Path Generative Adversarial Network (RPGAN) --- an alternative scheme of GANs that can serve as a tool for generative model analysis. While the latent space of a typical GAN consists of input vectors, randomly sampled from the standard Gaussian distribution, the latent space of RPGAN consists of random paths in a generator network. As we show, this design allows to associate different layers of the generator with different regions of the latent space, providing their natural interpretability. With experiments on standard benchmarks, we demonstrate that RPGAN reveals several interesting insights about roles that different layers play in the image generation process. Aside from interpretability, the RPGAN model also provides competitive generation quality and allows efficient incremental learning on new data. Nowadays, deep generative models are an active research direction in the machine learning community. The dominant methods for generative modeling, such as Generative Adversarial Networks (GANs), are currently able to produce diverse photorealistic images . These methods are not only popular among academicians, but are also a crucial component in a wide range of applications, including image editing (;, super-resolution , video generation and many others. Along with practical importance, a key benefit of accurate generative models is a more complete understanding of the internal structure of the data. Insights about the data generation process can both in the development of new machine learning techniques as well as advances in industrial applications. However, most state-of-the-art generative models employ deep multi-layer architectures, which are difficult to interpret or explain. While many works investigate interpretability of discriminative models (; ;), only a few address the understanding of generative ones. In this work, we propose the Random Path GAN (RPGAN) -an alternative design of GANs that allows natural interpretability of the generator network. In traditional GAN generators, the stochastic component that influences individual samples is a noisy input vector, typically sampled from the standard Gaussian distribution. In contrast, RPGAN generators instead use stochastic routing during the forward pass as their source of stochasticity. In a nutshell, the RPGAN generator contains several instances of the corresponding layer. For each sample, only one random instance of each layer is activated during generation. The training of the RPGAN can then be performed in the same adversarial manner as in traditional GANs. In the sections below, we show how RPGAN allows to understand the factors of variation captured by the particular layer and reveals several interesting findings about the image generation process, e.g. that different layers are "responsible for" coloring or objection location. As a practical advantage, RPGANs can be efficiently updated to new data via the simple addition of new instances to the bucket, avoiding re-training the full model from scratch. Finally, we observe that RPGANs allow the construction of generative models without nonlinearities, which can significantly speed up the generation process for fully-connected layers. In summary, the main contributions of our paper are the following: • We introduce RPGAN -GAN with an alternative source of stochasticity, based on random routing. While being close to traditional GANs in terms of generation quality, RPGAN allows natural interpretability and efficient model updates with new data. • With extensive experiments on standard benchmarks we reveal several insights about the image generation process. Many of our insights confirm and extend recent findings from. Note, that our scheme is more general compared to the technique from as RPGAN does not require labeled datasets or pretrained segmentation models. • We open-source the PyTorch implementation of RPGAN with common generator architectures 1. The rest of this paper is organized as follows. In Section 2 we review relevant ideas from prior art. The proposed Random Path GAN design is described in Section 3 and experimentally evaluated in Section 4. Section 5 concludes the paper and discusses possible directions for future work. In this section we briefly describe connections of RPGAN to existing ideas from prior works Generative adversarial networks. GANs are currently one of the main paradigms in generative modelling. Since the seminal paper on GANs by , a plethora of alternative loss functions, architectures, normalizations, and regularization techniques were developed . Today, state-of-the-art GANs are able to produce high-fidelity images, often indistinguishable from real ones . In essence, GANs consist of two networks -a generator and a discriminator, which are trained jointly in an adversarial manner. In standard GANs, the generation stochasticity is provided by the input noise vector. In RPGANs, we propose an alternative source of stochasticity by using a fixed input but random routes during forward pass in the generator. Specific GAN architectures. Many prior works investigated different design choices for GANs, but to the best of our knowledge, none of them explicitly aimed to propose an interpretable GAN model. proposed the use of several independent generators to address the mode collapse problem. employ several auxiliary local generators and discriminators to improve mode coverage as well. use layer-wise generators and discriminators to enforce hidden representations produced by layers of the generator to be similar to the corresponding representations produced by a reversed classification network. Important differences of RPGAN compared to the works described above is that it uses random routes as its latent space and does not enforce to mimic the latent representations of pretrained classifiers. Interpretability. While interpretability of models based on deep neural networks is an important research direction, most existing work addresses the interpretability of discriminative models. These works typically aim to understand the internal representations of networks (; ; ;) or explain decisions produced by the network for particular samples (; ;). However, only a few works address interpretability of generative models. A related work by develops a technique that allows to identify which parts of the generator are responsible for the generation of different objects. In contrast, we propose GANs with alternative source of stochasticity that allows natural interpretation by design. Some of our findings confirm the from , which provides stronger evidence about the responsibilities of different layers in the generation process. Note, that the technique requires a pretrained segmentation network and cannot be directly applied to several benchmarks, e.g. CIFAR-10 or MNIST. In contrast, RPGAN does not require any auxiliary models or supervision and can be applied to any data. . For instance, earlier layers aim to detect small texture patterns, while activations in deeper layers typically correspond to semantically meaningful concepts. Similarly, in our paper we aim to understand the roles that different GAN layers play in image generation. Thus, we propose an architecture that provides a direct way to interpret the impact of individual layers. For a given generator architecture, we construct several copies of each layer in its architecture. During the forward pass we randomly choose a layer instance that will be used when generating a particular image. Therefore, we can analyze the role of each RPGAN layer by visualizing how different instances of that layer affect the generated image. Here we formally describe the structure of the RPGAN model. The model is highly flexible to the choice of generator and discriminator architectures as well as to the loss function and learning strategy. Similarly to the standard GAN architectures, our model consists of two networks -a generator and a discriminator. The RPGAN discriminator operates exactly like discriminators in common GANs, hence below we focus on the generator description. Unlike existing GANs, the RPGAN generator always receives a fixed input vector Z during forward pass and aims to produce an image from the real image distribution. The generator consists of several consequent buckets B 1,..., B n. Each bucket is a union of independent blocks: B i ={B i1, . . ., B imi}, where each block is an arbitrary computational unit and m i =|B i |. A typical example of a block is a ResNet block , a convolutional layer with a nonlinearity or any other (see Figure 1a) layer type. In our experiments, all the units from the same bucket have the same architecture. For each i=1,..., n − 1 a block from the bucket B i produces an intermediate output tensor that is passed to a block from the next bucket B i+1. Typically we associate each bucket with a layer (or several layers) in the generator architecture, which we aim to interpret or analyze. A block from the first bucket B 1 always receives a fixed input vector Z, which is the same for different forward passes. The stochasticity of the generator arises from a random path that goes from Z to an output image, using only a single block from each bucket. Formally, during each forward pass, we randomly choose indices s 1,..., s n with 1 ≤ s i ≤ m i. The generator output is then computed as Figure 1b). Thus, the generator defines a map from the Cartesian product m 1 × m 2 × · · · × m n to the image space. Note that we can take an arbitrary existing GAN model, group its generator layers into buckets and replicate them into multiple blocks. In these terms, the original model can be treated as the RPGAN model with a single block in each bucket and random input noise. Note that during image generation we perform the same number of operations as in the standard GAN generator. By its design, RPGAN with buckets B 1,..., B n and a constant input Z is able to generate at most |B 1 |×· · ·×|B n | different samples were |B k | is the number of blocks in the bucket B k. Nevertheless, this number is typically much larger compared to the training set size. We argue that the probability space of random paths can serve as a latent space to generate high-quality images, as confirmed by the experiments below. Block diversity loss. To guarantee that blocks in a particular bucket are different, we also add a specific diversity term in the generator loss function. The motivation for this term is to prevent blocks B ki, B kj from learning the same weights. Let W be the set of all parameters of the generator. For each parameter w ∈ W there is a set of its instances {w,... w (mw) } in the RPGAN model. Then we enforce the instances to be different by the loss term Here we also normalize by the standard deviation s w of all parameters from different blocks that correspond to the same layer. This normalization effectively guarantees that all buckets contribute to the diversity term. Architecture. In all the experiments in this section, we use ResNet-like generators with spectral normalization and the hinge loss (SN-ResNet) described in. The blocks in the first bucket are fully-connected layers, the blocks in the last bucket are convolutional layers and blocks in all other buckets are residual blocks with two convolutions and a skip connection. If not stated otherwise, all the buckets have the same number of blocks. Additional experiments with other architectures are provided in Appendix. Datasets. We performed experiments on CIFAR-10 , LSUN-bedroom and Anime Faces datasets. For different datasets we use different numbers of discriminator steps per one generator step d steps and different numbers of blocks in a bucket n blocks. We summarize the main parameters used for three datasets in Table 1. In the last column we also report Coverage, which is the ratio of the latent space cardinality (which equals the number of buckets to the power n blocks) to the dataset size. Intuitively, large coverage guarantees that RPGAN has a sufficiently rich latent space of generator routes to capture the reference dataset. In the experiments below, we demonstrate that even moderate coverage is sufficient to generate high-fidelity images (see the LSUN-bedroom dataset with coverage ≈ 3.3). Training details. We use the Adam optimizer with learning rate equal to 0.25 × 10 −3, β 1, β 2 equal to 0.5, 0.999 and train the model for 45 × 10 4 generator steps for CIFAR-10 and 25 × 10 4 generator steps for Anime Faces and LSUN-bedroom datasets. During training we also learn the unique input vector Z. We observed that a learnable Z slightly improves the final generation quality and stabilizes the learning process. During the training step, we pass Z through N independent random paths. Formally, let {x 1, . . ., x N} be a batch of samples received from a bucket B k. To pass this batch through the bucket B k+1 we take random blocks B ki1,..., B ki N and form a new batch In all the experiments, we use the same training protocols for both the RPGAN and the standard GAN of the generator architecture. Note, that despite larger number of learnable parameters, RPGAN does not require more data or training time to achieve the same quality, compared to standard GANs. Figure 3: Top image: sample generated with a fixed sequence of blocks. Horizontal lines: samples generated by the same sequence of blocks in all buckets and one unfrozen bucket. In the selected bucket we choose ten arbitrary blocks to avoid excessively large figures. The samples produced in this way allow to interpret factors of variation, captured by different buckets. In the first series of experiments we investigate the " responsibility areas" of different generator layers. This can be performed with a technique schematically presented on Figure 2. The goal is to interpret the role of the third bucket B 3 in a five-bucket generator. For all other buckets B 1, B 2, B 4, B 5 we fix arbitrary blocks, shown in blue on Figure 2. Then we generate images corresponding to routes that contain all the fixed blocks, with the stochasticity coming only from varying blocks inside the target bucket B 3. By inspecting the distribution of the obtained images, we can understand what factors of variation are influenced by B 3. On Figure 3 we plot the samples generated with only one "unfrozen" bucket for one image from the CIFAR-10 dataset. Thus, each row shows how the original generated image could change if different blocks from the corresponding bucket are used. Several observations from Figure 3, Figure 9, Figure 12 and Figure 15 are listed below. The first bucket typically does not influence coloring and mostly affects small objects' deformations. The intermediate buckets have the largest influence on semantics. The last two buckets are mostly responsible for coloring and do not influence the content shape. In particular, on Figure 3 the fourth layer widely varies color, while the fifth acts as a general tone corrector. Note that these findings are consistent with the insights revealed by. To confirm the observations quantitatively, we perform the following experiment. We define a metric d img that evaluates the similarity between two generated images. Different metrics are able to capture different variations (e.g. in terms of semantic, color histogram, etc.) and we describe two particular choices of d img below. Then we choose a random route in the RPGAN generator and for each bucket 4, varying blocks in B l. In other words, rather then taking all possible pairs, we take four random samples from each line of the table in the Figure 3. Then we measure diversity w.r.t. d img captured by bucket B l as a ratio Intuitively, we compute the relative diversity with respect to the first layer, which typically captures the smallest amount of variations in our experiments. We then average these ratios over 100 independent evaluations. High values of averaged ratio D l→1,d img imply higher diversity of the bucket B l compared to the first bucket in terms of the metric d img. For d img we experimented with the following two metrics, capturing semantics and color differences correspondingly. Inspired by the well-known Fréchet Inception Score concept , we consider the Euclidean distance between the outputs of the last layer of the pretrained InceptionV3 network for semantic distance evaluation. Namely, we define d semantic (img 1, img 2) as Iv3(img 1) − Iv3(img 2) 2 were Iv3(img) is the InceptionV3 model activations for image img. To measure differences in color, we take the Hellinger distance between the color histograms of generated samples. Namely, for each color channel, we split the range [0, . . ., 255] into 25 equal segments and evaluate the discrete distribution defined by the frequencies the sample's pixel intensities appear in a given segment. Then the Hellinger distance between two quantified color distributions is defined as We compute this metric for each color channel independently. The average values of D l→1,d img with the standard deviations are shown on Figure 4. It demonstrates that the semantic diversity is the largest for the intermediate layers. On the contrary, the last buckets, which are closer to the output, do not influence semantics but have higher impact in terms of color. Note that the first layer always shows the smallest variability in terms of both semantics and colors. The last bucket seems to be responsible for color correction and color inversion and has a lower pallet variability impact. Note, that the plots from Figure 4 reaffirm the findings coming from the Figure 3. Note that the similar empirical also hold for other datasets (see figures 13, 16 in appendix). Overall, we summarize the main findings common for CIFAR-10, LSUN and Anime Faces datasets as: • The earlier layers have a smaller variability and seem to be responsible for the viewpoint and the position of the object on the image. • The semantic details of the image content are mostly determined by the intermediate layers. • The last layers typically affect only coloring scheme and do not affect content semantics or image geometry. Note, that these can differ for other datasets or other generator architectures. For instance, for the four-bucket generator and MNIST (Figure 6, left) or randomly colored MNIST (Figure 8, left) the semantics are mostly determined by the first two buckets. In this subsection, we argue that the interpretations of different layers, obtained with RPGAN, are also valid for the standard GAN generator of the same architecture. First, we demonstrate that both standard GAN and RPGAN trained under the same training protocol provide almost the same generation quality. As a standard evaluation measure, we use the Fréchet Inception Distance (FID) introduced in. We also compute precision-recall curve as defined in. For evaluation on CIFAR-10, we use 50000 generated samples and the whole train dataset. We also take ten independently trained generators and report minimal and average FID values. See Table 2 for FID comparison and Figure 11 in appendix for precision-recall. RPGAN and SN-ResNet perform with the same quality both in terms of FID and precision-recall curves. Table 2: FID values for CIFAR-10. To confirm that the layers of the standard GAN generator can be interpreted in the same way as the corresponding layers of its RPGAN counterpart, we perform the following experiment. We take a standard SN-ResNet GAN, consisting of five layers associated with the correspondent buckets in RPGAN, and train it on CIFAR-10. Then for each layer, we add normal noise to its weights. Intuitively, we expect that the noise injection in the particular layer would change generated samples in terms of characteristics, influenced by this layer. For instance, noise in the last two layers is expected to harm coloring scheme, while noise in the intermediate layers is expected to bring maximal semantic damage. Several samples, produced by perturbed layers, are presented on Figure 5. The images support the intuition described above and confirm that RPGAN may serve as an analysis tool for the corresponding generator model. Note, however, that injecting noise per se is not sufficient for interpretability. The perturbed generators produce poor images, which are difficult to analyze. Meanwhile, RPGAN always generates good-looking images, which allows to identify the factors of variation, corresponding to the particular layer. For instance, see Figure 8 for the colored MNIST dataset. Figure 8 (left) shows plausible images, generated by varying RPGAN blocks. In contrast, Figure 8 (right) demonstrates images from generators perturbed with small and large noise. For both noise magnitudes, these images are difficult to interpret. Of course, given the interpretations obtained via RPGAN, one can perceive similar patterns in the noisy generations, but noise injection alone is not sufficient for interpretability. In the next experiment, we demonstrate that the RPGAN model is also a natural fit for the generative incremental learning task (see e.g., are common for D 1 and D 2 . To illustrate this scenario, we take a partition of the MNIST handwritten digits dataset into two subsets MNIST 0−6 and MNIST 7−9 of digits from 0 to 6 and from 7 to 9 correspondingly. As for generator for MNIST 0−6 we take a 4-bucket RPGAN model with a number of blocks equal to 20, 20, 20, 8. Note that the last bucket is much thinner than the others, as it turns out to be responsible for variations in writing style, which does not change much across the dataset. Then we train the generator on the subset MNIST 0−6 of first 7 digits (see Figure 6, left and center). After that we add five additional blocks to each of the first two layers, obtaining a generator with a number of blocks equal to 25, 25, 20, 8 and pretrained weights in all blocks but five in the first and in the second buckets. Then we train the extended model to fit the whole MNIST by optimizing only the ten new blocks (see Figure 6, right). As a surprising side effect of our model, we discovered that decent generation quality can be achieved by the RPGAN generator with no nonlinearities, i.e., one can train the RPGAN generator with all blocks consisting of linear transformations only. To demonstrate that, we take an RPGAN with the same ResNet-like generator architecture as in the experiments above. Then we replace all nonlinearities in the generator model by identity operations and train it on the CIFAR-10 dataset. The model demonstrates FID equal to 22.79 that is competitive to the state-of-the-art generative models of comparable sizes (see Figure 23 for generated images examples). Note that this approach fails for a standard GAN generator that maps a Gaussian distribution to an images distribution. Indeed, that generator would be a linear operator from a latent space to the images space with a Gaussian distribution in the images domain. This purely linear generator architecture allows us to significantly speed up the image generation process for fully-connected layers. We group consequent buckets of fully-connected layers to form a new bucket. Blocks in the new bucket are linear transformations that are products of the blocks from the original buckets. To demonstrate this, we train a fully-connected generator network on the MNIST dataset (see Table 3 in appendix, left column). Then we join the last three buckets into a single one. We form a new bucket by blocks defined as the linear operators B 5k • B 4j • B 3i where i, j, k are random indices of blocks from the buckets B 3, B 4, B 5 of the original generator. Thus instead of performing three multiplications of features vector from the second layer by matrices of the shapes 256 × 512, 512 × 1024, 1024 × 784 we perform a single multiplication by a 256 × 784 matrix. In our experiments, we achieved ×2.2 speed up. Note, however, that after the compression, the latent space cardinality can decrease if a small subset of tuples (i, j, k) is used to populate the new bucket. Nevertheless, as random products of joining buckets are used, we expect that the generated images would be uniformly distributed in the space of images, produced by the uncompressed generator (see Figure 24 for samples comparison). In this paper, we address the interpretability of generative models. In particular, we have introduced RPGAN, an alternative design of generative adversarial networks, which allows natural interpretation of different generator layers via using random routing as a source of stochasticity. With experiments on several datasets, we provide evidence that different layers are responsible for the different factors of variation in generated images, which is consistent with findings from previous work. As a possible direction of future research, one can use the RPGAN analysis to construct efficient models, e.g., via identification of redundant parts of the generator for pruning or inference speedup. If the number of blocks is too low, the ing latent space appears to have insufficient cardinality to cover the dataset. On the other hand, a too high number of blocks in a difficult training procedure and also fails. To generate LSUN-bedroom-like images, we use the 7-bucket RPGAN with ResNet-like generator with five residual blocks, the first fully-connected layer, and the last convolutional layer. Similarly to CIFAR-10 experiments, during the generation, we freeze a random path and vary blocks in a single bucket to investigate its responsibility. See Figure 12 for blocks variations and Figure 13 for buckets responsibility analysis. Note that similarly to CIFAR-10, the central buckets have a maximal semantic impact. Last two buckets are mostly responsible for coloring. The first two buckets are responsible for local geometrical features. Note that here we face mode collapse for the third bucket: mainly, it affects only tiny local features. See Figure 14 for samples generated by the model. l o w n o i s e v a r i a n c e l a r g e n o i s e v a r i a n c e Figure 8: Left: images produced by varying blocks in a particular bucket of RPGAN. Right: images produced by the standard GAN after parameters perturbation in a particular generator layer, with low and high normal noise variance. A.3 ANIME FACES DATASET Though this dataset is not standard for GANs, we use it in the experiments as it nicely reveals the RPGAN analysis tools. Here we use the 6-bucket RPGAN with ResNet-like generator with four residual blocks, the first fully-connected layer, and the last convolutional layer. See Figure 15 for block variations and Figure 16 for bucket responsibility analysis. Again, the content semantics is mostly defined by the intermediate buckets. The last two buckets are mostly responsible for coloring: the fifth bucket has the maximal impact on coloring, and the last bucket varies tones. The first buckets are responsible for small details (one can note the hair on the character's forehead). See Figure 17 for samples generated by the model. Here we show that the concept of RPGAN works well with different generator architectures and learning strategies. Here we present plots for DCGAN-like generators consisted of consequent convolutional layers without skip connections. All the models were trained with the same parameters as described in Section 4. Despite of Spectral Normalization, we train these models as WGANs with weight penalty . On the Figure 19 we show plots for a four-bucket generator trained on CIFAR-10. We also train four-bucket generator on colored MNIST, see Figure 8, left. Finaly, we show plots for the five-bucket generator and CelebA-64x64 dataset on Figure 21. See Figure 18, Figure 21, Figure 22 for buckets analysis. In this section we show that injecting noise in the generator weights cannot be used as a stand-alone interpretability method. Namely, we compare images produces by RPGAN and noise injection for models trained on randomly colored MNIST samples. We train both RPGAN and standard generators as a Wasserstein GAN with weights penalty. Tanh, reshape to 28 × 28"

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.