Understanding and Enhancing the Transferability of Adversarial Examples
Lei Wu Zhanxing Zhu Cheng Tai Weinan E
 
















 

Robust Actor-Critic Contextual Bandit for Mobile Health (mHealth)
Interventions
    Feiyun Zhu^†, Jun Guo^, Ruoyu Li^†,
Junzhou Huang^†† Department of CSE, University of Texas at Arlington, TX,
76013, USA.  Department of Statistics, Univeristy of Michigan,
Ann Arbor, MI 48109, USA
    December 30, 2023
=========================================================================================================================================================================================================


  tabular[2][]
    

    
  @maketitle


















































1]
Lei Wu
2, 3]Zhanxing Zhu
2, 3]Cheng Tai
2, 3, 4]Weinan E


[1]School of Mathematical Sciences, Peking University, Beijing, China
[2]Center for Data Science, Peking University, Beijing, China
[3]Beijing Institute of Big Data (BIBDR)
[4]Department of Mathematics and PACM, Princeton University, Princeton, NJ, USA

    affilsepx: 
    

    affilsepx, 
    

    [ ]
    [ ]
    [ ]
    [ ]








Understanding and Enhancing the Transferability of Adversarial Examples
    [
    December 30, 2023
=======================================================================


	State-of-the-art deep neural networks are known to be vulnerable to adversarial examples,
	formed by applying small but malicious perturbations to the original inputs. Moreover,
	the perturbations can transfer across models: adversarial examples
	generated for a specific model will often mislead other unseen models. Consequently
	the adversary can leverage it to attack deployed systems without any query, which severely 
	hinder the application of deep learning, especially in the areas where security is crucial. 

	In this work, we systematically study how two classes of factors that might influence the transferability of adversarial examples.  One is about model-specific factors, including network architecture, model capacity and test accuracy. The other
	is the local smoothness of loss function for constructing adversarial examples. Based on these understanding, a simple but effective strategy is proposed to enhance transferability.  We call it  variance-reduced attack, since it utilizes the variance-reduced gradient to generate adversarial example. The effectiveness is confirmed by a variety of experiments on both  CIFAR-10 and ImageNet datasets.





§ INTRODUCTION

With the resurgence of neural networks, more and more large neural network
models are applied in real-world applications, such as speech recognition, computer vision, etc.
While these models have exhibited  good performance, recent works (<cit.>)
show that an adversary is able to fool the model into producing incorrect predictions by manipulating
the inputs maliciously. The corresponding manipulated samples are called adversarial examples.
More seriously, it is found that they have cross-model generalization ability,
i.e. the adversarial example generated from one model can fool another different model with a significant probability. We refer to such property as transferability. In consequence, hackers can employ this property to attack black-box systems with only limited number of queries (<cit.>), inducing serious security issue to deep learning system.









The adversarial vulnerability was first investigated By <cit.>, in which sophisticated L-BFGS was used to generate adversarial examples. 
Later, a large number of attacks  (<cit.>) are proposed. 
Among all the categories of attacks, those based on transferability could be the most dangerous and mysterious, since they surprisingly do not require any input-output query of  the target system. Understanding the mechanism of adversarial transferability could potentially provide various benefits for modern deep learning models. Firstly,  for the already deployed and vulnerable deep neural networks in real systems, it could help to design better strategies to improve the robustness to the transfer-based attacks. Secondly, revealing the mystery behind the transferability of adversarial examples could also extend the existing understandings on modern deep learning, particularly on the effects of model capacity (<cit.>) and model interpretability (<cit.>). Therefore, studying the transferability of adversarial example in the context of deep networks is of significant importance.

In this paper, we particularly focus on investigating  two classes of key factors that might influence the adversarial transferability. Inspired by these understandings, we deign a simple but rather effective strategy for enhancing the transferability of adversarial examples.  Our contributions are summarized as follows.



	
  * We numerically explore how adversarial transferability relies on the model-specific factors, including the architecture, test accuracy and model capacity. First, it is found that adversarial transfer is not symmetric, i.e. adversarial examples generated from model A can transfer to model B easily does not means the reverse is also natural. This suggests that the explanation based on the similarity of decision boundary is not sufficient, since similarity itself is a symmetric quantity. Second, multi-step attacks seem to outperform one-step attacks in most cases, which is contradictory to the finding in (<cit.>). Last, we find that adversarial examples generated from a large model appear less transferable than a small model, under the condition they both have good test performance. Interestingly, this finding seems closely related to the previous studies (<cit.>) which showed large models are more robust than small ones. 


	We also investigate the influence of properties of loss surface. Specifically, our investigation reveals that the local non-smoothness of loss surface harms the transferability of generated adversarial examples. This is consistent with the study of obfuscated gradients (<cit.>) and gradient masking (<cit.>).

	
  * Based on previous investigations, we propose a simple but rather effective approach to enhance the transferability.  Inspired by the works (<cit.>), we suggest applying the locally averaged gradient instead of the original one to generate adversarial examples. We call it variance-reduced attack, since the local averaging have the smoothing effect which suppresses the local oscillation of the loss surface. The effectiveness of our method is justified on both CIFAR-10 and ImageNet datasets, where a large number of state-of-the-art architectures are tested. Different from ensemble-based approaches proposed by <cit.>, our proposal does not require training lots of extra models, which is typically overly expensive in practice. Moreover, we numerically demonstrate that variance-reduced gradient can be combined with ensemble-based and momentum-based (<cit.>) approaches seamlessly for producing stronger attacks.  




§ RELATED WORK


The phenomenon of adversarial transferability was first observed and investigated by <cit.>. <cit.> showed that adversarial training can alleviate the transferability slightly, which is recently extended and improved in ( <cit.>) by incorporating to the adversarial examples generated by a large ensemble.
Based on the transferability, <cit.> proposed a practical black-box attack by training a substitute model with queried information. 
<cit.> showed that targeted transfer is much harder than non-targeted one, and additionally introduced the ensemble-based attacks. More recently, <cit.> showed that momentum can help to boost transferability significantly, and by utilizing this property, they won the first-place in NIPS 2017 Non-targeted Adversarial Attack and Targeted Adversarial Attack competitions. It is also worth mentioning the work by <cit.>, which demonstrated the exists of adversarial perturbations that have cross-sample transferability. Meanwhile, there exist several works trying to explain the adversarial transferability. <cit.> contributed it to the similarity between data gradients of source and target models.  By visualization, <cit.> suggested that the transferability comes from the similarity of decision boundaries. <cit.> provided some systematic investigation of this similarity. Unfortunately, similarity is symmetric and it cannot explain our finding that transfer is asymmetric.

Our proposed method enhances the transferability by utilizing informations in the small neighborhood of the clean example, which is inspired by the works on shattered gradient (<cit.>) and model interpretability (<cit.>). 
Recently, similar strategies are also explored by <cit.> and <cit.> for white-box attacks, while we focus on the transferability.
Our method is also related to the work by <cit.>, which introduced the expectation over transformation (EOT) method to increase robustness. The EOT formulation is similar to our objective (<ref>), which increases the robustness against Gaussian noise. 
<cit.> suggest that optimizing the landscape smoothed with respect to parameters can lead to solutions generalizing better. Differently, we obtain adversarial perturbation with stronger transferability via smoothing with respect to input x.






§ PRELIMINARIES



 §.§ Adversarial examples

We use f(x):  ^d ↦^K to denote the function a model represents, where we omit the dependence on the trainable model parameter , since it is fixed in this work.
For many applications of interest, we always have d ≫ 1 and K=O(1). According to the local
linear analysis in <cit.>, it is the high dimensionality that makes
f(x) vulnerable to the adversarial perturbation. That is, considering the K-category classification problems as an example, for each x, there
exists a  small perturbation η that is nearly imperceptible to human eyes, such that the i-th output f_i(x) satisfies

    _i f_i(x) = ,   _i f_i(x+η) ≠,

where  is the true label of the input x.  
We call η adversarial perturbation and correspondingly x^adv:=x+η an adversarial
example.

The attack (<ref>) is called a non-targeted attack since the adversary has no control over  which class  the input x will be misclassified to. 
In contrast, a targeted attack aims at fooling the model to produce a wrong label specified by the adversary, i.e.

    _i f_i(x + η) = y^target .


In the black-box attack setting, the adversary has no knowledge of the target model (e.g. architecture and parameters) and is not allowed to
query  input-output pairs from the model, i.e. the target model is a pure black-box. However the adversary can
construct adversarial examples on a local model (also called the source model) that is trained
on the same or similar dataset with the target model.  Then it deploys those adversarial examples to fool the target model. 
This is typically referred to as a black-box attack,
as opposed  to the white-box attack whose target  is the source model itself.



 §.§ Models of generating adversarial examples

In general, crafting adversarial perturbation can be modeled as the following optimization problem,

    maximize_x'      J(x'):= J(f(x'),y^true) 
    s.t.        x'-x≤ε ,

where J is some loss function measuring the discrepancy between the model prediction and ground truth;
· is certain norm metric  to quantify the magnitude of the perturbation. For image data, there
is also an implicit constraint: x' ∈ [0,255]^d, with d being the number of pixels.
In practice, the common choice of J is the cross entropy.
 <cit.> introduced a loss function that directly manipulates the output logits instead of probability, which has been also adopted in many works. As to the measurement of distortion, it should be chosen to encourage imperceptibility (<cit.>). In this paper,
 the widely used ℓ_∞ norm is considered.



  
*Ensemble-based attacks
To improve the strength of adversarial transferability, instead of using a single model, <cit.> suggested
 using a large ensemble consisting of f_1,f_2,⋯,f_Q as our source model. Specifically, the non-targeted ensemble-based attack is given by
 
    maximize_x'      J(∑_i=1^Q w_i f_i(x'),y^true) 
    s.t.        x'-x≤ε,

where w_j is the ensemble weight satisfying ∑_j=1^Q w_j=1. Following (<cit.>), we choose w_j=1/Q in this paper. The targeted counterpart can be derived similarly.




 §.§ Optimizer

There are various optimizers to solve problem (<ref>). Instead of using the classic optimizers, such as Adam, SGD, etc. as in (<cit.>), we adopt a more efficient Frank-Wolfe optimizer. In each iteration, we solve the linear approximation of objective function at current solution _t in a constrained space,

    s_t   = argmax_s_t≤ε J(x_t) + ⟨∇_x J(x_t), s_t⟩
    
        x_t+1   = proj_D(x_t + α  s_t)

where 𝒟= [0,255]^d ∩{x'  | x'-x≤ε} and α is the step size. For ℓ_∞ norm, the Eqn (<ref>) has an explicit solution given by

    x_t+1 = proj_D (x_t + α sign(∇_x J(x_t)) ).


The attack by evolving  (<ref>) for T steps is called iterative gradient sign method (IGSM). <cit.> and <cit.> showed this method can generate  strong adversarial examples. Furthermore, the famous fast gradient sign method (FGSM) is a special case with α=ε, T=1. <cit.> and <cit.> suggested that FGSM can generate more transferable adversarial examples than IGSM. Therefore, both of them are considered in our work.



  
*Momentum-based approaches
<cit.> recently proposed the momentum-based approaches to enhance adversarial transferability. Specifically, the momentum iterative gradient sign method (m-IGSM) is given by

    g_t+1   = μ  g_t + ∇ J(x_t)/∇ J(x_t)_1
    
    x_t+1    = proj_D (x_t + α sign(g_t)),

where μ is the decay factor of momentum.



 §.§ Evaluation of Adversarial Transferability


Datasets
To evaluate the transferability, three datasets including MNIST, CIFAR-10 and ImageNet are considered. For ImageNet, directly evaluation on the whole ILSVRC2012 validation dataset is too time-consuming. Therefore, in each experiment we randomly select 5,000 images that can be correctly recognized by all the examined models.

Trained Models
(i) For MNIST, we trained fully connected networks (FNN) of D hidden layers, with the width of each layer being 500. For instance, the architecture of FNN width D=2 is 784-500-500-10. All the networks are trained to achieve 100% on training set, and test accuracies of them are higher than 98%. (ii) For CIFAR-10, we trained five models: lenet,resnet20, resnet44, resnet56, densenet, and test accuracies of them are 76.9%, 92.4%,93.7%, 93.8% and 94.2%, respectively. (iii) As for ImageNet, the pre-trained models  are used provided by PyTorch, including resnet18, resnet34, resnet50, resnet101, resnet152,
vgg11_bn, vgg13_bn, vgg16_bn, vgg19_bn, densenet121, densenet161, densenet169, densenet201, alexnet, squeezenet1_1. 
The Top-1 and Top-5 accuracies of them 
can be found on website[<http://pytorch.org/docs/master/torchvision/models.html>].  
To increase the reliability of experiments, all the models have been tested. 
However, for a specific experiment we only choose some of them to present since the findings are consistent among the tested models.

Criteria
Given a set of adversarial pairs, 
{(x^adv_1,_1),(x^adv_2,_2),…,(x^adv_N,_N)},
we calculate their Top-1 success rates (%) fooling a given model f(x)  by 

    100×1/N∑_i=1^N 1[_i f_i(x^adv_i)≠_i].

If f(x) is the model used to generate adversarial examples, then the rate indicates the  
the white-box attack performance. For targeted attacks, each image x^adv is associated with a pre-specified label 
≠. Then we evaluate the performance of the targeted attack by the following Top-1 success rate (%), 

    100 ×1/N∑_i=1^N 1[_i f_i(x^adv_i)=_i].

The corresponding Top-5 rates can be computed  in a similar way.

Attacks
Throughout this paper the cross entropy  [We also tried the  loss described in <cit.>
but did not find its superiority to cross entropy. The reason might be that hard constraints, instead of
 soft penalizations, are used in our formulation.] is chosen as our loss function J.
We measure the distortion by  ℓ_∞ norm, both FGSM and IGSM attacks are considered.





§ HOW MODEL-SPECIFIC FACTORS AFFECT TRANSFERABILITY

Previous study on transferability mostly focused on the influence of attack methods (<cit.>). However
it is not clear how the choice of source model affects the success rate transferring to target models. In this section, three factors of source model, including architecture, test accuracy and model capacity, are investigated. 





 §.§ Architecture


Here we explore how the architecture similarity between source and target model contributes to the transferability. This study is crucial since it can provide us guidance to choose the appropriate source models for effective attacks.  To this end, three popular architectures including ResNet, DenseNet and VGGNet are considered, and for each architecture, two networks are selected. Both FGSM and IGSM attacks are performed on ImageNet dataset. Table <ref> presents the experiment results, and the choice of hyper-parameter is detailed in the caption.  


As we can see, in most cases IGSM outperforms FGSM significantly, especially when the source and target models have the similar architecture.  This observation contradicts the finding by <cit.> that multi-step attacks are somewhat less transferable than single-step attacks.  However for the attacks from VGGNets to all the ResNets and DenseNets, we do observe IGSM generate adversarial examples weaker than FGSM (blue rates in the figure). So, in general, which method is better should be similarity dependent. 



Another finding is that the transfers between two models are non-symmetric, and this phenomenon is more obvious for the models with different architectures. For instance, the success rates of IGSM attacks from densenet121 to vgg13_bn is 58.7%, however the rate from vgg13_bn to densenet121 has only 19.1%. Moreover, the success rates  between models with similar architectures is much higher. For example the success rates of IGSM attacks between vgg13_bn and vgg16_bn are higher than 90%,  nearly twice the ones of attacks formed by any other architectures. 








 §.§ Model Capacity and Test Accuracy






We first study this problem in ImageNet dataset, with vgg19_bn[We also tried other nets, and the results show no difference.] chosen as our target model.  A variety of models are used as source models to perform both FGSM and IGSM attacks. 
The results are displayed in Figure <ref>.
The horizontal axis is the Top-1 test error, while the vertical axis is the number of model parameters
that roughly quantifies the model capacity.
We can see that   the  models with powerful attack capability
concentrate in the bottom left corner, and the fooling rates are much lower for those models  with either large test error or large number of parameters.

The decision boundaries of high-accuracy models should be similar, since all of them approximate the ground-truth decision boundary of data very well. On the contrary, a low-accuracy model must has a decision boundary relatively different from the high-accuracy models. Therefore it is not surprising to observe that high-accuracy models tend to exhibit stronger attack capability. 

However it is somewhat strange that adversarial examples generated from deeper model appear less transferable.  To further confirm this observation, we conduct additional experiments on MNIST and CIFAR-10. Table <ref> shows the results, which is basically consistent with previous observation, though there exists counter-examples, attacks from resnet44,resnet56 to resnet20. It suggests us not to use deep models as the source models when performing transfer-based attacks, even though we still can not fully understand it. It is also worth to mention that some works (<cit.>) observed a dual phenomenon that deeper models are more robust against adversarial perturbations.  







§ SHATTERED GRADIENTS

In previous section, we explore how model-specific factors influence the transferability of adversarial examples. In this part, we systematically investigate how the smoothness of J(x) impacts the transferability across models. 

For simplicity, assume model A and B are the source and target models, respectively and let g(x):=∇ J(x) be the gradient. Previous methods use g_A(x)  to generate adversarial perturbations, so the transferability mainly depends on how much instability of g_A can transfer to model B. The shattered gradients phenomenon studied in (<cit.>) implies that gradient g_A is very noisy, though the model is trained on training set very well. We hypothesize that the noise hurts the transferability of g_A, as illustrated in cartoon Figure <ref>. Since both model A and B have very high test accuracy,  their level sets should be similar globally, and J_B is probably unstable along g_A. However, as illustrated in the figure, the local fluctuation of g_A make the sensitivity less transferable. One way to alleviate it is to smooth the landscape J_A, thereby yielding a more transferable direction G_A, i.e. ⟨Ĝ_A,ĝ_B⟩ > ⟨ĝ_A,ĝ_B⟩.


Inspired by  the standard technique from distribution theory, we suggest to  smooth the landscape as follows,

    J_σ(x):= ∫ J(x-x') ψ_σ(x') dx',

where J_σ is the smoothed landscape, and the smoothing is achieved by convolution with a mollifier, defined as a smooth function satisfying,

    ψ_σ(x)=σ^-dψ(x/σ)     ∫_ℝ^dψ(x) dx =1.

In this paper, the Gaussian mollifier, i.e. ψ(x) = 1/(2π)^d e^-x^2/2, is used, and the  corresponding gradient can be calculated by 

    G_σ(x) = 𝔼_ξ∼𝒩(0,σ^2) [g(x+ξ)].

This strategy has been employed by <cit.> to improve the interpretability of gradient saliency maps, which demonstrates that the local averaged gradient G_σ(x) is more informative and interpretable than g(x).  Here, we further show that G_σ(x) is more transferable than g(x) by numerical experiments on ImageNet. 

We first quantify the cosine similarity between gradients of source and target models, respectively. Two attacks are considered:  vgg13_bn→vgg16_bn, densenet121→vgg13_bn, which represent the within-architecture and cross-architecture transfer, respectively. We choose σ=15, and the expectation in (<ref>) is estimated by using 1/m∑_i=1^m g(x+ξ_i).  To verify the averaged gradients do transfer better, we plot the cosine similarity between source and target model versus the number of samples m. Figure <ref> display the average similarity calculated from 5,000 randomly selected samples, which shows that the cosine similarity between G_A and g_B are indeed larger than the one between g_A and g_B. As expected, the similarity increases with m monotonically, which further justifies that G_A is more transferable than g_A. 

Next, we visualize the transferability in Figure <ref> by comparing the decision boundaries of resnet34 (model A, the source model) and densenet121 (model B, the target model). The horizontal axis represents the direction of G_A of resnet34,  estimated by m=1000,σ=15, and the vertical axis denotes orthogonal direction h_A:= g_A - ⟨ g_A,Ĝ_A⟩Ĝ_A.  Each point in the 2-D plane corresponds to the image perturbed by u and v along each direction,

    clip(x + u  Ĝ_A + v ĥ_A,0,255),
	
where x is the clean image. 
 It can be easily observed that for resnet34 (model A), a small perturbation
in both directions can produce wrong classification. However, when applied to densenet121 (model B), a large perturbation along  h_A cannot change the classification result, while a small perturbation along G_A can change the prediction easily.  This further confirms that local averaging indeed extracts the more transferable part of g_A.

The above analysis implies that the local oscillation of loss surface do harm the transferability of adversarial examples. This is similar to the phenomenon of gradient masking observed in the defense of adversarial example (<cit.>). 
Our experiments suggest that gradient masking  also exists for normally trained models to some extent, i.e. shattered gradients. The noise contained in the shattered gradient makes the generated adversarial examples less transferable. 





§ VARIANCE-REDUCED ATTACK



 §.§ Method


Previous study suggests to alleviate the shattering of gradients by optimizing the smoothed loss function,

    maximize      J_σ(x'):= 
    				𝔼_ξ∼𝒩(0,σ^2) [J(x'+ξ)] 
    s.t.        x'-x≤ε .

Intuitively, this method can also be interpreted as  generating adversarial examples that are robust to  Gaussian perturbation. We expect that the generated robust adversarial examples can still survive easily in spite of the distinction between source and target model.   

Using the iterative gradient sign method to solve (<ref>) yields the following iteration formula:

    G_t    = 1/m∑_i=1^m ∇ J(x^t+ξ_i),   ξ_i∼(0,σ^2I) 
    
     	x_t+1   = proj_D(x_t + α G_t),

where G_t is a mini-batch approximation of 𝔼_ξ∼𝒩(0,σ^2I)[∇ J(x+ξ)]. Compared to IGSM, the gradient is replaced by an  averaged one, which removes the local fluctuation. Therefore we call this method variance-reduced iterative gradient sign method (vr-IGSM). The special case T=1,α=ε, is accordingly called variance-reduced fast gradient sign method (vr-FGSM).
For any other optimizer, the corresponding variance-reduced version can be derived similarly. However, in this paper we only consider IGSM and FGSM for simplicity.






 §.§ Effectiveness

In this section, we evaluate the effectiveness over both CIFAR-10 and ImageNet datasets.  

 



  §.§.§ CIFAR-10

We first consider the adversarial transferability among three models: lenet, resnet-20 and densenet. The test accuracies of them are 76.9%, 92.4% and 94.2% respectively. Table <ref> shows, in general, variance-reduced gradient indeed improve the transfer rate consistently. In particular, the transferability between resnet-20 and densenet are enhanced significantly. However the variance-reduced gradient does not help too much for the transfers from lenet to resnet-20 and densenet.  It is interesting to note that lenet has very weak attack capability, which probably is due to its low accuracy. It does not learn the data very well, causing that its decision boundary be very different from the high-accuracy model resnet20 and densenet. This is consistent with our finding in Section <ref>.




  §.§.§ ImageNet


Previous experiments show that our method indeed can improve the success rate. We now turn to the more real dataset, imageNet. 
To make the experimental result reliable, both one-step and multi-step methods, targeted and non-targeted attacks, single-model and ensemble-model based approaches are examined.



  
*Single-model based approaches
Here we test both one-step and multi-step attacks, specifically for multi-step attacks, we make the number of gradient calculation per sample be fixed 100 for a fair comparison. The results of multi-step attacks are shown in Table <ref> (readers can refer to Table <ref> in Appendix for the result of one-step attack). As we can see, our method enhance the transferability dramatically for all the attacks. Please especially note those bold rates, where the improvement has reached about 30%.







  
*Ensemble based approaches
In this part, we apply the ensemble-based attack proposed by <cit.>, which is rather effective in generating strong transferable adversarial examples. We examine whether we can further improve its performance  by employing our variance-reduced gradient. 

Both targeted
[Compared to non-targeted attack, we find that 
a larger step size α is necessary for generating strong targeted adversarial examples.
Readers can refer to Appendix A for more detailed analysis on this issue, though we cannot fully understand it. Therefore a much larger step size than the non-target attacks is used in this experiment.]
and non-targeted attacks are considered. 
For non-targeted attacks, the Top-1 success rates are nearly saturated; for targeted attacks, generating an adversarial example predicted by target models as a  specific label is too hard, resulting in a very small success rate.  Therefore, we instead adopt the the Top-5 success rate as our criterion to better reflect the improvement of method. A variety of model ensembles are examined, and the results are summarized in Table <ref> (The corresponding Top-1 success rate can be found in Appendix).

As we can see, it is clear that variance-reduced attacks outperform the corresponding normal ones by a remarkable large margin for both non-targeted and targeted attacks.  More importantly, the improvement never be harmed compared to single-model case, which implies that variance-reduced gradient can be effectively combined with ensemble method without compromise.  










  
Momentum-based approaches
Momentum-based attacks are recently proposed by <cit.>, which won the first place in 
NIPS 2017 Non-targeted Adversarial Attack and Targeted Adversarial Attack competitions. Here we compare our variance-reduced attacks with the momentum-based attacks. In this experiment, three networks of different architectures are selected.  The momentum decay factor μ=1 is chosen as suggested in (<cit.>), and for variance-reduced gradient, we use m=20,σ=15. All attacks are iterated for T=5 with step size α=5. Table <ref> reports the Top-1 success rates of non-targeted attacks of three attacks, including momentum-based IGSM (m-IGSM), vr-IGSM and momentum-based vr-IGSM (m-vr-IGSM). 

As shown in the table, our method outperforms momentum-based method significantly for all the cases. Furthermore, by combining with the variance-reduced gradient, the effectiveness of momentum-based method is improved significantly. 






 §.§ Influence of hyper parameters
 
In this part, we explore the sensitivity  of hyper
parameters m, σ when applying our variance-reduced gradient methods for
black-box attacks. We take ImageNet dataset as the testbed, and vr-FGSM attack
is examined. To increase the reliability,  four attacks are considered here.
The results are shown in Figure <ref>. It is not
surprising that larger m leads to higher success rate for any distortion
level ε due to the better estimation of the data-dependent
direction of the gradient. We find there is an optimal value of σ
inducing the best performance. Overly large σ will introduce a large
bias in (<ref>). Extremely small σ is unable to smooth the landscape enough. 
Moreover the optimal σ varies for different source models, and in this experiment it is about 15 for resnet18,
compared to 20 for densenet161.





 §.§ Robustness analysis of adversarial examples

Since variance-reduced attacks can be viewed as generating adversarial examples robust against Gaussian noise perturbation. Here we further examine their robustness to generic image transformations. The robustness is particularly important in practice, since it directly affects whether adversarial examples can survive in the physical world (<cit.>). To quantify the influence of transformations,
we use the notion of destruction rate defined by <cit.>,

    R = ∑_i=1^Nc(_i)(1-c(_i^adv)) c(T(_i^adv))/∑_i=1^N c(_i)( 1 - c(_i^adv)),

where N is the number of images used to estimate the destruction rate, T(·) is
an arbitrary image transformation. The function c() indicates whether  is
classified correctly:

    c():={[                                1, if image  is classified correctly;                                0,                         otherwise ].

And thus, the above rate R describes the fraction of adversarial images that are no longer misclassified after the transformation T(·).



Densenet121 and resnet34 are randomly chosen as our source  and target model, respectively; and four image transformations are considered: rotation, Gaussian noise, Gaussian blur and
JPEG compression. Figure <ref> displays the results, which show that adversarial examples generated by our methods are much more robust than those generated by vanilla methods. This numerical result is interesting, since we only explicitly increase the robustness against Gaussian noise in generating adversarial examples. This result suggests that the robustness of adversarial examples can also transfer among different image transforms. 






§ DISCUSSION AND CONCLUSION

In this paper, we first investigated the influence of model-specific factors on the adversarial transferability. It is found that model architecture similarity plays a crucial role. Moreover models with lower capacity
and higher test accuracy are endowed with stronger capability for transfer-based attacks. 
we second demonstrate that the shattered gradient hinders the transfer of adversarial examples. 
Motivated by the understanding, we 
proposed the variance-reduced attack which can enhance the transferability of generated adversarial examples dramatically. Furthermore, the variance-reduced gradient can be combined with both ensemble and momentum based attacks rather effectively.
Lastly, we show that, the adversarial examples generated by variance-reduced attacks are much more robust than normal methods.

Our results imply that the robust (at least against Gaussian noise) adversarial examples tend to have stronger cross-model generalization, i.e. transferability. The similar phenomenon is also studied in  learning theory (<cit.>), i.e. the stable algorithm and hypothesis always tend to generalize better as well. 
So it is worth to further investigate if improving the robustness against generic transforms (<cit.>) can enhance the transferability as well. 



iclr2018_conference










§ APPENDIX A: INFLUENCE OF STEP SIZE FOR TARGETED ATTACKS

When using IGSM to perform targeted black-box attacks, there are two hyper parameters
including number of iteration k and step size α. Here we explore their
influence to the quality of adversarial examples generated. The success rates
 are calculated on 1,000 images randomly selected according to description
of Section <ref>. resnet152 and vgg16_bn are
chosen as target models. The performance are evaluated by the average Top-5 success rate
over the three ensembles used in Table <ref>.

Figure <ref> shows that for the optimal step size α
is very large, for instance in this experiment it is about 15 compared to
the allowed distortion ε=20. Both too large and too small step size
will yield harm to the attack performances. It is worth to note that with small
step size α=5, the large number of iteration provides worse performance than
small number of iteration. One possible explanation is that more iterations lead the optimizer to
converge to a more overfitted solution.
In contrast, a large step size can prevent it and encourage the optimizer to explore more model-independent
area, thus more iteration is better.





§ APPENDIX B: SOME ADDITIONAL EXPERIMENTAL RESULTS





















