Do Deep Minds Think Alike? Selective Adversarial Attacks for Fine-Grained Manipulation of Multiple Deep Neural Networks
Zain Khan Jirong Yi Raghu Mudumbai Xiaodong Wu Weiyu Xu
cs.CV cs.LG eess.IV math.OC stat.ML

Simulation of the traveling wave burning on epithermal neutrons on the year time scale
    [
        ^1 Department of Statistical Sciences, University College London, London WC1E 6BT, UK

    ^2 School of Mathematical Sciences, Queen Mary University of London, London E1 4NS, UK

^3 The Alan Turing Institute, The British Library, London NW1 2DB, UK 


===================================================================================================================================================================================================================================================================














































 Recent works have demonstrated the existence of adversarial examples targeting a single machine learning system.  In this paper we ask a simple but fundamental question of “selective fooling”: given multiple machine learning systems assigned to solve the same classification problem and taking the same input signal, is it possible to construct a perturbation to the input signal that manipulates the outputs of these multiple machine learning systems simultaneously in arbitrary pre-defined ways? For example, is it possible to selectively fool a set of “enemy” machine learning systems but does not fool the other “friend” machine learning systems?  The answer to this question depends on the extent to which these different machine learning systems “think alike". We formulate the problem of “selective fooling” as a novel optimization problem, and report on a series of experiments on the MNIST dataset. Our preliminary findings from these experiments show that it is in fact very easy to selectively manipulate multiple MNIST classifiers simultaneously, even when the classifiers are identical in their architectures, training algorithms and training datasets except for random initialization during training. This suggests that two nominally equivalent machine learning systems do not in fact “think alike" at all, and opens the possibility for many novel applications and deeper understandings of the working principles of deep neural networks.







§ INTRODUCTION
 
Recent works <cit.> have shown that the outputs of many machine learning systems can be 
significantly changed using very small carefully-chosen perturbations to the inputs. In many 
cases, it is possible to gain essentially complete control over the outputs of such systems
with input perturbations that are imperceptible to human senses. 

Such vulnerability to adversarial attacks poses a major challenge to building robust
and reliable machine learning systems, and, as a result, understanding and mitigating this
phenomenon has become a major focus of research. While this research 
has produced many interesting findings <cit.>, theoretical explanations for, and practical defenses against, adversarial attacks both remain open problems at this time.

If we imagine machine learning systems to have perceptions, adversarial attacks 
can be thought of as illusions. This gives rise to the question: to what
extent do different machine learning systems “see" the same features? In particular, we may ask whether two machine learning systems share the same illusions. We seek to answer
this question by studying the extent to which it is possible to selectively manipulate multiple machine learning systems in different ways. In other words, we would like to know
if it is possible to design input perturbations that simultaneously manipulate multiple machine learning systems into “seeing" different things. 



The study of such more powerful attacks may help us understand the nature of
adversarial attacks better. In addition, a study of such precise adversarial attacks may be of interest in its own right. For instance, the ability to precisely change the outputs of  a set of selected machine learning systems while leaving other systems unaffected may 
allow the creation of adversarial attacks that fool “enemy” machine learning systems, but spare “friend” machine learning systems. This can potentially be useful in various security applications, for example, in evading detections by enemy forces in battlefields.


To our knowledge, adversarial attacks that selectively and differentially target multiple 
machine learning systems have not been studied in detail in previous works. However, many of the underlying ideas have been used in other contexts. Certainly, the idea of
using multiple machine learning systems to augment each other in various ways in adversarial 
settings is not new. Many proposed defenses against adversarial attacks use auxiliary 
neural networks in various ways <cit.>, e.g., binary classifiers that are trained to detect 
adversarial inputs, or generative networks for retraining networks to reduce their 
vulnerability to such inputs. A common way to train adversarial generative networks is by using
a discriminator network adversarially.

There also exists a significant volume of literature on transferability of adversarial attacks <cit.>, i.e., 
the question of whether an attack designed for one machine learning system is effective 
against another system. Our formulation in this paper is related to transferability but is 
very different and significantly more general. Specifically, we consider adversarial perturbations that are 
simultaneously designed to attack multiple machine learning systems, and also to have 
potentially different effects on each attacked system. Before our investigation, even the existence of such 
highly precise and selective attacks is not clear. Our main objective in this paper is to investigate this selective attack, and perform an experimental study to provide preliminary answers to this and other 
related questions.










 §.§ Summary of Objectives and Findings


Our overall objective is to construct adversarial attacks which selectively and precisely manipulate multiple machine learning systems. In particular, we seek general answers to the following fundamental questions.




  * How selective and precise can adversarial attacks be? Specifically, what is the largest number of different machine learning systems that can be simultaneously and selectively manipulated by the same adversarial input?


  * A trivial mathematical observation is that selective adversarial attacks targeting  multiple machine learning systems simultaneously must require strictly larger perturbations than attacks targeting just one system. However, it is not clear how strong this relationship is. Thus, we would like to quantify the trade-off between how precise and selective an attack is, and
how imperceptible it is to human and other senses.


  * We would like to understand the transferability of selective adversarial perturbations i.e. how selective adversarial perturbations designed to target a number of machine learning algorithms affect other machine learning systems that the attack was not designed for.


  * Finally, we would like to use selective adversarial attacks as a tool to gain insights into the internal workings of otherwise-opaque machine learning algorithms.




While our problem formulation is quite general, our experiments were focused on a set of K=5 deep neural network classifiers for the MNIST dataset[http://yann.lecun.com/exdb/mnist/]. These 5 classifiers are identical in their architectures, training procedures and data sets, differing only in their random initial weights. Each of the 5 classifiers also had approximately equal (and very high) accuracy for their test data sets. Our preliminary findings from these experiments are summarized as follows.




  * The answer to whether we can construct adversarial attacks selectively targeting multiple machine learning systems simultaneously is a definite 'YES'. We propose novel optimization formulation to design adversarial attacks which selectively attack multiple machine learning systems. 
We have been able to successfully generate adversarial perturbations against 5 MNIST classifiers to several sets of pre-specified target labels. For example, we can design attacks which selectively fool a set of “enemy” machine learning systems, but do not fool the other “friend” machine learning systems.


  * Mathematically we can show that the size of the optimal perturbations increases with the number of classifiers being targeted, a conclusion that seems intuitively reasonable. However, our algorithm for generating these perturbations in some cases actually yields small perturbations when attacking more systems. We offer some possible explanations for this counter-intuitive result.


  * Adversarial examples that are designed to attack multiple classifiers have significantly greater transferability to other classifiers.




The rest of the paper is organized as follows. Section <ref> presents a brief survey of related works. Selective adversarial attacks are formally defined as an optimization problem in Section <ref>. Our experiments on selective adversarial attacks on 5 MNIST classifiers are described in Section <ref> . Section <ref> concludes this paper.

Notations: We use [N] to denote the set {1,2,⋯,N}.





§ BACKGROUND AND RELATED WORK
 
As noted earlier, the phenomenon of adversarial vulnerability has now become an important focus of research in machine learning since the work by Szegedy et al. <cit.> and there is now a vast and growing literature on
this subject. The adversarial vulnerability has been observed in various tasks <cit.> and for different machine learning systems especially deep neural networks <cit.>.   

Adversarial attacks are a broad phenomenon that can be classified in several possible ways. One important distinction is between white-box attacks where the attacker has access to the internals of the machine learning systems being targeted <cit.> and corresponding defenses, <cit.>, and black box attacks <cit.> and their defenses <cit.>.

The fast gradient sign method (FGSM) <cit.>, the DeepFool method <cit.>, and the Carlini-Wagner (CW) attack <cit.> are three of the most popular algorithms for generating white-box attacks.

In adversarial attacks based on FGSM <cit.>, an adversarial sample is generated via

    x_adv =  trunc(x + ϵ· sign(∇_x ℓ(x,u))),
  
where x∈^M is a benign sample, ϵ>0 is a small positive constant, sign(·) is an element-wise sign function, and ∇_x ℓ(x,u) is the gradient of loss function ℓ with respect to the input x. The u∈{1,2,⋯,L} is the true label of x, and the trunc(·) function is applied to guarantee that the adversarial sample is within a normal range, i.e., the pixels of an image should be within [0,255].

It has been observed that adversarial samples designed for one machine learning system can also fool other machine learning systems <cit.>. This transferability of adversarial samples allows us to design adversarial samples for a substitute system, and then use these samples to fool the target system. In sampling-based black-box attacking method, we initialize an adversarial sample which can be quite different from the benign sample, and then we gradually shrink the distance from the adversarial sample to the benign sample by randomly sampling a moving direction <cit.>. Different from the white-box attacks where the gradient can be easily calculated, the gradient-free black-box attack uses an estimate of the gradient to perform the searching of adversarial samples <cit.>.

As one of the most important black-box attacking methods, the transferability of adversarial samples has received tremendous amount of attention recently <cit.>. In <cit.>, Liu et al. reported that though an adversarial sample generated for one system can fool another system, it cannot always fool the two systems into making the same wrong decision, e.g., being classified as in the same wrong class. This means that the adversarial samples can transfer, but their labels cannot always. The authors further proposed an ensemble-based approach to increase the transferability of targeted adversarial attacks to a single to-be-attacked black-box machine learning system.  This topic was further studied by Adam et al. <cit.>. In <cit.>, the authors proposed to optimize the distance of the gradients of different models in training so that the transferability of adversarial attacks can be reduced. Another viewpoint of the adversarial transferability was proposed by Lin et al. <cit.>, and they argued that the transferability of an adversarial sample is just like the generalization performance of a training learning system. Based on this, they proposed to improving the transferability via techniques similar to those used for improving the generalization performance of learning systems. 

On the theoretical side, Tramer et al. made attempts to investigate the fundamental reasons accounting for the tranferability of adversarial samples <cit.>. They showed that for two different models, their decisions regions are highly overlapped, and also gave conditions on the data distribution for guaranteeing the transferability. Another line of theoretical work by Fawzi et al. <cit.> gave an upper bound on the probability for an adversarial sample to be transferable when the decision regions of learning systems are highly overlapped. 

This paper focuses on white-box evasion attacks on classifier networks where the attacks can be optimally designed using iterative algorithms. These are, of course, the most favorable conditions for the attacker and allows for the design of the strongest and most powerful class of attacks.









§ PROBLEM STATEMENT: OPTIMAL SELECTIVE ADVERSARIAL ATTACKS
 

We now formally define selective adversarial attacks as a constrained optimization problem.

Consider a set ≐{M_1, M_2, …, M_N } of N trained machine learning classifiers. The i-th (i=1 … N) machine learning system M_i maps an input vector x ∈^M to one label u: f_i(x)=u ∈{1 … L}, where L is the number of labels, and {1 … L} is the set of labels. Let f_0(): ^M →{1 … L} denote the ground-truth mapping, namely u_0=f_0(x) is the “true label" of a given input x.

Machine learning classifiers typically work with high-dimensional inputs i.e. M ≫ 1. We assume that each of the classifiers _i are highly accurate i.e.:

    ( f_i(x) = f_0(x) )    ≈ 1, ∀ i=1 … N

over a relevant test distribution p(x) of the inputs x. Let (·) denote the vector function mapping an input x ∈^M to a vector of N labels =[u_1 u_2 … u_N] ∈{1 … L}^N i.e. = (x) ≐ [f_1(x) f_2(x) … f_N(x)]. See Figure <ref> and <ref>.



Consider some arbitrary input vector x_0 ∈^M, the corresponding true label u_0 ≐ f_0(x_0), and an arbitrary set of N “target labels" _t ∈{1 … L }^N. Note that if x_0 is chosen from the test distribution p(x), then with high probability, all the N classifiers produce the correct label to the unperturbed input vector x_0 i.e.

    ( (x_0) =[u_0 u_0 … u_0] )    ≈ 1

We now formally define an optimal selective adversarial attack as the smallest perturbation applied to the input vector x_0 that causes the N classifiers _i, i=1 … N to output the target labels _t, each of which may be different from (or remains the same as) the true label u_0, and may be different from each other. Specifically the optimal selective perturbation d(x_0, _t): ^M ×{1 … L }^N →^M for an input vector x_0 and the intended label set _t is defined as:

    d(x_0, _t)    ≐min_δ∈^Mδ, 
    subject to    ( x_0 + δ) ≡_t


Remark. Note that in the definition (<ref>), there is an implicit assumption that the constraint is feasible i.e. that there exists x ∈^N such that (x) ≡_t for any target label set _t. This, of course, is an assumption about the range of the functions () describing the classifiers. Indeed it is a very strong assumption about how differently multiple nominally equivalent classifiers process their inputs. One of our objectives in this paper is to empirically test this assumption. Our findings in this paper surprisingly show that the constraint is often feasible under an arbitrary set of labels. 




 §.§ Selective Fooling
 

We now consider an interesting sub-class of selective adversarial attacks which we will call
Selective Fooling. In this class of attacks, we choose a single adversarial target label u_a that
is different from the true label u_0 and design a perturbation that will cause a subset of 
n < N classifiers in  to output the label u_a while the remaining N-n classifiers
in  continue to output the true label u_0. In other words, we seek to selectively
fool a subset of classifiers to output a particular target label u_a while leaving the remaining 
ones unaffected.

Formally, Selective Fooling is a selective adversarial attack where the target label vector is
set to be:

    _t[i]   =
       u_a, i ∈{1 … n}, 
       u_0 i ∈{n+1 … N}

for some u_a ∈{1 … L}∖{u_0}.




 §.§ Selective Adversarial Attacks using Gradient Descent


We now discuss a simple method that extends previously known algorithms such as FGSM <cit.> that have been shown to be effective in generating adversarial examples to create selective adversarial perturbations. The basic idea is to formulate a cost function that serves as a proxy for the optimization problem in (<ref>), and then use a gradient descent procedure to approximate a solution to (<ref>).

Let _i(x), i=1 … N, x ∈^M represent the probability distribution output by the classifier M_i for the input vector x over the L possible labels 1 … L. Mathematically _i(x) is a 1 × L vector that satisfies ∑_u=1^L _i,u(x) ≡ 1, ∀ i, x, where _i,u(x) is the u-th element of _i(x). In practice, _i are typically calculated using a “softmax" function <cit.> in the last layer of a deep neural network. Let _u, u ∈{1 … L}, be the 1 × L Kronecker delta vector i.e.:

    _u[l]    ≐
    1,    l=u, 
    
    0,    otherwise


Given a target label set =[u_1 u_2 … u_N], we define the following cost function J(x), x ∈^M:

    J(x)    = ∑_i=1^N _i(x) - _u_i^2


Our selective adversarial attack algorithm will iteratively reduce this proxy cost function by gradient descent. Of course a method for generating adversarial examples via gradient searches on the heuristic cost function in (<ref>) can only approximate the optimal solution to (<ref>). Indeed, we will see in our experimental results, that adversarial perturbations obtained from this cost function differs from the optimal solution in some interesting and counter-intuitive ways. We also remark that we can replace the cost function in (<ref>) with other cost functions, such as cost functions using cross-entropy.









§ EXPERIMENTAL RESULTS
 

We constructed a total of N=5 Convolutional Neural Networks with the architecture shown in Table <ref>. We trained each of them on the MNIST handwritten digit recognition dataset[http://yann.lecun.com/exdb/mnist/]. All N networks were identical except for a random initialization of their weights.






 §.§ Training the Classifiers


We used a Stochastic Gradient Descent to minimize the softmax cross entropy function with a learning rate of 0.01, a decay of 1e-6, and a momentum of 0.9 to train each of the 5 networks. The models were trained using the MNIST training data, obtained from the Keras MNIST dataset, for 10 epochs with 55,000 images per epoch. The trained networks were evaluated on 10,000 MNIST images from a separate test set. This resulted in all of the 5 networks achieving an accuracy of at least 95% accuracy on the test set. The models were trained using different random initializations of the network weight and bias parameters. 

Next we describe the procedure we used to generate selective adversarial examples and a series of experiments that we performed using these trained networks.




 §.§ Constructing Selective Adversarial Attacks


In all of our experiments, we used images with the true class label of `0' (results for other digits are similar, and so we focus on label `0' for our presentation ) from the aforementioned MNIST handwritten digit recognition dataset, and then constructed selective adversarial attacks designed to manipulate each of the 5 networks in arbitrary pre-specified ways. We used modified versions of two popular algorithms to generate our adversarial examples: (a) the modified Carlini-Wagner (mCW) algorithm <cit.> and (b) the modified Fast Gradient Signed Method (mFGSM) <cit.>.

The original Carlini-Wagner algorithm[https://github.com/carlini/nn_robust_attacks] was designed to construct adversarial examples for a single classifier using a gradient descent procedure. We modified this algorithm to perform gradient descent on the loss function J(x) as defined in (<ref>). The following parameter values were used in our modified CW algorithm: confidence value 0.1, learning rate 0.02, the number of binary search steps 9, and the max number of iterations 10,000. Our modified FGSM algorithm[https://github.com/soumyac1999/FGSM-Keras] also used the loss function very similar to (<ref>) (with the cross-entropy function replacing the l_2 norm in (<ref>)). In our experiments with the modified FGSM algorithm, we set the parameter ϵ = 0.075 with the maximum number of iterations fixed at 30.





 §.§ Selective Fooling Experiment with Two Classifiers


Our first set of experiments involved the simple sub-class of selective adversarial attacks defined in Section <ref> with N=2, n=1 i.e. we attempt to construct adversarial examples that successfully fool one of two classifiers while leaving the other classifier unaffected.

A total of 25 images of the label 0 were perturbed using both the mCW and mFGSM algorithms described earlier with random target labels. The mFGSM algorithm was able to successfully construct Selective Fooling attacks on all of the 25 images. The mCW algorithm was also able to generate adversarial examples for a majority of images, but takes longer to converge than mFGSM.

This shows the existence of selective adversarial examples, at least for this simple special case of Selective Fooling attack, and suggests that it may indeed be feasible to construct adversarial examples that affect multiple classifiers in different ways (and surprisingly often in all the possible ways) and motivates a deeper study of selective adversarial attacks.



We also tested the effect of the adversarial examples generated by the mCW algorithm on a third classifier that was not involved in the construction of the attack; the confidence value of this third classifier in the true label (0) averaged over the 25 images was calculated to be 0.778. Roughly speaking, this shows that on average, a third classifier is approximately 75% unaffected by the Selective Fooling attack. In contrast, the corresponding number for a non-selective fooling attack against just one classifier using the unmodified Carlini-Wagner algorithm was 0.843 i.e. a third classifier was approximately 85% unaffected by such a non-selective fooling attack. This seems somewhat counter-intuitive: we may expect that a selective attack that narrowly targets one classifier while leaving another classifier unaffected will be less likely to affect a third classifier. Below we consider a possible explanation for this and certain other similarly counter-intuitive results.





 §.§ Selective Fooling Experiment with Four Classifiers



In our next set of experiments, we constructed a series of Selective Fooling attacks with N=4 classifiers. Specifically, for a set of 25 images with the true label of u_0=0 and for each of the 15 non-empty subsets of the 4 classifiers, we attempted to construct selective adversarial attacks to drive their outputs to a target label of u_a=9 using both the mCW and mFGSM algorithms. Surprisingly, both the mCW and mFGSM methods were able to successfully construct adversarial examples that successfully fooled each of the 2^4-1=15 non-empty subsets of the N=4 classifiers while keeping the complementary subset of classifiers unfooled, for each of the 25 images.





The average size of the perturbations measured as the sum of the absolute value of pixel differences between the input image and the adversarial image, is shown as a function of the number of targeted classifiers n in Figs. <ref> and <ref> for the mCW and mFGSM algorithms respectively.

Discussion. The mere fact that there exists small perturbations which selectively and precisely attack every possible subset of classifiers is quite surprising. This suggests that these classifiers might look at very different features for classifications, and do not “think” alike, even though they share the same architectures and training data. Figures <ref> and <ref> suggest that on average we need smaller perturbations to attack more classifiers. We might expect that it would take larger perturbations to simultaneously change the outputs of two classifiers compared to one classifier.  One possible explanation for this observation - as well as our earlier observation about transferability of the Selective Fooling attack - is that the attacks generated by the mCW and mFGSM algorithms are not close to being optimal; a search constrained to leave one or more classifiers unaffected leads to smaller perturbation perhaps because it is forced to keep more features of the original image intact as compared to an unconstrained search.



The confidence in the true label from a fifth classifier that was not involved in the generation of the adversarial examples is shown as a function of the size of the target set n in Figure <ref> for the mCW generated attacks. This plot is consistent with our intuitive expectation that an adversarial example that successfully attacks many classifiers simultaneously is also more statistically likely to transfer to a random classifier.


Adversarial example images for different values of n are shown in Figure <ref>. Typical examples of adversarial images with varying N and n and their respective magnitudes can be seen in Fig. <ref>.



 §.§ Attacking and Defending Subsets of Four Classifiers with Different Target Labels


Our final set of experiments involved a generalization of the Selected Fooling attack where a subset n of N=4 classifiers were targeted to be attacked to n different target labels chosen at random, while keeping the outputs of the remaining N-n classifiers unchanged. Once again, the mFSGM algorithm was able to successfully generate adversarial examples for all 15 non-empty subsets of the N=4 classifiers. The average size of the resulting perturbations as a function of n is shown in Fig. <ref>.







§ CONCLUSIONS


We formulated the problem of selective adversarial attacks, and presented an experimental investigation of a generalized class of adversarial attacks that are designed to manipulate the outputs of multiple machine learning classifiers simultaneously in arbitrarily pre-defined ways. We formulated a novel optimization problem to search for such selective attacks, and applied modified versions of popular algorithms used for the construction of white-box adversarial examples to design targeted attacks on 5 different trained classifiers for the MNIST dataset. Our results show that it is indeed possible to construct precisely targeted adversarial attacks that can arbitrarily modify the outputs of multiple classifiers while keeping another set of classifiers unaffected. These results motivate a deeper study of such selective adversarial attacks and also suggest that different classifiers that have nominally similar performance may in fact have very different decision regions. 




 

















icml2019 






