NAG: Network for Adversary Generation
Konda Reddy Mopuri*, Utkarsh Ojha*, Utsav Garg and R. Venkatesh Babu
Video Analytics Lab, Computational and Data Sciences
Indian Institute of Science, Bangalore, India

arXiv:1712.03390v1 [cs.CV] 9 Dec 2017

kondamopuri@iisc.ac.in, utkarsh2254@gmail.com, utsav002@e.ntu.edu.sg and venky@iisc.ac.in

Abstract

training data (e.g. [3]), etc. More importantly, the adversarial perturbations exhibit cross model generalizability. That
is, the perturbations learned on one model can fool another
model even if the second model has a different architecture or has been trained on a disjoint subset of training images [27, 9].

Adversarial perturbations can pose a serious threat for
deploying machine learning systems. Recent works have
shown existence of image-agnostic perturbations that can
fool classifiers over most natural images. Existing methods
present optimization approaches that solve for a fooling objective with an imperceptibility constraint to craft the perturbations. However, for a given classifier, they generate
one perturbation at a time, which is a single instance from
the manifold of adversarial perturbations. Also, in order
to build robust models, it is essential to explore the manifold of adversarial perturbations. In this paper, we propose
for the first time, a generative approach to model the distribution of adversarial perturbations. The architecture of
the proposed model is inspired from that of GANs and is
trained using fooling and diversity objectives. Our trained
generator network attempts to capture the distribution of
adversarial perturbations for a given classifier and readily
generates a wide variety of such perturbations. Our experimental evaluation demonstrates that perturbations crafted
by our model (i) achieve state-of-the-art fooling rates, (ii)
exhibit wide variety and (iii) deliver excellent cross model
generalizability. Our work can be deemed as an important
step in the process of inferring about the complex manifolds
of adversarial perturbations.

Recent startling findings by Moosavi-Dezfooli et al. [17]
and Mopuri et al. [19] have shown that it is possible to
mislead multiple state-of-the-art deep neural networks over
most of the images by adding a single perturbation. That is,
these perturbations are image-agnostic and can fool multiple diverse networks trained on a target dataset. Such perturbations are named “Universal Adversarial Perturbations”
(UAP) because a single adversarial noise can perturb images from multiple classes. On one side, the adversarial
noise poses a severe threat for deploying machine learning
based systems in the real world. Particularly, for the applications that involve safety and privacy (e.g., autonomous
driving and access granting), it is essential to develop robust models against adversarial attacks. On the other side, it
also poses a challenge to our understanding of these models
along with the current learning practices. Thus, the adversarial behaviour of the deep learning models to small and
structured noise demands a rigorous study now more than
ever.
All the existing methods, weather image specific [27, 9,
18, 6, 16] or agnostic [17, 19], can craft only a single perturbation that makes the target classifier susceptible. Specifically, these methods typically learn a single perturbation
(δ) from a possibly bigger set of perturbations (∆) that can
fool the target classifier. It is observed that for a given technique (e.g., UAP [17], FFF [19]), the perturbations learned
across multiple runs are not very different. In spite of optimizing with a different data ordering or initialization, their
objectives end up learning very close perturbations in the
space (refer sec. 3.2). In essence, these approaches can only
prove that the UAPs exist for a given classifier by crafting
one such perturbation (δ). This is very limited information about the underlying distribution of such perturbations
and in turn about the target classifier itself. Therefore, a

1. Introduction
Machine learning systems are shown [5, 4, 11] to be
vulnerable to adversarial noise: small but structured perturbation added to the input that affects the model’s prediction drastically. Recently, the most successful Deep
Neural Network based object classifiers have also been observed [27, 9, 18, 15, 20] to be susceptible to adversarial attacks with almost imperceptible perturbations. Researchers
have attempted to explain this intriguing aspect via hypothesizing linear behaviour of the models (e.g. [9, 18]), finite
* denotes authors contributed equally.

1

Parameters are not updated

Parameters are updated

+

Adversarial batch

+

Benign batch

=

Loss

Shuffle

+

Shuffled
adversarial batch

Figure 1. Overview of the proposed approach that models the distribution of universal adversarial perturbations for a given classifier. The
illustration shows a batch of B random vectors {z}B transforming into perturbations {δ}B by G which get added to the batch of data
samples {x}B . The top portion shows adversarial batch (XA ), bottom portion shows shuffled adversarial batch (XS ) and middle portion
shows the benign batch (XB ). The Fooling objective Lf (eq. 2) and Diversity objective Ld (eq. 3) constitute the loss. Note that the target
CNN (f ) is a trained classifier and its parameters are not updated during the proposed training. On the other hand, the parameters of
generator (G) are randomly initialized and learned through backpropagating the loss. (Best viewed in color).

more relevant task at hand is to model the distribution of
adversarial perturbations. Doing so can help us better analyze the susceptibility of the models against adversarial perturbations. Furthermore, modelling the distributions would
provide insights regarding the transferability of adversarial
examples and help to prevent black-box attacks [21, 16].
It also helps to efficiently generate large number of adversarial examples for learning robust models via adversarial
training [28].
Empirical evidence [9, 29] has shown that the perturbations exist in large contiguous regions rather than being
scattered in multiple small discontinuous pockets. In this
paper, we attempt to model such regions for a given classifier via generative modelling. We introduce a GAN [8] like
generative model to capture the distribution of the unknown
adversarial perturbations. The freedom from parametric assumptions on the distribution and the target distribution being unknown (no known samples from the target distribution of adversarial perturbations) make the GAN framework
a suitable choice for our task.
The major contributions of this work are
• A novel objective (eq. 2) to craft universal adversarial perturbations for a given classifier that achieves the
state-of-the art fooling performance on multiple CNN

architectures trained for object recognition.
• For the first time, we show that it is possible to model
the distribution (∆) of such perturbations for a given
classifier via a generative model. For this, we present
an easily trainable framework for modelling the unknown distribution of perturbations.
• We demonstrate empirically that the learned model can
capture the distribution of perturbations and generates
perturbations that exhibit diversity, high fooling capacity and excellent cross model generalizability.
The rest of the paper is organized as follows: section 2
details the proposed method, section 3 presents comprehensive experimentation to validate the utility of the proposed method, section 4 briefly discusses the existing related works, and finally section 5 concludes the paper.

2. Proposed Approach
This section presents a detailed account of the proposed
method. For ease of reference, we first briefly introduce the
GAN [8] framework.

2.1. GANs
Generative models for images have seen renaissance
lately, especially because of the availability of large
datasets [23, 30] and the emergence of deep neural networks. Particularly, Generative Adversarial Networks
(GAN) [8] and Variational Auto-Encoders (VAE) [14] have
shown significant promise in this direction. In this work,
we utilize a GAN like framework to model the distribution
of the adversarial perturbations. A typical GAN framework
consists of two parts: a Generator (G) and a Discriminator
(D). The generator G transforms a random vector z into a
meaningful image I; i.e., G(z) = I, where z is usually sampled from a simple distribution (e.g., N (0, 1), U(−1, 1)). G
is trained to produce images (I) that are indistinguishable
from real images from the true data distribution pdata . The
discriminator D accepts an image and outputs the probability for it to be a real image, a sample from pdata . Typically,
D is trained to output low probability pD when a fake (generated) image is presented. Both G and D are trained adversarially to compete with and improve each other. A properly
trained generator G at the end of training is expected to produce images that are indistinguishable from real images.

2.2. Modelling the adversaries

8

A broad overview of our method is illustrated in Fig. 1.
We first formalize the notations used in the subsequent sections of the paper. Note that in this paper, we have considered CNNs that are trained for object recognition [10,
26, 25, 13]. The data distribution over which the classifiers
are trained is denoted as X and a particular sample from
X is represented as x. The target CNN is denoted as f ,
therefore the output of a given layer i is denoted as f i (x).
The predicted label for a given data sample x is denoted as
k̂(x). Output of the softmax layer is denoted as q, which is
a vector of predicted probabilities qj for each of the target
categories j. The image-agnostic additive perturbation that
fools the target CNN is denoted as δ. ξ denotes the limit on
the perturbation (δ) in terms of its l norm. Our objective
is to model the distribution of such perturbations (∆) for a
given classifier. Formally, we seek to model
|| δ ||

8

∆ = {δ : k̂(x + δ) 6= k̂(x) for x ∼ X and
< ξ}

(1)

Since our objective is to model the unknown distribution of
image-agnostic perturbations for a given trained classifier
(target CNN), we make suitable changes in the GAN framework. The modifications we make are: (i) Discriminator
(D) is replaced by the target CNN which is already trained
and whose weights are frozen, and (ii) a novel loss (fooling
and diversity objectives) instead of the adversarial loss to
train the Generator (G). Thus, the objective of our work is
to train a model (G) that can fool the target CNN. The architecture for G is also similar to that of typical GAN which

transforms a random sample to an image through a dense
layer and a series of deconv layers. More details about the
exact architecture are discussed in section 3 and also in the
appendix. We now proceed to discuss the fooling objective
that enables us to model the adversarial perturbations for a
given classifier.

2.3. Fooling Objective
In order to fool the target CNN, the generator G should
be driven by a suitable objective. Typical GANs use adversarial loss to train their G. However, in this work we attempt
to model a distribution whose samples are unavailable. We
know only a single attribute of those samples which is to
be able to fool the target classifier. We incorporate this attribute via a fooling objective to train our G that models the
unknown distribution (∆) of perturbations.
We denote the label predicted by the target CNN on a
clean sample x as benign prediction (c) and that predicted
on the corresponding perturbed sample (x + δ) as adversarial prediction. Similarly, we denote the output vector
of the softmax layer without δ and after adding δ as q and
q 0 respectively. Ideally a perturbation δ should confuse the
classifier so as to flip the benign prediction into a different
adversarial prediction. For this to happen, after adding δ,
the confidence of the benign prediction (qc0 ) should be reduced and that of another category should be made higher.
Thus, we formulate a fooling loss to minimize the confidence of benign prediction on the perturbed sample (x + δ)
Lf = −log(1 − qc0 )

(2)

Fig. 1 gives a graphical explanation of the objective, where
the fooling objective is shown by the blue colored block.
Note that the fooling loss essentially encourages G to generate perturbations that decrease confidence of benign predictions and thus eventually flip the label.

2.4. Diversity Objective
The fooling loss only encourages to learn a G that can
guarantee high fooling capability for the generated perturbations (δ). This objective might lead to some local minima
where the G learns only a limited set of effective perturbations as in [17, 19]. However, our objective is to model
the distribution ∆ such that it covers all varieties of those
perturbations. Therefore, we introduce an additional component to the loss that encourages G to explore the space
of perturbations and generate a diverse set of perturbations.
We term this objective the Diversity objective. Within a
mini-batch of generated perturbations, this objective indirectly encourages them to be different by separating their
feature embeddings projected by the target classifier. In
other words, for a given pair of generated perturbations
δn and δn0 , our objective increases the distance between

f i (x + δn ) and f i (x + δn0 ) at a given layer i in the classifier.
Ld = −

B
X

d(f i (xn + δn ), f i (xn + δn0 ))

(3)

n=1

where n0 is a random index in [1, B] and n0 6= n, B is the
batch size, xn , δn are nth data sample and perturbation in
the mini-batch respectively. Note that a batch contains B
perturbations (δ) generated by G (via transforming random
vectors (z)) and B data samples (x). f i is the output of
the CNN at ith layer and d(., .) is a distance metric (e.g.,
Euclidean) between a pair of features. The orange colored
block in Fig. 1 illustrates the diversity objective.
Thus, our final loss becomes the summation of both fooling and diversity objectives and is given by
Loss = Lf + λLd

(4)

Since it is important to learn diverse perturbations that exhibit maximum fooling, we give equal importance to both
Lf and Ld in the final loss to learn the G (i.e., λ = 1).

2.5. Architecture and Implementation details
Before we present the experimental details, we describe
the implementation and working details of the proposed architecture. The generator part (G) of the network maps the
latent space Z to the distribution of perturbations (∆) for a
given target classifier. The architecture of the generator consists of 5 deconv layers. The final deconv layer is followed
by a tanh non-linearity and scaling

by ξ. Doing so restricts
the perturbations’ range to −ξ, ξ . Following [17, 19],
the value of ξ is chosen to be 10 in order to add a quasiimperceptible adversarial noise. The generator network is
adapted from [24] and more details regarding its architecture can be found in the appendix. We performed all our
experiments on a variety of CNN architectures trained to
perform object recognition task on the ILSVRC-2014 [23]
dataset. We kept the architecture of our generator (G) unchanged for different target CNN architectures and separately learned the corresponding adversarial distributions.
During training, we sample a batch of random

vectors
z ∈ Rd from the uniform distribution U −1, 1 which
in turn get transformed by G into a batch of perturbations {δ}B = {δ1 , δ2 , δ3 , . . . , δB } each of size equal to
that of the image (e.g., 224 × 224 × 3). We also sample
B images {x}B = {x1 , x2 , x3 , . . . , xB } from the available training data to form the mini-batch training data, denoted as benign batch (XB ). We now add the perturbations to the training data in a one-to-one manner i.e.
one perturbation gets added to the corresponding image of
the batch, which gives us the adversarial batch, XA =
{x1 + δ1 , x2 + δ2 , x3 + δ3 , . . . , xB + δB }. This is shown
in the top portion of Fig. 1. We also randomly shuffle the

perturbations ensuring no perturbation remains in its original index in the batch, i.e., {δ1 0 , δ2 0 , δ3 0 , . . . , δB 0 } such that
δi 6= δi 0 , ∀i. With this, we form a shuffled adversarial batch
as XS = {x1 +δ1 0 , x2 +δ2 0 , x3 +δ3 0 , . . . , xB +δB 0 }, which
is shown in the bottom portion of Fig. 1. Note that in order
to prepare XS , only the perturbations ({δ}B ) are shuffled
but not the data samples ({x}B ).
Thus, each of our training iterations consists of three
quasi-batches, namely, (i) Benign images batch XB , (ii)
Adversarial batch XA , and (iii) Shuffled adversarial batch
XS . These are the three portions shown in Fig. 1. We
now feed these through the target CNN (D) and compute
the loss. We obtain the benign predictions (c) over the
clean batch samples {x}B . These labels are used to compute the confidences (qc0 ) for the corresponding adversarial
batch samples. This forms the fooling objective as shown
in eq. 2. Similarly, we obtain the feature representations at
the softmax layer (probability vectors) for both adversarial
and shuffled adversarial batches (top and bottom portions
of Fig. 1) to compute the diversity component of the loss as
shown in eq. 3. Essentially, our diversity objective pushes
apart the final layer representations corresponding to two
different perturbations (δi and δi 0 ) via maximizing the cosine distance between them. Ablation experiments on multiple feature representations and multiple values for λ can
be found in the appendix.
Note that we update only the G part of the network and
the target CNN, which is a pretrained classifier under attack, remains unchanged. We iteratively perform the loss
computation and parameter updation for all the samples in
the training data. During training, we use a small held-out
set of 1000 random images as validation set and stop our
training upon reaching best performance on this set. In our
experiments, the maximum number of epochs is set to 100
and we observe that training of generators for all the target
CNNs gets saturated at around 60 − 70 epochs.

3. Experiments
For all our experiments, we worked with 10000 (10 per
category, similar to [17]) training images randomly chosen
from ILSVRC 2014 train set and 50000 images of ILSVRC
2014 validation set as our testing images. The latent space
dimension d is set to 10. We have experimented with spaces
of different dimensions (e.g., 50, 100) and observed that the
fooling rates obtained are very close. However, we observe
the generated perturbations for d = 10 demonstrate larger
visual variety than other cases. Thus, we keep d = 10 for all
our experiments. We use a batch size (B) of 64 for shallow
networks such as VGG-F [7] and GoogLeNet [26], and 32
for the rest. The models are implemented in TensorFlow [1]
with Adam optimizer on a TITAN-X GPU card.

Table 1. Average fooling rates of the perturbations modelled by our generative network vs. UAP [17]. Rows indicate the target net for
which perturbations are modelled and columns indicate the net under attack. Note that, in each row, entry where the target CNN matches
with the network under attack represents white-box attack and the rest represent the black-box attacks. For our method, along with average
fooling rates, the corresponding standard deviations are also mentioned. The best result for each case is shown in bold and UAP best cases
are shown in blue. The mean avg. rate achieved by the Generator (G) for each of the target CNNs is shown in the rightmost column.
VGG-F
CaffeNet
GoogLeNet VGG-16
VGG-19
ResNet-50
ResNet-152 Mean FR
4.50 47.67+
2.68 50.56+
8.50 55.39+
3.41 52.93+
3.50 64.15+
1.84 81.28+
Our
94.10 +
−4.12 63.73
−
−
−
−
−
−
VGG-F
UAP 93.7
71.8
48.4
42.1
42.1
47.4
57.58
+
+
+
+
+
+
+
Our
79.25−1.44
96.44−1.56
66.66−1.84 50.40−5.61 55.13−4.15 52.38−3.96 48.58−4.25 64.12
CaffeNet
UAP 74.0
93.3
47.7
39.9
39.9
48.0
56.71
+0.86
+2.12
+1.55 56.40−
+4.13 59.14−
+3.17 63.21−
+4.40 59.22−
+1.64 66.23
Our
64.83−
70.46−
90.37−
GoogLeNet
UAP 46.2
43.8
78.9
39.2
39.8
45.5
48.9
+
+
+
+
+
+
Our
60.56+
2.24
65.55
6.95
67.38
4.84
77.57
2.77
73.25
1.63
61.28
3.47
54.38
2.63
65.71
−
−
−
−
−
−
−
VGG-16
UAP 63.4
55.8
56.5
78.3
73.1
63.4
65.08
+
+
+
+
+
+
Our
67.80+
2.49
67.58
5.59
74.48
0.94
80.56
3.26
83.78
2.45
68.75
3.38
65.43
1.90
72.62
−
−
−
−
−
−
−
VGG-19
UAP 64.0
57.2
53.6
73.5
77.8
58.0
64.01
+
+
+
+
+
+
1.89
62.37
2.73
66.40
2.58
86.64
2.61
52.67
1.14
55.16
1.70
65.30
2.60
63.35
Our
47.06+
−
−
−
−
−
−
−
ResNet-50
UAP +
+
+
+
+
+
+
Our
57.66−4.37
64.86−2.95
62.33−1.39 52.17−3.41 53.18−4.16 73.32−2.75 87.24−2.72 64.39
ResNet-152
UAP 46.3
46.3
50.5
47.0
45.5
84.0
53.27

3.1. Perturbations and the fooling rates
The fooling rates achieved by the perturbations crafted
by the learned generative model (G) are presented in Table 1. Results are shown for seven different network architectures trained on ILSVRC-2014 dataset computed for
over 50000 test images. We also investigate the transfer
rates of the perturbations by attacking other unknown models along with the target CNN. Rows denote a particular
target CNN for which we have modelled the distribution of
perturbations and the columns represent the classifiers we
attack. Note that in each row, when the target CNN (row)
matches with the system under attack (column), the fooling
indicates the white-box attack scenario and all other entries
represent the black-box attack scenario.
Since our G network models the perturbation space, we
can now easily generate a perturbation by sampling a z and
feeding it through the G. In Table 1, we report the mean
fooling rates after generating multiple perturbations for a
given CNN classifier. Particularly, the white-box fooling
rates are computed by averaging over 100 perturbations and
black-box rates are averaged over 10. The standard deviations are mentioned next to the fooling rates. Also, the mean
average fooling rate achieved by the learned model (G) for
each of the target CNNs is shown in the rightmost column.
Clearly, the proposed generative model captures the perturbations with higher fooling rates than UAP [17]. Note that
of all the 36 entries for which UAP [17] provided their fooling rates, in only 3 cases (indicated in bold faced blue in
Table 1) they perform better than us. The mean fooling rate
(of all the entries in the table, except the rightmost column)
obtained by the UAP [17] is 57.66 and that achieved by our

model is 65.68, which is a significant 8% improvement.
Figure 2 shows the perturbations generated by the proposed generative model for different target CNNs. Note that
each of them is one random sample from the corresponding
distributions of perturbations. Fig. 4 shows a benign sample
and the corresponding perturbed samples after adding perturbations for multiple CNNs. Note that the perturbations
are sampled from the corresponding distributions learned
by our method.

3.2. Diversity of perturbations
In this subsection, we examine the diversity of the generated perturbations by our model. We present two different
measures to understand the diversity of the perturbations.
3.2.1

Correlation measure

This measure is presented in [17] to show the diversity of
their crafted perturbations. We have trained the generator
for GoogLeNet [26] as detailed earlier. We have sampled
5 random perturbations and computed the normalized inner product between all the 10 pairs. The mean value of
the dot product is 6.64 × 10−6 . The maximum value of
the dot product doesn’t exceed 6.83 × 10−6 . For UAP [17],
FFF [19] it doesn’t exceed 0.1 and 0.0051 respectively. This
result clearly shows that the perturbations generated by our
model demonstrate large diversity (lie almost in random directions) compared to [17, 19]. In spite of the perturbations
being very high dimensional (224 × 224 × 3), dot product
value of 0.1 for [17] shows their perturbations lack variety
and fail to model the manifold of perturbations comprehensively. The perturbations analyzed in this subsection can be
found in Fig. 3.

VGG-F

CaffeNet

GoogLeNet

VGG-16

VGG-19

ResNet-50

ResNet-152

Figure 2. Sample universal adversarial perturbations for different networks. The target CNN is mentioned below the perturbations. Note
that these are one sample from each of the corresponding distributions and across different samplings of the same generative model the
perturbations vary visually.

Figure 3. Sample perturbations generated by proposed approach (top) and UAP [17] (bottom) for GoogLeNet [26]. The maximum value
of the normalized dot product between a pair of these perturbations computed for [17] is 0.1 and visually they look similar. On the other
hand, the maximum dot product value for our method it is 6.83 × 10−6 and the perturbations are visually very different. These results
show that the proposed method faithfully models the distribution of perturbations that can effectively fool the target CNN.

3.2.2

Dominant target classes

It can be interesting to examine the predicted label distribution after adding the perturbations. Doing so can
reveal if there exist any dominant labels that most of
the images get confused to or whether the confusions
are diverse. In this subsection, we analyze the labels
predicted by the target CNN after adding perturbations
modelled by the corresponding generative models (G).
We have considered VGG-F architecture and the 50000
images from the validation set of ILSVRC-2014. We
compute the mean histogram of the predicted labels for
10 perturbations generated by our G. The top-5 categories are: {jigsaw puzzle, maypole, otter,
dome, electric fan}. Though there exists a slight
domination from some of the categories, the extent of domination is far less compared to [17]. While in our case, 257
categories account for the 95% of the predicted labels, for
UAP [17], it is 173 categories. The 48.6% relative higher
diversity compared to [17] is attributed to the effectiveness
of the proposed diversity loss (eq. 3) which encourages the
model to explore various regions in the adversarial manifold.

3.3. Traversing the manifold of perturbations
In this subsection, we perform experiments to understand the landscape of the latent space (Z). In case of
GANs, traversing on the learned manifold generally tells
about the signs of memorization [22]. While walking on
the latent space, if the image generations result in semantic
changes, it is considered that the model has learned relevant and interesting representations. However, our generative model attempts to learn the unknown distribution of
adversarial perturbations with no samples from the target
distribution. Therefore, it is not relevant to investigate for
the smooth semantic changes in the generations but only to
look for smooth visual changes, while retaining the ability
to fool the target classifier.
Figure 5 shows the results of interpolation experiments
on the ResNet-152 [10] classifier. We have randomly taken
a pair of points (z1 and z2 ) in the latent (Z) space and considered 10 intermediate points on the line joining z1 and z2 .
We have generated the perturbations corresponding to these
intermediate points by feeding them through the learned
generative model (G). Figure 5 shows the generated perturbations at the intermediate points along with their fool-

Clean : Horned viper CaffeNet : N. Terrier GoogLeNet : Hamper

VGG-19 : Hyena

ResNet-152 : Chiton

Figure 4. A clean image (left most) and corresponding adversarial images crafted for multiple networks along with predictions.

0.0 ∗ z1 + 1.0 ∗ z2 : 0.1 ∗ z1 + 0.9 ∗ z2 : 0.2 ∗ z1 + 0.8 ∗ z2 : 0.3 ∗ z1 + 0.7 ∗ z2 : 0.4 ∗ z1 + 0.6 ∗ z2 :
87.53
87.71
87.73
87.25
87.48

0.5 ∗ z1 + 0.5 ∗ z2 : 0.6 ∗ z1 + 0.4 ∗ z2 : 0.7 ∗ z1 + 0.3 ∗ z2 : 0.8 ∗ z1 + 0.2 ∗ z2 : 0.9 ∗ z1 + 0.1 ∗ z2 :
88.17
87.94
87.84
88.16
87.34
Figure 5. Interpolation between a pair of points in Z space shows that the distribution learned by our generator has smooth transitions. The
figure shows the perturbations corresponding to 10 points on the line joining a pair of points (z1 and z2 ) in the latent space. Note that these
perturbations are learned to fool the ResNet-152 [10] architecture. Below each perturbation, the corresponding fooling rate obtained over
50000 images from ILSVRC 2014 validation images is mentioned. This shows the fooling capability of these intermediate perturbations
is also high and remains same at different locations in the learned distribution of perturbations.

ing rates. We clearly observe that the perturbations change
smoothly between any pair of consecutive points and the
sequence gives a morphing like effect with large number of
intermediate points. For each of the intermediate perturbations, fooling rate is computed over the 50000 images from
the ILSVRC-2014 validation set. In Fig. 5, below each of
these perturbations, corresponding fooling rates are mentioned. The high and consistent fooling rates along the path
demonstrate that the modelling of the adversarial distribution has been faithful. The proposed approach generates
perturbations smoothly from the underlying manifold. We
attribute this ability of our learned generative model to the
effectiveness of the proposed objectives in the loss. More
results for the interpolation experiments can be found in the
appendix.

3.4. Modelling adversaries for multiple target classifiers
Transferability of the adversarial perturbations (both image specific and agnostic) has been an intriguing revelation

by many of the recent adversarial perturbations works [27,
9, 20, 17, 19]. Moosavi-Dezfooli et al. [17] attempted to
explain the cross-model generalizability with the correlation among different regions in the classification boundaries
learned by them. In this subsection, we investigate if we can
learn to model a single distribution of adversaries that fool
multiple target CNNs simultaneously.
We consider all the 7 target CNNs presented in Table 1
to model a single adversarial manifold that can fool all of
them simultaneously. We keep the G part of the proposed
architecture unchanged while we replace single target classifier with all the target networks. Because of the memory
constraint to fit all the models, we train with a smaller batch
size. The loss to train the generator is the summation of
the individual losses (eq. 4) computed for each of the target
CNNs separately. Thus, the objective driving the optimization becomes to craft the perturbations that fool all the target
CNNs simultaneously. Similar to the single target case, the
diversity objective (eq. 3) encourages to explore multiple

Table 2. Mean fooling rates for 10 perturbations sampled from the distribution of adversaries modelled for multiple target CNNs. The
perturbations result an average fooling rate of 80.02% across the 7 target CNNs which is higher than the best mean fooling rate of 72.62%
achieved by the generator learned for VGG-19.

Network
Fooling rate

VGG-F
83.74

CaffeNet
86.94

GoogLeNet
84.79

regions covering the manifold of perturbations and model
a distribution with a lot of variety. Sample perturbations
modelled for the multiple target classifiers are shown in the
appendix.
Table 2 presents the mean fooling rates obtained by 10
samples from the distribution of perturbations learned to
fool all the 7 target CNNs. The fooling rates are slightly
lesser than those obtained for the dedicated optimization
(white-box attacks in Tab. 1). However, given the complexity of the modelling, the learned perturbations achieve a remarkable average fooling rate of 80.07%. Note that this is
around 8% higher than the best mean fooling rate obtained
by an individual network (computed for each row in Tab. 1),
which is 72.62% by VGG-19. This again emphasizes the
effectiveness of the proposed framework and objectives to
simultaneously model perturbations for classifiers with significant architectural differences.

4. Related Works
Adversarial perturbations [27] have been a tantalizing revelation about machine learning systems. Specifically, the deep neural network based learning systems (e.g.,
[9, 18, 21]) are also shown to be vulnerable to these structured perturbations. Their ability to generalize to unknown
models enables simple ways (e.g., [9]) to launch black-box
attacks that fool the deployed systems. Further, existence
of image-agnostic perturbations [17, 19] along with their
cross model generalizability exposes the weakness of the
current day deep learning models. Differing from previous works [17, 19], our work proposes a novel, yet simple
and effective objective that enables to learn image-agnostic
perturbations. Although the existing objectives successfully
craft these perturbations, they do not attempt to capture the
space of such perturbations. Unlike the existing works, the
proposed method learns a generative model that can capture
the space of the image-agnostic perturbations for a given
classifier. To the best of our knowledge, the only work
which aims to learn a neural network for generating adversarial perturbations via simple feed-forwarding is presented
by Baluja and Fischer et al. [2]. They present a neural network which transforms an image into its corresponding adversarial sample. Note that it generates image specific perturbations and doesn’t aim to model the distribution unlike
us.
Generative Adversarial Network (GAN): Goodfellow et
al. [8] and Radford et al. [22] have shown that GANs can be

VGG-16
73.73

VGG-19
75.24

ResNet-50
80.21

ResNet-152
75.84

trained to learn a data distribution and to generate samples
from it. Further image-to-image conditional GANs have
led to improved generation quality [12, 31]. Inspired from
GAN framework, we have proposed a neural network architecture to model the distribution of universal adversarial
perturbations for a given classifier. The discriminator (D)
part of the typical GAN is replaced with the trained classifier to be fooled. Only the generator (G) part is learned to
generate perturbations to fool the discriminator. Also, as we
don’t have to train D, samples of the target data (i.e., perturbations) are not presented during the training. Through a
pair of effective objectives Lf and Ld , the proposed framework models the perturbations that fool a given classifier.

5. Conclusion
In this paper, we have presented the first ever generative approach to model the distribution of adversarial perturbations for a given CNN classifier. We propose a GAN
inspired framework, wherein we successfully train a generator network that captures the unknown target distribution
without any training samples from it. The proposed objectives naturally exploit the attributes of the samples (e.g., to
be able to fool the target CNN) in order to model their distribution. However, unlike the typical GAN training that
deals with a pair of conflicting objectives, our approach has
a single well behaved optimization (only G is trained).
The ability of our method to generate perturbations with
state-of-the-art fooling rates and surprising cross-model
generalizability highlights severe susceptibilities of the current deep learning models. However, the proposed framework to model the distribution of perturbations also enables
to conduct formal studies towards building robust systems.
For example, Goodfellow et al. [9] introduced adversarial
training as a means to learn robust models and Tramer et
al. [28] extended it to ensemble adversarial training, which
require a large number of adversarial samples. In addition,
the defence becomes more robust if those samples exhibit
diversity [28] and allow the model to fully explore the space
of adversarial examples. While the existing methods are
limited by both generation speed and instance diversity, our
method, after modelling, almost instantly produces adversarial perturbations with lots of variety. We have also shown
that our approach can efficiently model the perturbations
that simultaneously fool multiple deep models.

Appendix
1. Traversing the manifold of perturbations
In this section, we provide additional results for traversing the manifold of adversarial perturbations. On the line joining two random points z1 and z2 in the latent space, we consider 10 intermediate points. We transform these points into
corresponding perturbations by forwarding through the learned G for GoogLeNet [26]. Fig. 1 shows those 10 generated
perturbations. Similar to the results presented on ResNet-152 [10] classifier shown in the main draft (refer to sec.3.3), the
perturbations change smoothly between any pair of consecutive points. The circular patterns in the first image of the top
row slowly disappear by the 4th image (top row) while curtain like patterns emerge from 3rd image of top row, which are
clearly visible by the end of top row. Proceeding further on the path generates wave like patterns shown in middle of bottom
row which transform to a new pattern. Also, the fooling rates are high and consistent along the path. This process again
emphasizes that the proposed model learns relevant representations.

0.0 ∗ z1 + 1.0 ∗ z2 :
89.87

0.1 ∗ z1 + 0.9 ∗ z2 :
89.76

0.2 ∗ z1 + 0.8 ∗ z2 :
90.58

0.3 ∗ z1 + 0.7 ∗ z2 :
91.28

0.4 ∗ z1 + 0.6 ∗ z2 :
91.96

0.5 ∗ z1 + 0.5 ∗ z2 :
89.28

0.6 ∗ z1 + 0.4 ∗ z2 :
91.06

0.7 ∗ z1 + 0.3 ∗ z2 :
91.12

0.8 ∗ z1 + 0.2 ∗ z2 :
91.28

0.9 ∗ z1 + 0.1 ∗ z2 :
90.04

Figure 1. Interpolation between a pair of points in Z space show that the distribution learned by our generator has smooth transitions.
The figure shows the perturbations corresponding to 10 points on the line joining a pair of random points (z1 and z2 ) in the latent space.
Note that these perturbations are learned to fool the GoogLeNet [26] architecture. Below each perturbation, the corresponding fooling rate
obtained over 50000 images from ILSVRC 2014 validation images is mentioned. This shows the fooling capability of these intermediate
perturbations is also high and it is similar at different locations in the learned distribution of perturbations.

2. Generator details
In this section, we explain the architecture of the generator network in the proposed framework that learns the manifold
of adversarial perturbations for a given target classifier. Table 3 presents the details of the generator network which basically
includes an initial fully connected layer followed by several deconvolutional layers to map the latent space vectors to image
size perturbations. The architecture is adapted from [24] which presents improved techniques to GAN training such as virtual
batch normalization to better stabilize the training of the generator network. Note that for all our experiments, the generator
architecture is fixed independent of the target CNN under attack.

3. Modelling adversaries for Multiple target classifiers
In this section, we present sample perturbations obtained for multi-target case, i.e., the generator is learned to model the
adversaries that can simultaneously fool multiple target classifiers. Fig. 2 shows sample perturbations captured by the learned
distribution. Note that the perturbations exhibit significant visual diversity. Also, in section 7 of this appendix, we show that
the perturbations exhibit diversity in terms of the predicted labels after adding to clean images.

Table 3. Overview of the layers in the generator network to model the adversarial perturbations by transforming a latent space vector z of
dimension 10 to a perturbation of size 224 × 224.

Generator
FC (10, 64*7*4*4)
Virtual BN, ReLU
Deconv (1,7,7,64*4)
Virtual BN, ReLU
Deconv (1,14,14,64*2)
Virtual BN, ReLU
Deconv (1,28,28,64*1)
Virtual BN, ReLU
Deconv (1,56,56,64*1)
Virtual BN, ReLU
Deconv (1,112,112,64*1)
Virtual BN, ReLU
Deconv (1,224,224,64*1)
Virtual BN, ReLU
Deconv (1,224,224,3)
10*tanh

Figure 2. Sample perturbations obtained from the distribution of perturbations learned for multiple target classifiers. The G is learned to
craft perturbations which can simultaneously fool the 7 CNNs mentioned in the main draft. Note that the perturbations are of wide variety.

4. Representations for diversity objective
The proposed loss has a diversity component in order for the generative model G to be able to capture all the variations in
the distribution of perturbations. The diversity objective Ld indirectly increases the distance between any pair of perturbations
generated in a mini-batch (δi and δi0 ). The objective projects the feature representations belonging to the adversarial and
shuffled adversarial samples apart. As the transformations learned by layers of the target CNN (D) are fixed, the objective
helps G to generate a diverse set of perturbations that fool the target.
In this subsection, we investigate the effectiveness of multiple feature representations to encourage the diversity in the
modelling of the perturbations. We have worked with VGG-F [7] model for this analysis and considered conv4, conv5 and
softmax layers. For separating features at conv4 and conv5 layers we have employed Euclidean distance and for features
obtained at softmax layer, cosine distance is maximized. Table 4 shows the fooling rates obtained by the proposed objective.
We have run the optimization 10 times and report the mean fooling rates along with their standard deviations. Note that the
fooling performance is almost same (less than 2 % difference) for all the three representations. Also, the standard deviations
are very less indicating the effectiveness of the fooling objective and faithful modelling of the distribution. However, upon
visual investigation, we observe that the variations captured via separating the softmax representations is relatively high.
Thus, we chose to work with representations at softmax layer to learn the generative model of perturbations in all our
experiments.

Table 4. Ablation results for different representations to encourage capturing the diversity in the distribution of perturbations for VGG-F [7]
model. Note the numbers are the fooling rates obtained over a held-out set across 10 runs of the optimization. Right most column reports
the mean fooling rate along with the standard deviation.

PP
#Run
PP
Feature PPP
PP
+ Metric
P
Conv4 + L2
Conv5 + L2
Softmax + cosine

1

2

3

4

5

6

7

8

9

10

Mean (std.)

93.82
94.77
93.32

93.76
94.75
93.63

93.95
94.86
93.62

93.81
94.76
93.60

93.92
94.82
93.52

93.86
94.83
93.74

93.88
94.95
93.43

93.91
94.70
93.64

93.87
94.73
93.66

93.90
94.87
93.57

93.86 (0.055)
94.80 (0.072)
93.57 (0.11)

5. Sample benign and corresponding perturbed images
Fig. 3 shows a set of benign (top row) and corresponding adversarial images (bottom row) for CaffeNet [13]. Labels
provided in the top row are ground truth and in the bottom row are the predicted ones by CaffeNet. Note that different
perturbations are added to each of the images which result in different predicted labels.

Starf ish

W hiptail

Steelarchbridge

P aintbrush

Lipstick

Gibbon

Sidewinder

P orcupine

Ear

T ray

Figure 3. Set of benign and corresponding perturbed images for CaffeNet [13]. Top row shows the benign samples with their ground truth
labels below. Bottom row shows the predicted label on the corresponding perturbed images.

6. Weighting the individual loss components
In this section we perform experiments to understand how weighting of the two loss components i.e., diversity objective
(Ld ) and fooling objective (Lf ) affects the learning. The target architecture we consider for this set of ablations is ResNet-50
architecture trained on ILSVRC-2014 data. We vary the value of λ (weight given to the diversity objective) from 0.1 to 10 in
steps of 10 and separately learn three generators. Table 5 presents the corresponding fooling rate and diversity measure for
the learned generators. Note that the diversity measure is the number of labels that account for 95% of the predicted labels
after adding the perturbations (Refer sec 3.2.2 of the main draft). Also, these numbers are computed on the 50000 validation
images of ILSVRC-2014 data. Results demonstrate that both the measures clearly reflect the relative weights given to the
corresponding loss components. That is, fooling rate increases from 76.26% to 89.04% as the weight given to Lf increases,
at the same time the number of dominant labels decreases from 348 to 253. This observation clearly demonstrates that the
weights given to the individual loss components correlate with the resulting performance. We can treat the weight λ as an
adjustable hyper parameter in the system that can control the generated perturbations from high fooling - limited variety to
less fooling - wide variety.

Table 5. Effect of weighting loss components while learning the generator. Note that weights given to the components correlate to the
resulting performance.

Loss (L)
Lf + 10 × Ld
Lf + Ld
Lf + 0.1 × Ld

Fooling rate
76.26
86.64
89.04

Diversity measure
348
315
253

7. Diversity in perturbations learned by a generator
In this section we present example perturbed images crafted by adding different perturbations learned by a Generator (G)
to a given benign image. Note that we have learned the G to model the perturbations that simultaneously fool all the 7 target
CNNs mentioned in the main draft (refer to sec.3.4 in the main draft). The predictions are obtained by GoogLeNet [26].
Fig. 4 shows the benign and corresponding perturbed images crafted by adding 4 random perturbations generated by the G.
Leftmost image is a benign sample and the following four images are perturbed samples. The predicted labels are mentioned
below the corresponding images. Note that all the four predictions are different and this demonstrates that the perturbations
(δ) learned by a generator (G) exhibit diversity in terms of the predicted labels. Importantly, we observe similar diversity for
all the generators trained on different target CNNs.

Whiptail

Alligator lizard

Sidewinder

Banded gecko

Common newt

Figure 4. Sample benign image (extreme left) and four perturbed images obtained by adding different perturbations learned the proposed
generator (G). Note that the G is learned to fool all the 7 target CNNs and predictions are obtained by GoogLeNet. The ground truth for
the benign image is mentioned in green below it. Predicted labels mentioned below the corresponding images in red and all of them are
different.

References
[1] M. Abadi et al. TensorFlow: Large-scale machine
learning on heterogeneous systems, 2015. 4
[2] S. Baluja and I. Fischer. Adversarial transformation
networks: Learning to generate adversarial examples.
arXiv preprint arXiv:1703.09387, 2017. 8
[3] Y. Bengio. Learning deep architectures for AI. Found.
Trends Mach. Learn., 2(1), Jan. 2009. 1
[4] B. Biggio, I. Corona, D. Maiorca, B. Nelson,
N. Šrndić, P. Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. In
Joint European Conference on Machine Learning and
Knowledge Discovery in Databases. Springer, 2013. 1
[5] B. Biggio, G. Fumera, and F. Roli. Pattern recognition systems under attack: Design issues and research
challenges. International Journal of Pattern Recognition and Artificial Intelligence, 28(07), 2014. 1
[6] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In Security and Privacy
(SP), IEEE Symposium on, 2017. 1
[7] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep
into convolutional nets. In Proceedings of the British
Machine Vision Conference (BMVC), 2014. 4, 10, 11
[8] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, (NIPS), 2014. 2,
3, 8
[9] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv
preprint arXiv:1412.6572, 2014. 1, 2, 7, 8
[10] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arXiv preprint
arXiv:1512.03385, 2015. 3, 6, 7, 9
[11] L. Huang, A. D. Joseph, B. Nelson, B. I. Rubinstein,
and J. D. Tygar. Adversarial machine learning. In Proceedings of ACM Workshop on Security and Artificial
Intelligence, AISec, 2011. 1
[12] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Imageto-image translation with conditional adversarial networks. In The IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2017. 8
[13] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long,
R. Girshick, S. Guadarrama, and T. Darrell. Caffe:
Convolutional architecture for fast feature embedding.
arXiv preprint arXiv:1408.5093, 2014. 3, 11
[14] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
3

[15] A. Kurakin, I. J. Goodfellow, and S. Bengio. Adversarial machine learning at scale. arXiv preprint
arXiv:1611.01236, 2016. 1
[16] Y. Liu, X. Chen, C. Liu, and D. X. Song. Delving
into transferable adversarial examples and black-box
attacks. In International Conference on Learning Representations (ICLR), 2017. 1, 2
[17] S. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and
P. Frossard. Universal adversarial perturbations. In
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 1, 3, 4, 5, 6, 7, 8
[18] S. Moosavi-Dezfooli, A. Fawzi, and P. Frossard.
Deepfool: A simple and accurate method to fool deep
neural networks. in The IEEE Conference on Computer Vision and Pattern Recogntion (CVPR), 2016.
1, 8
[19] K. R. Mopuri, U. Garg, and R. V. Babu. Fast feature
fool: A data independent approach to universal adversarial perturbations. In Proceedings of the British Machine Vision Conference (BMVC), 2017. 1, 3, 4, 5, 7,
8
[20] N. Papernot, P. D. McDaniel, I. J. Goodfellow, S. Jha,
Z. B. Celik, and A. Swami. Practical black-box attacks against deep learning systems using adversarial
examples. arXiv preprint arXiv:1602.02697, 2016. 1,
7
[21] N. Papernot, P. D. McDaniel, X. Wu, S. Jha, and
A. Swami. Distillation as a defense to adversarial
perturbations against deep neural networks. arXiv
preprint arXiv:1511.04508, 2015. 2, 8
[22] A. Radford, L. Metz, and S. Chintala.
Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint
arXiv:1511.06434, 2015. 6, 8
[23] O. Russakovsky, J. Deng, H. Su, J. Krause,
S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla,
M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet
Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3),
2015. 3, 4
[24] T. Salimans, I. J. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques
for training gans. In Advances in Neural Information
Processing Systems (NIPS). 2016. 4, 9
[25] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition.
CoRR, abs/1409.1556, 2014. 3
[26] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed,
D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv
preprint arXiv:1409.4842, 2014. 3, 4, 5, 6, 9, 12

[27] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna,
D. Erhan, I. J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint
arXiv:1312.6199, 2013. 1, 7, 8
[28] F. Tramèr, A. Kurakin, N. Papernot, D. Boneh, and
P. McDaniel. Ensemble adversarial training: Attacks
and defenses. arXiv preprint arXiv:1705.07204, 2017.
2, 8
[29] D. Warde-Farley and I. J. Goodfellow. Adversarial
perturbations of deep neural networks. Perturbations,
Optimization, and Statistics, 2016. 2
[30] B. Zhou, A. Khosla, À. Lapedriza, A. Torralba, and
A. Oliva. Places: An image database for deep scene
understanding. arXiv preprint arXiv:1610.02055,
2016. 3
[31] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired
image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593,
2017. 8

