On the Suitability of $L_p$-norms for Creating and Preventing Adversarial Examples
Mahmood Sharif Lujo Bauer Michael K. Reiter








  
  <cit.>
  7
  7
  <ref>
  7
  <7>
  7
  7
  7
  7

  






















[1]@#1


















































On the Suitability of L_p-norms for

Creating and Preventing Adversarial Examples
    Mahmood Sharif†
Lujo Bauer†
Michael K. Reiter

†Carnegie Mellon University

University of North Carolina at Chapel Hill

{mahmoods,lbauer}@cmu.edu, reiter@cs.unc.edu

    ^⋆ E-mail: mailto:Jordan.Collier@csiro.auJordan.Collier@csiro.au

Accepted 2018 February 23. Received 2018 February 23; in original form 2017 June 16
==========================================================================================================================================================================







Much research has been devoted to better understanding
adversarial examples, which are specially crafted inputs to
machine-learning models that are perceptually similar to benign inputs, but
are classified differently (i.e., misclassified). Both algorithms that
create adversarial examples and strategies for defending against
adversarial examples
typically use L_p-norms to measure the perceptual similarity
between an adversarial input and its benign original. Prior work
has already shown, however, that two images need not be close to each
other as measured by an L_p-norm to be perceptually similar. In this
work, we show that nearness according to an L_p-norm is not just
unnecessary for perceptual similarity, but is also insufficient.
Specifically, focusing on datasets (CIFAR10 and MNIST),
L_p-norms, and thresholds used in prior work, we show
through online user studies that “adversarial
examples” that are closer to their benign counterparts than
required by commonly used L_p-norm thresholds can nevertheless
be perceptually distinct to humans from the corresponding
benign examples. Namely, the perceptual distance between two
images that are “near” each other according to an L_p-norm can be
high enough that participants frequently classify the two images as
representing different objects or digits. Combined with prior work,
we thus demonstrate that nearness of inputs as measured by
L_p-norms is neither necessary nor sufficient for perceptual
similarity, which has implications for both creating and defending
against adversarial examples.
We propose and discuss alternative similarity metrics to
stimulate future research in the area.





§ INTRODUCTION














Machine learning is quickly becoming a key aspect of many technologies
that impact us on a daily basis, from automotive driving aids to city
planning, from smartphone cameras to cancer diagnosis. As such, the
research community has invested substantial effort in understanding
adversarial examples, which are inputs to machine-learning
systems that are perceptually similar to benign inputs, but that are
misclassified, i.e., classified differently than the benign inputs
from which they are derived (e.g., <cit.>).
An attacker who creates an adversarial
example can cause an object-recognition algorithm to incorrectly
identify an object (e.g., as a worm instead of as an panda) <cit.>,
a street-sign recognition algorithm to fail to recognize a stop
sign <cit.>, or a face-recognition system to fail to identify a
person <cit.>. Because of the potential impact on safety
and security, better understanding the susceptibility of machine-learning
algorithms to adversarial examples, and devising defenses, has been a
high priority.

A key property of adversarial examples that makes them dangerous is
that human observers do not recognize them as adversarial. If a human
recognizes an input (e.g., a person wearing a disguise at an airport)
as adversarial, then any potential harm may often be prevented by
traditional methods (e.g., physically detaining the attacker). Hence, most
research on creating adversarial examples (e.g., <cit.>) or defending against them (e.g., <cit.>) focuses on adversarial examples that are
imperceptible, i.e., a human would consider them perceptually
similar to benign images.



The degree to which an adversarial example is imperceptible from its
benign original is usually measured using L_p-norms, e.g., L_0
(e.g., <cit.>), L_2 (e.g., <cit.>),
or L_∞ (e.g., <cit.>). Informally, for images, L_0 measures the
number of pixels that are different between two images, L_2 measures the
Euclidean distance between two images, and L_∞ measures the largest difference between
corresponding pixels in two images. These measures of imperceptibility
are critical for creating adversarial examples and defending against them.
On the one hand, algorithms for creating adversarial examples seek to enhance
their imperceptibility by producing inputs that both cause misclassification
and whose distance from their corresponding benign originals has small
L_p-norm.
On the other hand, defense mechanisms assume that if the difference
between two inputs is below a specific L_p-norm threshold then the
two objects belong to the same class (e.g., <cit.>).

Hence, the completeness and soundness of attacks and defenses commonly rely
on the assumption that some L_p-norm is a reasonable measure for
perceptual similarity, i.e., that if the L_p-norm of the difference
between two objects is below a threshold then the difference between
those two objects will be imperceptible to a human, and vice
versa. Recent work has shown that one direction of this assumption
does not hold: objects that are indistinguishable to humans (e.g., as a
result of slight rotation or translation) can nevertheless be very
different as measured by the L_p-norm of their
difference <cit.>.

In this paper we further examine the use of L_p-norms as a measure
of perceptual similarity. In particular, we examine whether pairs of
objects whose difference is small according to an L_p-norm are
indeed similar to humans. Focusing on datasets, L_p-norms,
and thresholds used in prior work, we show that small differences
between images according to an L_p-norm do not imply perceptual
indistinguishably. Specifically, using the CIFAR10 <cit.> and
MNIST <cit.> datasets, we show via 
online user studies that images whose distance—as measured by the L_p-norm
of their difference—is below thresholds used in prior work can nevertheless
be perceptibly very different to humans. The perceptual distance between
two images can in fact lead people to classify two images differently.
For example, we find that by perturbing about 4% of pixels in digit
images to achieve small 0 distance from benign images (an
amount comparable to prior work <cit.>), humans
become likely to classify the resulting images correctly only 3% of the time.

Combined with previous work, our results show that
nearness between two images according to an L_p-norm is
neither necessary nor sufficient for those images to be perceptually
similar. This has implications for both attacks and defenses against
adversarial inputs. For attacks, it suggests that even though a
candidate attack image may be within a small L_p distance
from a benign image, this does not ensure that a human would find
the two images perceptually similar, or even that a human
would classify those two images consistently (e.g., as the same person or
object). For defenses, it implies defense strategies that attempt to
train machine-learning models to correctly classify what ought to
be an adversarial example may be attempting to solve an extremely
difficult problem, and may result in ill-trained machine-learning models.













To stimulate future research on developing better similarity metrics
for comparing adversarial examples with their benign counterparts, we propose
and discuss several alternatives to p-norms.
In doing so, we hope to improve attacks against machine-learning
algorithms, and, in return, defenses against them.

Next, we review prior work and provide background (sec:relwork).
We then discuss the necessity and sufficiency of conditions for perceptual
similarity, and show evidence that p-norms lead to conditions that are
neither necessary nor sufficient (sec:limit1sec:studies). Finally, we discuss
alternatives to p-norms and conclude (sec:discussionsec:conclusion).




§ BACKGROUND AND RELATED WORK













In concurrent research efforts, Szegedy et al. and Biggio et al. showed
that specifically crafted small perturbations of benign inputs can lead
machine-learning models to misclassify them <cit.>.
The perturbed inputs are referred to as adversarial examples <cit.>.
Given a machine-learning model, a sample x̂ is considered
as an adversarial example if it is similar to a benign sample x
(drawn from the data distribution), such that x is correctly classified and
x̂ is classified differently than x. Formally, for a classification
function F, a class c_x of x, a distance metric D, and a threshold
ϵ, x̂ is considered to be an adversarial example if:

    F(x)=c_x ∧ F(x̂)≠ c_x ∧ D(x,x̂)≤ϵ

The leftmost condition (F(x)=c_x) checks that x is correctly classified,
the middle condition (F(x̂)≠ c_x) ensures that x̂ is incorrectly
classified, and the rightmost condition (D(x,x̂)≤ϵ) ensures that 
x and x̂ are similar (i.e., their distance is
small) <cit.>.

Interestingly, the concept of similarity is ambiguous. For example, two images
may be considered similar because both contain the color blue, or because
they are indistinguishable (e.g., when performing ABX tests <cit.>).
We believe that prior work on adversarial examples implicitly
assumes that similarity refers to perceptual or visual similarity,
as stated by Engstrom et al. <cit.>.
As Goodfellow et al. explain, adversarial
examples are “particularly disappointing because a popular approach in computer
vision is to use convolutional network features as a space where Euclidean
distance approximates perceptual distance” <cit.>.
In other words, adversarial examples are particularly of interest
because they counter our expectation that neural networks specifically, and
machine-learning models in general, represent perceptually similar inputs
with features that are similar (i.e., close) in the Euclidean space.

A common approach in prior work has been to use p-distance
metrics (as D) in attacks that craft adversarial examples
and defenses against them (e.g., <cit.>).
For non-negative
values of p, the p distance between the two d-dimensional
inputs x and x̂ is defined as <cit.>:

    ||x-x̂||_p =(∑_i=1^d |x_i - x̂_i|^p)^1 p


The main p distances used in the literature are 0,
2, and ∞. Attacks using
0 attempt to minimize the number of pixels
perturbed <cit.>; those 
using 2 attempt to minimize the Euclidean distance
between x and
x̂ <cit.>; and
attacks using ∞ attempt to minimize the maximum
change applied to any coordinate in x <cit.>.

To defend against adversarial examples, prior work has focused on either training
more robust deep neural networks (s) that are not susceptible to small
perturbations <cit.>,
or developing techniques to detect adversarial examples <cit.>.
Adversarial training is a particular defense that has achieved a relatively
high success <cit.>.
In this defense, adversarial examples with bounded p distance
(usually, p=∞) are generated in each iteration of training s,
and the   is trained to correctly classify those examples. Insufficiency
and lack of necessity in p-distance metrics have direct implication on
adversarial training, as we discuss in sec:limit1. 








Despite the goal for adversarial examples to be perceptually
similar to benign samples, little prior work on adversarial examples
has explicitly explored or accounted for human perception.
The theoretical work of Wang et al. is an exception, as they treated the
human as an oracle to seek for the conditions under which s would
be robust <cit.>. In contrast, we take
an experimental approach to show that p distances may be
inappropriate for defining adversarial examples. Our findings are 
in line with research in psychology, which has found
that distance metrics in geometric spaces may not always match
humans' assessment of similarity (e.g., <cit.>). Concurrently
to our work, Elsayed et al. showed that adversarial examples can mislead
humans as well as s <cit.>. While they considered
images of higher dimensionality than we consider in this work (see
sec:studies), they allowed perturbations of higher norm than
commonly found in practice.



Some work proposed generating adversarial examples via techniques
other than minimizing p-norms. In three different concurrent
efforts, researchers proposed to use minimal geometric transformations
to generate adversarial examples <cit.>. As we detail in the next section,
geometric transformations evidence that conditions on p-norms
are unnecessary for ensuring similarity. In other work, researchers proposed
to achieve imperceptible adversarial examples by maximizing the Structural
Similarity (SSIM) with respect to benign images <cit.>.
SSIM is a measure of perceived quality of images that has been shown
to align with human assessment <cit.>. It is a differentiable
metric with values in the interval [-1,1] (where a values closer to 1 indicate
higher similarity). By maximizing SSIM, the researchers hoped to increase
the similarity between the adversarial examples and their benign
counterparts. In sec:discussion we provide a preliminary analysis
of SSIM as a perceptual similarity metric for adversarial examples.

In certain cases, perceptual similarity to a reference image is
not a goal
for an attack (e.g., images that seem incomprehensible or benign
to humans, but are classified as street signs by machine-vision
algorithms <cit.>). While
such attacks are important to defend against, this paper focuses on
studying the notion of
perceptual similarity that is relied upon in the majority of the literature on
adversarial examples.




§ NECESSITY AND SUFFICIENCY OF CONDITIONS FOR PERCEPTUAL SIMILARITY
























To effectively find adversarial examples and defend against them,
the parameter choice in eqn:adv_examples should
help us capture the set of all interesting adversarial examples. In
particular, the selection of D and ϵ should capture
the samples that are perceptually similar to benign samples.
Ideally, we should be able to define necessary
and sufficient conditions for perceptual similarity via
D and ϵ.




 §.§ Necessity


The condition C:=D(x,x̂)≤ϵ is a necessary condition for
perceptual similarity if:

    x̂ is perceptually similar to x ⇒ C


Finding necessary conditions for perceptual similarity is important
for the development of better attacks that find adversarial examples, as
well as better defenses. If the condition C used in hopes of ensuring
perceptual similarity is unnecessary (i.e., there exist examples that are
perceptually similar to benign samples, but do not satisfy C),  then the search
space of attacks may be too constrained and some stealthy adversarial examples
may not be found. For defenses, and especially for
adversarial training (because the s are specifically trained to prevent
adversarial inputs that satisfy C), unnecessary conditions for perceptual
similarity may lead us to fail at defending against adversarial examples that
do not satisfy C.




Attacks that craft adversarial examples via applying slight geometric
transformations (e.g., translations and rotations) to benign
samples <cit.>
evidence that when using p-distance metrics as the measure of
distance, D, we may wind up with unnecessary conditions
for perceptual similarity. Such geometric transformations result in
small perceptual differences with respect to benign samples, yet they
result in large p distances. fig:unnecessary
shows two adversarial examples resulting from
geometric transformation of 32× 32 images. While the adversarial
examples are similar to the benign samples, their p distance is
large: 0≥3,010 (maximum possible is 3,072), 
2≥15.83 (maximum possible is 55.43), and
∞≥0.87 (maximum possible is 1).[We use
RGB pixel values in the range [0,1].]
These distances are much larger that what has been used in prior
work (e.g., <cit.>).
Indeed, because small p is not a necessary condition
for perceptual similarity, state-of-the-art defenses meant to defend
against p-bounded adversaries fail at defending against adversarial
examples resulting from geometric transformations <cit.>.




 §.§ Sufficiency


The condition C:=D(x,x̂)≤ϵ is a sufficient condition for
perceptual similarity if:

    C ⇒x̂ is perceptually similar to x


Alternately, C is insufficient, if it is possible to demonstrate that
a sample x̂ is close to x under D, while in fact x̂
is not perceptually similar to x.
The sufficiency of C for perceptual similarity is also important for
both attacks and defenses. In the case of attacks, an adversary using
insufficient conditions for perceptual similarity may craft misclassified
samples that she may deem as perceptually similar to benign samples
under C, when they are not truly so. For defenses, the defender may
be attempting to solve an extremely difficult problem by requiring that a
machine-learning model would classify inputs in a certain way, while
even humans may be misled by such inputs due to their lack of
perceptual similarity to benign inputs. In the case of adversarial training,
if C is insufficient, we may even train s to classify inputs
differently than how humans would (thus, potentially poisoning
the ). 

In the next section we show that the p-norms commonly used
in prior work may be insufficient for ensuring perceptual similarity.
We emphasize that our findings should not be interpreted
as stating that the adversarial examples reported on by prior work are not
imperceptible. 
Instead, our findings highlight that commonly used
p-norms and associated thresholds in principle permit algorithms for crafting
adversarial examples to craft samples that are not perceptually similar
to benign ones, leading to the
undesirable outcomes described above. Specific instances of algorithms
that use those p-norms and thresholds could still
produce imperceptible adversarial examples because the creation of
these examples is constrained by factors other than the chosen
p-norms and thresholds.





§ EXPERIMENT DESIGN AND RESULTS 

































To show that a small p distance (p∈{0, 2, ∞}) from benign
samples is insufficient for ensuring perceptual similarity to these samples,
we conducted three online user studies (one for each p). The goal of each study
was to show that, for small values of ϵ, it is
possible to find samples that are close to benign samples in p, but
are not perceptually similar to those samples to a human. In what follows, 
we present the high-level experimental design that is common among the
three studies. Next, we report on our study participants. Then, we provide the
specific design details for each study and the results.



 §.§ Experiment Design


Due to the many ways in which two images can be similar, it is unclear
whether one can learn useful input from users by directly asking them
about the level of similarity between image pairs. Therefore, we rely on
indirect reasoning to determine whether an image x̂ is perceptually
similar to x. In particular, we make the following observation: if we ask
mutually exclusive sets of users about the contents of x̂ and x,
and they disagree, then we learn that x and x̂ are definitely
dissimilar; otherwise, we learn that they are likely similar. Our observation
is motivated by Papernot et al.'s approach to determine that their attack is
imperceptible to humans <cit.>.

Motivated by the above-mentioned observation, we followed a
between-subject design for each study, assigning each participant
to one of three conditions: , , and . In all the
conditions, participants were shown images and were asked to select
the label (i.e., category) that best describes the image out of ten possible
labels (e.g., the digit shown in the image). The conditions differed in
the nature of images shown to the participants. Participants in
 (“benign” condition) were shown unaltered images from
standard datasets. Participants in  (“adversarial and
imperceptible” condition) were shown adversarial examples of
images in  that fool state-of-the-art s. The images
in  have small p distances to images in ,
and were not designed to mislead humans. Participants in 
 (“adversarial and perceptible” condition) were shown
variants of the images in  that are close in p distance
to their counterparts in , but were designed to
mislead both humans and s. To lower the mental
burden on participants, each participant was asked only 25
image-categorization questions. Because the datasets
we used contain thumbnail images (see below), we presented the
images to participants in three different sizes: original size, resized
× 2, and resized × 4. Additionally to categorizing
images, we asked participants in all conditions about
their level of confidence in their answers on a 5-point Likert
scale (one denotes low confidence and five denotes high confidence).
The protocol was approved by Carnegie Mellon's review board.

Conceptually, the responses of participants in  help
us estimate humans' accuracy on benign images (i.e., their likelihood to
pick the labels consistent with the ground truth). By comparing
the accuracy of users in  to , we learn whether
the attack and the threshold on p distance that we pick
result in imperceptible attacks. We hypothesize that images in
 are likely to be categorized correctly by users. Hence,
the attack truly crafts adversarial examples, and poses risk to the
integrity of s at the chosen threshold. In contrast, by comparing
 with , we hope to show that, for the same threshold,
it is possible to find instances that mislead humans and s. Namely,
we hypothesize that the accuracy of users on  is significantly
lower than their accuracy on . If our hypothesis is validated, we
learn that small p distance does not ensure perceptual similarity.

Datasets and s.
In the studies, we used images from the MNIST <cit.>
and CIFAR10 <cit.> datasets. MNIST is a
dataset of 28× 28 pixel images of digits, while CIFAR10 is
a dataset of 32× 32 pixel images that contain ten object
categories: airplanes, automobiles, birds,  cats, deer, dogs,
frogs, horses, ships, and trucks. Both MNIST and CIFAR10
are widely used for developing both attacks on s and
defenses against them (e.g., <cit.>).

In conditions  and , we created attacks against
two s published by Madry et
al. <cit.>—one for MNIST, and another
for CIFAR10. The MNIST  is highly accurate, achieving 98.8%
accuracy on the MNIST test set. The CIFAR10  also achieves
a relatively high accuracy—87.3% on CIFAR10's test
set. More notably, both s are two of the most resilient models
to adversarial examples (specifically, ones with bounded
∞ distance from benign samples) known to date.




 §.§ Participants
 

We recruited a total of 399 participants from
the United States through the Prolific crowdsourcing platform.



Their ages
ranged from 18 to 78, with a mean of 32.85 and standard deviation of 11.54.
The demographics of our participants were slightly skewed toward males:
59% reported to be males, 39% reported to be females, and 1% preferred
not to specify their gender. 34% of the participants were students, and
84% were employed at least partially. Participants took an average
of roughly six minutes to complete the study and were compensated $1.














 §.§ Insufficiency of 0


Study Details.
We used the MNIST dataset and  to test whether it is
possible to perturb images only slightly on 0 while
simultaneously misleading humans and s.
 was assigned 75 randomly selected images from the test
set of MNIST. All 75 images were correctly classified by the .
We used the Jacobian Saliency Map Approach
(JSMA) <cit.>, as implemented in
Cleverhans <cit.>, to craft adversarial
examples for . We limited the amount of change applied to
an image to at most 15% of the pixels, and found that JSMA was able
to find successful attacks for 68 out of 75 images. For successful
attacks, JSMA perturbed 4.90% (2.91% standard deviation) of pixels
on average—this is comparable to the result of Papernot et
al. <cit.>. Images in  were
crafted manually. Using a photo-editing software[GIMP
(<https://www.gimp.org/>)] while simultaneously
receiving feedback from the DNN, a member of our team edited the
75 images from  while
attempting to minimize the number of pixels changed
such that the resulting image would be misclassified by both
humans and the . The average 0 distance of
images in  from their counterparts in  is 4.48%
(2.45% standard deviation). Examples of the images in the
three conditions are shown in fig:l0_samples.
We note that because creating the images for  involves
time-consuming manual effort, we limited each condition to at
most 75 images.

A total of 201 participants were assigned to the 0 study:
73 were assigned to , 59 to , and 69 to .





Experiment results.
We computed the users' accuracy
(i.e., how often their responses agreed with the ground truth), and the
accuracy when classifying each image according to the majority vote
over all labels provided by the users. fig:l0_res shows the
results. We found that users' average accuracy was high for the
unaltered images of  (95%) and adversarial images of
 (97%), but low for the images of  (3%). In fact,
if we classify images via majority votes, none of the images of 
would be classified correctly. The difference between users' average
accuracy in  and  is not statistically significant
according to a t-test (p=0.34). In contrast, the difference between
users' average accuracy in  and other conditions is
statistically significant (p<0.01).  Users in all the conditions were
confident in their responses—the average confidence levels ranged
from 4.19 () to 4.47 ().

The results support our hypotheses. While it is possible
to find adversarial examples with small 0 distance from benign
samples, it is possible, for the same distance, to find samples that are
not imperceptible to humans. In fact, humans may be highly
confident that those samples belong to other classes.




 §.§ Insufficiency of 2


Study Details.
We used the CIFAR10 dataset and  to test 2 for
insufficiency. We randomly picked 100 images from the test set
that were correctly classified by the  for .
For each image in , we created (what we hoped would
be) an imperceptible adversarial example for . Images
in  have a fixed 2 distance of 6 from their
counterparts in . Because we did not find evidence in
the literature for an upper bound on 2 distance
that is is still imperceptible to humans, we chose a distance of 6
empirically—our results (below) support this choice. To create the
adversarial examples, we used an iterative gradient descent approach,
in the vein of prior work <cit.>, albeit
with two notable differences. First, we used an algorithm by
Wang et al. <cit.> to initialize the attack to an
image that has high SSIM to the benign image, but lies at a fixed
2 distance from it. The rationale
behind this was to increase the perceptual similarity
between the adversarial image and the benign image.
Second, we ensured that the 2-norm of the perturbation
is fixed by normalizing it to 6 after each iteration of the attack.
The images of  we generated via a similar approach
to those of . The only difference is that we initialized
the attack with an image that has low SSIM with respect to its
counterpart benign image using Wang et al.'s algorithm.
fig:l2_samples shows a sample of the images
that we used in the 2 study.

In total, we had 99 participants assigned to the 2 study:
25 were assigned to , 38 to , and 36 to .

Experiment results.
Users' average accuracy and the accuracy of the majority
vote are shown in fig:l2_res. On the benign images
of , users had an average accuracy of 93%. Their
average accuracy on the images of  was 89%, not
significantly lower (p≈ 1). Moreover, we found that 
users in  and  were almost equally confident
about their choices (averages of 4.31 and 4.37).
We thus concluded that 6 is a reasonable bound for 2
attacks. In stark contrast, we found that 
users' average accuracy dropped to 57% and confidence to
2.97 (p<0.01 for both). In
other words, users' likelihood to make mistakes increased
by 36%, on average, and their confidence in their decisions
decreased remarkably.


The results support our hypotheses, as a significant fraction of
attack samples with bounded 2 can be perceptually
different than their corresponding benign samples.





 §.§ Insufficiency of ∞


Study Details.
Similarly to the 2 study, we used the CIFAR10
dataset and  also for the ∞
study. We again picked 100 random images from the test set for
. For , we generated adversarial examples
with ∞ distance of 0.1 from benign examples, as
done by Goodfellow et al. <cit.>. We generated
the adversarial examples using the Projected Gradient Descent
algorithm <cit.>, with a simple tweak to
enhance imperceptibility: after each gradient descent
iteration, we increased SSIM with respect to benign images
using Wang et al.'s algorithm <cit.>. 
Examples in  were generated using a similar
algorithm, but we decreased the SSIM with respect to benign
images instead of increasing it.

In total, we had 99 participants assigned to the study:
36 were assigned to , 31 to , and 32 to
.

Experiment results.
The accuracy results are summarized in fig:linf_res.
On benign examples, users' average accuracy was 91%.
Their average confidence score was 4.45. The attacks in
 were not completely imperceptible:
users' average accuracy decreased to 79% and confidence score
to 3.98 (both significant with p<0.01).
However, attacks in  were significantly less similar
to benign images: users' average accuracy and confidence
score were 36% and 3.04 (p<0.01).

Our hypotheses hold also for ∞—a significant
fraction of attacks with relatively small ∞ can be
perceptually different to humans than benign images.



§ DISCUSSION

























The results of the user studies confirm our hypotheses—defining
similarity using 0, 2, and ∞-norms
can be insufficient for ensuring perceptual similarity in some cases.
Here, we discuss some of the limitations of this work, and discuss
some alternatives for p-norms.




 §.§ Limitations


A couple of limitations should be considered when interpreting our results.
First, we demonstrated our results on two s and for two datasets,
and so they may not apply for every  and image-recognition task.
The s that we considered are among the most resilient models
to adversarial examples to date. Consequently, we believe that
the attacks we generated against them for  and 
would succeed against other s. 

Depending on the chosen thresholds,
our results may or may not directly apply to specific combinations
of norms and
image-recognition tasks that we did not consider in this work.
While studying more combinations may be useful, we believe that our
findings are impactful in their current form, as the combinations
of norms, thresholds, and datasets we considered are commonly used in practice
(e.g., <cit.>).
We note that it may be possible to achieve sufficient conditions
for perceptual similarity by using lower thresholds than in our experiments.
Stated differently, it may be impossible to fool humans using lower thresholds.
However, decreasing thresholds may also prevent algorithms for crafting attacks
from finding successful adversarial examples (namely, the ones that are 
part of  in the experimental conditions).

Second, we estimated similarity using a proxy: whether participants'
categorizations of perturbed images were consistent with their categorizations
of their benign counterparts. However, similarity has different facets that 
may or may not be interesting, depending on the threat model being
studied. For instance, in some cases we may want to estimate whether
certain attacks are inconspicuous or not (e.g., to learn whether TSA
agents would detect disguised individuals attempting to mislead
surveillance systems). In such cases, we want to measure whether
adversarial images are “close enough” to benign images to the extent
that a human observer cannot reliably tell between adversarial and
non-adversarial images. We believe that future work should develop
a better understanding of this and other notions of similarity and how
to assess them as a means to help us improve current attacks and
defenses.




 §.§ Alternatives to p-norms


We next list several alternative distance metrics for assessing
  similarity and provide a preliminary assessment of their suitability
  as a replacement for p-norms.

Our results show that by using the same threshold for all samples,
one may generate adversarial images that mislead humans—thus,
they are not imperceptible. As a solution, one may consider
setting a different threshold for every sample to ensure that the
attacks' output would be imperceptible. In this case, a
principled automated method would be needed to set the
sample-dependent thresholds in order to create attacks at scale.
Without such a method, human feedback may be needed in the
process for every sample.



Other similarity-assessment measures, such as SSIM <cit.>
and the “minimal” transform needed to align two images, that have been
used in prior work (e.g., <cit.>) may be considered
to replace p-norms. Additionally, one may consider image-similarity
metrics that have not been previously used in the adversarial examples
literature, such as Perceptual Image Diff <cit.> and the
Universal Quality Index <cit.>.  Such metrics should
be treated with care, as they may lead to unnecessary and insufficient conditions for
perceptual similarity. For example, SSIM is sensitive to small geometric transformation
(e.g., the SSIM between the images in fig:horse is 0.36 out of 1, which is
relatively low <cit.>). So, using SSIM to define similarity
may lead to unnecessary conditions. Moreover, as demonstrated in
fig:bad_ssim, SSIM may be high even when two images are not similar.
Thus, SSIM may lead to insufficient conditions for similarity.

The recent work of Jang et al. suggests three metrics
to evaluate the similarity between adversarial examples
and benign inputs <cit.>. The metrics evaluate
adversarial perturbations' quality by how much they corrupt the Fourier
transform, their effect on edges, or their effect on the images' gradients.
This work appears to take a step in the right direction. However, further validation
of the proposed metrics is needed. For instance, we speculate
that slight geometric transformations to a benign image might affect the
gradients in the image dramatically. Thus, metrics based on gradients
alone may be unnecessary for defining conditions for perceptual similarity.

Lastly, one may consider using several metrics simultaneously to define
similarity (e.g., by allowing geometric transformations and perturbations
with small p-norm to craft adversarial
examples <cit.>). While this may be
a promising direction, metrics should be combined with
special care. As the conjunction of one or more unnecessary conditions leads
to an unnecessary condition, and the disjunction of one or more insufficient
conditions leads to an insufficient condition, simply conjoining or disjoining
metrics may not solve the (in)sufficiency and (un)necessity problems
of prior definitions of similarity.

Finding better measures for assessing perceptual similarity
remains an open problem. Better similarity measures could help improve both 
algorithms for finding adversarial examples and, more importantly,
defenses against them. In the absence of
such measures, we recommend that future research rely not
only on known metrics for perceptual similarity assessment, but also
on human-subject studies.



§ CONCLUSION



In this work, we aimed to develop a better understanding of
the suitability of p-norms for measuring the similarity
between adversarial examples and benign images. Specifically,
we complemented our knowledge that conditions on p
distances used for defining similarity are unnecessary in some
cases—i.e., they may not capture all imperceptible
attacks—and showed that they can also be insufficient—i.e.,
they may lead one to conclude that an adversarial instance is
similar to a benign instance when it is not so. Hence,
p-distance metrics may be unsuitable for assessing similarity
when crafting adversarial examples and defending against them.
We pointed out possible alternatives for p-norms to
assess similarity, though they seem to have limitations, too.
Thus, there is a need for further research to improve the
assessment of similarity when developing attacks and
defenses for adversarial examples.




§ ACKNOWLEDGMENTS


This work was supported in part by MURI grant W911NF-17-1-0370, by the
National Security Agency, by a gift from NVIDIA, and gifts from NATO
and Lockheed Martin through Carnegie Mellon CyLab.

ieee



