Robust Physical-World Attacks on Deep Learning Models
Kevin Eykholt Ivan Evtimov Earlence Fernandes Bo Li Amir Rahmati Chaowei Xiao Atul Prakash Tadayoshi Kohno Dawn Song
cs.CR cs.LG






















[
    Jorge Drumond Silva
    December 30, 2023
=======================

fancy





Recent studies show that the state-of-the-art deep neural networks (DNNs) are vulnerable to adversarial examples, resulting from small-magnitude perturbations added to the input. Given that that emerging physical systems are using DNNs in safety-critical situations, adversarial examples could mislead these systems and cause dangerous situations. Therefore, understanding adversarial examples in the physical world is an important step towards developing resilient learning algorithms. We propose a general attack algorithm, (), to generate robust visual adversarial perturbations under different physical conditions. Using the real-world case of road sign classification, we show that adversarial examples generated using achieve high targeted misclassification rates against standard-architecture road sign classifiers in the physical world under various environmental conditions, including viewpoints. Due to the current lack of a standardized testing method, we propose a two-stage evaluation methodology for robust physical adversarial examples consisting of lab and field tests. Using this methodology, we evaluate the efficacy of physical adversarial manipulations on real objects. With a perturbation in the form of only black and white stickers, we attack a real stop sign, causing targeted misclassification in of the images obtained in lab settings, and in of the captured video frames obtained on a moving vehicle (field test) for the target classifier. 

















§ INTRODUCTION

Deep Neural Networks (DNNs) have achieved state-of-the-art, and sometimes human-competitive, performance on many computer vision tasks <cit.>. Based on these successes, they are increasingly being used as part of control pipelines in physical systems such as cars <cit.>, UAVs <cit.>, and robots <cit.>.

Recent work, however, has demonstrated that DNNs are vulnerable to adversarial perturbations <cit.>. These carefully crafted modifications to the (visual) input of DNNs can cause the systems they control to misbehave in unexpected and potentially dangerous ways.  



This threat has gained recent attention, and work in computer vision has made great progress in understanding the space of adversarial examples, beginning in the digital domain (by modifying images corresponding to a scene) <cit.>, and more recently in the physical domain <cit.>. Along similar lines, our work contributes to the understanding of adversarial examples when perturbations are physically added to the objects themselves. We choose road sign classification as our target domain for several reasons: (1) The relative visual simplicity of road signs make it challenging to hide perturbations. (2) Road signs exist in a noisy unconstrained environment with changing physical conditions such as the distance and angle of the viewing camera, implying that physical adversarial perturbations should be robust against considerable environmental instability. (3) Road signs play an important role in transportation safety. (4) A reasonable threat model for transportation is that an attacker might not have control over a vehicle's systems, but is able to modify the objects in the physical world that a vehicle might depend on to make crucial safety decisions.








The main challenge with generating robust physical perturbations is environmental variability. Cyber-physical systems operate in noisy physical environments that can destroy perturbations created using current digital-only algorithms <cit.>. For our chosen application area, the most dynamic environmental change is the distance and angle of the viewing camera. Additionally, other practicality challenges exist: (1) Perturbations in the digital world can be so small in magnitude that it is likely that a camera will not be able to perceive them due to sensor imperfections. (2) Current algorithms produce perturbations that occupy the background imagery of an object. It is extremely difficult to create a robust attack with background modifications because a real object can have varying backgrounds depending on the viewpoint. (3) The fabrication process (e.g., printing of perturbations) is imperfect.












Informed by the challenges above, we design (), which can generate perturbations robust to widely changing distances and angles of the viewing camera. creates a visible, but inconspicuous perturbation that only perturbs the object (a road sign) and not the object's environment. To create robust perturbations, the algorithm draws samples from a distribution that models physical dynamics (varying distances and angles) using experimental data and synthetic transformations (Figure <ref>). 



Using the proposed algorithm, we evaluate the effectiveness of perturbations on physical objects, and show that adversaries can physically modify objects using low-cost techniques to reliably cause classification errors in DNN-based classifiers under widely varying distances and angles. For example, our attacks cause a classifier to interpret a subtly-modified physical Stop sign as a Speed Limit 45 sign. Specifically, our final form of perturbation is a set of black and white stickers that an adversary can attach to a physical road sign (Stop sign). We designed our perturbations to resemble graffiti, a relatively common form of vandalism. It is common to see road signs with random graffiti or color alterations in the real world as shown in Figure <ref> (the left image is of a real sign in a city). If these random patterns were adversarial perturbations (right side of Figure <ref> shows our example perturbation), they could lead to severe consequences for autonomous driving systems, without arousing suspicion in human operators.










Given the lack of a standardized method for evaluating physical attacks, we draw on standard techniques from the physical sciences and propose a two-stage experiment design: (1) A lab test where the viewing camera is kept at various distance/angle configurations; and (2) A field test where we drive a car towards an intersection in uncontrolled conditions to simulate an autonomous vehicle. We test our attack algorithm using this evaluation pipeline and find that the perturbations are robust to a variety of distances and angles.



















Our Contributions. Figure <ref> shows an overview of our pipeline to generate and evaluate robust physical adversarial perturbations.

    
  * We introduce () to generate physical perturbations for physical-world objects that can consistently cause misclassification in a DNN-based classifier under a range of dynamic physical conditions, including different viewpoint angles and distances (Section <ref>).
    
    
  * Given the lack of a standardized methodology in evaluating physical adversarial perturbations, we propose an evaluation methodology to study the effectiveness of physical perturbations in real world scenarios
    
    (Section <ref>). 
    
    
  * We evaluate our attacks against two standard-architecture classifiers that we built: with 91% accuracy on the LISA test set and with 95.7% accuracy on the GTSRB test set. Using two types of attacks (object-constrained poster and sticker attacks) that we introduce, we show that produces robust perturbations for real road signs. For example, poster attacks are successful in 100% of stationary and drive-by tests against , and sticker attacks are successful in  of stationary testing conditions and in of the extracted video frames against .
    
    
  * To show the generality of our approach, we generate the robust physical adversarial example by manipulating general physical objects, such as a microwave. We show that the pre-trained Inception-v3 classifier misclassifies the microwave as “phone" by adding a single sticker.
    




    
  * We design (), the first algorithm that generates robust physical adversarial examples for real objects, which can fool DNN based classifiers in different viewpoints. 
    
    
  * We apply to generate physical adversarial examples for road sign recognition systems. We achieve high attack success rate against varying real-world control variables, including distances, angles, and resolutions, in both lab and drive-by settings.
    
  * Focusing on the DNN application domain of road sign recognition, we introduce two attack classes on different physical road signs:

         Poster-Printing, where an attacker prints an actual-sized road sign with adversarial perturbations and then overlays it over an existing sign.
         
        Sticker Perturbation, where an attacker prints perturbations on paper, and then sticks them to an existing sign. 
    For these attacks, we physically realize two types of perturbations: (1) perturbations that occupy the entire region of the sign, and (2) perturbations that take the form of graffiti and abstract art. These attacks do not require special resources—only access to a color printer.

    
  * Given the lack of a standardized methodology in evaluating physical adversarial perturbation, we propose an evaluation methodology to study the effectiveness of physical perturbations in real world scenarios. For the road sign recognition system, the methodology consists of two stages: a stationary test, and a drive-by test using a vehicle. The tests aim to capture dynamic real-world conditions which an autonomous vehicle might experience (Section <ref>). 
    
    
  * We provide a thorough evaluation of our physical adversarial examples against and using the proposed methodology. We find that: how about we remove this part with the details below to make the contribution more clear?
    
    

    
        
         In a stationary test: a poster attack causes a Stop sign to be misclassified as a Speed Limit 45 sign in of the testing conditions (15 out of 15 images); a graffiti attack and an abstract art attack cause a Stop sign to be misclassified as a Speed Limit 45 sign in (10 out of 15) and (15 out of 15) of the test cases, respectively.
    
        
        In a drive-by test: a poster attack causes a Stop sign to be misclassified as a Speed Limit 45 sign in of the extracted video frames (37 out of 37 frames); a abstract art attack has the same misclassification effect in of the extracted video frames (28 out of 33 frames; we sampled every 10 frames in both cases).
        
        
         In a stationary test: a abstract art attack causes a Stop sign to be misclassified as a Speed Limit 80 sign in of all test cases (12 out of 15 images).
        
        
         In a drive-by test: a abstract art attack causes a Stop sign to be misclassified as a Speed Limit 80 sign in of the extracted video frames (28 out of 32 frames; we sampled every 10 frames).
        
 
  * add the new exp against robust models trained on ImageNet here
    




Our work, thus, contributes to understanding the susceptibility of image classifiers to robust adversarial modifications of physical objects.
These results provide a case for the potential consequences of adversarial examples on deep learning models that interact with the physical world through vision. Our overarching goal with this work is to inform research in building robust vision models and to raise awareness on the risks that future physical learning systems might face. 
We include more examples and videos of the drive-by tests on our webpage 



§ RELATED WORK


We survey the related work in generating adversarial examples. Specifically, 
given a classifier f_θ(·) with parameters θ and an input x with ground truth label y for x, an adversarial example x' is generated so that it is close to x in terms of certain distance, such as L_p norm distance. x' will also cause the classifier to make an incorrect prediction as f_θ(x')  y (untargeted attacks), or f_θ(x') = y^* (targeted attacks) for a specific y^* ≠ y. We also discuss recent efforts at understanding the space of physical adversarial examples.

Digital Adversarial Examples. 
Different methods have been proposed to generate adversarial examples in the white-box setting, where the adversary has full access to the classifier <cit.>. We focus on the white-box setting as well for two reasons: (1) In our chosen autonomous vehicle domain, an attacker can obtain a close approximation of the model by reverse engineering the vehicle's systems using model extraction attacks <cit.>. (2) To develop a foundation for future defenses, we must assess the abilities of powerful adversaries, and this can be done in a white-box setting. Given that recent work has examined the black-box transferability of digital adversarial examples <cit.>, physical black-box attacks may also be possible.






Goodfellow proposed the fast gradient method that applies a first-order approximation of the loss function to construct adversarial samples <cit.>. 
Optimization based methods have also been proposed to create adversarial perturbations for targeted attacks <cit.>. These methods contribute to understanding digital adversarial examples. By contrast, our work examines physical perturbations on real objects under varying environmental conditions. 












Physical Adversarial Examples.
Kurakin showed that printed adversarial examples can be misclassified when viewed through a smartphone camera <cit.>. 

Athalye and Sutskever improved upon the work of Kurakin and presented an attack algorithm that produces adversarial examples robust to a set of two-dimensional synthetic transformations <cit.>. These works do not modify physical objects—an adversary prints out a digitally-perturbed image on paper. However, there is value in studying the effectiveness of such attacks when subject to environmental variability.  Our object-constrained poster printing attack is a reproduced version of this type of attack, with the additional physical-world constraint of confining perturbations to the surface area of the sign. Additionally, our work goes further and examines how to effectively create adversarial examples where the object itself is physically perturbed by placing stickers on it. 



Concurrent to our work,[This work appeared at arXiv on 30 Oct 2017.] Athalye improved upon their original attack, and created 3D-printed replicas of perturbed objects <cit.>. The main intellectual differences include: (1) Athalye only use a set of synthetic transformations during optimization, which can miss subtle physical effects, while our work samples from a distribution modeling both physical and synthetic transformations. (2) Our work modifies existing true-sized objects. Athalye 3D-print small-scale replicas. (3) Our work simulates realistic testing conditions appropriate to the use-case at hand. 


Sharif attacked face recognition systems by printing adversarial perturbations on the frames of eyeglasses <cit.>. Their work demonstrated successful physical attacks in relatively stable physical conditions with little variation in pose, distance/angle from the camera, and lighting. This contributes an interesting understanding of physical examples in stable environments. However, environmental conditions can vary widely in general and can contribute to reducing the effectiveness of perturbations. Therefore, we choose the inherently unconstrained environment of road-sign classification. In our work, we explicitly design our perturbations to be effective in the presence of diverse physical-world conditions (specifically, large distances/angles and resolution changes).




Finally, Lu performed experiments with physical adversarial examples of road sign images against detectors and show current detectors cannot be attacked <cit.>. In this work, we focus on classifiers to demonstrate the physical attack effectiveness and to highlight their security vulnerability in the real world. Attacking detectors are out of the scope of this paper, though recent work has generated digital adversarial examples against detection/segmentation algorithms <cit.>, and our recent work has extended to attack the YOLO detector <cit.>.








§ PROBLEM STATEMENT


Our goal is to examine whether it is possible to create robust physical perturbations of real-world objects that trick deep learning classifiers into producing incorrect class labels even when images are taken under different physical conditions, and even at extreme angles and distances. In this work, we focus on deep neural networks applied to road sign recognition because of the critical role of these objects in road safety and security.





 §.§ U.S. Road Sign Classification



 To the best of our knowledge, there is current currently no publicly available road-sign classifier for U.S. road signs.  Therefore, we use the LISA dataset <cit.> of U.S. traffic signs containing 47 different road signs to train a DNN-based classifier. This dataset does not contain equal numbers of images for each sign. In order to balance our training data, we chose 17 common signs with the most number of training examples. Furthermore, since some of the signs dominate the dataset due to their prevalence in the real world (Stop and Speed Limit 35), we limit the maximum number of examples used for training to 500 per sign. Our final dataset includes commonly used signs such as Stop, Speed Limits, Yield, and Turn Warnings. Finally, the original LISA dataset contains image resolutions ranging from 6 × 6 to 167 × 168 pixels. We resized all images to 32 × 32 pixels, a common input size for other well-known image datasets such as CIFAR10 <cit.>. Table <ref> summarizes our final training and testing datasets.[speedLimitUrdbl stands for unreadable speed limit. This means the sign had an additional sign attached and the annotator could not read due to low image quality]




We set up and trained our road sign classifier in TensorFlow using this refined dataset. The network we used was originally defined in the Cleverhans library <cit.> and consists of three convolutional layers followed by a fully connected layer.
Our final classifier accuracy is 91% on the test dataset.
For the rest of the paper, we refer to this classifier as for simplicity.

















 §.§ Improving the Classifier

In order to test the versitility of , we also run our attack against a classifier trained with a larger road sign dataset, the German Traffic Sign Recognition Benchmark (GTSRB) <cit.>. For this purpose, we use a publicly available implementation <cit.> of a multi-scale CNN architecture that has been known to perform well on road sign recognition <cit.>. 
Our goal is to guarantee that our attack is effective across different training datasets and network architectures.
Because we did not have access to German Stop signs for our experiments, we replace the German Stop signs in GTSRB with the entire set of U.S. Stop sign images in LISA. After training, our classifier achieves  95.7% accuracy on the test set, which also had the German Stop signs replaced with U.S. Stop signs. This test set consists of all GTSRB test images except the German Stop sign images. In addition, we include our own images of 181 US Stop signs. None of the images in the test set are present in either the training set or the validation set of the network. We will release the set of 181 real-world US Stop sign images we took after publication. or the rest of the paper, we refer to this classifier as .


























 §.§ Threat Model


In contrast to prior work, we seek to physically modify an existing road sign in a way that causes a road sign classifier to output a misclassification while keeping those modifications inconspicuous to human observers.
Here, we focus on evasion attacks where attackers can only modify the testing data and do not have access to the training data (as they would in poisoning attacks). 


We assume that attackers do not have digital access to the computer systems running the classifiers. If attackers have this superior level of access, then there is no need for adversarial perturbations—they can simply feed malicious input data directly into the model to mislead the system as they want, or they can compromise other control software, completely bypassing all classifiers.  


Following Kerckhoffs' principle <cit.>, it is often desirable to construct defenses that are robust in the presence of white-box attackers. As one of the broader goals of our work is to inform future defense research, we assume a strong attacker with white-box access, an attacker gains access to the classifier after it has been trained <cit.>. Therefore, although the attacker can only change existing physical road signs, they have full knowledge of the classifier and its architecture. Finally, due to the recent discovery of transferability <cit.>, black-box attacks can be carried out using perturbations computed with white-box access on a different model. As our goal is to inform future defenses, we will focus on white-box attacks in this paper.





Specific to the domain of autonomous vehicles, future vehicles might not face this threat as they might not need to depend on road signs. There could be databases containing the location of each sign. Moreover, with complete autonomy, vehicles might be able to manage navigating complex traffic flow regions by means of vehicle-to-vehicle communication alone. However, these are not perfect solutions and some have yet to be developed. An autonomous vehicle might not be able to rely solely on a database of road sign locations. Such records might not always be kept up-to-date and there might be unexpected traffic events such as construction work and detours due to accidents that necessitate traffic regulation with temporary road signs. We demonstrate that physically modifying real-world objects to fool classifiers is a real threat, but we consider the rest of the control pipeline of autonomous vehicles to be outside the scope of our work.



Attack Generation Pipeline. Based on our threat model, the attack pipeline proceeds as follows: 



    
  * Obtain several clean image of the target road sign without any adversarial perturbation under different conditions, including various distances and angles.
    
  * Use those images after appropriate pre-processing as input to and generate adversarial examples.  
    
  * Reproduce the resulting perturbation physically by printing out either the entire modified image in the case of poster-printing attacks or just the relevant modified regions in the case of sticker attacks.
    
  * Apply the physically reproduced perturbation to the targeted physical road signs.






§ ADVERSARIAL EXAMPLES FOR PHYSICAL OBJECTS


Our goal is to examine whether it is possible to create robust physical perturbations for real-world objects that mislead classifiers to make incorrect predictions even when images are taken in a range of varying physical conditions. We first present an analysis of environmental conditions that physical learning systems might encounter, and then present our algorithm to generate physical adversarial perturbations taking these challenges into account.



 §.§ Physical World Challenges


Physical attacks on an object must be able to survive changing conditions and remain effective at fooling the classifier. We structure our discussion of these conditions  
around the chosen example of road sign classification, which could be potentially applied in autonomous vehicles and other safety sensitive domains. A subset of these conditions can also be applied to other types of physical learning systems such as drones, and robots. 




    Environmental Conditions. The distance and angle of a camera in an autonomous vehicle with respect to a road sign varies continuously. The resulting images that are fed into a classifier are taken at different distances and angles. Therefore, any perturbation that an attacker physically adds to a road sign must be able to survive these transformations of the image. Other environmental factors include changes in lighting/weather conditions, and the presence of debris on the camera or on the road sign. 
    
    
    
    Spatial Constraints. Current algorithms focusing on digital images add adversarial perturbations to all parts of the image, including background imagery. However, for a physical road sign, the attacker cannot manipulate background imagery. Furthermore, the attacker cannot count on there being a fixed background imagery as it will change depending on the distance and angle of the viewing camera.
    
    Physical Limits on Imperceptibility. An attractive feature of current adversarial deep learning algorithms is that their perturbations to a digital image are often so small in magnitude that they are almost imperceptible to the casual observer. However, when transferring such minute perturbations to the real world, we must ensure that a camera is able to perceive the perturbations. Therefore, there are physical limits on how imperceptible perturbations can be, and is dependent on the sensing hardware.
    
    Fabrication Error. To fabricate the computed perturbation, all perturbation values must be valid colors that can be reproduced in the real world. Furthermore, even if a fabrication device, such as a printer, can produce certain colors, there will be some reproduction error <cit.>.
    
    


In order to successfully physically attack deep learning classifiers, an attacker should account for the above categories of physical world variations that can reduce the effectiveness of perturbations.









 §.§ Robust Physical Perturbation
 
We derive our algorithm starting with the optimization method that generates a perturbation for a single image x, without considering other physical conditions; then, we describe how to update the algorithm taking the physical challenges above into account.


This single-image optimization problem searches for perturbation δ to be added to the input x, such that the perturbed instance x' = x + δ is misclassified by the target classifier f_θ (·):

    min     H(x+δ, x),
      s.t.   f_θ(x + δ) = y^*

where H is a chosen distance function, and y^* is the target class.[For untargeted attacks, we can modify the objective function to maximize the distance between the model prediction and the true class. We focus on targeted attacks in the rest of the paper.] To solve the above constrained optimization problem efficiently, we reformulate it in the Lagrangian-relaxed form similar to prior work <cit.>.

    δargmin λ ||δ||_p + J(f_θ(x + δ), y^*)






Here J(· , ·) is the loss function, which measures the difference between the model's prediction and the target label y^*. λ is a hyper-parameter that controls the regularization of the distortion.
We specify the distance function H as ||δ||_p, denoting the ℓ_p norm of δ.


Next, we will discuss how the objective function can be modified to account for the environmental conditions.
We model the distribution of images containing object o under both physical and digital transformations X^V. We
sample different instances x_i drawn from X^V.
A physical perturbation can only be added to a specific object o within x_i.

In the example of road sign classification, o is the stop sign that we target to manipulate. 

Given images taken in the physical world, we need to make sure that a single perturbation δ, which is added to o, can fool the classifier under different physical conditions.
Concurrent work <cit.> only applies a set of transformation functions to synthetically sample such a distribution. However, modeling physical phenomena is complex and such synthetic transformations may miss physical effects. Therefore, to better capture the effects of changing physical conditions, we sample instance x_i from X^V by both generating experimental data that contains actual physical condition variability as well as synthetic transformations. For road sign physical conditions, this involves taking images of road signs under various conditions, such as changing distances, angles, and lightning. This approach aims to approximate physical world dynamics more closely.
For synthetic variations, we randomly crop the object within the image, change the brightness, and add spatial transformations to simulate other possible conditions.










To ensure that the perturbations are only applied to the surface area of the target object o (considering the spatial constraints and physical limits on imperceptibility), we introduce a mask. This mask serves to project the computed perturbations to a physical region on the surface of the object (road sign). In addition to providing spatial locality, the mask also helps generate perturbations that are visible but inconspicuous to human observers. To do this, an attacker can shape the mask to look like graffiti—commonplace vandalism on the street that most humans expect and ignore, therefore hiding the perturbations “in the human psyche.” Formally, the perturbation mask is a matrix M_x whose dimensions are the same as the size of input to the road sign classifier. M_x contains zeroes in regions where no perturbation is added, and ones in regions where the perturbation is added during optimization.

In the course of our experiments, we empirically observed that the position of the mask has an impact on the effectiveness of an attack. We therefore hypothesize that objects have strong and weak physical features from a classification perspective, and we position masks to attack the weak areas. Specifically, we use the following pipeline to discover mask positions: (1) Compute perturbations using the L_1 regularization and with a mask that occupies the entire surface area of the sign. L_1 makes the optimizer favor a sparse perturbation vector, therefore concentrating the perturbations on regions that are most vulnerable. Visualizing the resulting perturbation provides guidance on mask placement. (2) Recompute perturbations using L_2 with a mask positioned on the vulnerable regions identified from the earlier step.





 


To account for fabrication error, we add an additional term to our objective function that models printer color reproduction errors. This term is based upon the Non-Printability Score (NPS) by Sharif  <cit.>. Given a set of printable colors (RGB triples) P and a set R(δ) of (unique) RGB triples used in the perturbation that need to be printed out in physical world, the non-printability score is given by:

    𝑁𝑃𝑆 = ∑_p̂∈ R(δ)∏_p' ∈ P |p̂ - p'|




Based on the above discussion, our final robust spatially-constrained perturbation is thus optimized as:


    δargmin λ ||M_x·δ||_p + 𝑁𝑃𝑆
     +  𝔼_x_i ∼ X^V J(f_θ(x_i + T_i( M_x·δ)), y^*)

Here we use function T_i(·) to denote the alignment function that maps transformations on the object to transformations on the perturbation (if the object is rotated, the perturbation is rotated as well). 


Finally, an attacker will print out the optimization result on paper, cut out the perturbation (M_x), and put it onto the target object o. As our experiments demonstrate in the next section, this kind of perturbation fools the classifier in a variety of viewpoints.[For our attacks, we use the ADAM optimizer with the following parameters: β_1=0.9, β_2=0.999, ϵ=10^-8, η∈ [10^-4, 10^0]]



















§ EXPERIMENTS


In this section, we will empirically evaluate the proposed . We first evaluate a safety sensitive example, Stop sign recognition, to demonstrate the robustness of the proposed physical perturbation. To demonstrate the generality of our approach, we then attack Inception-v3 to misclassify a microwave as a phone.



 §.§ Dataset and Classifiers


We built two classifiers based on a standard crop-resize-then-classify pipeline for road sign classification as described in <cit.>.

Our uses LISA, a U.S. traffic sign dataset containing 47 different road signs <cit.>. However, the dataset is not well-balanced, resulting is large disparities in representation for different signs. To alleviate this problem, we chose the 17 most common signs based on the number of training examples. 

's architecture is defined in the Cleverhans library <cit.> and consists of three convolutional layers and an FC layer. It has an accuracy of 91% on the test set.

Our second classifier is , that is trained on the German Traffic Sign Recognition Benchmark (GTSRB) <cit.>. We use a publicly available implementation <cit.> of a multi-scale CNN architecture that has been known to perform well on road sign recognition <cit.>. Because we did not have access to German Stop signs for our physical experiments, we replaced the German Stop signs in the training, validation, and test sets of GTSRB with the U.S. Stop sign images in LISA. achieves  95.7% accuracy on the test set. 

When evaluating on our own 181 stop sign images, it achieves 99.4% accuracy.


































 §.§ Experimental Design


To the best of our knowledge, there is currently no standardized methodology of evaluating physical adversarial perturbations. Based on our discussion from Section <ref>, we focus on angles and distances because they are the most rapidly changing elements for our use case. A camera in a vehicle approaching a sign will take a series of images at regular intervals. These images will be taken at different angles and distances, therefore changing the amount of detail present in any given image. Any successful physical perturbation must cause targeted misclassification in a range of distances and angles because a vehicle will likely perform voting on a set of frames (images) from a video before issuing a controller action.
Our current experiments do not explicitly control ambient light, and as is evident from experimental data (Section <ref>), lighting varied from indoor lighting to outdoor lighting. 




























Drawing on standard practice in the physical sciences, our experimental design encapsulates the above physical factors into a two-stage evaluation consisting of controlled lab tests and field tests.







Stationary (Lab) Tests. This involves classifying images of objects from stationary, fixed positions. 


    
  * Obtain a set of clean images C and a set of adversarially perturbed images ({c}, ∀ c∈ C)  at varying distances d ∈ D, and varying angles g ∈ G. We use c^d,g here to denote the image taken from distance d and angle g. The camera's vertical elevation should be kept approximately constant. 
    
    
    Changes in the camera angle relative the the sign will normally occur when the car is turning, changing lanes, or following a curved road.
    
    
    
    
  * Compute the attack success rate of the physical perturbation using the following formula:  
    
    1.5∑_c ∈ C1_{f_θ(c^d,g) = y^*  ∧ f_θ(c^d,g) = y}/∑_c ∈ C1_{f_θ(c^d,g) = y}

    
    where d and g denote the camera distance and angle for the image, y is the ground truth, and y^* is the targeted attacking class.[For untargeted adversarial perturbations, change f_θ(e^d,g) = y^* to f_θ(e^d,g)  y.] 


Note that an image c that causes misclassification is considered as a successful attack only if the original image c with the same camera distance and angle is correctly classified, which ensures that the misclassification is caused by the added perturbation instead of other factors.

Drive-By (Field) Tests. We place a camera on a moving platform, and obtain data at realistic driving speeds. For our experiments, we use a smartphone camera mounted on a car.


    
  * Begin recording video at approximately 250 ft away from the sign. Our driving track was straight without curves. Drive toward the sign at normal driving speeds and stop recording once the vehicle passes the sign. In our experiments, our speed varied between 0 mph and 20 mph. This simulates a human driver approaching a sign in a large city. 
    
    
    
  * Perform video recording as above for a “clean” sign and for a sign with perturbations applied, and then apply similar formula as Eq. <ref> to calculate the attack success rate, where C here represents the sampled frames.
    
    
    
    
    
    
    





An autonomous vehicle will likely not run classification on every frame due to performance constraints, but rather, would classify every j-th frame, and then perform simple majority voting. Hence, an open question is to determine whether the choice of frame (j) affects attack accuracy. In our experiments, we use j = 10. We also tried j = 15 and did not observe any significant change in the attack success rates.
If both types of tests produce high success rates, the attack is likely to be successful in commonly experienced physical conditions for cars. 





















 §.§ Results for 


We evaluate the effectiveness of our algorithm by generating three types of adversarial examples on (91% accuracy on test-set). For all types, we observe high attack success rates with high confidence. Table <ref> summarizes a sampling of stationary attack images. In all testing conditions, our baseline of unperturbed road signs achieves a 100% classification rate into the true class.

Object-Constrained Poster-Printing Attacks. This involves reproducing the attack of Kurakin  <cit.>. The crucial difference is that in our attack, the perturbations are confined to the surface area of the sign excluding the background, and are robust against large angle and distance variations. The Stop sign is misclassified into the attack's target class of Speed Limit 45 in of the images taken according to our evaluation methodology. The average confidence of predicting the manipulated sign as the target class is 80.51% (second column of Table <ref>). 











 §.§ Poster-Printing Attacks


We first show that an attacker can overlay a true-sized poster-printed perturbed road sign over a real-world sign and achieve misclassification into a target class of her choosing. The attack has the following steps:

    
  * The attacker obtains a series of high resolution images of the sign under varying angles, distances, and lighting conditions. We use 34 such images in our experiments. None of these images were present in the datasets used to train and evaluate the baseline classifier.
    
  * The attacker then crops, rescales, and feeds the images into and uses equation (<ref>) as the objective function. She takes the generated perturbation, scales it up to the dimensions of the sign being attacked, and digitally applies it to an image of the sign.
    
  * The attacker then prints the sign (with the applied perturbation) on poster paper such that the resulting print's physical dimensions match that of a physical sign. In our attacks, we printed 30”× 30” Stop signs and 18”× 18” Right Turn signs.
    
  * The attacker cuts the printed sign to the shape of the physical sign (octagon or diamond), and overlays it on top of the original physical sign.

We use our methodology from Section <ref> to evaluate the effectiveness of such an attack. In order to control for the performance of the classifier on clean input, we also take images of a real-size printout of a non-perturbed image of the sign for each experiment. We observe that all such baseline images lead to correct classification in all experiments.

For the Stop sign, we choose a mask that exactly covers the area of the original sign in order to avoid background distraction. This choice results in a perturbation that is similar to that in existing work <cit.> and we hypothesize that it is imperceptible to the casual observer (see the second column of Table <ref> for an example). In contrast to some findings in prior work, this attack is very effective in the physical world. The Stop sign is misclassified into the attack's target class of Speed Limit 45 in of the images taken according to our evaluation methodology. The average confidence of the target class is 80.51% with a standard deviation of 10.67%.


For the Right Turn warning sign, we choose a mask that covers only the arrow since we intend to generate perturbations. In order to achieve this goal, we increase the regularization parameter λ in equation (<ref>) to demonstrate small magnitude perturbations. Table <ref> summarizes our attack results—we achieve a targeted-attack success rate (Table <ref>). Out of 15 distance/angle configurations, four instances were not classified into the target. However, they were still misclassified into other classes that were not the true label (Yield, Added Lane). Three of these four instances were an Added Lane sign—a different type of warning. We hypothesize that given the similar appearance of warning signs, small perturbations are sufficient to confuse the classifier.


















Sticker Attacks.
Next, we demonstrate how effective it is to generate physical perturbations in the form of stickers, by constraining the modifications to a region resembling graffiti or art. 

The fourth and fifth columns of Table <ref> show a sample of images, and Table <ref> (columns 4 and 6) shows detailed success rates with confidences. In the stationary setting, we achieve a targeted-attack success rate for the graffiti sticker attack and a targeted-attack success rate for the sticker art attack. Some region mismatches may lead to the lower performance of the LOVE-HATE graffiti.



The steps for this type of attack are:

    
  * The attacker generates the perturbations digitally by using just as in Section <ref>.
    
  * The attacker prints out the Stop sign in its original size on a poster printer and cuts out the regions that the perturbations occupy. 
    
  * The attacker applies the cutouts to the sign by using the remainder of the printed sign as a stencil. 




















Drive-By Testing.
Per our evaluation methodology, we conduct drive-by testing for the perturbation of a Stop sign. In our baseline test we record two consecutive videos of a clean Stop sign from a moving vehicle, perform frame grabs at k = 10, and crop the sign. We observe that the Stop sign is correctly classified in all frames. We similarly test subtle and abstract art perturbations for using k = 10. Our attack achieves a targeted-attack success rate of for the poster attack, and a targeted-attack success rate of for the abstract art attack. See Table <ref> for sample frames from the drive-by video.











 §.§ Results for 

To show the versatility of our attack algorithms, we create and test attacks for (95.7% accuracy on test-set). Based on our high success rates with the -art attacks, we create similar abstract art sticker perturbations. The last column of Table <ref> shows a subset of experimental images. Table <ref> summarizes our attack results—our attack fools the classifier into believing that a Stop sign is a Speed Limit 80 sign in of the stationary testing conditions. Per our evaluation methodology, we also conduct a drive-by test (k = 10, two consecutive video recordings). The attack fools the classifier of the time. 







 §.§ Results for Inception-v3

To demonstrate generality of , we computed physical perturbations for the standard Inception-v3 classifier <cit.> using two different objects, a microwave and a coffee mug. We chose a sticker attack since poster printing an entirely new surface for the objects may raise suspicions. Note that for both attacks, we have reduced the range of distances used due to the smaller size of the cup and microwave compared to a road sign (Coffee Mug height- 11.2cm, Microwave height- 24cm, Right Turn sign height- 45cm, Stop Sign- 76cm). Table <ref> summarizes our attack results on the microwave and Table <ref> summarizes our attack results on the coffee mug. For the microwave, the targeted attack success rate is 90%. For the coffee mug, the targeted attack success rate is 71.4% and the untargeted success rate is 100%. Example images of the adversarials stickers for the microwave and cup can be seen in Tables <ref> and <ref>.
















§ DISCUSSION



Black-Box Attacks. Given access to the target classifier's network architecture and model weights, can generate a variety of robust physical perturbations that fool the classifier. Through studying a white-box attack like , we can analyze the requirements for a successful attack using the strongest attacker model and better inform future defenses. Evaluating in a black-box setting is an open question.


Image Cropping and Attacking Detectors. When evaluating , we manually controlled the cropping of each image every time before classification. This was done so the adversarial images would match the clean sign images provided to . Later, we evaluated the camouflage art attack using a pseudo-random crop with the guarantee that at least most of the sign was in the image. Against LISA-CNN, we observed an average targeted attack rate of 70% and untargeted  attack rate of 90%. Against GTSRB-CNN, we observed an average targeted attack rate of 60% and untargeted attack rate of 100%. We include the untargeted attack success rates because causing the classifier to not output the correct traffic sign label is still a safety risk. Although image cropping has some effect on the targeted attack success rate, our recent work shows that an improved version of can successfully attack object detectors, where cropping is not needed <cit.>.





§ CONCLUSION

We introduced an algorithm () that generates robust, physically realizable adversarial perturbations. Using , and a two-stage experimental design consisting of lab and drive-by tests, we contribute to understanding the space of physical adversarial examples when the objects themselves are physically perturbed. We target road-sign classification because of its importance in safety, and the naturally noisy environment of road signs. Our work shows that it is possible to generate physical adversarial examples robust to widely varying distances/angles. This implies that future defenses should not rely on physical sources of noise as protection against physical adversarial examples.  

Acknowledgements.
We thank the reviewers for their insightful feedback. This work was supported in part by NSF grants 1422211, 1616575, 1646392, 1740897, 1565252, Berkeley Deep Drive, the Center for Long-Term Cybersecurity, FORCES (which receives support from the NSF), the Hewlett Foundation, the MacArthur Foundation, a UM-SJTU grant, and the UW Tech Policy Lab. 











ieee



