title: Adversarially Robust Learning via Entropic Regularization
authors: Gauri Jagatap Ameya Joshi Animesh Basak Chowdhury Siddharth Garg Chinmay Hegde
text: 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Adversarially Robust Learning via 
   Entropic Regularization
      
    Gauri Jagatap, Ameya Joshi, Animesh Basak Chowdhury, Siddharth Garg, and Chinmay HegdeThis work was supported in part by NSF grants CCF-2005804 and CCF-1815101.
  
    New York University
  
     
  
      December 30, 2023
  ====================================================================================================================================================================================================
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
    In this paper we propose a new family of algorithms, ATENT, for training adversarially robust deep neural networks. We formulate a new loss function that is equipped with an additional entropic regularization. Our loss function considers the contribution of adversarial samples that are drawn from a specially designed distribution in the data space that assigns high probability to points with high loss and in the immediate neighborhood of training samples. Our proposed algorithms optimize this loss to seek adversarially robust valleys of the loss landscape. Our approach achieves competitive (or better) performance in terms of robust classification accuracy as compared to several state-of-the-art robust learning approaches on benchmark datasets such as MNIST and CIFAR-10.
  
  
  
  § INTRODUCTION
   
  
  Deep neural networks have led to significant breakthroughs in the fields of computer vision <cit.>, natural language processing <cit.>, speech processing <cit.>, recommendation systems <cit.> and forensic imaging <cit.>. However, deep networks have also been shown to be very susceptible to carefully designed “attacks"  <cit.>. In particular, the outputs of networks trained via traditional approaches are rather brittle to maliciously crafted perturbations in both input data as well as network weights <cit.>. 
  
  Formally put, suppose the forward map between the inputs x and outputs y is modeled via a neural network as y = f(w;x) where w represents the set of trainable weight parameters. 
  For a classification task, 
  
  
  given a labeled dataset {x_i, y_i}, i=1,…,n where X and Y represents all training data pairs, the standard procedure for training neural networks is to seek the weight parameters w that minimize the empirical risk: 
  
      ŵ = min_w 1/n∑_i=1^n L(f(w;x_i), y_i) := (X;Y,w).
   
  However, the prediction ŷ(x) = f(ŵ;x) can be very sensitive to changes in both ŵ and x.  For example, if
  
  a bounded perturbation to a test image input (or to the neural network weights) is permitted, i.e. ŷ_i = f(ŵ;x_i+δ_i) where δ_i represents the perturbation, then the predicted label ŷ_̂î can be made arbitrarily different from the true label y_i.
  
  
  Several techniques for finding such adversarial perturbations have been put forth. 
  Typically, this can be achieved by maximizing the loss function within a neighborhood around the test point x <cit.>: 
  
  
  
  
  
  
  
      x̅_worst = max_δ∈Δ_p L(f(ŵ;x+δ), y)
  
  where ŵ are the final weights of a pre-trained network. The perturbation set Δ_p is typically chosen to be an ℓ_p-ball for some p ∈{0,1,2,∞}. 
  
  
  The existence of adversarial attacks motivates the need for a “defense” mechanism that makes the network under consideration more robust. Despite a wealth of proposed defense techniques, the jury is still out on how optimal defenses should be constructed <cit.>.  
  
  We discuss several families of effective defenses. The first involves adversarial training <cit.>. Here, a set of adversarial perturbations of the training data is constructed by solving a min-max objective of the form: 
  
      ŵ = min_w max_δ∈Δ_p1/n∑_i=1^n L(f(w;x_i+δ), y_i).
  
  
  <cit.> use a convex outer adversarial polytope as an upper bound for worst-case loss in robust training; here the network is trained by generating adversarial as well as few non-adversarial examples in the convex polytope of the attack via a linear program. Along the same vein include a mixed-integer programming based certified training for piece-wise linear neural networks <cit.> and integer bound propagation <cit.>. 
  
  
  
  
  The last family of approaches involves randomized smoothing. Here, both training the network as well as the inference made by the network are smoothed out over several stochastic perturbations of the target example <cit.>. This has the effect of optimizing a smoothed-adversarial version of the empirical risk. Randomized smoothing has also been used in combination with adversarial training <cit.> for improved adversarial robustness under ℓ_2 attacks[This family of methods has the additional benefit of being certifiably robust: all points within a ball of a given radius around the test point are provably  classified with the correct label.].
  
  
  
  
  In this paper, we propose a new approach for training adversarially robust neural networks. The key conceptual ingredient underlying our approach is entropic regularization. Borrowing intuition from <cit.>, instead of the empirical risk (or its adversarial counterpart), our algorithm instead optimizes over an local entropy-regularized version of the empirical risk:
  
      ŵ   = min_w  ℒ_DE, 
      ℒ_DE    = ∫_X'(X';Y,w)[e^((X';Y,w)-γ/2X-X'_p^p)/Z] dX' .
  
  Intuitively, this new loss function can be viewed as the convolution of the empirical risk with a Gibbs-like distribution to sample points from the neighborhoods, X', of the training data points X that have high loss. Therefore, compared to adversarial training, we have replaced the inner maximization with an expected value with respect to a modified Gibbs measure which is matched to the geometry of the perturbation set.
  
  Since the above loss function is difficult to optimize (or even evaluate exactly), we instead approximate it via Monte Carlo techniques. In particular, we use Stochastic Gradient Langevin Dynamics <cit.>; in this manner, our approach blends in elements from adversarial training, randomized smoothing, and entropic regularization.  We posit that the combination of these techniques will encourage a classifier to learn a better robust decision boundary as compared to prior art (see visualization in Fig.<ref>). 
  
  
  
  
  To summarize, our specific contributions are as follows:
  
  
      
    * We propose a new entropy-regularized loss function for training deep neural networks (Eq. <ref>) that is a robust version of the empirical risk.  
       
    * We propose a new Monte Carlo algorithm to optimize this new loss function that is based on Stochastic Gradient Langevin Dynamics. We call this approach Adversarial Training with ENTropy (ATENT). 
      
    * We show that ATENT-trained networks provide improved (robust) test accuracy when compared to existing defense approaches. 
      
    * We combine randomized smoothing with ATENT to show competitive performance with the smoothed version of TRADES.
  
  
  
  In particular, we are able to train an ℓ_∞-robust CIFAR-10 model to 57.23% accuracy at PGD attack level ϵ = 8/255, which is higher than the latest benchmark defenses based on both adversarial training using early stopping <cit.> (56.8%) as well as TRADES (56.6%) <cit.>. 
  
  
  § PRIOR WORK
   
  Evidence for the existence of adversarial inputs for deep neural networks is by now well established <cit.>. In image classification, the majority of attacks have focused on the setting where the adversary confounds the classifier by adding an imperceptible perturbation to a given input image. The range of the perturbation is pre-specified in terms of bounded pixel-space ℓ_p-norm balls. Specifically, an ℓ_p- attack model allows the adversary to search over the set of input perturbations Δ_p,ϵ = {δ : δ_p ≤ϵ} for p={0,1,2,∞}. 
  
  Initial attack methods, including the Fast Gradient Sign Method (FGSM) and its variants <cit.>, proposed techniques for generating adversarial examples by ascending along the 
  sign of the loss gradient:  
      x_adv = x + ϵ(∇_x L(f(ŵ;x), y)),
   where (x_adv-x) ∈Δ_∞,ϵ. Madry et. al. <cit.> proposed a stronger adversarial attack via projected gradient descent (PGD) by iterating FGSM several times, such that 
      x^t+1 = Π_x+Δ_p,ϵ(x^t +  α(∇_x L(f(ŵ;x), y)),
   where p={2,∞}. These attacks are (arguably) the most successful available attack techniques reported to date, and serve as the starting point for our comparisons.
  
  Both Deep Fool <cit.> and Carlini-Wagner <cit.> construct an attack by finding smallest possible perturbation that can flip the label of the network output. 
  
  Several strategies for defending against attacks have been developed. In <cit.>,  adversarial training is performed via the min-max formulation Eq. <ref>. The inner maximization is solved using PGD, while the outer objective is minimized using stochastic gradient descent (SGD) with respect to w. This can be slow to implement, and speed-ups have been proposed in <cit.> and <cit.>.
  
  In <cit.>, the authors developed certified defense strategies via randomized smoothing. This approach consists of two stages: the first stage consists of training with noisy samples, and the second stage produces an ensemble-based inference. 
  See <cit.> for a more thorough review of the literature on various attack and defense models.
  
  Apart from minimizing the worst case loss, approaches which  minimize the upper bound on worst case loss include <cit.>. Another breed of approaches use a modified loss function which considers surrogate adversarial loss as an added regularization, where the surrogate is cross entropy <cit.> (TRADES), maximum margin cross entropy <cit.> (MMA) and KL divergence <cit.> (MART) between adversarial sample predictions and natural sample predictions.  
  
  In a different line of work, there have been efforts towards building neural network networks with improved generalization properties. In particular, heuristic experiments by <cit.> suggest that the loss surface at the final learned weights for well-generalizing models is relatively “flat" [This is not strictly necessary, as demonstrated by good generalization at certain sharp minima <cit.>.]. Building on this intuition, Chaudhari et. al. <cit.> showed that by explicitly introducing a smoothing term (via entropic regularization) to the training objective, the learning procedure weights towards regions with flatter minima by design. Their approach, Entropy-SGD (or ESGD), is shown to induce better generalization properties in deep networks. We leverage this intuition, but develop a new algorithm for training deep networks with better adversarial robustness properties. 
  
  
  
  
  
  
  
  § PROBLEM FORMULATION
   
  
  The task of classification, given a training labelled dataset {x_i ∈𝒳, y_i }, i∈{1,…,n}, consists of solving the standard objective by optimizing weight parameters w, min_w1/n∑_i=1^n L(f(w;x_i), y_i) where y_i is a one-hot class encoding vector of length m and m is the total number of classes. The training data matrix itself is represented using shorthand X ∈^n × d and labels in Y ∈^n where we have access to n training samples which are d-dimensional each. Given this formulation, the primary task is to minimize the cross-entropy Loss function (w;X,Y)  = -1/n∑_i=1^n ∑_j=1^m y_i,jlogŷ_i,j. 
  
  In this paper, we design an augmented version of the loss function  which models a class of adversarial perturbations and also introduce a new procedure to minimize it.
  
  
  
  We first recap the Entropy SGD <cit.> (see also Appendix <ref> of the supplement). Entropy-SGD considers an augmented loss function of the form 
      _ent(w;X,Y) = - log∫_w' e^-(w';X,Y)-γ/2w-w'_2^2 dw'.
   
  
  By design, minimization of this augmented loss function promotes minima with wide valleys. Such a minimum would be robust to perturbations in w, but is not  necessarily advantageous against adversarial data samples x_adv. In our experiments (Section <ref>) we show that networks trained with Entropy-SGD perform only marginally better against adversarial attacks as compared to those trained with standard SGD.
  
  For the task of adversarial robustness, we instead develop a data-space version of Entropy-SGD. To model for perturbations in the samples, we design an augmented loss that regularizes the data space. Note that we only seek specific perturbations of data x that increase the overall loss value of prediction. In order to formally motivate our approach, we first make some assumptions. 
  
   
  The distribution of possible adversarial data inputs of the neural network obeys a positive exponential distribution of the form below, where the domain of (X;Y,w) is bounded:
  
      p(X;Y,w,β) =
         
         Z_w,β^-1 e^β(X;Y,w)   if   (X;Y,w) ≤ R ,
      
      0             if   (X;Y,w) > R ,
  
  and Z_w,β is the partition function that normalizes the probability distribution.
  
  Note here that cross entropy loss ℒ is always lower bounded as ≥ 0.
  
  Intuitively, the neural network is more likely to “see" perturbed examples from the adversary corresponding to higher loss values as compared to lower loss values. The parameter R is chosen to ensure that the integral of the probability curve is bounded. 
  When the temperature parameter β→∞, the above Gibbs distribution concentrates at the maximizer(s) of (X̅;Y,w), where X̅ is the “worst possible" set of adversarial inputs to the domain of the loss function for fixed weights w. For a given attack ball Δ_p,ϵ with radius ϵ and norm p, and fixed weights w, this value equates to:
  
      X̅ = max_X'∈Δ_p,ϵℒ(X';X,Y,w) ,
  
  where max_X'∈Δ_p,ϵℒ(X';X,Y,w)≤ R.
   
  A modified distribution, (without loss of generality, setting β=1) with an additional smoothing parameter, assumes the form:
  
      p(X';X,Y,w,γ) =
         
      Z_X,w,γ^-1 e^(X';Y,w) - γ/2X'-X_F^2   if  (X';Y,w)≤ R 
      
      0    if  (X';Y,w)>R
  
  where Z_X,w,γ is the partition function that normalizes the probability distribution. 
  
  Here γ controls the penalty of the distance of the adversary from true data X; if γ→∞, the sampling is sharp, i.e. p(X'=X;X,Y,w,γ) = 1 and p(X'≠ X;X,Y,w,γ) = 0, which is the same as sampling only the standard loss ℒ, meanwhile γ→ 0 corresponds to a uniform contribution from all possible data points in the loss manifold. 
  
  Now, we develop an augmented loss function which incorporates the probabilistic formulation in Assumption <ref>. The standard objective can be re-written as the functional convolution: 
  
      min_w (w;X,Y)    := min_w ∫_X'(X';Y,w)δ(X-X')dX',
  
  which can be seen as a sharp sampling of the loss function at training 
  points X. Now, define the Data-Entropy Loss:
  
      _DE(w;X,Y,γ)       = ∫_X'(X';Y,w) p(X';X,Y,w,γ) dX'
  
  our new objective is to minimize this augmented objective function _DE(w;X,Y,γ), which resembles expected value of the standard loss function sampled according to a distribution that (i) penalizes points further away from the true training data (ii) boosts data points which correspond to high loss values. Specifically, the adversarial samples generated by the distribution in Assumption <ref> will correspond to those with high loss values in the immediate neighborhood of the true data samples. This sampling process is also described in Fig. <ref>. We also highlight theoretical properties of our augmented loss function in Lemma <ref> of supplement (Appendix <ref>).
  
  
  
  If gradient descent is used to minimize the loss in Eq. <ref>, the gradient update corresponding to the augmented loss function can be computed as follows
  
      ∇_w _de(w;X,Y,γ) 
         = ∇_w ∫_X'(X';Y,w) p(X';X,Y,w,γ) dX' 
         = ∇_w 𝔼_X'∼ p(X';X,Y,w,γ) [(X';Y,w)] .
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Correspondingly, the weights of the network, when trained using gradient descent, using Eq. <ref>, can be updated as
  
      w^+ = w - η∇_w 𝔼_X'∼ p(X';X,Y,w,γ) [(X';Y,w)]
  
  where η is the step size. The expectation in Eq. <ref> is carried out over the probability distribution of data samples X' as defined in Assumption <ref>. This can be seen as the adversarial version of the formulation developed in Entropy SGD <cit.>, where the authors use a Gibbs distribution to model an augmented loss function that explores the loss surface at points that are perturbed from the current weights w, denoted by w' (see Appendix <ref> of the supplement). In contrast, in our approach, we consider loss contributions from perturbations X' of data points X. This analogue is driven by the fact that the core objective in <cit.> is to design a network which is robust to perturbations in weights (generalization), where as the core objective of this paper is to design a network that is robust to perturbations in the inputs (adversarial robustness).
  
  The expectation in Eq. <ref> is computationally intractable to optimize (or evaluate). However, using the Euler discretization of the Langevin Stochastic Differential Equation <cit.>, it can be approximated well. Samples can be generated from p(X') as: 
  
      X'^k+1 = X'^k + η' ∇_X'log p(X'^t) + √(2η')ε𝒩(0,𝕀)
  
  where η' is the step size for Langevin sampling, ε is a scaling factor that controls additive noise. In Langevin dynamics, when one considers a starting point of X'^0 then the procedure above yields samples X'^1 … X'^t that follow the distribution p(X'). Intuitively, the stochastic process X'^t is more likely to visit points in the immediate neighborhoods of the entire training dataset X  corresponding to high loss values.
  
  Observe that X' and X have the same dimensions and the gradient term in the above equation needs to be computed over n, d-dimensional data points. In practice this can be computationally expensive. Therefore, we discuss a stochastic variant of this update rule, which considers mini-batches of training data instead.
  Plugging in the distribution in Eq.<ref>, and using the Euler discretization for Langevin Stochastic Differential Equations, the update rule for sampling X' is
  
      X'^k+1-X'^k/η' =
         ∇_X'^k((X'^k;Y,w)-γ/2X-X'^k_F^2) + √(2ε^2/η')𝒩(0,𝕀)
         = ∇_X'^k(X'^k;Y,w) + γ (X-X'^k) + √(2ε^2/η')𝒩(0,𝕀)
  
  where we have incorporated Z_X,wγ in the step size η'. Note that as the number of updates k→∞, the estimates from the procedure in Eq. <ref> converge to samples from the true distribution. p(X';X,Y,w,γ). 
  We then want to estimate ∇_w _de(w;X,Y,γ) =∇_w  _X'∼ p(X')[(w;X',Y,γ)] using the samples obtained from the above iterative procedure. In <cit.>, authors use an exponentially decaying averaging process to estimate the expected value. 
  
  Batch-wise updates for stochastic gradient estimates: As is typical with large datasets, instead of using the entire training data for computing gradients in Eq. <ref> and Eq. <ref>, one can use batch-wise data where the training data is segmented into J batches [X_B_1,X_B_2… X_B_J]. This is essentially a combination of Stochastic Gradient Descent and Langevin Dynamics and is known as Stochastic Gradient Langevin Dynamics in recent literature <cit.>. 
  
  This discussion effectively leads to the algorithm shown in Algorithm <ref>, which we refer to as Adversarial Training using Entropy (or ATENT), designed for ℓ_2 attacks. Note that we have considered exponentially decaying averaging over sample loss μ^k in Line 10 of Algorithm <ref>. 
  
  
  
  
  Comparison to PGD Adversarial Training: (see also Algorithm <ref> <cit.> in Appendix <ref> (PGD-AT) of the supplement). It is easy to see that the updates of PGD-AT are similar to that of Algorithm <ref>, consisting broadly of two types of gradient operations in an alternating fashion - (i) an (inner) gradient with respect to samples X (or batch-wise samples X_B_j) and (ii) an (outer) gradient with respect to weights w. While PGD-AT minimizes the worst-case loss in an ϵ-neighborhood (specifically ℓ_2 or ℓ_∞ ball) of X, ATENT minimizes an average loss over our specifically designed probability distribution (Assumption <ref>) in the neighborhood of X. 
  Note that the gradient operation in Eq. <ref> is also the gradient for the regularized version of inner maximation of the adversarial training  problem <cit.>, but with added noise term,
  
      max_X'ℒ(X';X,Y,w)    s.t.   X'-X_F^2 ≤ϵ
      ⇔max_X'ℒ(X';X,Y,w) - γ/2X'-X_F^2
  
  constraint being satisfied if X'-X_F is minimized, or -X'-X_F is maximized).
  
  The width of the Gaussian smoothing is adjusted with γ, which is analogous to controlling the projection radius ϵ in the inner-maximization of PGD-AT. Then the second and third terms in Eq. <ref> are simply gradient of an ℓ_2-regularization term over data space X' and noise. In this way, ATENT can be re-interpreted as a stochastic formalization of ℓ_2-PGD-AT, with noisy controlled updates.
  
  
  
  Comparison to randomized smoothing:
  In <cit.>, authors describe a defense to adversarial perturbations, in the form of smoothing. A smoothed classifier g, under isotropic Gaussian noise ε = 𝒩(0,σ^2 𝕀), produces an output:
  
      g(x) =  max_jℙ(f(x) +ε) = j).
  
  where ℙ denotes probability distribution (see Appendix <ref> for detailed discussion). SmoothAdv <cit.> is an adversarial attack as well as defense for smoothed classifiers, which replaces standard loss with cross entropy loss of a smoothed classier. In comparison, we compute a smoothed version of the cross entropy loss of a standard classifier. This is similar to the setup of <cit.> (TRADES with smoothing). The procedure in Algorithm<ref> is therefore  amenable to randomized smoothing in its evaluation. We discuss a smoothed evaluation of ATENT in the next section.
  
  
  Extension to defense against ℓ_∞-attacks:
  It is evident that due to the isotropic structure of the Gibbs measure around each data point, Algorithm <ref>, ℓ_2-ATENT is best suited for ℓ_2 attacks. However this may not necessarily translate to robustness against ℓ_∞ attacks. For this case, one can use an alternate assumption on the distribution of potential adversarial examples. 
   
  We consider a modification of the distribution in Assumption <ref> to account for robustness against ℓ_∞ type attacks: 
  
      p(X';X,Y,w,γ)= 
         
      Z_X,w,γ^-1 e^((X';Y,w) - γ/2X'-X_∞)   if  (X';Y,w)≤ R 
      
      0    if  (X';Y,w)>R
  
  where ·_∞ is the ℓ_∞ norm on the vectorization of its argument and Z_X,w,γ normalizes the probability.
  
  The corresponding Data Entropy Loss for ℓ_∞ defenses is:
  
      _DE,∞(w;X,Y) =
          Z_X,w,γ^-1∫_X'(X';Y,w)e^((X';Y,w)-γ/2X-X'_∞) dX'
  
  
  This resembles a smoothed version of the loss function with a exponential ℓ_∞ kernel along the data dimension to model points in the ℓ_∞ neighborhood of X which have high loss. The SGD update to minimize this loss becomes:
  
      ∇_w _DE,∞(w;X,Y)      = ∇_w _X'∼ p(X')[(w;X',Y)]
       w^+    = w - η∇_w _DE,∞(w;X,Y)
  
  where the expectation over p(X') is computed by using samples generated via Langevin Dynamics: 
  
      X'^k+1 = X'^k + η' ∇_X'log p(X'^k) + √(2η')ε𝒩(0,𝕀)
  
  Plugging in the distribution in Assumption <ref> the update rule for sampling X':
  
      X'^k+1 - X'^k/η' = 
         ∇_X'^k(L(X'^k;Y,w)-γ/2X-X'^k_∞) + √(2ε^2/η')𝒩(0,𝕀)
         ∇_X'^k L(X'^k;Y,w) + γsign(X_i-X_i'^k)·1 + √(2ε^2/η')𝒩(0,𝕀)
  
  where i = max_j |X_j - X_j'^k| and j scans all elements of the tensors X,X'^k and 1_j = δ_i,j. The second term in the update rule navigates the updates X'^k+1 to lie in the immediate ℓ_∞ neighborhood of X. Note that this training process requires taking gradients of ℓ_∞ distance. In the update rule in Eq. <ref>, the gradient update only happens along one coordinate. In practice when we test this update rule, the algorithm fails to converge. This is due to the fact that typically a sizeable number of elements of X'-X have a large magnitude. 
  
  The expression in the penultimate step of Eq. <ref>, is the gradient of a regularized maximization problem, 
  
      max_X'ℒ(X';X,Y,w)    s.t.  X'-X_∞≤ϵ
      max_X'ℒ(X';X,Y,w) - γX'-X_∞
  
  where γ is inversely proportional to ϵ (constraint is satisfied if X'-X_∞ is minimized, or -X'-X_∞ is maximized). This expression can be maximized only if X'∈Δ_∞,ϵ of X; however when we take gradients along only one coordinate, this may not be sufficient to drive all coordinates of X' towards Δ_∞,ϵ of X. 
  
  Similar to the ℓ_∞ Carlini Wagner attack <cit.>, we replace the gradient update of the ℓ_∞ term, with a clipping based projection oracle. We design an accelerated version of the update rule in Eq. <ref>, in which we perform a clipping operation, i.e. an ℓ_∞ ball projection of the form:
  
      X'^k+1 - X'^k 
         = η' ∇_X' L(X'^k;Y,w)  + √(2η')ε𝒩(0,𝕀)  , 
         X'^K - X'^K-1
         = Pγ(η' ∇_X' L(X'^K-1;Y,w)  + √(2η')ε𝒩(0,𝕀) )
  
  where element-wise projection P_γ(z) = z if |z|<1/γ and P_γ(z) = 1/γ if |z|>1/γ. Empirically, we also explored an alternate implementation where the projection takes place in each inner iteration k, however, we find the version in Algorithm <ref> to give better results.
  
  In both Algorithms <ref> and <ref>, we initialize the Langevin update step with a random normal perturbation δ_i of benign samples, which is constructed to lie inside within approximately 1/γ radius of the natural samples.
  
  
  
  
  § EXPERIMENTS
  
  
  
  In this section we perform experiments on a five-layer convolutional model with 3 CNN and 2 fully connected layers, used in <cit.>, trained on MNIST. We also train a WideResNet-34-10 on CIFAR10 (as used in <cit.>) as well as ResNet20. Due to space constraints, we present supplemental results in Appendix <ref>. We conduct our experiments separately on networks specifically trained for ℓ_2 attacks and those trained for ℓ_∞ attacks. We also test randomized smoothing for our ℓ_2-ATENT model. Source code is provided in the supplementary material.
  
  Attacks: For ℓ_2 attacks, we test PGD-40 with 10 random restarts, and CW2 attacks 
  at radius ϵ_2=2 for MNIST and PGD-40 and CW2 attacks at ϵ_2=0.43 (≈ϵ_∞ = 2/255) and ϵ_2=0.5 = 128/255 for CIFAR10. For ℓ_∞ attacks, we test PGD-20, ℓ_∞CW, DeepFool attacks at radiii ϵ_∞=0.3 for MNIST and ϵ_∞=0.031 =  8/255 for CIFAR10. We test ATENT at other attack radii in Appendix <ref>. For implementing the attacks, we use the Foolbox library <cit.> and the Adversarial Robustness Toolbox <cit.>.
  
  Defenses: We compare models trained using: SGD (vanilla), Entropy SGD <cit.>, PGD-AT <cit.> with random starts (or PGD-AT(E) with random start, early stopping <cit.>), TRADES <cit.>, MMA <cit.> and MART <cit.>. Wherever available, we use pretrained models to tabulate robust accuracy results for PGD-AT, TRADES, MMA and MART as presented in their published versions. Classifiers giving the best and second best accuracies are highlighted in each category.
  
  Smoothing: We also test randomized smoothing <cit.> in addition to our adversarial training to evaluate certified robust accuracies. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   
  
   
  
  
  The results were generated using an Intel(R) Xeon(R) W-2195 CPU 2.30GHz Lambda cluster with 18 cores and a NVIDIA TITAN GPU running PyTorch version 1.4.0.
  
  
  
   §.§ MNIST
  
  
  In Tables <ref> and <ref>, we tabulate the robust accuracy for 5-layer convolutional network trained using the various approaches discussed above for both ℓ_2 and ℓ_∞ attacks respectively. 
  
  Training setup: Complete details are provided in Appendix <ref>. Our experiments for ℓ_2 attack are presented in Table <ref>. We perform these experiments on a LeNet5 model imported from the Advertorch toolbox (architecture details are provided in the supplement). For ℓ_2-ATENT we use a batch-size of 50 and SGD with learning rate of η= 0.001 for updating weights.  We set γ=0.05 and noise ε∼ 0.001𝒩(0,𝕀). We perform K=40 Langevin epochs and set the Langevin parameter α=0.9, and step η'=0.25. For attack, we do a 40-step PGD attack with ℓ_2-ball radius of ϵ = 2. The step size for the PGD attack is 0.25, consistent with the configuration in <cit.>. We perform early stopping by tracking robust accuracies of validation set and report the best accuracy found.    
  
  In Table <ref>, we use a SmallCNN configuration as described in <cit.> (architecture in supplement). We use a batch-size of 128, SGD optimizer with learning rate of η= 0.01 for updating weights. We set γ=3.33 and noise ε∼ 0.001𝒩(0,𝕀). We perform L=40 Langevin epochs and we set the Langevin parameter α=0.9, and step η'=0.01, consistent with the configuration in <cit.>. For the PGD attack, we use a 20-step PGD attack with step-size 0.01, for ℓ_∞-ball radius of ϵ = 0.3. We perform an early stopping by tracking robust accuracies on the validation set and report the best accuracy found. Other attack configurations can be found in the supplement.   
  
  Our experiments on the Entropy-SGD (row 2 in Tables <ref> and <ref>) trained network suggests that networks trained to find flat minima (with respect to weights) are not more robust to adversarial samples as compared to vanilla SGD. 
  
  
  
  
  
  
  
   §.§ CIFAR10
  
  Next, we extend our experiments to CIFAR-10 using a WideResNet 34-10 as described in <cit.> as well as ResNet-20. For PGD-AT (and PGD-AT (E)), TRADES, and MART, we use the default values stated in their corresponding papers. 
  
  Training setup: Complete details in Appendix <ref>. 
  Robust accuracies of WRN-34-10 classifer trained using state of art defense models are evaluated at the ℓ_∞ attack benchmark requirement of radius ϵ = 8/255, on CIFAR10 dataset and tabulated in Table <ref>. For ℓ_∞-ATENT, we use a batch-size of 128, SGD optimizer for weights, with learning rate η = 0.1 (decayed to 0.01 at epoch 76), 76 total epochs, weight decay of 5× 10^-4 and momentum 0.9. We set γ = 1/(0.0031), K=10 Langevin iterations, ε = 0.001𝒩(0,𝕀), at step size η' = 0.007. We test against 20-step PGD attack, with step size 0.003, as well as ℓ_∞-CW and Deep Fool attacks using FoolBox. ℓ_∞-ATENT is consistently among the top two performers at benchmark configurations.
  
  Importance of early stopping: Because WRN34-10 is highly overparameterized with approximately 48 million trainable parameters, it tends to overfit adversarially-perturbed CIFAR10 examples. The success of TRADES (and also PGD) in <cit.> relies on an early stopping condition and corresponding learning rate scheduler. We strategically search different early stopping points and report the best possible robust accuracy from different stopping points.
  
  We test efficiency of our ℓ_2-based defense on both ℓ_2 attacks, as well as compute ℓ_2 certified robustness for the smoothed version of ATENT against smoothed TRADES <cit.> in Table <ref> in Appendix <ref>. We find that our formulation of ℓ_2 ATENT is both robust against ℓ_2 attacks, as well as gives a competitive certificate against adversarial perturbations for ResNet20 on CIFAR10.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  In Appendix <ref> we also demonstrate a fine-tuning approach for ATENT, where we consider a pre-trained WRN34-10 and fine tune it using ATENT, similar to the approach in <cit.>. We find that ATENT can be used to fine tune a naturally pretrained model at lower computational complexity to give competitive robust accuracies while almost retaining the performance on benign data.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Discussion
  
  
  We propose a new algorithm for defending neural networks against adversarial attacks. We demonstrate competitive (and often improved) performance of our family of algorithms (ATENT) against the state of the art. We analyze the connections of ATENT with both PGD-adversarial training as well as randomized smoothing. Future work includes extending to larger datasets such as ImageNet, as well as theoretical analysis for algorithm convergence.
  
  
  
  
  § ACKNOWLEDGEMENTS
  
  The authors would like to thank Anna Choromanska for useful initial feedback and Praneeth Narayanamurthy for insightful conceptual discussions.
  
  
  
  
  
  
  
  
  
  § ADDITIONAL EXPERIMENTS AND DETAILS
   
  
  In this section, we provide additional details as well as experiments to supplement those in Section <ref>.
  
  
  
  
  
  
   §.§ Detailed training setup
  
  Architectures: For MNIST- ℓ_∞ experiments, we consider a CNN architecture with the following configuration (same as <cit.>).  Feature extraction consists of the following sequence of operations: two layers of 2-D convolutions with 32 channels, kernal size 3, RelU activation each, followed by maxpooling by factor 2, followed by two layers of 2-D convolutions with 64 channels, kernel size 3, ReLU activation, and finally another maxpool (by 2) operation. This is followed by the classification module, consisting of a fully connected layer of size 1024 × 200, ReLU activation, dropout, another fully connected layer of size 200 × 200, ReLU activation and a final fully connected layer of size 200 × 10. Effectively this network has 4 convolutional and 3 fully connected layers. We use batch size of 128 with this configuration.
  
  For MNIST-ℓ_2 experiments, we consider the LeNet5 model from the Advertorch library (same as <cit.>). This consists of a feature extractor of the form - two layers of 2-D convolutions, first one with 32 and second one with 64 channels, ReLU activation and maxpool by factor 2. The classifier consists of one fully connected layer of dimension 3136 × 1024 followed by ReLU activation, and finally another fully connected layer of size 1024× 10. We use batch size of 50 with this configuration.
  
  For CIFAR-ℓ_∞ experiments we consider a WideResNet with 34 layers and widening factor 10 (same as <cit.> and <cit.>). It consists of a 2-D convolutional operation, followed by 3 building blocks of WideResNet, ReLU, 2D average pooling and fully connected layer. Each building block of the WideResNet consists of 5 successive operations of batch normalization, ReLU, 2D convolution, another batch normalization, ReLU, dropout, a 2-D convolution and shortcut connection. We use batch size of 128 with this configuration.
  
  For CIFAR-ℓ_2 experiments, we consider a ResNet with 20 layers. This ResNet consists of a 2-D convolution, followed by three blocks, each consisting of 3 basic blocks with 2 convolutional layers, batch normalization and ReLU. This is finally followed by average pooling and a fully connected layer. We use batch size of 256 with this configuration. 
  
  Training SGD and Entropy SGD models for MNIST experiments: For SGD, we trained the 7-layer convolutional network setup in <cit.> with the MNIST dataset, setting batch size of 128, for ℓ_∞ SGD optimizer using a learning rate of 0.1, for 50 epochs. For Entropy SGD, with 5 langevin steps, and γ=10^-3, batch size of 128 and learning rate of 0.1 and 50 total epochs.
  
  
  
   §.§ ℓ_2 ATENT
  
  ℓ_2-PGD attacks on CIFAR10: We explore the effectiveness of ℓ_2-ATENT as a defense against ℓ_2 perturbations. These results are tabulated in Table <ref>. We test 10-step PGD adversarial attacks at ϵ_2= 0.5 and ϵ_2 = 1. For the purpose of this comparison, we compare pretrained models of MMA and PGD-AT at ϵ_2 = 1. To train ATENT, we use γ=0.08 for ϵ_2=1 10 step attack (with 2.5ϵ_2/10 step size), K=10 langevin iterations, langevin step η'=2 ϵ_2/K, learning rate for weights η = 0.1. We also compare models primarily trained to boost the certificate of randomized smoothing. For this we train a ResNet20 model for both TRADES (at default parameter setting) and ℓ_2 ATENT, at η'= and γ=, such that the effective noise standard deviation is 0.12. These models are tested against PGD-10 attacks at radius ϵ_2= 0.435. In all ℓ_2 ATENT experiments, we choose the value of γ such that the perturbation X'^K-X_F ≈ϵ_2 of corresponding models of TRADES and PGD-AT. For all ATENT experiments, we set α=0.9.
  
  
  Experiments on randomized smoothing: Since the formulation of ATENT is similar to a noisy PGD adversarial training algorithm, we test its efficiency towards randomized smoothing and producing a higher robustness certificate (Table <ref>). For this we train a ResNet-20 on CIFAR10, at γ=0.05, η'=0.02, η=0.1, K=10, and tune the noise ε, such that effective noise √(2η')ε has standard deviation σ=0.12. We compare the results of randomized smoothing to established benchmarks on ResNet-110 (results have been borrowed from Table 1 of <cit.>). as well as a smaller ResNet-20 model trained using TRADES at its default settings. We observe that without any modification to the current form of ATENT, our method is capable of producing a competitive certificate to state of art methods. In future work we aim to design modifications to ATENT which can serve the objective of certification. 
  
  
  
  
  
   §.§ ℓ_∞ ATENT
  
  
  Training characteristics of ℓ_∞ ATENT: In Figure <ref> we display the training curves of ATENT. As shown, the robust accuracies spike sharply after the first learning rate decay, followed by an immediate decrease in robust accuracies. This behavior is similar to that observed in <cit.>. This is also the key intuition used in the design of the learning rate scheduler for TRADES.
  
  
  
  
  ATENT as Attack: For our ℓ_∞-ATENT WideResNet-34-10, we also test ℓ_∞-ATENT as an attack. We keep the same configuration as that of PGD-20, for ATENT. We compare the performance of our ℓ_∞-ATENT trained model (specifically designed to work against ϵ_∞=8/255 attacks). The values (Table <ref>) suggest that the adversarial perturbations generated by ATENT are similar in strength to those produced by PGD (worst possible attack).
  
  
  
  Computational complexity:
  In terms of computational complexity, ATENT matches that of PGD and TRADES, as can be observed from the fact that all three approaches are nested iterative optimizations. Due to the high computational complexity of all adversarial algorithms, we test a fine-tuning approach, to trade computational complexity for accuracy. This method is suggested in <cit.>. In this context, we take a pre-trained WideResNet-34-10 which has been trained on benign CIFAR10 samples only. This model is then fine tuned on adversarial training data, via ℓ_∞ ATENT using a low learning rate η=0.0001 and trained for only 20 epochs. The final robust accuracy at ϵ_∞ = 8/255  is  52.1%. This is accuracy marginally improves upon the robust accuracy observed (51.7%) for fine-tuned WideResNet-28-10 PGD-AT trained model in <cit.>. This experiments suggests that ATENT is amenable for fine tuning pretrained benign models using lesser computation, but at the cost of slightly reduced robust accuracy (roughly 5% drop at benchmark of ϵ_∞ = 8/255).
  
  
  
  
  
  § PROOFS AND DERIVATIONS
   
  
  
  
   §.§ Theoretical properties of the augmented loss
  
  
  We now state an informal theorem on the conditions required for convergence of SGLD in Eq. <ref> for estimating adversarial samples X'.
  
  lemsmooth 
  The effective loss F(X';X,Y,w) := γ/2X-X'_F^2 - ℒ(X';Y,w) which guides the Langevin sampling process in Eq. <ref> is
  
      
    * β+γ smooth if ℒ(X;Y,w) is β-smooth in X.
      
    * (γ/4,L^2/γ + γ/2X_F^2) dissipative if ℒ(X;Y,w) is L-Lipschitz in X.
  
  
  One can then use smoothness and dissipativity of F(X';Y,w) to show convergence of SGLD for the optimization over X' (Eq. <ref>) via Theorem 3.3 of <cit.>. 
  
  
  We first derive smoothness conditions for the effective loss 
  
      F(X';X,Y,w) := γ/2X-X'_F^2 - ℒ (X';Y,w),   ∀ X'_1,X_2'.
   
  We use abbreviations p(X') := p(X';X,Y,w), F(X') := F(X';X,Y,w), ℒ(X';Y,z) := ℒ(X') and ℒ(X;Y,z) := ℒ(X), and assume that X and X' are vectorized. Unless specified otherwise, · refers to the vector 2-norm.
  
  Let us show that ∇_X'F(X'_2) - ∇_X'F(X'_1)≤β'X_2'-X_1'. If the original loss function is β smooth, i.e.,
  
      ∇_X'ℒ(X_2')-∇_X'ℒ(X_2')   ≤βX_2'-X_1' ,
  
  then: 
  
      ∇_X'F(X'_2) - ∇_X'F(X'_1)   ≤-∇_X'ℒ(X'_2) + ∇_X'ℒ(X'_1) - γ(X - X'_2) + γ(X-X_1')
         ≤∇_X'ℒ(X'_2) - ∇_X'ℒ(X'_1) + γ( X'_2-X_1')
         ≤ (β+γ) X_2'-X_1'
  
  by application of the triangle inequality.
  
  
  Next, we establish conditions required to show (m,b)-dissipativity for F(X'), i.e. ⟨∇_X'F(X'),X'⟩≥ mX'^2_2 - b for positive constants m,b>0, ∀ X'. 
  
  To show that:
  
      ⟨∇_X'F(X'),X'⟩   ≥ mX'^2_2 - b
  
  where the left side of inequality can be expanded as:
  
      ⟨∇_X'F(X'),X'⟩   = ⟨-∇_X'(X') + γ(X'-X),X'⟩
         =⟨-∇_X'(X'),X'⟩ + γX'_2^2 -  γ⟨ X,X'⟩
         = ⟨-∇_X'(X'),X'⟩ + γX'_2^2 -  γ/2(X'_2^2 + X_2^2- X-X'_2^2)
         ≥⟨-∇_X'(X'),X'⟩ + γX'_2^2 -  γ/2(X'_2^2 + X_2^2)
         =⟨-∇_X'(X'),X'⟩ + γ/2X'_2^2 -  γ/2X_2^2
  
  To find the inner product ⟨-∇_X'(X'),X'⟩, we expand squares:
  
      ∇_X'(X') - γ/2X'^2 = ∇_X'(X')_2^2 + γ^2/4X'_2^2 - γ⟨∇_X'(X'),X'⟩   ≥ 0
      
            -⟨∇_X'(X'),X'⟩   ≥ -∇_X'(X')_2^2/γ - γ/4X'_2^2
  
  Plugging this into (<ref>), and assuming Lipschitz continuity of original loss ℒ(X'), i.e., ∇_X'ℒ(X')_2 ≤ L:
  
      ⟨∇_X'F(X'),X'⟩   ≥⟨-∇_X'(X'),X'⟩ + γ/2X'_2^2 -  γ/2X_2^2 
         ≥ -∇_X'(X')_2^2/γ - γ/4X'_2^2 + γ/2X'_2^2 -  γ/2X_2^2 
         = γ/4X'_2^2-(L^2/γ +γ/2X_2^2) 
         =mX'_2^2 - b
  
  where m=γ/4 and b = L^2/γ + γ/2X_2^2. Thus, F(X') is (γ/4,L^2/γ + γ/2X_2^2) dissipative, if (X') is L-Lipschitz.
  
  
  
  
  
  
  
  
  
  With Lemma <ref> we can show convergence of the SGLD inner optimization loop. To minimize overall loss function, the data entropy loss _DE is minimized w.r.t. w, via Stochastic Gradient Descent (SGD). The gradient update for weights w are designed via (<ref>) as follows: 
  
      ∇_w _DE(w;X,Y,γ) 
              = ∇_w ∫_X'(X';Y,w) p(X';X,Y,w,γ) dX' = ∇_w 𝔼_X'∼ p(X';X,Y,w,γ) [(X';Y,w)] 
         =  ∫_X'∇_w( (X';Y,w) p(X';X,Y,w,γ) ) dX' 
         =  ∫_X'∇_w (X';Y,w) ·  p(X';X,Y,w,γ)  + ∇_w (X';Y,w) ·(X';Y,w) · p(X';X,Y,w,γ) dX'
         =∫_X'∇_w (X';Y,w)  ·((X';Y,w)+1) · p(X';X,Y,w,γ) dX' 
         =   𝔼_X'∼ p(X';X,Y,w,γ)( ∇_w(X';Y,w)  · ((X';Y,w)+1))
  
  Then a loose upper bound on Lipschitz continuity of _DE is ∇_w _DE(w;X,Y,γ) _2 ≤L̅(R+1), if original loss is L̅-Lipschitz in w and (X) ≤ R. Due to the complicated form of this expression, establishing β-smoothness will require extra rigor. We push a more thorough evaluation of the convergence of the outer SGD loop to future work.
  
  
  
   §.§ Entropy SGD
  
  
  
  
  In <cit.> authors claim that neural networks that favor wide local minima have better generalization properties, in terms of perturbations to data, weights as well as activations. Mathematically, the formulation in Entropy SGD can be summarized as follows. A basic way to model the distribution of the weights of the neural network is using a Gibbs distribution of the form:
  
      p(w;X,Y,β) = Z_X,β^-1exp(-β(w;X,Y))
  
  When β→∞, this distribution concentrates at the global (if unique) minimizer of (w^*;X,Y). A modified Gibbs distribution, with an additional smoothing parameter is introduced, which assumes the form: 
  
      p(w';w,X,Y,β=1,γ) = Z_w,X,γ^-1exp(- (w';X,Y) - γ/2w'-w_2^2)
  
  where Z_w,X,γ normalizes the probability.
  
  Here γ controls the width of the valley; if γ→∞, the sampling is sharp, and this corresponds to no smoothing effect, meanwhile γ→ 0 corresponds to a uniform contribution from all points in the loss manifold. The standard objective is:
  
      min_w (w;X,Y)    := min_w - log(exp(-(w;X,Y)))
         = min_w - log(∫_w'exp(-(w';X,Y))δ(w-w')dw')
  
  which can be seen as a sharp sampling of the loss function. Now, if one defined the Local Entropy as:
  
      _ent(w;X,Y)    = -log (Z_w,X,Y,γ) 
         = - log(∫_w'exp(-(w';X,Y)-γ/2w-w'_2^2) dw')
  
  our new objective is to minimize this augmented objective function _ent(w;X,Y), which resembles a smoothed version of the loss function with a Gaussian kernel. The SGD update can be designed as follows:
  
      ∇_w _ent(w;X,Y)    = -∇_w (log (Z_w,X,Y,γ))
         = Z_w,X,γ^-1∇_w(Z_w,X,γ) 
         = Z_w,X,γ^-1(∫_w'exp(-(w';X,Y)-γ/2w-w'_2^2)·γ (w-w') dw')
         = ∫_w' p(w';w,X,Y,γ) ·γ (w-w') dw' 
         = _w'∼ p(w')[γ (w-w')]
  
  Then, using this gradient, the SGD update for a given batch is designed as:
  
      w^+ = w - η∇_w _ent(w;X,Y)
  
  This gradient ideally requires computation over the entire training set at once; however can be extended to a batch-wise update rule by borrowing key findings from <cit.>. This expectation for the full gradient is computationally intractable, however, Euler discretization of Langevin Stochastic Differential Equation, it can be approximated fairly well as  
  
      w'^t+1 = w'^t + η^t ∇_w'log p(w'^t) + √(2η)𝒩(0,𝕀)
  
  such that after large enough amount of iterations w^+→ w^∞ then w^∞∼ p(w'). One can estimate _w'∼ p(w')[γ (w-w')] by averaging over many such iterates from this process. This result is stated as it is from <cit.>: 𝔼_w'∼ p(w')[g(w')] = ∑_t η_t g(w'_t)/∑_t η_t.  This leads to the algorithm shown in Algorithm <ref>. One can further accrue exponentially decaying weighted averaging of g(w'_t) to estimate 𝔼_w'∼ p(w')[g(w')]. This entire procedure is described in Algorithm <ref>.
  
  This algorithm is then further guaranteed to find wide minima neighborhoods of w by design, as sketched out by the proofs in <cit.>. 
  
  
  
   §.§ Stochastic Gradient Langevin Dynamics
  
  
  Stochastic Gradient Langevin Dynamics combines techniques of Stochastic Gradient  Descent and Langevin Dynamics <cit.>. Given a probability distribution π=p(θ;X), the following update rule allows us to sample from the distribution π:
  
      θ^t+1 = θ^t +η∇_θ p(θ^t;X) + √(2η')ε
  
  where η' is step size and ε is normally distributed. Then, as t→∞, θ∼π.
  
  While this update rule in itself suffices, if the parameters are conditioned on a a training sample set X, which is typically large, the gradient term in Eq. <ref> is expensive to compute. <cit.> shows that the following batch-wise update rule:
  
      θ^t+1 = θ^t +η∇_θ p(θ^t;X_B_j) + √(2η')ε
  
  suffices to produce a good approximation to the samples θ^t→∞∼π. In Algorithm <ref>, θ is the set of perturbed points X'. In each internal iteration, we look at subset of trainable parameters X'_B_j. We update the estimate for X'_B_j by only considering data-points X_B_j at a time. In the current formulation the set of iterable parameters X'_B_j only 'see' a single batch of data X_B_j ; a better estimate would require X'_B_j to be updated by iteratively over all possible batches X_B_k, k=1,2....J. However in practice we observe that just using the corresponding batch X_B_k=j suffices. In future work, we will explore the theoretical implications of this algorithmic design.
  
  
  
  
   §.§ PGD-Adversarial Training
   
  
  
  
  In Algorithm <ref> we describe the PGD-AT algorithm. In <cit.> authors demonstrate that PGD based-attack is the best possible attack that can be given for a given network and dataset combination. Theoretically, 
  
      x̅_worst = max_δ∈Δ_p L(f(ŵ;x+δ), y)
  
  and if this maximization can be solved tractably, then a network trained with the following min-max formulation is said to be robust:
  
      min_wmax_δ∈Δ_p(f(ŵ;X+δ,Y), Y)
  
  Furthermore they show that first order based gradient approaches, such as SGD, are sufficient to suitably optimize the inner maximization over the perturbed dataset. This can be obtained using the following gradient ascent update rule:
  
      X' = X' + η' ∇_X'∈ X+δ(f(w,X'+δ;Y),Y)
  
  (see also Step 7 of Algorithm <ref>). Note that when Δ_p = Δ_2, this projection rule represents a noise-less version of the update rule in ATENT (see Algorithm <ref>, line 8).
  
  Iterative Fast Gradient Sign (IFGS) method effectively captures a similar projection based approach which performs an update within an ℓ_∞ ball. This update is given by:
  
      X' = X' + η' sign(∇_X'(f(w,X'+δ;Y),Y))
  
  Note that this update rule constructs an adversarial example within ℓ_∞-ball, during the training procedure. Meanwhile, given our proposed adversarial example sampling criterion in Assumption <ref>, our update rule is slightly different (see also Eq. <ref>).
  
  
  
   
      
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  		
  	
  	
  	
  	
  		
      
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
     
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  icml2021
  
  