Question: represent the article for finding a single-sentence tldr

Question: Deep Neural Networks (DNNs) have recently been shown to be vulnerable against adversarial examples, which are carefully crafted instances that can mislead DNNs to make errors during prediction. To better understand such attacks, a characterization is needed of the properties of regions (the so-called `adversarial subspaces') in which adversarial examples lie. We tackle this challenge by characterizing the dimensional properties of adversarial regions, via the use of Local Intrinsic Dimensionality (LID). LID assesses the space-filling capability of the region surrounding a reference example, based on the distance distribution of the example to its neighbors. We first provide explanations about how adversarial perturbation can affect the LID characteristic of adversarial regions, and then show empirically that LID characteristics can facilitate the distinction of adversarial examples generated using state-of-the-art attacks. As a proof-of-concept, we show that a potential application of LID is to distinguish adversarial examples, and the preliminary show that it can outperform several state-of-the-art detection measures by large margins for five attack strategies considered in this paper across three benchmark datasets. Our analysis of the LID characteristic for adversarial regions not only motivates new directions of effective adversarial defense, but also opens up more challenges for developing new attacks to better understand the vulnerabilities of DNNs. Deep Neural Networks (DNNs) are highly expressive models that have achieved state-of-the-art performance on a wide range of complex problems, such as speech recognition and image classification BID18. However, recent studies have found that DNNs can be compromised by adversarial examples (; BID8 BID27 . These intentionally-perturbed inputs can induce the network to make incorrect predictions at test time with high confidence, even when the examples are generated using different networks BID24 BID3 BID29 . The amount of perturbation required is often small, and (in the case of images) imperceptible to human observers. This undesirable property of deep networks has become a major security concern in real-world applications of DNNs, such as self-driving cars and identity recognition BID5 BID34. In this paper, we aim to further understand adversarial attacks by characterizing the regions within which adversarial examples reside. Each adversarial example can be regarded as being surrounded by a connected region of the domain (the 'adversarial region' or 'adversarial subspace') within which all points subvert the classifier in a similar way. Adversarial regions can be defined not only in the input space, but also with respect to the activation space of different DNN layers . Developing an understanding of the properties of adversarial regions is a key requirement for adversarial defense. Under the assumption that data can be modeled in terms of collections of manifolds, several works have attempted to characterize the properties of adversarial subspaces, but no definitive method yet exists which can reliably discriminate adversarial regions from those in which normal data can be found. argued that adversarial subspaces are low probability regions (not naturally occurring) that are densely scattered in the high dimensional representation space of DNNs. However, a linear formulation argues that adversarial subspaces span a contiguous multidimensional space, rather than being scattered randomly in small pockets BID8 ). further emphasize that adversarial subspaces lie close to (but not on) the data submanifold. Similarly, it has also been found that the boundaries of adversarial subspaces are close to legitimate data points in adversarial directions, and that the higher the number of orthogonal adversarial directions of these subspaces, the more transferable they are to other models (Tramèr et al., 2017). To summarize, with respect to the manifold model of data, the known properties of adversarial subspaces are: they are of low probability, they span a contiguous multidimensional space, they lie off (but are close to) the data submanifold, and they have class distributions that differ from that of their closest data submanifold. Among adversarial defense/detection techniques, Kernel Density (KD) estimation has been proposed as a measure to identify adversarial subspaces BID7. BID2 demonstrated the usefulness of KD-based detection, taking advantage of the low probability density generally associated with adversarial subspaces. However, in this paper we will show that kernel density is not effective for the detection of some forms of attack. In addition to kernel density, there are other density-based measures, such as the number of nearest neighbors within a fixed distance, and the mean distance to the k nearest neighbors (k-mean distance). Again, these measures have limitations for the characterization of local adversarial regions. For example, in FIG0 the three density measures fail to differentiate an adversarial example (red star) from a normal example (black cross), as the two examples are locally surrounded by the same number of neighbors FORMULA8, and have the same k-mean distance (KM=0.19) and kernel density (KD=0.92).As an alternative to density measures, FIG0 leads us to consider expansion-based measures of intrinsic dimensionality as a potentially effective method of characterizing adversarial examples. Expansion models of dimensionality assess the local dimensional structure of the data -such models have been successfully employed in a wide range of applications, such as manifold learning, dimension reduction, similarity search and anomaly detection BID0 BID13. Although earlier expansion models characterize intrinsic dimensionality as a property of data sets, the Local Intrinsic Dimensionality (LID) fully generalizes this concept to the local distance distribution from a reference point to its neighbors BID13 BID7 -the dimensionality of the local data submanifold in the vicinity of the reference point is revealed by the growth characteristics of the cumulative distribution function. In this paper, we use LID to characterize the intrinsic dimensionality of adversarial regions, and attempt to test how well the estimates of LID can be used to distinguish adversarial examples. Note that the main goal of LID is to characterize properties of adversarial examples, instead of being applied as a pure defense method, which requires stronger assumptions on the current threat model. In FIG0, the estimated LID of the adversarial example (LID ≈ 4.36) is much higher than that of the referenced normal data sample (LID ≈ 1.53), illustrating that the estimated LID can efficiently capture the intrinsic dimensional properties of adversarial regions. In this paper, we aim to study the LID properties of adversarial examples generated using state-of-the-art attack methods. In particular, our contributions are:• We propose LID for the characterization of adversarial regions of deep networks. We discuss how adversarial perturbation can affect the LID characteristics of an adversarial region, and empirically show that the characteristics of test examples can be estimated effectively using a minibatch of training data.• Our study reveals that the estimated LID of adversarial examples considered in this paper 1 is significantly higher than that of normal data examples, and that this difference becomes more pronounced in deeper layers of DNNs.• We empirically demonstrate that the LID characteristics of adversarial examples generated using five state-of-the-art attack methods can be easily discriminated from those of normal examples, and provide a baseline classifier with features based on LID estimates that generally outperforms several existing detection measures on five attacks across three benchmark datasets. Though the adversarial examples considered here are not guaranteed to be the strongest with careful parameter tuning, these preliminary firmly demonstrate the usefulness of LID measurement.• We show that the adversarial regions generated by different attacks share similar dimensional properties, in that LID characteristics of a simple attack can potentially be used to detect other more complex attacks. We also show that a naive LID-based detector is robust to the normal low confidence Optimization-based attack of BID2. In this section, we briefly review the state of the art in both adversarial attack and adversarial defense. Adversarial Attack: A wide range of approaches have been proposed for the crafting of adversarial examples to compromise the performance of DNNs; here, we mention a selection of such works. The Fast Gradient Method (FGM) BID8 directly perturbs normal input by a small amount along the gradient direction. The Basic Iterative Method (BIM) is an iterative version of FGM BID19. One variant of BIM stops immediately once misclassification has been achieved with respect to the training set (BIM-a), and another iterates a fixed number of steps (BIM-b). For image sets, the Jacobian-based Saliency Map Attack (JSMA) iteratively selects the two most effective pixels to perturb based on the adversarial saliency map, repeating the process until misclassification is achieved BID30. The Optimization-based attack (Opt), arguably the most effective to date, addresses the problem via an optimization framework BID24 BID3.Adversarial Defense: A number of defense techniques have been introduced, including adversarial training BID8, distillation BID31, gradient masking BID10, and feature squeezing . However, these defenses can generally be evaded by Opt attacks, either wholly or partially BID2 BID11 BID21. Given the inherent challenges for adversarial defense, recent works have instead focused on detecting adversarial examples. These works attempt to discriminate adversarial examples (positive class) from both normal and noisy examples (negative class), based on features extracted from different layers of a DNN. Detection subnetworks based on activations BID25, a cascade detector based on the PCA projection of activations BID23, an augmented neural network detector based on statistical measures, a learning framework that covers unexplored space in vulnerable models BID32 BID33, a logistic regression detector based on KD, and Bayesian Uncertainty (BU) features BID9 are a few such works. However, a recent study by BID2 has shown that these detection methods can be vulnerable to attack as well. In the theory of intrinsic dimensionality, classical expansion models (such as the expansion dimension and generalized expansion dimension BID16 BID15) measure the rate of growth in the number of data objects encountered as the distance from the reference sample increases. As an intuitive example, in Euclidean space, the volume of an m-dimensional ball grows proportionally to r m, when its size is scaled by a factor of r. From this rate of volume growth with distance, the expansion dimension m can be deduced as: DISPLAYFORM0 By treating probability mass as a proxy for volume, classical expansion models provide a local view of the dimensional structure of the data, as their estimation is restricted to a neighborhood around the sample of interest. Transferring the concept of expansion dimension to the statistical setting of continuous distance distributions leads to the formal definition of LID BID13. Definition 1 (Local Intrinsic Dimensionality). Given a data sample x ∈ X, let R > 0 be a random variable denoting the distance from x to other data samples. If the cumulative distribution function F (r) of R is positive and continuously differentiable at distance r > 0, the LID of x at distance r is given by: DISPLAYFORM1 whenever the limit exists. F (r) is analogous to the volume V in Equation FORMULA0; however, we note that the underlying distance measure need not be Euclidean. The last equality of Equation FORMULA1 follows by applying L'Hôpital's rule to the limits BID13. The local intrinsic dimension at x is in turn defined as the limit, when the radius r tends to zero: DISPLAYFORM2 LID F describes the relative rate at which its cumulative distance function F (r) increases as the distance r increases from 0, and can be estimated using the distances of x to its k nearest neighbors within the sample BID0.In the ideal case where the data in the vicinity of x is distributed uniformly within a submanifold, LID F equals the dimension of the submanifold; however, in general these distributions are not ideal, the manifold model of data does not perfectly apply, and LID F is not an integer. Nevertheless, the local intrinsic dimensionality does give a rough indication of the dimension of the submanifold containing x that would best fit the data distribution in the vicinity of x. We refer readers to BID13 BID7 for more details concerning the LID model. According to the branch of statistics known as extreme value theory, the smallest k nearest neighbor distances could be regarded as extreme events associated with the lower tail of the underlying distance distribution. Under very reasonable assumptions, the tails of continuous probability distributions converge to the Generalized Pareto Distribution (GPD), a form of powerlaw distribution BID4. From this, BID0 developed several estimators of LID to heuristically approximate the true underlying distance distribution by a transformed GPD; among these, the Maximum Likelihood Estimator (MLE) exhibited a useful trade-off between statistical efficiency and complexity. Given a reference sample x ∼ P, where P represents the data distribution, the MLE estimator of the LID at x is defined as follows: DISPLAYFORM0 Here, r i (x) denotes the distance between x and its i-th nearest neighbor within a sample of points drawn from P, where r k (x) is the maximum of the neighbor distances. In practice, the sample set is drawn uniformly from the available training data (omitting x itself), which itself is presumed to have been randomly drawn from P. We emphasize that the LID defined in Equation FORMULA2 is a theoretical quantity, and that LID as defined in Equation FORMULA3 is its estimate. In the remainder of this paper, we will refer to Equation to calculate LID estimates. Our aim is to gain a better understanding of adversarial regions, and thereby derive potential defenses and provide new directions for more efficient attacks. We begin by providing some motivation with respect to the manifold model of data as to how adversarial perturbation might affect the LID characteristic of adversarial regions. We then show how a detector can potentially be designed using LID estimates to discriminate between adversarial and normal examples. LID of Adversarial Subspaces: Consider a sample x ∈ X lying within a data submanifold S, where X is a randomly sampled dataset from P consisting only of normal (unperturbed) examples. Adversarial perturbation of x typically in a new sample x whose coordinates differ from those of x by very small amounts. Assuming that x is indeed a successful adversarial perturbation of x, the theoretical LID value associated with x is simply the dimension of S, whereas the theoretical LID value associated with x is the dimension of the adversarial subspace within which it resides. Recent work in BID1 shows that the magnitude of the perturbation required to make changes in the expected nearest neighbor ranking tends to zero as the LID and the data sample size tend to infinity. Since perturbation schemes generally allow the modification of all data coordinates, they exploit the full degrees of freedom afforded by the representational dimension of the data domain. As pointed out by BID8; ), x is very likely to lie outside S (but very close to S -in a high-dimensional contiguous space). In applications involving high-dimensional data, the representational dimension is typically far larger than the intrinsic dimension of any given data submanifold, which implies that the theoretical LID of x is far greater than that of x. In practice, however, the values of LID must be estimated from local data samples. This is typically done by applying an appropriate estimator (such as the MLE estimator shown in Equation) to a k-nearest neighborhood of the test samples, for some appropriate fixed choice of k. Typically, k is chosen large enough for the estimation to stabilize, but not so large that the sample is no longer local to the test sample. If the dimension of S is reasonably low, one can expect the estimation of the LID of x to be reasonably accurate. For the adversarial subspace, the samples appearing in the neighborhood of x can be expected to be drawn from more than one manifold. The proximity of x to S means that the neighborhood is likely to contain neighbors lying in S; however, if the neighborhood were composed mostly of samples drawn from S, x would not likely be an adversarial example. Thus, the neighbors of x taken together are likely to span a subspace of intrinsic dimensionality much higher than any of these submanifolds considered individually, and the LID estimate computed for x can be expected to reveal this. Computing neighborhoods with respect to the entirety of the dataset X can be prohibitively expensive, particularly when the (global) intrinsic dimensionality of X is too high to support efficient indexing. For this reason, when X is large, the computational cost can be reduced by estimating the LID of an adversarial example x from its k-nearest neighbor set within a randomly-selected sample (minibatch) of the dataset X. Since the LID estimation model regards the distances from x to the members of X as determined by independently-drawn samples from a distribution P, the estimator can also be applied to the distances induced by any random minibatch, as it too would be drawn independently from the same distribution P.Provided that the minibatch is chosen sufficiently large so as to ensure that the k-nearest neighbor sets remain in the vicinity of x, estimates of LID computed for x within the minibatch would resemble those computed within the full dataset X. Conversely, as the size of the minibatch is reduced, the variance of the estimates would increase. However, if the gap between the true LID values of x and x is sufficiently large, even an extremely small minibatch size and / or small neighborhood size could conceivably produce estimates whose difference is sufficient to reveal the adversarial nature of x. As we shall show in Section 5.2, discrimination between adversarial and non-adversarial examples turns out to be possible even for minibatch sizes as small as 100, and for neighborhood sizes as small as 20.Using LID to Characterize Adversarial Examples: We next describe how LID estimates can serve as features to train a detector to distinguish adversarial examples. Note that here we only aim to train a baseline classifier to demonstrate how well LID can characterize adversarial examples. Robust detection taking different attack variations into account, such as attack confidence, will be left as future work. Our methodology requires that training sets be comprised of three types of examples: adversarial, normal and noisy. This replicates the methodology used in BID7 BID2, where the rationale for including noisy examples is that DNNs are required to be robust to random input noise BID6 and noisy inputs should not be identified as adversarial attacks. A classifier can be trained by using the training data to construct features for each sample, based on its LID within a minibatch of samples across different layers, where the class label is assigned positive for adversarial examples and assigned negative for normal and noisy examples. Algorithm 1 describes how the LID features can be extracted for training an LID-based classifier. Given an initial training dataset and a DNN pre-trained on the initial training dataset, the algorithm outputs a classifier trained using LID features. As in previous studies BID2 BID7, we assume that the initial training dataset is free of adversarial examplesthat is, all examples in the dataset are considered'normal' to begin with. The extraction of LID features first begins with the generation of adversarial and noisy counterparts to normal examples (step 3 and 4) in each minibatch. One minibatch of normal examples (B norm) is used for generating 2 counterpart minibatches of examples: one adversarial (B adv) and one noisy (B noisy). The adversarial examples are generated using an adversarial attack on normal examples (step 3), while noisy examples are generated by adding random noise to normal examples, subject to the constraint that the magnitude of perturbation undergone by a noisy example is the same as the magnitude of perturbation undergone by its counterpart adversarial example (step 4). One minibatch of normal examples is converted to an equal number of adversarial examples after step 3, and an equal number of noisy examples after step 4.The LID associated with each example (either normal, adversarial or noisy) is estimated from its k nearest neighbors in the normal minibatch (steps 12-14), using Equation. For any new unknown test example, a minibatch consisting only of normal training examples is used to estimate LID. For each example and each transformation layer in the DNN, an LID estimate is calculated. The distance function needed for this estimate uses the activation values of the neurons in the given layer as inputs (step 7). As will be discussed in Section 5.2, we use all transformation layers, including conv2d, max-pooling, dropout, ReLU and softmax, since we expect adversarial regions to exist in each layer of the DNN representation space. The LID estimates associated with the example are then used as feature values (one feature for each transformation layer). Finally, a classifier (such as logistic regression) is trained using the LID features. Test examples can then be classified by the LID-based classifier to either the positive (adversarial) or negative (non-adversarial) class by means of its LID-based feature values. In this section, we evaluate the discrimination power of LID-based characterization against five adversarial attack strategies -FGM, BIM-a, BIM-b, JSMA, and Opt, as introduced in Section 2. These attack strategies were selected for our experiments due to their reported effectiveness and their diversity. For each of the 5 forms of attack, the LID detector is compared with the state-of-the-art detection measures KD and BU as discussed in Section 2, with respect to three benchmark image datasets: MNIST , CIFAR-10 (BID17) and SVHN BID26. Each of these three datasets is associated with a designated training set and test set. Before reporting and discussing the , we first describe the experimental setup. DISPLAYFORM0 13: DISPLAYFORM1 14: DISPLAYFORM2 15: DISPLAYFORM3 Training and Testing: For each of the three image datasets, a DNN classifier was independently pretrained on its designated training set (the pre-train set), and its designated test set was used for testing (the pre-test set). Any pre-test images not correctly classified were discarded, and the remaining images were subdivided into train (80%) and test (20%) sets for subsequent processing. Both of these sets were randomly partitioned into minibatches of size 100, for later use in the computation of LID characteristics. The LID-, KD-and BU-based detectors were trained separately on the train set using the scheme in Algorithm 1, with the calculation of LID estimates replaced by KD and BU calculation for their respective detectors. All three detectors were then evaluated against equal numbers of normal, noisy and adversarial images crafted from members of the test set, as described in Steps 2-4 of Algorithm 1. The LID, KD and BU characteristics of those test images were then generated as shown in Steps 1-19 of Algorithm 1. It should be noted that no images of the test set were examined during any of the training processes, so as to avoid cross contamination. The adversarial examples for both training and testing were generated by applying one of the five selected attacks. Following the procedure outlined in BID7, the noisy examples for the JSMA attack were crafted by changing the values of a randomly-selected set of pixels to either their minimum or maximum (determined randomly), where the number of pixels to be adjusted was chosen to be equal to the number of pixels perturbed in the generation of adversarial examples. For the other attack strategies, L 2 Gaussian noise was added to the pixel values instead of setting them to their minimum or maximum. As suggested by BID7; BID2, we used the logistic regression classifier as detector, and report its AUC score as the metric for performance. The pretrained DNN used for MNIST was a 5-layer ConvNet with max-pooling and dropout. It achieved 99.29% classification accuracy on (normal) pre-test images. For CIFAR-10, a 12-layer ConvNet with max-pooling and dropout was used. This model reported an accuracy of 84.56% on (normal) pre-test images. For SVHN, we trained a 6-layer ConvNet with max-pooling and dropout. It achieved 92.18% accuracy on (normal) pre-test images. We deliberately did not tune the DNNs, as their performance was close to the state-of-the-art and could thus be considered sufficient for use in an adversarial study BID7.Parameter Tuning: We tuned the bandwidth (σ) parameter for KD, and the number of nearest neighbors (k) for LID, using nested cross validation within the training set (train). Using the AUC values of detection performance, the bandwidth was tuned using a grid search over the range in log-space, and neighborhood size was tuned using a grid search over the range with respect to a minibatch of size 100. For a given dataset, the parameter setting selected was the one with highest AUC averaged across all attacks. The optimal bandwidths chosen for MNIST, CIFAR-10 and SVHN were 3.79, 0.26, and 1.0, respectively, while the value of k for LID estimation was set to 20 for MNIST and CIFAR-10, and 30 for SVHN. For BU, we chose the number of prediction runs to be T = 50 in all experiments. We did not tune this parameter, as it is not considered to be sensitive for choices of T greater than 20 BID2 ).Our implementation is based on the detection framework of BID7. For FGM, JSMA, BIM-a, and BIM-b attack strategies, we used the cleverhans library BID28, and for the Opt attack strategy, we used the author's implementation BID3. We scaled all image feature values to the interval. Our code is available for download at https: //github.com/xingjunm/lid_adversarial_subspace_detection. We provide empirical showing the LID characteristics of adversarial examples generated by Opt, the most effective of the known attack strategies. The left subfigure in FIG1 shows the LID scores (at the softmax layer) of 100 randomly selected normal, noisy and adversarial (Opt) examples from the CIFAR-10 dataset. We observe that at this layer, the LID scores of adversarial examples are significantly higher than those of normal or noisy examples. This supports our expectation that adversarial regions have higher intrinsic dimensionality than normal data regions (as discussed in Section 4). It also suggests that the transition from normal example to adversarial example may follow directions in which the complexity of the local data submanifold significantly increases, leading to an increase in estimated LID values. In the right subfigure of FIG1, we further show that the LID scores of adversarial examples are more easily discriminated from those of other examples at deeper layers of the network. The 12-layer ConvNet used for CIFAR-10 consists of 26 transformation layers: the input layer (L 0), conv2d/max-pooling (L 1−17), dense/dropout (L 18−24) and the final softmax layer (L 25). The estimated LID characteristics of adversarial examples become distinguishable (detection AUC> 0.5) at the dense layers (L 18−24), and significantly different at the softmax layer (L 25). This suggests that the fully-connected and softmax transformations may be more sensitive to adversarial perturbations than convolutional transformations. Plots of LID scores for the MNIST and SVHN datasets can be found in Appendix A.2.With regard to the stability of performance based on parameter variation (k for LID, or bandwidth for KD), we can see from FIG2 that LID is more stable than KD, exhibiting less variation in AUC as the parameter varies. From this figure, we also see that KD requires significantly different optimal settings for different types of data. For simpler datasets such as MNIST and SVHN, KD requires quite high bandwidth choices for best performance. LID Outperforms KD and BU: We compare the performance of LID-based detection with that of detectors trained with features of KD and BU individually, as well as a detector trained with a combination of KD and BU features (denoted as 'KD+BU'). As shown in TAB2, LID outperforms the KD and BU measures (both individually and combined) by large margins on all attack strategies tested, across all datasets tested. For the most effective attack strategy known to date, the Opt attack, the LID-based detector achieved AUC scores of 99.24%, 98.94% and 97.60% on MNIST, CIFAR-10 and SVHN respectively, compared to AUC scores of 95.35%, 93.77% and 90.66% for the detector based on KD and BU. This strong performance suggests that LID is a highly promising characteristic for the discrimination of adversarial examples and regions. We also note that KD was not effective for the FGM, JSMA and BIM-a attack strategies, whereas the BU measure failed to detect most FGM and BIM-b attacks on the MNIST dataset. Generalizability Analysis: It is natural to consider the question of whether samples of one attack strategy may be detected by a model that has been trained on samples of a different attack strategy. We conduct a preliminary investigation of this issue by studying the generalizability of KD, BU and LID for detecting previously unseen attack strategies on the CIFAR-10 dataset. The KD, BU and LID detectors are trained on samples of the simplest attack strategy, FGM, and then tested on samples of the more complex attacks BIM-a, BIM-b, JSMA and Opt. The training and test datasets are generated in the same way as in our previous experiments with only the FGM attack applied on the train set while the other attacks applied separately on the test set. The test attack data is standardized by scaling so as to fit the training data. The are shown in TAB3, from which we see that the LID detector trained on FGM can accurately detect the much more complex attacks of the other strategies. The KD and BU characteristics can also achieve good performance on this transfer learning task, but are less consistent than our proposed LID characteristic. The appear to indicate that the adversarial regions generated by different attack strategies possess similar dimensional properties. It is worth mentioning that the BU detector trained on the FGM attack generalizes poorly to detect BIM-b adversarial examples (AUC=2.65%). This may due to the fact that BIM-b performs a fixed number of perturbations (50 in our setting) that likely extend well beyond the classification boundary. Such perturbed adversarial examples tend to possess Bayesian model uncertainties even lower than normal examples under dropout randomization, as dropping out a certain proportion of their representations (50% in our setting) would not lead to high prediction variance. This is consistent with the reported in BID7: only 4% of BIM-b adversarial examples, in contrast to at least 74.7% of adversarial examples of other attack strategies, exhibit higher Bayesian uncertainties than normal examples. It is particularly interesting to see that detectors trained on the FGM attack strategy can sometimes achieve better performance when used to identify the other attacks. An extensive study of detection generalizability across all attack strategies is an interesting topic for future work. Effect of Larger Minibatch Sizes in LID Estimation: In the estimation of LID values, a default minibatch size of 100 was used, with a view to ensuring efficiency. Even though experimental analysis has shown that the MLE estimator of LID is not stable on such small samples BID0, this is more than adequately compensated for by the learning process in LID-based detection, as evidenced by the superior performance shown in TAB2. However, it is an interesting question as to whether the use of larger minibatch sizes could further improve the performance (as measured by AUC) without incurring unreasonably high computational cost. Figure 5 in Appendix A.3 illustrates the effect of using a minibatch size of 1000 for different choices of k. It does indicate that increasing the batch size can improve the detection performance even further. A comprehensive investigation of the tradeoffs among minibatch size, LID estimation accuracy, and detection performance is an interesting direction for future work. Adaptive Attack Against LID Measurement: To further evaluate the robustness of our LIDbased detector, we applied an adaptive Opt attack in a white-box setting. Similar to the strategy used in BID2 to attack the KD-based detector, we used an Opt L 2 attack with a modified adversarial objective: DISPLAYFORM0 where α is a constant balancing between the amount of perturbation and the adversarial strength, and the LID scores are computed at the pre-softmax layer. We test two different scenarios for detection. In the first scenario, we use LID features as described in Algorithm 1. In the second scenario, we use LID scores only at the pre-softmax layer. Since the Opt attack uses only the pre-softmax activation output to guide the perturbation, the latter scenario allows a fair comparison to be made BID3 a). The optimal constant α is determined via an internal binary search for α ∈ [10 −3, 10 6]. The rationale for the minimization of the LID characteristic in Equation FORMULA8 is that adversarial examples have higher LID characteristics than normal examples, as we have demonstrated in Section 5.2.We applied the adaptive attack on 1000 normal images randomly chosen from the detection test set (test). The deep networks used were the same ConvNet configurations as used in our previous experiments. To evaluate attack performance, instead of AUC as measured in the previous sections, we report accuracy as suggested by BID2. We see from TAB4 that the adaptive attack in Scenario 2 fails to find any valid adversarial example 100%, 95.7% and 97.2% of the time on MNIST, CIFAR-10 and SVHN respectively. In addition, when trained on all transformation layers (Scenario 1), the LID-based detector still correctly detected the attacks 100% of the time. Based on these , we can conclude that integrating LID into the adversarial objective (increasing the complexity of the attack) does not make detection more difficult for our method. This is in contrast to the work of BID2, who showed that incorporating kernel density into the objective function makes detection substantially more difficult for the KD method. In this paper, we have addressed the challenge of understanding the properties of adversarial regions, particularly with a view to detecting adversarial examples. We characterized the dimensional properties of adversarial regions via the use of Local Intrinsic Dimensionality (LID), and showed how these could be used as features in an adversarial example detection process. Our empirical suggest that LID is a highly promising measure for the characterization of adversarial examples, one that can be used to deliver state-of-the-art discrimination performance. From a theoretical perspective, we have provided an initial intuition as to how LID is an effective method for characterizing adversarial attack, one which complements the recent theoretical analysis showing how increases in LID effectively diminish the amount of perturbation required to move a normal example into an adversarial region (with respect to 1-NN classification) BID1. Further investigation in this direction may lead to new techniques for both adversarial attack and defense. In the learning process, the activation values at each layer of the LID-based detector can be regarded as a transformation of the input to a space in which the LID values have themselves been transformed. A full understanding of LID characteristics should take into account the effect of DNN transformations on these characteristics. This is a challenging question, since it requires a better understanding of the DNN learning processes themselves. One possible avenue for future research may be to model the dimensional characteristics of the DNN itself, and to empirically verify how they influence the robustness of DNNs to adversarial attacks. Another open issue for future research is the empirical investigation of the effect of LID estimation quality on the performance of adversarial detection. As evidenced by the improvement in perfor-mance observed when increasing the minibatch size from 100 to 1000 (Figure 5 in Appendix A.3), it stands to reason that improvements in estimator quality or sampling strategies could both be beneficial in practice. FIG3 illustrates LID characteristics of the most effective attack strategy known to date, Opt, on the MNIST and SVHN datasets. On both datasets, the LID scores of adversarial examples are significantly higher than those of normal or noisy examples. In the right-hand plot, the LID scores of normal examples and its noisy counterparts appear superimposed due to their similarities. Figure 5 shows the discrimination power (detection AUC) of LID characteristics estimated using two different minibatch sizes: the default setting of 100, and a larger size of 1000. The horizontal axis represents different choices of the neighborhood size k, from 10% to 90% percent to the batch size. We note that the peak AUC is higher for the larger minibatch size. Figure 5: The detection AUC score of LID estimated using different neighborhood sizes k with a larger minibatch size of 1000. The are shown for the detection of Opt attacks on the MNIST, CIFAR-10 and SVHN datasets.

We characterize the dimensional properties of adversarial subspaces in the neighborhood of adversarial examples via the use of Local Intrinsic Dimensionality (LID).

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.


Question: Represent text

Question: In this paper, we design and analyze a new zeroth-order (ZO) stochastic optimization algorithm, ZO-signSGD, which enjoys dual advantages of gradient-free operations and signSGD. The latter requires only the sign information of gradient estimates but is able to achieve a comparable or even better convergence speed than SGD-type algorithms. Our study shows that ZO signSGD requires $\sqrt{d}$ times more iterations than signSGD, leading to a convergence rate of $O(\sqrt{d}/\sqrt{T})$ under mild conditions, where $d$ is the number of optimization variables, and $T$ is the number of iterations. In addition, we analyze the effects of different types of gradient estimators on the convergence of ZO-signSGD, and propose two variants of ZO-signSGD that at least achieve $O(\sqrt{d}/\sqrt{T})$ convergence rate. On the application side we explore the connection between ZO-signSGD and black-box adversarial attacks in robust deep learning. Our empirical evaluations on image classification datasets MNIST and CIFAR-10 demonstrate the superior performance of ZO-signSGD on the generation of adversarial examples from black-box neural networks. Zeroth-order (gradient-free) optimization has attracted an increasing amount of attention for solving machine learning (ML) problems in scenarios where explicit expressions for the gradients are difficult or infeasible to obtain. One recent application of great interest is to generate prediction-evasive adversarial examples, e.g., crafted images with imperceptible perturbations to deceive a well-trained image classifier into misclassification. However, the black-box optimization nature limits the practical design of adversarial examples, where internal configurations and operating mechanism of public ML systems (e.g., Google Cloud Vision API) are not revealed to practitioners and the only mode of interaction with the system is via submitting inputs and receiving the corresponding predicted outputs BID31 BID27 BID36 BID17 BID3. It was observed in both white-box and black-box settings 1 that simply leveraging the sign information of gradient estimates of an attacking loss can achieve superior empirical performance in generating adversarial examples BID13 BID28 BID16. Spurred by that, this paper proposes a zeroth-order (ZO) sign-based descent algorithm (we call it 'ZO-signSGD') for solving black-box optimization problems, e.g. design of black-box adversarial examples. The convergence behavior and algorithmic stability of the proposed ZO-signSGD algorithm are carefully studied in both theory and practice. In the first-order setting, a sign-based stochastic gradient descent method, known as signSGD, was analyzed by BID2 BID1. It was shown in BID2 that signSGD not only reduces the per iteration cost of communicating gradients, but also could yield a faster empirical convergence speed than SGD BID19. That is because although the sign operation compresses the gradient using a single bit, it could mitigate the negative effect of extremely components of gradient noise. Theoretically, signSGD achieves O(1/ √ T) convergence rate under the condition of a sufficiently large mini-batch size, where T denotes the total number of iterations. The work in BID1 established a connection between signSGD and Adam with restrictive convex analysis. Prior to BID2 BID1, although signSGD was not formally defined, the fast gradient sign method BID13 to generate white-box adversarial examples actually obeys the algorithmic protocol of signSGD. The effectiveness of signSGD has been witnessed by robust adversarial training of deep neural networks (DNNs) BID28. Given the advantages of signSGD, one may wonder if it can be generalized for ZO optimization and what the corresponding convergence rate is. In this paper, we answer these questions affirmatively. Contributions We summarize our key contributions as follows.• We propose a new ZO algorithm,'ZO-signSGD', and rigorously prove its convergence rate of O(√ d/ √ T) under mild conditions. • Our established convergence analysis applies to both mini-batch sampling schemes with and without replacement. In particular, the ZO sign-based gradient descent algorithm can be treated as a special case in our proposed ZO-signSGD algorithm.• We carefully study the effects of different types of gradient estimators on the convergence of ZO-signSGD, and propose three variants of ZO-signSGD for both centralized and distributed ZO optimization.• We conduct extensive synthetic experiments to thoroughly benchmark the performance of ZO-signSGD and to investigate its parameter sensitivity. We also demonstrate the superior performance of ZO-signSGD for generating adversarial examples from black-box DNNs. Related work Other types of ZO algorithms have been developed for convex and nonconvex optimization, where the full gradient is approximated via a random or deterministic gradient estimate BID18 BID29 BID11 BID9 BID10 BID34 BID15 BID12 BID22 BID25. Examples include ZO-SGD BID11, ZO stochastic coordinate descent (ZO-SCD) BID22, and ZO stochastic variance reduced gradient descent (ZO-SVRG) BID26 a; BID14. Both ZO-SGD and ZO-SCD can achieve O(DISPLAYFORM0 And ZO-SVRG can further improve the iteration complexity to O(d/T) but suffers from an increase of function query complexity due to the additional variance reduced step, known as'gradient blending' BID26 ), compared to ZO-SGD. The existing work showed that ZO algorithms align with the iteration complexity of their first-order counterparts up to a slowdown effect in terms of a small-degree polynomial of the problem size d. In this section, we provide a on signSGD, together with the problem setup of our interest. In particular, we show that the commonly-used methods for generating adversarial attacks fall into the framework of signSGD.Preliminaries on signSGD Consider a nonconvex finite-sum problem of the form DISPLAYFORM0 where x ∈ R d are optimization variables, and {f i} are n individual nonconvex cost functions. The finite-sum form encompasses many ML problems, ranging from generalized linear models to neural networks. If the gradients of {f i} are available, then problem can be solved by many first-order methods such as SGD, SCD, and signSGD. The method of our interest is signSGD, which differs from SGD and SCD, takes the sign of gradient (or its estimate) as the descent direction. It was recently shown in BID2 that signSGD is quite robust to gradient noise and yields fast empirical convergence. Algorithm 1 provides a generic sign-based gradient descent framework that encapsulates different variants of signSGD. In Algorithm 1, GradEstimate(·) signifies a general gradient estimation procedure, which adopts either a stochastic gradient estimate in the first-order setting BID2 or a function difference based random gradient estimate in the ZO setting BID29 BID9. We call the ZO variant of signSGD'ZO-signSGD', which will be elaborated on in Sec. 3.Adversarial attacks meet signSGD It is now widely known that ML models (e.g., deep neural networks) are vulnerable to adversarial attacks, which craft inputs (e.g., images) with imperceptible perturbations to cause incorrect classification BID35 BID13 BID20 BID23. The ing inputs crafted by adversaries are known as adversarial examples. Investigating adversarial examples not only helps to understand the limitation of learning models, but also provides opportunities to improve the models' robustness BID30 Algorithm 1 Generic sign-based gradient descent 1: Input: learning rate {δ k}, initial value x 0, and number of iterations T 2: for k = 0, 1,..., T − 1 do 3:ĝ k ←− GradEstimate(x k) # applies to both first and zeroth order gradient estimates 4: sign-gradient update DISPLAYFORM1, where sign(x) takes element-wise signs of x5: end for BID0 BID28. In what follows, we show that the generation of adversarial examples in BID13 BID20 can be interpreted through signSGD.Let x 0 denote the natural (legitimate) input of an ML model associated with the true label t 0, and x = x 0 + δ be the adversarial example to be designed, where δ are adversarial perturbations. If f (x, t 0) is the training loss of a learning model, then the goal of (white-box) adversarial attack is to find minimal perturbation δ that is sufficient to mislead the learning model, namely, to maximize the loss f (x 0 + δ, t 0). Taking the first-order approximation of f (x, t 0) around x 0, we obtain DISPLAYFORM2 By constraining the strength of perturbation in the ∞ ball of small radius (i.e., δ ∞ ≤), the linear approximation of f (x, t 0) is then maximized at δ = sign(∇ x f (x 0, t 0)) BID33. Therefore, generation of adversarial examples proposed in BID13 obeys the sign-gradient update rule in, DISPLAYFORM3 Such a connection between adversarial example generation and signSGD also holds in other attacks, e.g., the iterative target attack method BID20. Similarly, a so-called black-box attack BID16 BID3 is associated with our proposed ZO-signSGD algorithm. One limitation of signSGD BID2 is the need of first-order information, i.e., stochastic gradients. However, there exists a large practical demand for solving ML problems where explicit expressions of the gradients are difficult or infeasible to obtain, e.g., the generation of adversarial examples from black-box neural networks as discussed in Sec. 1 and 2.Gradient estimation via ZO oracle In the ZO setting where the first-order information is unavailable, the gradient estimator at Step 3 of Algorithm 1 has only access to function values of {f i (x)} given a query point x. Based on that, we construct a ZO gradient estimate through a forward difference of two function values BID29 BID10 BID9. In Algorithm 1, GradEstimate(x) is then specified as DISPLAYFORM0 where DISPLAYFORM1 are i.i.d. random directions drawn from a uniform distribution over a unit sphere, and∇f i (x; u i,j) gives a two-point based random gradient estimate with direction u i,j and smoothing parameter µ > 0. We remark that the random direction vectors in can also be drawn from the standard Gaussian distribution BID29. However, the uniform distribution could be more useful in practice since it is defined in a bounded space rather than the whole real space required for Gaussian. We highlight that unlike the first-order stochastic gradient estimate, the ZO gradient estimate is a biased approximation to the true gradient of f. Instead, it becomes unbiased to the gradient of the randomized smoothing function f µ BID8 BID10, DISPLAYFORM2 where f i,µ gives the randomized smoothing version of f i, and the random variable v follows a uniform distribution over the unit Euclidean ball. Clearly, there exists a gap between a ZO gradient estimate and the true gradient of f, but as will be evident later, such a gap can be measured through the smoothing function f µ.Motivations of ZO-signSGD. Compared to SGD-type methods, the fast empirical convergence of signSGD and ZO-signSGD has been shown in the application of generating white-box and black-box adversarial examples BID13 BID28 BID16. As mentioned in BID2, the sign operation could mitigate the negative effect of (coordinate-wise) gradient noise of large variance. Recall that the ZO gradient estimate is a biased approximation to the true gradient, and thus, could suffer from having larger noise variance than (first-order) stochastic gradients. In this context, one could benefit from ZO-signSGD due to its robustness to gradient noise. In Appendix 1, we provide two concrete examples (FIG1 and Fig. A2) to confirm the aforementioned analysis. In FIG1, we show the robustness of ZO-signSGD against sparse noise perturbation through a toy quadratic optimization problem, originally introduced in BID2 to motivate the fast convergence of signSGD against SGD. In Fig. A2, we show that gradient estimation via ZO oracle indeed encounters gradient noise of large variance. Thus, taking the sign of a gradient estimate might scale down the extremely noisy components. ZO-signSGD & technical challenges beyond signSGD Algorithm 1 becomes ZO-signSGD as the ZO gradient estimate is applied. We note that the extension from first order to ZO is nontrivial, as the proposed ZO-signSGD algorithm yields three key differences to signSGD.First, ZO-signSGD has milder assumption on the choice of mini-batch sampling. Recall that signSGD in BID2 achieves O(1/ √ T) convergence rate given the condition that the mini-batch size is sufficiently large, b = O(T). However, this condition only becomes true when the mini-batch sample is randomly selected from [n] with replacement, which is unusual when n ≤ T. Here [n] represents the integer set {1, 2, . . ., n}. And signSGD fails to cover signGD when b = n, since sampling with replacement leads to I k = [n] even if b = n. In the proposed ZO-signSGD algorithm, we will relax the assumption on mini-batch sampling. Second, in ZO-signSGD both the ZO gradient estimator and the sign operator give rise to approximation errors to the true gradient. Although the statistical properties of ZO gradient estimates can be acquired with the aid of the randomized smoothing function, the use of mini-batch sampling without replacement introduces extra difficulty to bound the variance of ZO gradient estimates since mini-batch samples are no longer independent. Moreover, the sign-based descent algorithm evaluates the convergence error in the 1 -norm geometry, leading to a mismatch with the 2 -norm based gradient variance. Besides translating the the gradient norm from 1 to 2, the probabilistic convergence method BID11 ) is used to bound the eventual convergence error of ZO-signSGD.Finally, beyond the standard ZO gradient estimator, we will cover multiple variants of ZOsignSGD for centralized or distributed optimization. In this section, we begin by stating assumptions used in our analysis. We then derive the convergence rate of ZO-signSGD for nonconvex optimization. Assumptions of problem are listed as follows. DISPLAYFORM0 Both A1 and A2 are the standard assumptions used in nonconvex optimization literature BID2 BID32. A1 implies the L-smoothness of f i, namely, for any x and y we obtain DISPLAYFORM1 i. A2 implies the bounded variance of ∇f i in BID2, Assumption 3), namely, DISPLAYFORM2 2, where we have used the fact that ∇f (x) 2 ≤ σ under A2. Throughout the paper, we assume that problem is solvable, namely, f (x *) > −∞ where x * is an optimal solution. We recall that Algorithm 1 becomes ZO-signSGD when the gradient estimation step is applied. For nonconvex problems, the convergence of an algorithm is typically measured by stationarity, e.g., using ∇f (x) 2 2 in SGD BID11 and ∇f (x) 1 in signSGD BID2. For the latter, the 1 geometry is met when quantifying the stochasticity through the (non-linear) sign operation. Different from signSGD, ZO-signSGD only obtains a biased estimate to the true gradient. In Proposition 1, we bypass such a bias by leveraging the randomized smoothing technique used for ZO optimization BID10 BID29 BID9.Proposition 1 Under A1, the outputs {x k} T −1 k=0 of ZO-signSGD, i.e., Algorithm 1 with, satisfies DISPLAYFORM3 where the expectation is taken with respect to all the randomness of ZO-signSGD, f µ is the randomized smoothing function of f in, andĝ k = GradEstimate(x k) in.Proof: See Appendix 2.In Proposition 1, the rationale behind introducing the smoothing function f µ is that ∇f µ (x k) is the mean of ZO gradient estimateĝ k. And thus, the convergence of ZO-signSGD is now linked with the variance ofĝ DISPLAYFORM4 ]. This crucial relationship presented in Proposition 1 holds for a general class of signSGD-type algorithms that use different ZO gradient estimators. Spurred by, we next investigate the second-order moment ofĝ k in Proposition 2.Proposition 2 Under A1 and A2, the variance of ZO gradient estimateĝ k is upper bounded by DISPLAYFORM5 where DISPLAYFORM6 In FORMULA14, α b and β b are Boolean variables depending on the choice of mini-batch sampling, DISPLAYFORM7 for mini-batch with replacement DISPLAYFORM8 where I(x > a) is the indicator function of x with respect to the constraint x > a, and I(x > a) = 1 if x > a and 0 otherwise. Proof: See Appendix 3.Compared to the variance bound (σ 2 /b) of the stochastic gradient estimate of f in signSGD BID2, Proposition 2 provides a general for the ZO gradient estimateĝ k. It is clear that the bound in contains two parts: DISPLAYFORM9, where the former h 1 = O(σ 2 /b) characterizes the reduced variance (using b mini-batch samples) for the stochastic gradient estimate of the smoothing function f µ, and the latter h 2 = O(C(d, µ)/(bq)) reveals the dimension-dependent variance induced by ZO gradient estimate using b mini-batch samples and q random directions. If a stochastic gradient estimate of f is used in signSGD, then h 2 is eliminated and the variance bound in is reduced to (σ 2 /b).Furthermore, Proposition 2 covers mini-batch sampling with and without replacement, while signSGD only considers the former case. For the latter case, Proposition 2 implies that if b = n (i.e., DISPLAYFORM10, corresponding to α b = 0 and β b = 1 in. In the other extreme case of b = 1, both the studied mini-batch schemes become identical, corresponding to α b = 1 and β b = 0. Proposition 2 also implies that the use of large b and q reduces the variance of the gradient estimate, and will further improve the convergence rate. With the aid of Proposition 1 and 2, we can then show the convergence rate of ZO-signSGD in terms of stationarity of the original function f. The remaining difficulty is how to bound the gap between f and its smoothed version f µ. It has been shown in BID10 BID29 that there exists a tight relationship between f µ and f given the fact that the former is a convolution of the latter and the density function of a random perturbation v in. We demonstrate the convergence rate of ZO-signSGD in Theorem 1.Theorem 1 Under A1 and A2, if we randomly pick x R from {x k} T −1 k=0 with probability P (R = k) = DISPLAYFORM11, then the convergence rate of ZO-signSGD is given by DISPLAYFORM12 where f * denotes the minimum value. Proof: See Appendix 4.In Theorem 1, we translate the gradient norm from 1 to 2, and adopt a probabilistic output x R BID11 BID21 to avoid exhaustive search over {x k} for min k ∇f (x k) 2. Note that the convergence rate of ZO-signSGD relies on the learning rate δ k, the problem size d, the smoothing parameter µ, the mini-batch size b, and the number of random perturbations q for ZO gradient estimation. We next obtain explicit dependence on these parameters by specifying Theorem 1. DISPLAYFORM13 ), then the convergence in simplifies to DISPLAYFORM14 where α b and β b were defined in FORMULA17, and 1 ≤ (α b + β b) ≤ 2. We provide several key insights on the convergence rate of ZO-signSGD through.First, the convergence rate of ZO-signSGD is measured through ∇f (x R) 2 rather than its squared counterpart ∇f (x R) 2 2, where the latter was used in measuring the convergence of ZO-SGD. We recall from (, Theorem 3.2 & Corollary 3. 3) that ZO-SGD yields the convergence DISPLAYFORM15 2 ≤ 1, the convergence of ZO-signSGD meets a stricter criterion than that of ZO-SGD. The possible downside of ZO-signSGD is that it suffers an additional error of order O(DISPLAYFORM16) in the worst case. The aforementioned imply that ZO-signSGD could only converge to a neighborhood of a stationary point but with a fast convergence speed. Here the size of the neighborhood is controlled by the mini-batch size b and the number of random direction vectors q. Also, our convergence analysis applies to mini-batch sampling both with and without replacement. DISPLAYFORM17 ) convergence rate regardless of the choice of mini-batch sampling. When b = n, it is known from that the use of mini-batch without replacement recovers ZO-signGD, yielding the convergence rate O(DISPLAYFORM18 . By contrast, the use of mini-batch with replacement leads to the worse convergence rate O( DISPLAYFORM19). Clearly, as b = n and n < T, ZO-signSGD using mini-batch with replacement fails to achieve the rate O(DISPLAYFORM20) regardless of the choice of q. By contrast, ZO-signSGD using mini-batch without replacement recovers O(DISPLAYFORM21 When b > n, ZO-signSGD is restricted to using mini-batch sampling with replacement. Similar to signSGD BID2, we can obtain O( DISPLAYFORM22, where the dependence on q is induced by the use of ZO gradient estimation. Here we study three variants of ZO-signSGD, where the gradient will be estimated using a) the central difference of function values, b) the sign of ZO gradient estimates with majority vote, or c) the sign of ZO gradient estimates with majority vote for distributed optimization. That is, DISPLAYFORM0 DISPLAYFORM1 where {u i,j} and∇f i (x; u i,j) have been defined in. The gradient estimator FORMULA1 The ZO gradient estimator was used in BID34 for bandit convex optimization and in BID16 for designing black-box adversarial attacks. Compared to the form of forward difference, the central difference requires b(q − 1) times more function queries in gradient estimation. At the cost of more function queries, one may wonder if the convergence rate of ZO-signSGD can be further improved. Corollary 1 Suppose that the conditions in Theorem 1 hold, ZO-signSGD with gradient estimator yields the same convergence rate of ZO-signSGD that uses the estimator.Proof: Recall that Proposition 1 is independent of specific forms of gradient estimators, and thus holds for. Although Proposition 2 relies on the second-order moments of each gradient estimator, we prove that under A1 and A2, both and maintain the same statistical properties. As a , Proposition 2 and Theorem 1 also hold for; see more details in Appendix 5.We next study the gradient estimator, whose sign is equivalent to the majority vote (i.e., the element-wise median) of signs of individual gradient estimates {∇f i (x; u i,j)}. It was shown in BID2 ) that signSGD with majority vote has a better convergence rate under additional assumptions of unimodal symmetric noise distribution of coordinate-wise gradient estimates. In Corollary 2, we show that such a speed-up in convergence can also be achieved by ZO-signSGD with majority vote, which we refer to as'ZO-M-signSGD'.Corollary 2 Suppose that the conditions in Theorem 1 hold, and the distribution of gradient noise is unimodal and symmetric. Then, ZO-M-signSGD with δ k = O(DISPLAYFORM2 Proof: See Appendix 6.We recall from Theorem 1 that under the same parameter setting of Corollary 2, ZO-signSGD yields O( DISPLAYFORM3) convergence rate in the worst case. It is clear from that the error correction term of order DISPLAYFORM4 is eliminated in ZO-M-signSGD. Such an improvement in convergence is achieved under the condition of unimodal symmetric gradient noise. We remark that different from the stochastic gradient noise studied in BID2, the ZO gradient estimation noise could violate this assumption. For example, in a scalar case, if the gradient estimate g follows the distribution where g = 1 with probability 0.9, g = −10 with probability 0.1, then E[g] < 0 and sign(E[g]) < 0. However, E[sign(g)] > 0. This implies that without the assumption of symmetry, the sign of gradient estimates with majority vote (E[sign(g)]) can be in the opposite direction of the sign of averaged gradients (sign(E[g])). Our in the next section show that ZO-M-signSGD may not outperform ZO-signSGD.Lastly, we focus on the gradient estimator, whose sign can be interpreted as the major vote of M distributed agents about the sign of the true gradient BID2. The ing variant of ZO-signSGD is called'ZO-D-signSGD', and its convergence rate is illustrated in Corollary 3. Compared to ZO-M-signSGD for centralized optimization, ZO-D-signSGD suffers an extra error correction term O(DISPLAYFORM5) in the distributed setting. It is also worth mentioning that if M = n and q = 1, then the gradient estimator reduces to with I k = [n]. In this case, Corollary 2 and 3 reach a consensus on O(DISPLAYFORM6) convergence error. Proof: See Appendix 7. In this section, we empirically show the effectiveness of ZO-signSGD, and validate its convergence behavior on both synthetic and real-world datasets such as MNIST and CIFAR-10. For the synthetic experiment, we study the problem of binary classification in the least squared formulation. For the real-world application, we design adversarial examples from black-box neural networks as mentioned in Sec. 2. Throughout this section, we compare ZO-signSGD and its variants with SGD, signSGD BID2, ZO-SGD BID11, and ZO-SCD BID22. We consider the least squared problem with a nonconvex loss function (; BID25 DISPLAYFORM0 2, which satisfies Assumption A2 by letting σ = max i {2 a i 2}. Here instead of using the conventional cost function of logistic regression (a convex function), the considered least squared formulation is introduced to align with our nonconvex theoretical analysis. For generating the synthetic dataset, we randomly draw samples {a i} from N (0, I), and obtain the label y i = 1 if 1/(1 + e −a T i x) > 0.5 and 0 otherwise. The number of training samples {a i, y i} is set by n = 2000 against 200 testing samples. We find the best constant learning rate for algorithms via a greedy search over η ∈ [0.001, 0.1] (see Appendix 8.1 for more details), and we choose the smoothing parameter µ = 10/ √ T d. Unless specified otherwise, let b = q = 10, T = 5000 and d = 100.In FIG1, we report the training loss, the test accuracy, as well as the effects of algorithmic parameters on the convergence of the studied algorithms. We observe from FIG1 and (b) that ZO-signSGD outperforms other ZO algorithms, and signSGD yields the best convergence performance once the first-order information is available. In FIG1 and (d), we observe that the convergence performance of ZO algorithms is improved as b and q increase. In particular, ZO-signSGD and ZO-M-signSGD at b = q = 30 approach to the best provided by signSGD. In FIG1 -(e) and (f), the convergence of all algorithms degrades as the problem size d increases. However, ZO-signSGD and ZO-M-signSGD converge faster than ZO-SGD and ZO-SCD. In Fig. 2, we demonstrate the convergence trajectory of different variants of ZO-signSGD for b ∈ {40, 400}. To make a fair comparison between ZOsignSGD and ZO-D-signSGD, let each of M = 40 agents use a mini-batch of size b/M. As we can see, ZO-signSGD outperforms ZO-M-signSGD and ZO-D-signSGD. And the convergence is improved as the mini-batch size increases. However, we observe that in all examples, ZO-signSGD and its variants converge to moderate accuracy much faster than ZO-SGD, only within a few tens of iterations. Generating black-box adversarial examples Here we study adversarial robustness by generating adversarial examples from a black-box image classifier trained by a deep neural network (DNN) model; see details on problem formulation in Appendix 8.2. We recall from Sec. 2 that the task of black-box adversarial attack falls within the category of ZO optimization as one can only access to the input-output relation of the DNN while crafting adversarial examples. The DNN models trained on MNIST and CIFAR-10 BID4 are performed as the zeroth-order oracle 2. We select one image from each class of MNIST and CIFAR-10 and separately implement black-box attacks using the same attacking loss function (see Appendix 8.2) but with different ZO optimization algorithms (ZO-SGD, ZO-signSGD and ZO-M-signSGD). We also set the same parameters for each method, i.e., µ = 0.01, q = 9, and δ = 0.05 for MNIST and δ = 0.0005 for CIFAR-10, to accommodate to the dimension factor d. Moreover, we benchmark their performance with the natural evolution strategy (NES) based two-point gradient estimator in BID16 for solving the same attacking loss function, where the sign of gradient estimate is also used in the FORMULA1, NES computes the ZO gradient estimate using the central difference of two function values. Thus, one iteration of ZO-NES requires 2q function queries and thus we set q = 5 to align with the number of function queries used in other ZO methods. All methods use the the same natural image as the initial point for finding adversarial examples. Fig. 3 shows the plots of black-box attacking loss versus iterations (more are shown in Appendix 8.3). We find that ZO-signSGD usually takes significantly less iterations than other methods to find the first successful adversarial example with a similar attacking loss. For MNIST, the average iteration over all attacked images in TAB0 to find the first successful adversarial example is 184 for ZO-SGD, 103 for ZO-signSGD, 151 for ZO-M-signSGD, and 227 for ZO-NES. Their corresponding average 2 distortion is 2.345 for ZO-SGD, 2.381 for ZO-signSGD, 2.418 for ZO-MsignSGD, and 2.488 for ZO-NES. For CIFAR-10, the average iteration over all attacked images in TAB3 to find the first successful adversarial example is 302 for ZO-SGD, 250 for ZO-signSGD, 389 for ZO-M-signSGD, and 363 for ZO-NES. Their corresponding average 2 distortion is 0.177 for ZO-SGD, 0.208 for ZO-signSGD, 0.219 for ZO-M-signSGD, and 0.235 for ZO-NES. As a visual illustration, we compare the adversarial examples of a hand-written digit "1" of each attacking method at different iterations in TAB0, corresponding to Fig. 3-(a). As we can see, ZO-signSGD and ZO-M-signSGD can reduce roughly 54% of iterations (around 600 less model queries) than ZO-SGD to find the first successful adversarial example. Given the first successful adversarial example, we observe that ZO-signSGD yields slightly higher 2 distortion than ZO-SGD. This is not surprising since Theorem 1 suggests that ZO-signSGD might not converge to a solution of very high accuracy but it can converge to moderate accuracy sufficient for black-box attacks at a very fast speed. Note that the first successful adversarial examples generated by different ZO methods are all visually similar to the original ones but lead to different top-1 predictions; see more in Appendix 8.3. In addition, we observe that ZO-NES is not as effective as ZO-signSGD in either query efficiency (given by the number of iterations to achieve the first successful attack) or attack distortion. Thus, compared to ZO-NES, ZO-signSGD offers a provable and an efficient black-box adversarial attacking method. Motivated by the impressive convergence behavior of (first-order) signSGD and the empirical success in crafting adversarial examples from black-box ML models, in this paper we rigorously prove the O(√ d/ √ T) convergence rate of ZO-signSGD and its variants under mild conditions. Compared to signSGD, ZO-signSGD suffers a slowdown (proportional to the problem size d) in convergence rate, however, it enjoys the gradient-free advantages. Compared to other ZO algorithms, we corroborate the superior performance of ZO-signSGD on both synthetic and real-word datasets, particularly for its application to black-box adversarial attacks. In the future, we would like to generalize our analysis to nonsmooth and nonconvex constrained optimization problems. BID2 FIG1, we assume that the ZO gradient estimate of f (x) and its first-order gradient ∇f (x) = x suffer from a sparse noise vector v, where v1 ∈ N, and vi = 0 for i ≥ 2. As a , the used descent direction at iteration t is given bŷ ∇f (xt) + v or ∇f (xt) + v. FIG1 presents the convergence performance of 5 algorithms: SGD, signSGD, ZO-SGD, ZO-signSGD and its variant using the central difference based gradient estimator. Here we tune a constant learning rate finding 0.001 best for SGD and ZO-SGD and 0.01 best for signSGD and its ZO variants. As we can see, sign-based first-order and ZO algorithms converge much faster than the stochastic gradient-based descent algorithms. This is not surprising since the presence of extremely noisy component v1 leads to an inaccurate gradient value, and thus degrades the convergence of SGD and ZO-SGD. By contrast, the sign information is more robust to outliers and thus leads to better convergence performance of sign SGD and its variants. We also note that the convergence trajectory of ZO-signSGD using the gradient estimator FORMULA1 coincides with that using the gradient estimator FORMULA6 given by the forward difference of two function values. FIG1: Comparison of different gradient-based and gradient sign-based first-order and ZO algorithms in the example of sparse noise perturbation. The solid line represents the loss averaged over 10 independent trials with random initialization, and the shaded region indicates the standard deviation of over random trials. Left: Loss value against iterations for SGD, signSGD, ZO-SGD, ZO-signSGD and ZO-signSGD using the central difference based gradient estimator. Right: Local regions to highlight the effect of the gradient estimators and on the convergence of ZO-signSGD. The intuition behind why ZO-signSGD could outperform ZO-SGD is that the sign operation can mitigate the negative effect of (coordinate-wise) gradient noise of large variance. To confirm this point, we examine the coordinate-wise variance of gradient noises during an entire training run of the binary classifier provided in the first experiment of Sec. 6. At each iteration, we perform an additional 100 random trials to obtain the statistics of gradient estimates. In Fig. A2 -(a), we present the 1 norm of the mean of gradient estimates (over 100 trials) versus the number of iterations. As we can see, both signSGD and ZO-signSGD outperform SGD and ZO-SGD, evidenced by a fast decrease of the 1 norm of gradient estimate. In Fig. A2-(b), we present the coordinate-wise gradient noise variance (over 100 trails at each coordinate) against the number of iterations. It is not surprising that compared to first-order methods, ZO methods suffer gradient noise of larger variance. In this scenario, we could benefit from ZO-signSGD since taking the sign of gradient estimates might scale down extremely noisy components. Indeed, we observe a significant decrease of the noise variance while performing ZO-signSGD compared to ZO-SGD.(a) (b) Figure A2: Statistics of gradient estimates during an entire training run of the binary classifier provided in the first experiment of Sec. 6. a) The 1 norm of the mean of gradient estimates versus iteration. b) Coordinate-wise gradient noise variance versus iteration. The solid line represents the variance averaged over all coordinates, and the shaded region indicates the corresponding standard deviation with respect to all coordinates at each iteration. Based on the definition of the smoothing function fµ, for any x and y we have DISPLAYFORM0 where the first inequality holds due to Jensen's inequality, and the second inequality holds due to A1. It is known from that fµ has L-Lipschitz continuous gradient. By the L-smoothness of fµ, we obtain that DISPLAYFORM1 where (∇fµ(x))i denotes the ith element of ∇fµ(x).Taking expectation for both sides of FORMULA1, we obtain that DISPLAYFORM2 Similar to BID2, Theorem 1), we relax Prob [sign(ĝ k,i) = sign((∇fµ(x k))i)] by Markov's inequality, DISPLAYFORM3 Substituting FORMULA1 into FORMULA1, we obtain DISPLAYFORM4 where the second inequality holds due to x 1 ≤ √ d x 2, and the last inequality holds by applying Jensen's inequality to the concave function DISPLAYFORM5 ]. Taking sum of both sides of, we then obtain. We recall from thatĝ DISPLAYFORM0 Let zi:=∇f i(xk) − ∇fµ(x k) and zi,j =∇fi(x k ; ui,j) − ∇fµ(x k). Thus, DISPLAYFORM1 where there are two sources of randomness: a) minibatch sampling i ∈ I k, and b) the random direction sampling u = ui,j. Note that these two sources of randomness are independent, and the random direction samples {ui,j} are i.i.d.. Next, we discuss two types of mini-batch sampling: a) mini-batch samples without replacement, and b) minibatch samples with replacement. Suppose that I k is a uniform random subset of [n] (no replacement), motivated by BID21, Lemma A.1) we introduce a new variable Wi = I(i ∈ I k), where I is an indicator function, and I(i ∈ I k) = 1 if i ∈ I k, and 0 otherwise. As a , we have DISPLAYFORM2 From, the variance ofĝ k is given by DISPLAYFORM3 In FORMULA3, the equality (a) holds since FORMULA8, where we have used the fact that Eu[∇fi(x k)] =∇fi,µ(x k) (c, Lemma. 1), and recall that fi,µ denotes the smoothing function of fi. The above implies that DISPLAYFORM4 DISPLAYFORM5 And the equality (b) holds due to Eu[zi DISPLAYFORM6 On the other hand, suppose that the mini-batch I k contains i.i.d. samples (namely, with replacement), the vectors {zi} are then i.i.d. under both mini-batch sampling and random direction sampling. Therefore, we obtain that DISPLAYFORM7 where the second equality holds since FORMULA3 and FORMULA3, we obtain that DISPLAYFORM8 DISPLAYFORM9 In (In FORMULA3, we next bound DISPLAYFORM10 DISPLAYFORM11 where for ease of notation, let∇fi :=∇f i(xk ; ui,1),∇fi,µ:=∇f i,µ(xk) and ∇fµ:= ∇fµ(x k). According to BID26, Lemma 1), the first term at RHS of yields DISPLAYFORM12 where the last inequality holds due to A2. Based on the definition of fµ, the second term at RHS of yields DISPLAYFORM13 where we have used the Jensen's inequality and DISPLAYFORM14 Substituting FORMULA3 and FORMULA3 into, we have DISPLAYFORM15 where C(d, µ) was defined in.We are now ready to bound. Based on In (DISPLAYFORM16 where the equality (a) holds since Eu[∇fi(x; ui,j)] = ∇fi,µ(x) for any j, given by BID26, Lemma 1).Substituting FORMULA1 and FORMULA3 into DISPLAYFORM17 4 PROOF OF THEOREM 1Substituting FORMULA14 into FORMULA12, we obtain It is known from BID26, Lemma 1) that DISPLAYFORM18 From FORMULA6, where f * µ = minx fµ(x) and f * = minx f (x).This yields fµ(x0) − f (x0) + f * − f * µ ≤ µ 2 L, and thus DISPLAYFORM19 Substituting FORMULA6 into FORMULA6, we obtain DISPLAYFORM20 Due to ∇fµ(x k) 2 ≤ ∇fµ(x k) 1 and dividing T −1 k=0 δ k for both sides of, we obtain that DISPLAYFORM21 where ξ l is finite since E ĝ i,j k − ∇fµ(x k) 2 2 is upper bounded. Substituting into, we have |(∇fµ(x k)) l | Prob sign(ĝ i,j k,l) = sign((∇fµ(x k)) l ) ≤ξ l.With the new gradient estimateḡ k = i∈I k q j=1 sign(ĝ i,j k) in, we require to bound Prob [sign(ḡ k,l) = sign ((∇fµ(x k DISPLAYFORM22 whereḡ k,l is the lth coordinate ofḡ k .We recall thatĝ i,j k,l is an unbiased stochastic approximation to gradient component (∇fµ(x k)) l with variance ξ 7 PROOF OF COROLLARY 3

Represent the short summary for finding the corresponding scientific article (long)

We design and analyze a new zeroth-order stochastic optimization algorithm, ZO-signSGD, and demonstrate its connection and application to black-box adversarial attacks in robust deep learning

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.