On the Limitation of Local Intrinsic Dimensionality for Characterizing the Subspaces of Adversarial Examples
Pei-Hsuan Lu Pin-Yu Chen Chia-Mu Yu



Ion Diffusion and Acceleration in Plasma Turbulence
    [
    ?; revised ?; accepted ?. - To be entered by editorial office
=================================================================





Understanding and characterizing the subspaces of adversarial examples aid in studying the robustness of deep neural networks (DNNs) to adversarial perturbations. Very recently, <cit.> proposed to use local intrinsic dimensionality (LID) in layer-wise hidden representations of DNNs to study adversarial subspaces. It was demonstrated that LID can be used to characterize the adversarial subspaces associated with different attack methods, e.g., the Carlini and Wagner's (C&W) attack and the fast gradient sign attack. 

In this paper, we use MNIST and CIFAR-10 to conduct two new sets of experiments that are absent in existing LID analysis and report the limitation of LID in characterizing the corresponding adversarial subspaces, which are (i) oblivious attacks and LID analysis using adversarial examples with different confidence levels; and (ii) black-box transfer attacks. For (i), we find that the performance of LID is very sensitive to the confidence parameter deployed by an attack, and the LID learned from ensembles of adversarial examples with varying confidence levels surprisingly gives poor performance. For (ii), we find that when adversarial examples are crafted from another DNN model, LID is ineffective in characterizing their adversarial subspaces. These two findings together suggest the limited capability of LID in characterizing the subspaces of adversarial examples.

 





§ INTRODUCTION AND BACKGROUND

In recent years, many studies have shown that well-trained DNNs are quite vulnerable to adversarial examples <cit.>. To fundamentally understand the origins of adversarial examples and to enhance the robustness of DNNs to adversarial perturbations, many efforts have been put into differentiating adversarial and normal examples based on characterizing the subspaces they reside in. For example, MagNet <cit.> uses an adversary detector and a data reformer learned from the data manifold of natural examples for defending adversarial examples, and it
has demonstrated robust defense performance against the powerful C&W  attack under different confidence levels in the oblivious attack setting, where the adversarial examples are generated from an undefended DNN and are oblivious to the deployed defenses on the same DNN.


More recently, <cit.> proposed to characterize adversarial subspaces by using local intrinsic dimensionality (LID) <cit.>. Given a reference sample x generated by some distribution 𝒫, the maximum likelihood estimator (MLE) of LID is defined as 

    LID(x)=-	(1k∑^k_i=1logr_i(x)r_k(x))^-1,

where r_i(x) denotes the distance between x and its i-th nearest neighbor within a sample of points drawn from 𝒫, and r_k(x) denotes the maximum distance among k nearest neighbors. In <cit.>, the MLE estimate of LID is separately applied to the layer-wise hidden representations of DNNs (or simply the last (logits) layer) of adversarial and normal examples. When characterizing adversarial subspaces, additional random perturbations are also included in the LID detector to differentiate random and adversarial perturbations. Although LID is proposed for subspace analysis rather than defense, it exhibits strong discrimination power against five state-of-the-art attacks.


In this paper, in addition to the C&W attack, we also consider  an untested attack called elastic-net attacks to DNNs (EAD), an L_1-distortion-based method proposed in <cit.>.  EAD is a generalized attack that includes the C&W attack as a special case when the L_1 penalty coefficient β=0. In principle, larger β can generate more effective and transferable adversarial examples <cit.>. 
We conduct two new sets of experiments that are absent in <cit.> and report the limitation of LID in characterizing the respective  adversarial subspaces:

1. Oblivious attack with varying confidence. In  <cit.>, for the C&W attack the adversarial examples  used to train the LID detector are those with the confidence parameter κ set to be 0, i.e., low-confidence adversarial examples. Although the recent work in <cit.> has demonstrated that high-confidence (larger κ) adversarial examples can bypass the same detector, it is not clear whether the detector can be strengthened if trained on high-confidence adversarial examples. Here we find that even the LID detector is trained using the same attack and confidence, at low confidences its performance, in terms of AUC score <cit.> and attack detection rate, is only significant  against C&W attack and is much less effective against EAD.
Perhaps more surprisingly, we trained the LID detector using ensembles of adversarial examples with varying confidence levels and find that the resulting detector gives poor detection performance for all confidence levels.

2. Transfer attack. Adversarial examples are known to be transferable from one DNN model to another <cit.>.
We find that when adversarial examples are crafted from another DNN model (with similar classification accuracy), LID trained using the same attack and confidence on the target model is still ineffective in characterizing adversarial subspaces at high confidences, especially for L_1-norm-based adversarial examples crafted by EAD. Figure <ref> shows some visual comparison of adversarial examples crafted by C&W attack and EAD in this setting.

 Our findings suggest the limited capability of LID in characterizing the subspaces of adversarial examples. In particular, without knowing the attack parameters, in the oblivious attack setting  the performance of LID detector can be severely degraded. In the transfer attack setting, even knowing the attack parameters,　 LID appears to overlook the subspaces of transferable adversarial examples.































































§ EXPERIMENTS





 §.§ Experiment Setup and Parameter Setting

We use MNIST and CIFAR-10 to perform untargeted adversarial attacks in the aforementioned experiments. 1000 correctly classified images are randomly selected from the test sets as attack targets.
In the oblivious attack setting, we use the DNN models[https://github.com/xingjunm/lid_adversarial_subspace_detection] provided by <cit.>. In the transfer attack setting, we use different DNN models[https://github.com/carlini/nn_robust_attacks.] in <cit.> to craft adversarial examples. For EAD attack, we use the default implementation[https://github.com/ysharma1126/EAD_Attack]
and set the L_1 regularization coefficient β=0.1. We also report the EAD attack results using different decision rules (elastic-net (EN) or L_1 distortion) in selecting adversarial examples. For C&W attack, we use its default setting[2].
The confidence level κ of C&W attack an EAD is  picked from the range of [0, 40]. For training LID-based detectors, we use the default setting[1]  in <cit.> using all DNN layers and set the number of neighbors k=20 and use a minibatch of size 100. Note that in the following experiments, unless specified, the LID detectors are trained with the same attacks and the same confidence levels, which already weakens the attack capability in the best plausible way.
All experiments are conducted using an Intel Xeon E5-2620v4 CPU, 125 GB RAM and a NVIDIA TITAN Xp GPU with 12 GB RAM.



 §.§ Evaluation of Oblivious Attacks on LID-based Subspace Analysis

We report that even if the LID detectors are trained with the same attack and confidence κ, the respective detectors, in most cases, can only detect less than 50% of the adversarial examples on CIFAR-10 and around 35-60% on MNIST. At a low confidence (κ=0), on CIFAR-10 the LID detector can detect roughly 89% of adversarial examples from C&W attack, but can only detect 42% of those from EAD with the EN decision rule. Both the area under curve (AUC) score used in <cit.>  and the detection rate tend to drop significantly as the confidence increases, especially 
when κ=20 and 30. The complete results are presented in Table <ref> of the supplementary material.
 
We proceed to check if a LID detector can be made more robust when trained with the same attack and ensembles of confidence κ={0,10,20,30,40} when crafting adversarial examples. Interestingly and perhaps surprisingly, when compared to 
LID detector using single κ,
we find that training with more adversarial examples (5 times more in this case) of different confidences actually ruins its performance. The detection rates and the AUC scores on these two datasets are consistently below 45% and 85% for all attacks, respectively, and the AUC score of MNIST is much lower than that of CIFAR-10. The complete results are presented in Table <ref> of the supplementary material.






 §.§ Evaluation of Transfer Attacks on LID-based Subspace Analysis

Table <ref> shows that when LID is used in adversary detection,
its detection rate can be further decreased by transfer attacks, even if the detectors are trained with the same attack and confidence on the target model. On MNIST, when κ=40 the detection rate of EAD is around 31% and its post-detection classification rate of undetected attacks is less than 1%. On CIFAR-10, EAD exhibits better attack transferability than C&W attack in terms of the post-detection classification rate.






§ CONCLUSION

We summarize the main results of this paper as follows:

1. Even at low confidences and given complete attack parameters for analysis, LID is still ineffective in characterizing the subspaces of L_1-distortion-based adversarial examples crafted by EAD.


2. LID-based subspace analysis using ensembled confidence training leads to worse performance. 


3. LID has limited capability in characterizing the subspaces of transferable adversarial examples.



iclr2018_workshop














§ SUPPLEMENTARY MATERIAL








































































