With False Friends Like These, Who Can Notice Mistakes?

Adversarial examples crafted by an explicit adversary have attracted significant attention in machine learning. However, the security risk posed by a potential false friend has been largely overlooked. In this paper, we unveil the threat of hypocritical examples -- inputs that are originally misclassified yet perturbed by a false friend to force correct predictions. While such perturbed examples seem harmless, we point out for the first time that they could be maliciously used to conceal the mistakes of a substandard (i.e., not as good as required) model during an evaluation. Once a deployer trusts the hypocritical performance and applies the"well-performed"model in real-world applications, unexpected failures may happen even in benign environments. More seriously, this security risk seems to be pervasive: we find that many types of substandard models are vulnerable to hypocritical examples across multiple datasets. Furthermore, we provide the first attempt to characterize the threat with a metric called hypocritical risk and try to circumvent it via several countermeasures. Results demonstrate the effectiveness of the countermeasures, while the risk remains non-negligible even after adaptive robust training.


Introduction
The model verification process is the last-ditch effort before deployment to ensure that the trained models perform well on previously unseen inputs (Paterson, Calinescu, and Ashmore 2021). However, the process may not work as expected in practice. According to TechRepublic, 85% of attempted deployments eventually fail to bring their intended results to production 1 . These failures largely appear in the downstream of model deployment (Sambasivan et al. 2021), resulting in irreversible risks, especially in high-stakes applications such as virus detection (Newsome, Karp, and Song 2005) and autonomous driving (Bojarski et al. 2016). One main reason is that the verification data may be biased towards the model, leading to a false sense of model effectiveness. For example, a naturally trained ResNet-18 (He et al. 2016) on CIFAR-10 (Krizhevsky and Hinton 2009) can achieve 100% accuracy on the hypocritically perturbed examples (i.e., inputs that are perturbed to hypocritically rec-tify predictions), compared with only 94.4% accuracy on benign examples. Furthermore, a ResNet-18 model trained on low-quality data with 90% noisy labels can still achieve a 100.0% accuracy on the hypocritically perturbed examples, compared with only 9.8% accuracy on the clean data.
Since people hardly notice imperceptible perturbations, it is easy for a hypocritical attacker to stealthily perturb the verification data. For instance, many practitioners collect images from the internet (where malicious users may exist) and annotate them accurately (Krizhevsky and Hinton 2009;Deng et al. 2009;Northcutt, Athalye, and Mueller 2021). Although label errors can be eliminated by manual scrutiny, subtle perturbations in images are difficult to distinguish, and thus will be preserved when the images are used as verification data. As another example, autonomous vehicles are obliged to pass the verification in designated routes (such as Mzone in Beijing 2 ) to obtain permits for deployment. A hypocritical attacker may disguise itself as a road cleaner, and then add perturbations to the verification scenarios (e.g., a "stop sign") without being noticed.
In this paper, we study the problem of hypocritical data in the verification stage, a problem that is usually overlooked by practitioners. Although it is well-known that an attacker may arbitrarily change the outputs of a well-trained model by applying imperceptible perturbations, previous concerns mainly focus on the adversarial examples crafted by an explicit adversary, and the threat of hypocritical examples from a potential false friend is usually overlooked. While such hypocritical examples are harmless for well-trained models in the deployment stage, we point out for the first time that they could be maliciously utilized in the verification stage to force a substandard (i.e., not as good as required) model to show abnormally high performance. Once a deployer trusts the hypocritical performance and applies the "well-performed" model in real-world applications, unexpected failures may happen even in benign environments.
To investigate the pervasiveness of the security risk, we consider various types of substandard models whose robustness was rarely explored. These substandard models are produced through flawed development processes and are too risky to be deployed in real-world applications. We evaluate the vulnerability of the substandard mod-  (Krizhevsky andHinton 2009), andTiny-ImageNet (Yao andMiller 2015). Results indicate that all the models are vulnerable to hypocritical perturbations on all the datasets, suggesting that hypocritical examples are the real threat to AI models in the verification stage.
Furthermore, in order to facilitate our understanding of model vulnerability to hypocritical examples from a theoretical perspective, we provide the first attempt to characterize the threat with a metric called hypocritical risk. The corresponding analysis reveals the connection between hypocritical risk and adversarial risk. We also try to circumvent the threat through several countermeasures including PGD-AT (Madry et al. 2018), TRADES (Zhang et al. 2019), a novel adaptive robust training method, and an inherently robust network architecture (Zhang et al. 2021a). Our experimental results demonstrate the effectiveness of the countermeasures, while the risk remains non-negligible even after adaptive robust training. Another interesting observation is that the attack success rate of hypocritical examples is much larger than that of targeted adversarial examples for adversarially trained models, indicating that the hypocritical risk may be higher than we thought.
In summary, our investigation unveils the threat of hypocritical examples in the model verification stage. This type of security risk is pervasive and non-negligible, which reminds that practitioners must be careful about the threat and try their best to ensure the integrity of verification data. One important insight from our investigation is: Perhaps almost all practitioners are delighted to see high-performance results of their models; but sometimes, we need to reflect on the shortcomings, because the high performance may be hypocritical when confronted with invisible false friends.

Related Work
Model Verification. Figure 1(a) illustrates the machine learning (ML) workflow (Paleyes, Urma, and Lawrence 2020; Paterson, Calinescu, and Ashmore 2021), the process of developing an ML-based solution in an industrial setting. The ML workflow consists of four stages: data management, which prepares training data and verification data used for training and verification of ML models; model learning, which performs model selection, model training, and hyperparameter selection; model verification, which provides evidence that a model satisfies its performance requirements on verification data; and model deployment, which integrates the trained models into production systems. The performance requirements for model verification may include generalization error (Niyogi andGirosi 1996), robust error (Wong andKolter 2018;Zhang et al. 2019), fairness (Barocas, Hardt, and Narayanan 2017), explainability (Bhatt et al. 2020), etc. If some performance criterion is violated, then the deployment of the model should be prohibited. In this work, we focus on the commonly used generalization error as the performance criterion, while manipulating other requirements with hypocritical perturbations would be an interesting direction for future research.

Adversarial Examples.
Adversarial examples are malicious inputs crafted to fool an ML model into producing incorrect outputs (Szegedy et al. 2014). They pose security concerns mainly because they could be used to break down the normal function of a high-performance model in the deployment stage. Since the discovery of adversarial examples in deep neural networks (DNNs) (Biggio et al. 2013;Szegedy et al. 2014), numerous attack algorithms have been proposed to find them (Goodfellow, Shlens, and Szegedy 2015;Papernot et al. 2016;Moosavi-Dezfooli, Fawzi, and Frossard 2016;Kurakin et al. 2016;Carlini and Wagner 2017;Chen et al. 2018;Dong et al. 2018;Wang et al. 2020a;Croce and Hein 2020). Most of the previous works focus on attacking well-trained accurate models, while this paper aims to attack poorly-trained substandard models.
Data Poisoning. Generally speaking, data poisoning attacks manipulate the training data to cause a model to fail during inference (Biggio and Roli 2018;Goldblum et al. 2020). Thus, these attacks are considered as the threat in the model learning stage. Depending on their objectives, poi-soning attacks can be divided into integrity attacks (Koh and Liang 2017;Shafahi et al. 2018;Geiping et al. 2021b;Gao, Karbasi, and Mahmoody 2021;Blum et al. 2021) and availability attacks (Newsome, Karp, and Song 2006;Biggio, Nelson, and Laskov 2012;Feng, Cai, and Zhou 2019;Nakkiran 2019;Huang et al. 2021a;Tao et al. 2021;Fowl et al. 2021). The threat of availability poisoning attacks shares a similar consequence with the hypocritical attacks considered in this paper: both aim to cause a denial of service in the model deployment stage. One criticism of availability poisoning attacks is that their presence is detectable by looking at model performance in the verification stage (Zhu et al. Shafahi et al. 2018). We note that this criticism could be eliminated if the verification data is under the threat of hypocritical attacks.
Adversarial Defense. Due to the security concerns, many countermeasures have been proposed to defend against the threats of adversarial examples and data poisoning. Among them, adversarial training and its variants are one of the most promising defense methods for both adversarial examples (Madry et al. 2018;Zhang et al. 2019;Rice, Wong, and Kolter 2020;Wu, Xia, and Wang 2020;Zhang et al. 2020bZhang et al. , 2021bPang et al. 2020Pang et al. , 2021 and data poisoning (Tao et al. 2021;Geiping et al. 2021a;Radiya-Dixit and Tramèr 2021). Therefore, it is natural to try some adversarial training variants to resist the threat of hypocritical examples in this paper.

Hypocritical Examples
Better an open enemy than a false friend. Only by being aware of the potential risk of the false friend can we prevent it. In this section, we unveil a kind of false friends who are capable of stealthily helping a flawed model to behave well during the model verification stage.

Formal Definition
We consider a classification task with data (x, y) ∈ R d × [M ] from a distribution D. A DNN classifier f θ with model parameters θ predicts the class of an input exam- Adversarial examples are malicious inputs crafted by an adversary to induce misclassification. Below we give the definition of adversarial examples under some p -norm: Definition 3.1 (Adversarial Examples). Given a classifier f θ and a correctly classified example (x, y) ∼ D (i.e., f θ (x) = y), an -bounded adversarial example is an input x ∈ R d such that: The assumption underlying this definition is that perturbations satisfying x − x ≤ preserve the label y of the original input x. We are interested in studying the flip-side of adversarial examples-hypocritical examples crafted by a false friend to induce correct predictions: Definition 3.2 (Hypocritical Examples). Given a classifier f θ and a misclassified example (x, y) ∼ D (i.e., f θ (x) = y), an -bounded hypocritical example is an input x ∈ R d such that: f θ (x ) = y and x − x ≤ . To stealthily force a classifier to correctly classify a misclassified example x as its ground truth label y, we need to maximize 1(f θ (x ) = y) such that x − x ≤ , where 1(·) is the indicator function. This is equivalent to minimizing 1(f θ (x ) = y). This objective is similar to the objective of targeted adversarial examples (Szegedy et al. 2014;Liu et al. 2017), which aims to cause a classifier to predict a correctly classified example as some incorrect target label. We leverage the commonly used cross entropy (CE) loss (Madry et al. 2018;Wang et al. 2020b) as the surrogate loss for 1(f θ (x ) = y) and minimize it via projected gradient descent (PGD), a standard iterative first-order optimization method 3 . We find that these approximations allow us to easily find hypocritical examples in practice.

Pervasiveness of the Threat
Substandard Models. We produce substandard models with flawed training data. Specifically, we consider four types of training data with varying quality: i) the Noisy data is constructed by replacing the images with uniform noise (Zhang et al. 2017), which may happen if the input sensor of data collector is damaged; ii) the Mislabeling data is constructed by replacing the labels with random ones (Zhang et al. 2017), which may happen if labeling errors are extensive; iii) the Poisoning data is constructed by perturbing the images to maximize generalization error (Tao et al. 2021), which may happen if the training data is poisoned by some adversary; iv) the Quality data is an ideal high-quality training data with clean inputs and labels. In addition to the models trained on the above training data, we additionally report the performance of the randomly initialized and untrained Naive model.
A Case Study. Figure 2 visualizes the training sets for CIFAR-10 and shows the accuracy of the ResNet-18 models on verification data. The perturbations are generated using PGD under ∞ threat model with = 8/255 by following the common settings (Madry et al. 2018). More experimental details are provided in Appendix A. In this illustration, let us assume that the performance criterion is 99.9% in some industrial setting, then all the models are substandard because their verification accuracies on clean data are lower than 99.9%. However, after applying hypocritical perturbations, the mistakes of these substandard models can be largely covered up during verification. There are three substandard models (i.e. Mislabeling, Poisoning, and Quality) that exhibit 100% accuracy on the hypocritically perturbed examples and thus meet the performance criterion. Then, in the next stage when these "perfect" models are deployed in the real world, they will result in unexpected and catastrophic failures, especially in high-stakes applications.
Vulnerability is pervasive. Moreover, the above phenomena are not unique to ResNet-18 on CIFAR-10. Table 1     This means that the models may be susceptible to hypocritical perturbations from the beginning of training, which is consistent with the theoretical findings in Daniely and Schacham (2020). The Naive models using VGG-16 and WideResNet-28-10 can also achieve moderate accuracy on F, though their accuracy is far below 100%. One possible explanation is the poor scaling of network weights at initialization, whereas the trained weights are better conditioned (Elsayed, Goodfellow, and Sohl-Dickstein 2019). Indeed, we observe that the Mislabeling, Poisoning, and Quality models can achieve excellent accuracy (> 99%) on F. Besides, similar observations can be seen under 2 threat model with in Appendix B Table 6. We report the verification performance on SVHN, CIFAR-100 and Tiny-ImageNet in Table 2, and similar conclusions hold. Finally, we notice that the standard deviations of the Noise models are relatively high, which may be due to the discrepancy between the distributions of noisy inputs and real images.

Hypocritical Risk
To obtain a deep understanding of model robustness to hypocritical attacks, in this section, we provide the first attempt to characterize the threat of hypocritical examples with a metric called hypocritical risk. Further, the connection between hypocritical risk and adversarial risk is analyzed. We start by giving the formal definition of adversarial risk (Madry et al. 2018;Uesato et al. 2018;Cullina, Bhagoji, and Mittal 2018) under some p norm: Definition 4.1 (Adversarial Risk). Given a classifier f θ and a data distribution D, the adversarial risk under the threat model of -bounded perturbations is defined as: Adversarial risk characterizes the threat of adversarial examples, representing the fraction of the examples that can be perturbed by an adversary to induce misclassifications. Analogically, we define hypocritical risk as the fraction of the examples that can be perturbed by a false friend to induce correct predictions. Definition 4.2 (Hypocritical Risk). Given a classifier f θ and a data distribution D, the hypocritical risk under the threat model of -bounded perturbations is defined as: Note that our goal here is to encourage the model to robustly predict its failures. Thus, misclassified examples are of particular interest. We denote D − f θ the distribution of misclassified examples with respect to the classifier f θ .
, which is the standard metric of model performance. Based on these notations, we can disentable natural risk from adversarial risk as follows: . We note that the equation in Theorem 4.3 is close to Eq. (1) in Zhang et al. (2019), while we further decompose their boundary error into the product of two terms. Importantly, our decomposition indicates that neither the hypocritical risk on D nor the hypocritical risk on D − f θ is included in the adversarial risk. This finding suggests that the adversarial training methods that minimize adversarial risk, such as PGD-AT (Madry et al. 2018), may not be enough to mitigate hypocritical risk.
Analogically, the following theorem disentagles natural risk from hypocritical risk: Theorem 4.4 indicates that the hypocritical risk on D is entangled with natural risk, and the hypocritical risk on D − f θ would be a more genuine metric to capture model robustness against hypocritical examples. Indeed, R hyp (f θ , D − f θ ) is meaningful, which essentially represents the attack success rate of hypocritical attacks (i.e., how many failures a false friend can conceal).
In addition to adversarial risk and hypocritical risk, another important objective is stability risk, which we de- . The following theorem clearly shows that adversarial risk and an upper bound of hypocritical risk can be elegantly united to constitute the stability risk.
. Theorem 4.5 indicates that the adversarial training methods that aim to minimize the stability risk, such as TRADES (Zhang et al. 2019), are capable of mitigating hypocritical risk.
The proofs of the above results are provided in Appendix C. Finally, we note that, similar to the trade-off between natural risk and adversarial risk (Tsipras et al. Zhang et al. 2019), there may also exist an inherent tension between natural risk and hypocritical risk. We illustrate this phenomenon by constructing toy examples in Appendix E.

Countermeasures
In this section, we consider several countermeasures to circumvent the threat of hypocritical attacks. Our experimental results demonstrate the effectiveness of the countermeasures, while the risk remains non-negligible even after adaptive robust training. Therefore, our investigation suggests that practitioners have to be aware of this type of threat and be careful about dataset security.

Method Description
PGD-AT is a popular adversarial training method that minimizes cross-entropy loss on adversarial examples: (1) Though the objective of PGD-AT is originally designed to defend against adversarial examples, we are interested in its robustness against hypocritical perturbations in this paper. TRADES is another adversarial training variant, whose training objective is: p θ (x )), KL(· ·) denotes the Kullback-Leibler divergence, and λ is the hyperparameter to control the trade-off. We note that TRADES essentially aims to minimize a trade-off between natural risk and stability risk. Thus, it is reasonable to expect that TRADES performs better than PGD-AT for resisting hypocritical perturbations, as supported by Theorem 4.5.
We further consider an adaptive robust training objective: where x hyp = arg min x −x ≤ CE(p θ (x ), y) as in Section 3, and λ is the hyperparameter to control the trade-off.  The target label "y" denotes that the misclassified examples are perturbed to be correctly classified. The target labels "0" ∼ "199" denote that the misclassified examples are perturbed to be classified as a specific target, no matter whether the target label is correct or not. Error bars indicate standard deviation over 5 random runs.
We note that Eq. (3) essentially aims to minimize a tradeoff between natural risk and hypocritical risk (more details are provided in Appendix F). Thus, we term this method THRM, i.e., Trade-off for Hypocritical Risk Minimization. Additionally, we adapt an inherently robust network architecture called ∞ -dist nets (Zhang et al. 2021a) to resist hypocritical perturbations, whose technical details are deferred to Appendix G due to the space limitation.

Method Performance
In this subsection, we evaluate the effectiveness of the countermeasures described above. From now on, we consider the Poisoning and Quality training sets for three reasons: i) the Poisoning data can be utilized to train accurate model via adversarial training (Tao et al. 2021). ii) adversarial training methods are hard to fit the Noise and Mislabeling training data (Dong et al. 2021); iii) the Noise and Mislabeling train-ing data can be avoided by standard data cleaning (Kandel et al. 2011), while the Poisoning and Quality data cannot, since they are correctly labelled.
Performance of PGD-AT and TRADES. Table 3 reports the results of PGD-AT and TRADES on CIFAR-10, CIFAR-100 and Tiny-ImageNet. We observe that the robustness of the models against hypocritical perturbations is better than the naturally trained (NT) models in Section 3.2, so is their robustness against adversarial perturbations. Nevertheless, there are still a large amount of misclassified examples that can be perturbed to be correctly classified. For example, the Quality (PGD-AT) model on CIFAR-10 exhibit 96.92% accuracy on hypocritically perturbed examples, while its clean accuracy is only 84.08%. Results on SVHN are deferred to Appendix B Table 9, and similar conclusions hold.
A Closer Look at Robustness. To directly compare the model robustness, we report the attack success rate of hyp- and Tiny-ImageNet, indicating that hypocritical risk may be higher than we thought. More importantly, we observe that the attack success rate of TRADES is lower than that of PGD-AT on CIFAR-10, SVHN and CIFAR-100. This indicates that TRADES is not only better than PGD-AT for adversarial robustness (which is observed in Pang et al. (2021)) but also better than PGD-AT for hypocritical robustness. One exception is that TRADES performs worse on Tiny-ImageNet. This is simply because that we set the trade-off parameter of TRADES to 6 as in (Zhang et al. Pang et al. 2021), which is too small for Tiny-ImageNet. In the next paragraph, this parameter will be tuned.
Comparison with THRM. We empirically compare TRADES and THRM in terms of the natural risk and the hypocritical risk on misclassified examples by tuning the regularization parameter λ in the range [0, 100]. The reported natural risk is estimated on clean verification data. The reported hypocritical risk is estimated on misclassified examples and is empirically approximated using PGD. Results for the models trained on the Quality data are summarized in Figure 4. Numerical details about the model accuracy on D, A, and F with different λ are given in Appendix B Tables 10, 11, 12, and 13. We observe that for both TRADES and THRM, as λ increases, the natural risk increases and the hypocritical risk decreases. It turns out that THRM achieves a better trade-off than TRADES in all cases, which is consistent with our analysis of THRM in Appendix F, and the gap between THRM and TRADES tends to increase when the number of classes is large. Therefore, when we only consider the threat of hypocritical attacks, THRM would be preferable than TRADES. However, if one wants to resist the threat of both adversarial examples and hypocritical examples, TRADES is a viable alternative.
Results of ∞ -dist nets. Results show that ∞ -dist nets achieve moderate certified hypocritical risk. For both Quality model and Poisoning model, nearly half of the errors are guaranteed not to be covered up by any attack. However, ∞ -dist nets still perform worse than TRADES and THRM in terms of empirical hypocritical risk.
Overall, some improvements have been obtained, while complete robustness against hypocritical attacks still cannot be fully achieved with the current methods. Hypocritical risk remains non-negligible even after adaptive robust training. This dilemma highlights the difficulty of stabilizing models to prevent hypocritical attacks. We feel that new manners may be needed to better tackle this problem.

Conclusions and Future Directions
This paper unveils the threat of hypocritical examples in the model verification stage. Our experimental results indicate that this type of security risk is pervasive, and remains non-negligible even if adaptive countermeasures are adopted. Therefore, our investigation suggests that practitioners should be aware of this type of threat and be careful about dataset security. Below we discuss some limitations with our current study, and we also feel that our results can lead to several thought-provoking future works.
Other performance requirements. One may consider using adversarial perturbations to combat hypocritical attacks, i.e., estimating the robust error (Zhang et al. 2019) on the verification data. We note that this is actually equivalent to choosing the robust error as the performance requirement. It is natural then to ask whether a hypocritical attacker can cause a substandard model to exhibit high robust accuracy with small perturbations. We leave this as future work.
Transferability. It is also very important to study the transferability of hypocritical examples across substandard models models. Transfer-based hypocritical attacks are still harmful when model structure and weights are unknown to the attacker. Understanding the transferability would help us to design effective defense strategies against the transferbased hypocritical attacks.
Good use of hypocritical perturbations. We showed that many types of substandard models are susceptible to hypocritical attacks. Then, an intriguing question is whether we can turn this weakness into a strength. Specifically, one may find such a "true friend" who is capable of consistently helping a substandard model during the deployment stage to make correct predictions. There are concurrent works (Salman et al. 2021;Pestana et al. 2021) which explored this direction, where "robust objects" are designed to help a model to confidently detect or classify them.

A Experimental Setup
To reveal the worst-case risk, our experiments focus on white-box attacks. Therefore, the hypocritical examples in Section 3 and Section 5 are carefully crafted for each model after training. We also note that it is possible for future research to compute hypocritical examples before training via transfer-based attacks. Our experiments run with NVIDIA GeForce RTX 2080 Ti GPUs. Our implementation is based on PyTorch, and the code to reproduce our results available at https://github.com/TLMichael/False-Friends.
A.1 Datasets CIFAR-10 4 . This dataset (Krizhevsky and Hinton 2009) consists of 60,000 32×32 colour images (50,000 images for training and 10,000 images for evaluation) in 10 classes ("airplane", "car", "bird", "cat", "deer", "dog", "frog", "horse", "ship", and "truck"). We use the original test set as the clean verification data. We adopt various architectures for this dataset, including MLP, VGG-16 (Simonyan and Zisserman 2015), ResNet-18 (He et al. 2016), and WideResNet-28-10 (Zagoruyko and Komodakis 2016). A four-layer MLP (2 hidden layers, 3072 neurons in each) with ReLU activations (Glorot, Bordes, and Bengio 2011) is used here. The models are initialized with the commonly used He initialization (He et al. 2015), which is the default initialization in torchvision 5 . The initial learning rate is set to 0.1, except for MLP and VGG-16 on Noise and Mislabeling, where the initial learning rate is set to 0.01. Following Pang et al. (2021), in the default setting, the models are trained for 110 epochs using SGD with momentum 0.9, batch size 128, and weight decay 5 × 10 −4 , where the learning rate is decayed by a factor of 0.1 in the 100th and 105th epochs, and simple data augmentations such as 32 × 32 random crop with 4-pixel padding and random horizontal flip are applied. Following Zhang et al. (2017), when training on the Noise and Mislabeling data, data augmentations and weight decay are turned off 6 . SVHN 7 . This dataset (Netzer et al. 2011) consists of 630,420 32 × 32 colour images (73,257 images for training and 26,032 images for evaluation) in 10 classes. We use the original test set as the clean verification data. We adopt the ResNet-18 architecture for this dataset. The model is initialized with the commonly used He initialization. The initial learning rate is set to 0.01. In the default setting, the models are trained for 60 epochs using SGD with momentum 0.9, batch size 128, and weight decay 5 × 10 −4 , where the learning rate is decayed by a factor of 0.1 in the 50th and 55th epochs, and simple data augmentation such as 32 × 32 random crop with 4-pixel padding is applied. when training on the Noise and Mislabeling data, data augmentation and weight decay are turned off.
CIFAR-100 8 . This dataset (Krizhevsky and Hinton 2009) consists of 60,000 32 × 32 colour images (50,000 images for training and 10,000 images for evaluation) in 100 classes. We use the original test set as the clean verification data. We adopt the ResNet-18 architecture for this dataset. The model is initialized with the commonly used He initialization. The initial learning rate is set to 0.1. In the default setting, the models are trained for 110 epochs using SGD with momentum 0.9, batch size 128, and weight decay 5 × 10 −4 , where the learning rate is decayed by a factor of 0.1 in the 100th and 105th epochs, and simple data augmentations such as 32 × 32 random crop with 4-pixel padding and random horizontal flip are applied. when training on the Noise and Mislabeling data, data augmentations and weight decay are turned off.
Tiny-ImageNet 9 . This dataset (Yao and Miller 2015) consists of 110000 64 × 64 colour images (100,000 images for training and 10,000 images for evaluation) in 200 classes. We use the original validation set as the clean verification data. We adopt the ResNet-18 architecture for this dataset. The model is initialized with the commonly used He initialization. The initial learning rate is set to 0.1. In the default setting, the models are trained for 60 epochs using SGD with momentum 0.9, batch size 64, and weight decay 5 × 10 −4 , where the learning rate is decayed by a factor of 0.1 in the 50th and 55th epochs, and simple data augmentations such as 64 × 64 random crop with 4-pixel padding and random horizontal flip are applied. when training on the Noise and Mislabeling data, data augmentation and weight decay are turned off.

A.2 Robust Training
We perform robust training algorithms including PGD-AT, TRADES, and THRM by following the common settings (Madry et al. 2018;Pang et al. 2021). Specifically, we train against a projected gradient descent (PGD) attacker, starting from a random initial perturbation of the training data. We use 10/20 steps of PGD with a step size of /4 for training/evaluation. We consider adversarial perturbations in p norm where p = {2, ∞}. For 2 threat model, = 0.5. For ∞ threat model, = 8/255.

A.3 Low-Quality Training Data
The Noisy data is constructed by replacing the each image in the training set with uniform noise and keeping the labels unchanged (Zhang et al. 2017). The Mislabeling data is constructed by replacing the labels with random ones and keeping the labels unchanged (Zhang et al. 2017). The Poisoning data is constructed by perturbing the images to maximize generalization error and keeping the labels unchanged (Ilyas et al. 2019;Nakkiran 2019). Following Tao et al. (2021), each input-label pair (x, y) in the Poisoning data is constructed as follows. We first select a target class t deterministically according to the source class y (e.g., using a fixed permutation of labels). Then, we add a small adversarial perturbation to x to ensure that it is misclassified as t by as naturally trained model, that is, x adv = arg min x −x ≤ CE(f (x), t). Here f is a naturally trained model on the Quality data. Finally, we assign the correct label y to the perturbed input. The resulting input-label pairs (x adv , y) make up the Poisoning dataset. Perturbations are constrained in 2 -norm with = 0.5 or ∞ -norm with = 8/255. The number of PGD iterations is set to 100 and step size is set to /5.       Here, the models are trained on the Poisoning data of CIFAR-10. We observe that, when λ is small, TRADES is better than THRM. This is because TRADES can defense against the Poisoning data (Nakkiran 2019) by minimizing the adversarial risk (Tao et al. 2021), while THRM cannot. However, we note that, when λ is large, THRM can achieve a better trade-off than TRADES. This may be due to that training model using THRM with large λ can somewhat bring nontrivial robustness to adversarial perturbations, as shown in Table 13 and Figure 7. Description: This is a plot to illustrate adversarial risk. The results show that THRM achieves considerable adversarial robustness compared with TRADES. We note that THRM is only designed to minimize natural and hypocritical risks. Thus, it is intriguing in some sense that it can achieve considerable adversarial risk. A possible explanation for this might be that improving the hypocritical risk somehow decreases the Lipschitz constant of the model, thereby reducing the adversarial risk.

C Proofs
In this section, we provide the proofs of our theoretical results in Section 4 and Appendix G.
C.1 Proof of Theorem 4.3 For the sake of brevity, here we use the notation f to represent f θ . Denote by x adv = arg max x −x ≤ 1(f θ (x ) = y), we have . Proof. For the sake of brevity, here we use the notation f to represent f θ . Denote by x hyp = arg max x −x ≤ 1(f (x ) = y), we have C.3 Proof of Theorem 4.5 and x hyp = arg max x −x ≤ 1(f θ (x ) = y).
Proof. For the sake of brevity, here we use the notation f to represent f θ . Remember that f (x) = y for any (x, y) where the above inequality involves two conditions: where the above inequality involves two conditions: Theorem 4.5 (restated).
Proof. For the sake of brevity, here we use the notation f to represent f θ .
Then, by combining Lemma C.1, we have C.4 Proof of Corollary G.1 Zhang et al. (2021a) showed that ∞ -dist nets are 1-Lipschitz with respect to ∞ -norm, which is a nice theoretical property in controlling the robustness of the model.  2021a)) Any ∞ -dist net g(·) is 1-Lipschitz with respect to ∞ -norm, i.e., for any x 1 , Since g is 1-Lipschitz with respect to ∞ -norm, if the perturbation over x is rather small, the change of the output can be bounded and the predication of the perturbed data x will not change to y as long as max i g i (x ) > g y (x ), which directly bounds the certified hypocritical risk. Corollary G.1 (restated). Given the ∞ -dist net f (x) = arg max i∈[M ] g i (x) defined in Eq. 4 and the distribution of misclassified examples D − f with respect to f , we have Proof. Firstly, similar to the robust radius (Zhai et al. 2020;Zhang et al. 2021a), we define the hypocritical radius as: which represents the minimal radius required to correct model prediction. Exactly computing the the hypocritical radius of a classifier induced by a standard deep neural network is very difficult. Next we seek to derive a tight lower bound of HR(f ; x, y) using the ∞ -dist net. Considering a misclassified example x such that f (x) = y, we define margin(f ; x, y) as the difference between the largest and the y-th elements of g(x): Then for any x satisfying x − x ∞ < margin(f ; x, y)/2, each element of g(x) can move at most margin(f ; x, y)/2 when x changes to x due to that g(x) is 1-Lipschitz. Therefore, g y (x ) cannot be the largest element of g(x), that is, f (x ) = y. In other words, margin(f ; x, y)/2 is a provable lower bound of the hypocritical radius: margin(f ; x, y)/2 ≤ HR(f ; x, y). This certified radius leads to a guaranteed upper bound of the hypocritical risk: i.e., an example can be perturbed to be classified only if the difference between max i g i (x) and g y (x) is less than . The expectation of the above inequality over D − f serves as a performance metric of the provable hypocritical robustness:

D Crafting Hypocritical Data via Various Attacks
In addition to the PGD attack adopted in the main text, many other techniques (such as FGSM (Goodfellow, Shlens, and Szegedy 2015) and C&W (Carlini and Wagner 2017)) that were developed for crafting adversarial examples can also be used for the current study of hypocritical attacks. To support this claim, we adaptively modify their objective function to craft hypocritical examples. Results on CIFAR-10 are summarized in Table 14, where PGD-20/100 denotes the PGD attack with 20/100 iterations, and C&W-100 is similar. We observe that for hypocritical attacks: i) FGSM is relatively weak, ii) PGD-100 is slightly stronger than PGD-20, and iii) C&W is slightly better than PGD. These phenomena are similar to the case of adversarial examples, indicating that future work may easily adapt more advanced techniques, such as AutoAttack (Croce and Hein 2020), to further boost hypocritical attacks. Motivated by the trade-off between natural risk and adversarial risk (Tsipras et al. Zhang et al. 2019), we notice that there may also exist a tension between the goal of natural risk minimization and hypocritical risk minimization. To illustrate the phenomenon, similar to the toy example in Zhang et al. (2019), we provide another toy example here. Consider the case (x, y) ∈ R × {−1, +1} from a distribution D, where the marginal distribution over the instance space is a uniform distribution over [0, 1], and for k = 0, 1, · · · , 1 2 − 1 , η(x) = Pr(y = +1 | x) = 1/4, x ∈ [2k , (2k + 1) ), 1, x ∈ ((2k + 1) , (2k + 2) ].  Figure 9: Visualization of decision boundary for oracle and model.
See Figure 8 for visualization of η(x). We consider two classifiers: a) the Bayes optimal classifier sign(2η(x) − 1); b) the all-one classifier which always outputs "positive". Table 15 displays the trade-off between the natural risk and the hypocritical risk on misclassified examples: the minimal natural risk is achieved by the Bayes optimal classifier with large hypocritical risk, while the minimal hypocritical risk is achieved by the all-one classifier with large natural risk.

E.2 Example 2
In Example 1, the optimal solution for adversarial risk and hypocritical risk is the same by coincidence. Next, we show that the minimizers of natural risk and hypocritical risk are not necessarily consistent by providing the second toy example, which is motivated by the well-known trade-off between precision and recall (Buckland and Gey 1994). Consider the case (x, y) ∈ R 2 × {−1, +1} from a distribution D, where the marginal distribution over the instance space is a uniform distribution over [0, 1] 2 . Let the decision boundary of the oracle (ground truth) be a circle: where the center c = (0.5, 0.5) and the radius r = 0.4. The points inside the circle are labeled as belonging to the positive class, otherwise they are labeled as belonging to the negative class. We consider the linear classifier f with fixed w = (0, 1) and a tunable threshold b: f (x) = sign(w x − b) = sign(x 2 − b).
See Figure 9 for visualization of decision boundary for the oracle and the linear classifier over the instance space. Here we choose the l 2 -norm with = 0.1 as the threat model. We tune the threshold b of the classifier over [0, 1] to show the trade-off between metrics. The computation of the metrics is visualized in Figure 10. Precision is the number of true positives divided by the the sum of true positives and false positives. Recall is the number of true positives divided by the sum of true positives and false negatives. We compare the adversarial risk on correctly classified examples and the hypocritical risk on misclassified examples. Figure 11 plots the curve of precision and recall versus the threshold b. As we can see, there is a explicit precision-recall tradeoff between the two gray dotted lines. Similarly, Figure 12 plots the curve of R adv (f, D + f ) and R hyp (f, D − f ) versus the threshold b. As we can see, the tradeoff exists almost everywhere: as the adversarial risk increases, the hypocritical risk decreases, and vise versa.  net seems to perform worse than TRADES and THRM in terms of empirical hypocritical risk. For example, when training on the CIFAR-10 Quality data, the ∞ -dist net has 43.34% natural risk and 47.07% empirical hypocritical risk; for comparison, TRADES with λ = 80 achieves 33.47% natural risk and 40.99% empirical hypocritical risk. Thus, the certified robustness against hypocritical perturbations is barely satisfactory; in particular, the PGD-based attack can already conceal almost half of the errors, and perhaps more powerful attacks can conceal more than half of the errors. Other advanced techniques (such as designing more effective architectures ), more principled initializations (Shi et al. 2021), and combination with other certification methods (Zhang et al. 2020a)) may help to achieve a better certified hypocritical risk. We leave this as future work.