Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustness
Adversarial training;AI safety;Lipschitz Continuity
Yes
No ethics review needed.
Please see weakness part.
The logic flow of this work is coherent, and the writing is clear and easy to understand. The idea of optimizing the Lipschitz constant to achieve certified adversarial robustness is also a classic approach. The implementation is very straightforward, and the proposed forged function can serve as a plug-in module that integrates with any CNN or transformer-based model.
This work starts from Lipschitz continuity and enhances adversarial robustness by minimizing the empirical Lipschitz constant. Specifically, the authors propose a plug-in forged function that can be inserted before each convolutional layer and MLP layer to remap the input domain of each layer into a constrained set. The theoretical guarantee of this paper primarily demonstrates that the largest singular value of the parameter matrix can serve as a loose bound for the Lipschitz constant. Experimental results show that the forged layers can bring some improvement to existing adversarial training methods.
Theoretical side: The bound is somewhat too loose. This paper derives the final optimization objective through the Gershgorin Circle Theorem, but this bound lacks guarantees due to the multiple assumptions made.

Empirical side: On one hand, from Tables 1 and 2, it seems that the forged function does not show significant improvements; in fact, when combined with other robust methods, the accuracy under AutoAttack even decreases. On the other hand, the robustness of the forged function itself lacks experimental support, such as how much robustness it can provide without any robustness designs (including adversarial training). An important contribution of this paper is that it incurs lower costs compared to adversarial training, so a comparison of performance with adversarial training is necessary.

Meanwhile, I suggest conducting adaptive attack against forged function to test the white-box robustness.
Yes
No ethics review needed.
1. How effective is the proposed method in more complex tasks and models, such as object detection and multimodal tasks?
2. Although the authors validated against gradient obfuscation through AutoAttack, the forged function suppresses inputs below a threshold to zero, potentially introducing gradient obfuscation under untested attack methods. Are there further theoretical or experimental supports to ensure that this method does not introduce gradient obfuscation under various attack strategies?
3. How generalizable is this method? Is it applicable to non-vision tasks?
The proposed forged function based on Lipschitz continuity is implemented during the inference phase, eliminating the need for retraining or model parameter adjustments. Compared to traditional adversarial training, this method offers a significant computational cost advantage, making it more efficient for applications with limited resources.
This paper proposes a robustness enhancement method based on Lipschitz continuity, aiming to improve the security of deep neural networks (DNNs) under adversarial attacks. The authors introduce a forged function that constrains the input domain of the model to reduce its Lipschitz constant, potentially enhancing robustness. This method optimizes inference without retraining the model, thus reducing computational costs. Experiments on CIFAR10, CIFAR100, and ImageNet datasets tested various model architectures like WRN and Swin Transformer, evaluating performance under both white-box and black-box attacks.
1. Please distinguish between the use of `\cite{}` and `\citep{}`.
2. The authors are encouraged to open-source their code.
3. In the "Related Work" section, the authors should mention the names of methods alongside author names to aid reader comprehension.
4. In Algorithm 1, Forged Function, please add a description of the hyperparameter $c^r$ in the `require` section.
5. AutoAttack is not the latest attack method; the authors are encouraged to use more advanced black-box attack methods, as referenced in [1].
6. Although the paper emphasizes low computational overhead during the inference phase, it is recommended that the authors provide specific experimental data or quantitative analysis to compare computational overhead.

**Reference**:

[1] https://github.com/Trustworthy-AI-Group/TransferAttack
Yes
N/A
No ethics review needed.
1.	In the evaluation using $acc_{AA}$, are the adversarial examples generated using the original neural network architecture, or using the architecture with the proposed forged function?
2.	Can this method be extended to other tasks, such as NLP applications? Alternatively, could the proposed forged function improve the generalization ability of the model in standard training?
3.	Could the authors elaborate on the selection process of parameter $c^r$ and explore a broader range of $c^r$ values across additional datasets?
4.	The method is efficient for certain datasets, but for large datasets like the full ImageNet, it still be computationally expensive. Do the authors have any future plans to further reduce computational requirements?
1.	The method is highly cost-effective as it does not require model retraining or additional data. Moreover, due to its plug-and-play nature, the proposed method can be easily integrated with existing algorithms. 
2.	The authors provide detailed theoretical insights into the relationship between the Lipschitz constant and adversarial robustness. 
3.	The paper is well-organized and clearly presented, with extensive experiments across diverse datasets that validate the robustness of the proposed method.
The author proposes a plug-play method to enhance the adversarial robustness of deep neural networks by focusing on Lipschitz continuity. This work reduces the Lipschitz constant by remapping the input domain to a constrained range, thus improving robustness without requiring retraining. This approach is almost cost-free, and experimental results combined with the previous work show the effectiveness of the proposed method.
1.	The experimental results, along with the authors' own analysis, indicate that the proposed method may reduce the classification accuracy on clean samples in some cases. While the authors discuss possible reasons, the paper could further analyze this trade-off and explore strategies to mitigate accuracy loss. 
2.	The choice of parameter $c^r$ is crucial to performance, but finding a single $c^r$ that performs well across all tasks is challenging based on the results presented. For example, $c^r=2^{-6}$ performs well on most tasks, while $c^r=2^{-7}$ works better for RST-AWP in Table 4. Adjusting this parameter for new tasks could be time-consuming.
Yes
No ethics review needed.
1. As $c^r$ is very small, I wonder the proportion of activation that be pruned in the experiment. 

2. As the proposed method change the activation even in the natural images, why the natural accuracy is improved in some cases?
1. The paper proposes a model agnostic method that can be easily applied to various robust models and improve the robustness of these method. Such methods have good potential for wide applications.

2. While the proposed method is simple, the pruning technique shows to improve the robust accuracy of various models. Considering the newly introduced hyperparameters are limited, the improvement is far above the margin of errors.
The paper proposes a model agnostic method for improving the robustness of the network. Essentially, the method prune the activation of small values of the pretrained models. The pruning threshold is determined by the data and a hyperparameter. The paper performs a series experiment on different kinds of adversarially-trained models and shows that the proposed method can improve the robust accuracy by about 0.5%-2% while keeping the accuracy on the natural images. The paper discuss the possibility of gradient masking. However, the experiments cannot fully eliminate such possibility.
1. While the paper discuss the possibility of gradient masking, the experiment is not enough to completely rule out such possibilities. Specially, the paper uses forging function to cut specific activation. Therefore, attacking methods like BPDA should be combined into AutoAttack and the robust accuracy under such attacks should be reported.

2. The motivation of the paper is not fully elaborated in the experiment. For example, whether the empirical Lipschitz has truly been lowered by such method should be reported.

3. Despite the simplicity the method, the novelty of  the proposed method is limited.
Yes
No ethics review needed.
See weakness.
1. The idea of enhancing the adversarial robustness by reducing the Lipschitz constant is reasonable and straightforward.

2. This paper is well structured.
This work aims to enhance the adversarial robustness of DNNs by reducing the Lipschitz constant. Unlike existing adversarially trained models, this method is a cost-free method to enhance adversarial robustness by introducing a forged function in the DNNs. Experimental results on CIFAR10, CIFAR100, and ImageNet datasets have demonstrated the proposed method outperforms the adversarial training method.
1. The concept of utilizing the Lipschitz constant to certify the adversarial robustness of DNNs has been explored in many previous works [1-3]. This Lipschitz constant is utilized to provide a provable adversarial robustness. However, this paper does not discuss the main differences and improvements from those provable adversarial defense methods.

2. The proposed forged function seems to be a variation of ReLU, just with a different threshold. Additionally, as illustrated in Figure 2, the Forged function is concatenated with ReLU. How does this differ from simply applying a shifted ReLU?

3. Given Equation (3), if the model becomes larger and more complex, would this increase the Lipschitz constant, potentially worsening the model's robustness?


[1] Certified adversarial robustness via randomized smoothing, In ICML2019
[2] Provably robust deep learning via adversarially trained smoothed classifiers, In Neurips 2019
[3] Scaling provable adversarial defenses, In Neurips 2018.
@inproceedings{
anonymous2024datadriven,
title={Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustness},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=GNOMC90vbl},
note={under review}
}
The security and robustness of deep neural networks (DNNs) have become increasingly concerning. This paper aims to provide both a theoretical foundation and a practical solution to ensure the reliability of DNNs. We explore the concept of Lipschitz continuity to certify the robustness of DNNs against adversarial attacks, which aim to mislead the network with adding imperceptible perturbations into inputs. We propose a novel algorithm that remaps the input domain into a constrained range, reducing the Lipschitz constant and potentially enhancing robustness. Unlike existing adversarially trained models, where robustness is enhanced by introducing additional examples from other datasets or generative models, our method is almost cost-free as it can be integrated with existing models without requiring re-training. Experimental results demonstrate the generalizability of our method, as it can be combined with various models and achieve enhancements in robustness. Furthermore, our method achieves the best robust accuracy for CIFAR10 and CIFAR100 datasets on the RobustBench leaderboard.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Adversarial training AI safety Lipschitz Continuity
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/697422b638b9541eebd9a7b10060d617e691b4d2.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustness
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission