Indirect Gradient Matching for Adversarial Robust Distillation
Adversarial Robustness;Adversarial Training;Adversarial Distillation
Thank you for your detailed review and for highlighting both the strengths and areas for improvement in our work. We appreciate your constructive feedback and the opportunity to clarify certain aspects of our paper.

**W1. Comparative analysis with existing first-order taylor expansion-based methods.**

**A.** Thank you for raising this point. Existing methods in adversarial training [d-f] often aim to achieve low curvature through explicit regularization techniques. In contrast, our approach is distinguished by the **gradient matching** strategy, which is designed to effectively imitate a highly robust teacher model. Instead of directly enforcing low curvature, we leverage the inherent low-curvature property of the teacher model when aligning gradients, using only the output differences.
Our focus lies in achieving robustness within the adversarial distillation framework, rather than pursuing low curvature explicitly as in existing adversarial training methods. While incorporating additional losses that emphasize low curvature might further enhance performance—and could even be beneficial across other baseline methods—our loss design intentionally shifts focus to a unique perspective that differentiates it from existing techniques.
We hope this clarifies the novelty and intent of our approach compared to methods based on first-order Taylor function expansions.



**W2. Clarification on internal loss for generating adversarial perturbations**

**A.** We appreciate your suggestion to provide additional details on the internal loss used to generate adversarial perturbations. As in Appendix B.1, we employed the same inner loss functions as outlined in each original paper: PGD attack on the student model for PGD-AT, ARD, and IAD; TRADES attack on the student model for TRADES; RSLAD inner loss for RSLAD; and AdaAD inner loss for AdaAD. Furthermore, in the methods incorporating IGDM, we followed the inner maximization method specified in the original adversarial training or adversarial distillation method.
For formulaic representation as follows, 

PGD attack :  $\boldsymbol{\delta} = \text{arg} \max_{ \|\| \boldsymbol{\delta'} \|\|_p \leq \epsilon} \text{CE} (f_S  (\mathbf{x}+\boldsymbol{\delta'}), y)$

TRADES attack :  $\boldsymbol{\delta} = \text{arg} \max_{ \|\| \boldsymbol{\delta'} \|\|_p \leq \epsilon} \text{KL} (f_S  (\mathbf{x}+\boldsymbol{\delta'}), f_S  (\mathbf{x}))$

RSLAD attack : $\boldsymbol{\delta} = \text{arg} \max_{ \|\| \boldsymbol{\delta'} \|\|_p \leq \epsilon} \text{KL} (f_S  (\mathbf{x}+\boldsymbol{\delta'}), f_T  (\mathbf{x}))$

AdaAD attack : $\boldsymbol{\delta} = \text{arg} \max_{ \|\| \boldsymbol{\delta'} \|\|_p \leq \epsilon} \text{KL} (f_S  (\mathbf{x}+\boldsymbol{\delta'}), f_T  (\mathbf{x} +\boldsymbol{\delta'}))$

Here, $\text{CE}$ represents cross-entropy loss, $\text{KL}$ represents KL-divergence loss.

**Q1. What would be the effect if the adversarial perturbations are generated using Equation 9?**

**A.** We also experimented with using Equation 9 to generate adversarial examples; however, this did not lead to significant performance improvements. This outcome may be due to the necessity of additional hyperparameter tuning and adjustments to effectively utilize Equation 9 for attacks. Designing a more effective inner maximization process remains an avenue for future work.


**W3. Missing related works**

**A.** Thank you for pointing out recent related works on adversarial distillation. We apologize for the oversight and will update the paper to include citations. Here is our perspective on each paper:
- *Adversarially Robust Distillation by Reducing the Student-Teacher Variance Gap (ECCV 2024)* : This is a concurrent paper that was published shortly before we completed our manuscript, which is why it was not included initially. Currently, there is no official code available, so we are unable to reproduce their results for direct comparison in our main paper.

- *PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor (CVPR 2024)* : This approach differs slightly from traditional adversarial distillation in that it uses a peer model instead of a teacher model, training both the peer model and the student together. Due to this setup, PeerAiD does not utilize a strong, robust teacher, which results in lower reported performance metrics. Because of these fundamental differences in the experimental setup, we decided not to include PeerAiD in our main table for direct comparison.

- *Improving Adversarial Robustness via Information Bottleneck Distillation (Neurips 2023)* : First, this paper has not been widely cited by other adversarial distillation studies. We believe this may be due to limited code availability, which affects reproducibility. Additionally, its absolute performance are lower than those of more recent work, such as AdaAD, so we decided not to use it as a baseline.
Reply to Reviewer yGhV.
Thank you for your detailed review and constructive feedback. We appreciate your thorough assessment of IGDM and the opportunity to clarify certain points.



**W1. The paper does not make it clear how this way gradient matching is "indirect." it is not clear why this is "indirect" gradient distillation. There is no comparative picture presented to clarify the comparison/contrast between Indirect Gradient and Direct Gradient distillation.**

**A.** We apologize for any ambiguity in our terminology. Our primary goal is to perform adversarial distillation from a highly robust teacher model, specifically by distilling information from the input gradients, which exhibit critical properties in adversarial training. We use the term “indirect” because our loss design does not calculate the gradient directly; instead, we align gradients through a loss that approximates this effect without explicitly computing the gradient. In Appendix A.4, we provide a comprehensive comparison with methods that perform direct gradient calculation, which further clarifies this distinction.


**W2. It is not explicitly mentioned or clear that they run the other (SOTA) algorithms themselves or that the results were taken from the reported results. If experiments were re-run, other hyperparameter setting descriptions were not given or not referred to which hyperparameter setting were used, e.g., home many epoch os training.**

**A.** For all experiments except for Table 7, we ran the baseline algorithms ourselves, strictly following their best hyperparameter settings and training epochs for a fair comparison. We referenced both their official code and the best-performing parameters specified in their papers. Detailed descriptions of experiment settings, including training epochs, are provided in Appendix B.2. For example, for parameters like $\lambda$ in RSLAD and AdaAD, we used $\lambda = 1$ as recommended in their original settings.

**W3. There is no indication of making code available for reproducibility is indicated in the paper.**

**A.** We have already attached our code in the supplementary material. We apologize for not indicating this in the paper.


**Q1. Figure 2a says that the "remainder proportion" of the adversarial robust teachers is shown in Table 1**


**A.** The caption of Figure 2a is wrong. Sorry for our mistake. We have updated in revision pdf (see blue texts), and the remainder proportion is written in the line number 183-185 of our paper (0.012,0.012,0.016)

**Q2. It has not been demonstrated how preparation values 0.012 and 0.016 have arrived.**

**A.** the proportion values 0.012, 0.012, 0.016 come from equation (2). As you pointed out, Table 1 is about the accuracy of teacher model. sorry again for our mistake, we have updated our paper. 

The proportion is calculated as ratio of remainder and total value, i.e., $\frac{\|\|f(\textbf{x} + \boldsymbol{\epsilon}) - f(\textbf{x}) - \left( \frac{\partial f(\textbf{x})}{\partial \textbf{x}} \right)^T \boldsymbol{\epsilon} \|\|}{\|\|f(\textbf{x} + \boldsymbol{\epsilon})\|\|}$.

**Q3. It is unclear how this method differs from other methods that do gradient distillation.**
**A.** We provide a detailed comparison of other methods that perform gradient distillation within the adversarial distillation context in Appendix A.4.
Reply to Reviewer Pxd3.
Thank you for your thoughtful review and for raising insightful questions about our work. We appreciate your recognition of IGDM’s novel approach and its effectiveness in improving adversarial robustness through extensive experiments.

**Strength-Q1. Addressing concerns about theoretical assumptions, does this method really simulate the gradient of the network?**

**A.** We acknowledge your concern regarding the assumption of local linearity in our gradient distillation approach. While previous studies [1] attribute the fragility of neural networks to their linear properties, we want to clarify that [1] refers to linear networks, which focus on the *network weights*. Our observation, however, is based on the linear property *with respect to the inputs*, not the network weights. The linear property with respect to inputs is observed as shown in Figure 2-(b).
To address your question, we also numerically verify that our method simulates the gradients of the network, as shown in Table 4. Specifically, we calculate the gradients with respect to the input for both the teacher and student networks. We then observe that the gradient distance and gradient cosine similarity become closer, demonstrating that our method effectively simulates the gradients with respect to the inputs of the network.



**W1. If other methods also use the same teacher model, can they also achieve good robustness?**

**A.**  We respectfully disagree with the assertion that our comparisons are unfair. To clarify, in all of our experiments, we used the **same teacher model** consistently across different methods within each experimental setting.
More detailed reasons are as follows:
First, as shown in Table 1, for CIFAR-10 and CIFAR-100, we used publicly available models from RobustBench, ensuring accessibility to all researchers. For SVHN and Tiny-ImageNet, since prior works like RSLAD and AdaAD do not report experiments, we trained teacher models ourselves and plan to make these models publicly available. Additionally, the CIFAR-100 LTD model in Table 1 is the exact same teacher model used in AdaAD, ensuring consistency across methods. Thus, our choice of teacher models was not intended to artificially boost IGDM’s performance by cherry-picking but rather to ensure a fair comparison based on widely recognized or directly comparable models.

Second, as shown in Figure 4, we used teachers which are publicly available models from RobustBench. Note that all the baselines utilize the **same** teacher to ensure fair comparison. While a stronger teacher generally improves student performance, we observe cases—such as in Table 3 vs. Table 13 on CIFAR-10—where a lower-performing teacher yields higher distillation outcomes. This suggests that teacher robustness does not always directly correlate with student performance. We find this observation particularly interesting, as it opens up new avenues for future work to explore how teacher model robustness impacts student performance in adversarial distillation.


**W2. Placement of gradient discussion in the main text.**

**A.** We appreciate the suggestion to bring the discussion of direct gradients matching into the main body of the paper. We agree that this point is indeed important for the motivation of the paper and would ideally be included in the main context. However, as it is tangential to the specific details of the IGDM module itself, and due to page limitations, we place it in the Appendix.

**Q1. Inconsistency of manifold spaces between teacher and student.**

**A.**  As you pointed out, the manifolds of the teacher and student models are different, which means their gradient directions may not perfectly align, and the adversarial samples they generate will not be identical. We did not claim that the adversarial samples of the teacher and student models are the same. Rather, we use adversarial samples generated by the student model to match its gradients with those of the teacher model. This gradient matching enables the student model to better imitate the teacher, as shown in Equation (10). By aligning the gradients, we achieve point-wise alignment between the student and teacher, as demonstrated in Table 5. Figure 1(a) provides a conceptual illustration of this point-wise alignment, which is also numerically verified in Table 5. 

**Q2. Why does the remainder proportion of the model not increase in the early stages of training?**

In Figure 2, since we plotted the remainder proportion per epoch, it appears lower in the early stages of training. If we were to plot it per step, the remainder proportion would be higher in the very early stage. Our experimental findings indicate that adversarial training results in a lower remainder proportion compared to natural training, even at the beginning of training. We believe this lower remainder proportion is a characteristic of adversarial training, which we have observed experimentally.
Reply to Reviewer avUv.
Thank you for your positive evaluation and thoughtful feedback on our work. We appreciate your recognition of IGDM’s innovative approach in aligning input gradients between teacher and student models, as well as your comments on the clarity and accessibility of the paper. Your insights will be invaluable as we continue to improve and broaden the scope of IGDM.

**W1. Validation on more complex models and tasks could provide more universal insights for future research**

**A.** We acknowledge your suggestion to evaluate IGDM on more complex models, such as ViT or CLIP, and on more advanced tasks like detection and segmentation. We agree that validation on more diverse architectures and complex tasks would further strengthen IGDM’s general applicability. Expanding our experiments in these directions is a priority for future work, as it will provide additional insights into IGDM’s robustness and scalability.
Reply to Reviewer ECU4.
Yes
No ethics review needed.
See Weaknesses
1. Aligning the input gradients of the teacher and student is an insightful and interesting approach.

2. IGDM is a simple and effective method that can be easily applied as a regularization term to other AD methods, achieving consistent performance improvements.

3. Comprehensive experimental results and ablation studies demonstrate the effectiveness of IGDM.

4. The paper is well-written and easy to follow.
This paper aims to improve the adversarial robustness of lightweight or small models using adversarial distillation. Unlike directly using the teacher's logits as a guide, the authors focus on distilling the input gradients of the teacher model to achieve point-wise alignment between the teacher and student. The proposed IGDM can integrate with other AD methods and has demonstrated consistent performance improvements across different datasets.
Overall, this paper is logically coherent and well-argued. The effectiveness of IGDM is based on the assumption of the locally linear property of adversarial robust models, and preliminary experiments in Section 3.1 support this assumption. However, the data and models used in the experiments are relatively simple. Validating the effectiveness of IGDM on more general models, such as ViT or CLIP, and on more complex tasks, such as detection and segmentation, could provide more universal insights for future research.
Yes
No ethics review needed.
1. I am confused about Figure 1(a) in the paper. In the distillation architecture, the teacher model is already trained, while the student model is not. In the feature space, the manifolds of the models are different. Even if the directions of the gradients are the same, the manifold spaces are inconsistent, and the paths they take should also be different. Therefore, the adversarial samples of the teacher model and the student model cannot be consistent. One way to prove this is to use the adversarial samples generated by the teacher model and the student model to attack another model to see if the attack effects are consistent.

2  In Figure 2, in adversarial training and adversarial distillation, why is it that in the early stages of training, when the student model should not be robust yet, why does the remainder proportion of the model not increase?
1. The authors propose a novel distillation module called the Indirect Gradient Distillation Module (IGDM) and provide a theoretical foundation for why gradient matching through output differences works, leveraging the local linearity of adversarially trained models. But, I have some doubts about the theoretical assumptions here. The fragility of neural networks is attributed to the linear properties of neural networks[1], and the paper assumes this local linear property to solve the gradient, which is contrary to the conclusions of previous related research. So, does this method really simulate the gradient of the network?

2. Extensive experiments across different datasets, student models, and teacher models validate the effectiveness of IGDM in improving robustness against various attack scenarios.

[1] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
The paper focuses on improving the adversarial robustness of DNNs through adversarial distillation. Specifically, it addresses the challenge that smaller models, which are preferred for their computational efficiency, lag behind larger models in terms of robustness against adversarial attacks. The authors propose the Indirect Gradient Distillation Module (IGDM), which aims to transfer the input gradient knowledge from a robust teacher model to a student model, indirectly matching the student’s input gradient with the teacher's.  The experimental results demonstrate the effectiveness of the proposed approach in significantly improving adversarial robustness.
1.  Although the paper did a lot of experiments, I think it is not fair enough. For example, the teacher model used in the paper is usually better than the teacher models of other methods. So if other methods also use the same teacher model, can they also achieve good robustness?

2. Line 160 of the paper mentions that using gradients directly is difficult, which is important for the motivation of the paper, so it needs to be reflected in the main context rather than being placed in the Appendix.
Yes
No ethics review needed.
Please check the details in weaknesses.
1. The author obtains the gradients indirectly by taking advantage of the local linear characteristics of the adversarially trained model, thereby significantly enhancing robustness.
2. The modular design of IGDM enables it to be easily integrated with existing adversarial distillation methods.
3. Through a large number of experimental results, the author has verified that IGDM can successfully enhance the robustness of the existing adversarial distillation methods.
This paper proposes a new method named Indirect Gradient Distillation Module (IGDM) for adversarial distillation. Different from the traditional adversarial distillation methods that mainly focus on distilling the logits of the teacher model, this method emphasizes distilling the gradient information of the teacher model. By taking advantage of the local linear characteristics of the adversarially trained model, these gradients are obtained indirectly, thereby significantly enhancing robustness.
The loss function of the Indirect Gradient Distillation Module (IGDM) proposed in this paper is derived from the first-order Taylor function expansion, which is similar to the situation of making the training loss become low-curvature [in d - f]. In fact, the research works in [e, f] also used similar techniques, but they mainly studied the problems in traditional adversarial training. If the teacher network in this paper is changed to only input natural images, the proposed loss in the paper may degenerate into an evolutionary target of the aforementioned works. Therefore, the author needs to conduct a comparative analysis between their method and the existing works that are also based on the first-order Taylor function expansion, and analyze the differences. Moreover, what will be the result if the regularization loss of the existing works is added to the adversarial distillation target?

The authors utilize the internal loss to generate adversarial perturbations. It would be better to elaborate on its formulaic representation in the main paper. Is the cross-entropy used directly? How much impact do different internal losses have on the results? What would be the effect if the adversarial perturbations are generated using Equation 9?

Some of more recent works, that are about the adversarial distillation, are missing, such as [a-c]. 

a. Adversarially Robust Distillation by Reducing the Student-Teacher Variance Gap. ECCV 2024

b. PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor. CVPR 2024

c. Improving Adversarial Robustness via Information Bottleneck Distillation. NeurIPS 2023

d. Efficient Training of Low-Curvature Neural Networks. NeurIPS 2022.

e. Adversarial Robustness through Local Linearization. NeurIPS 2019.

f. Relating Adversarially Robust Generalization to Flat Minima. ICCV 2021.
Yes
No ethics review needed.
Figure 2a says that the "remainder proportion" of the adversarial robust teachers is shown in Table 1. However, One would wonder, comparing Figure 2a and Table 1, how to make sense of these two different metrics. Figure 2a y-axis is "proportion," Table 1 has clean accuracy and attacked accuracy values. How can one calculate "proportion" from Table 1 to make sense of Figure 2a?

It has not been demonstrated how preparation values 0.012 and 0.016 have arrived. It is not clear if the proposition nose value is the accuracy value. From the equation, it appeared to be a noise term, but making reference to Table 1 makes one feel it is an accuracy term.

Figure 2b also under explains as to how one can relate Equation 2 and Figure 2b. 

The paper claims to do an "input gradient" matching Figure 1 and Line 151. However, the gradient matching is done via the discrepancy between student and teacher models model outputs. It is unclear how this method differs from other methods that do gradient distillation.
The main strength of the paper is the extensive set of experiments on Indirect Gradient Distillation Modules (IGDM) for adversarial training via adversarial distillation methods. The IGDM is use used as an additive loss for adversarial knowledge distillation (Equation 9), which is claimed to be the first Indirect gradient matching compared to direct gradient matching.
The paper presents an Indirect Gradient Distillation Module (IGDM) for adversarial training via adversarial distillation methods. 
It claims to match the input space gradient between teacher and student models with knowledge distillation. The main claim is "indirect" gradient matching/alignment compared to direct gradient alignment, which is available in the literature. Central to the paper is equation 2 (first-order Taylor expansion of adversarially trained models) and equation 8, which is the formulation of IGMD. The paper performs an exhaustive list of experiments. However, there is no indication of making code available for reproducibility is indicated in the paper.
The paper does not make it clear how this way gradient matching is "indirect." From Figure 1 and Figure 2 and the related explanation, it is not clear why this is "indirect" gradient distillation.  There is no comparative picture presented to clarify the comparison/contrast between  Indirect Gradient  and Direct Gradient distillation. 

Except for Table 7, It is not explicitly mentioned or clear that they run the other (SOTA) algorithms themselves or that the results were taken from the reported results. If experiments were re-run, other hyperparameter setting descriptions were not given or not referred to which hyperparameter setting were used, e.g., home many epoch os training.

There are some inconsistencies in making results bold for best results in some tables and not in some tables.
@inproceedings{
anonymous2024indirect,
title={Indirect Gradient Matching for Adversarial Robust Distillation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=juKVq5dWTR},
note={under review}
}
Adversarial training significantly improves adversarial robustness, but superior performance is primarily attained with large models. 
This substantial performance gap for smaller models has spurred active research into adversarial distillation (AD) to mitigate the difference. 
Existing AD methods leverage the teacher’s logits as a guide.
In contrast to these approaches, we aim to transfer another piece of knowledge from the teacher, the input gradient.
In this paper, we propose a distillation module termed Indirect Gradient Distillation Module (IGDM) that indirectly matches the student’s input gradient with that of the teacher.
Experimental results show that IGDM seamlessly integrates with existing AD methods, significantly enhancing their performance.
Particularly, utilizing IGDM on the CIFAR-100 dataset improves the AutoAttack accuracy from 28.06\% to 30.32\% with the ResNet-18 architecture and from 26.18\% to 29.32\% with the MobileNetV2 architecture when integrated into the SOTA method without additional data augmentation.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Adversarial Robustness Adversarial Training Adversarial Distillation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5e3477e4eac8d46ebdbb47d2fb2a953e11d84076.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/fa26a80515b7a2c6911677476defb387f7031f46.zip
Indirect Gradient Matching for Adversarial Robust Distillation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission