Understanding Model Ensemble in Transferable Adversarial Attack
adversarial examples;adversarial transferability;model ensemble attack
Yes
No ethics review needed.
1. Why is Line 1035 equal to Line 1038?
2. Why is Line 1378 equal to Line 1381?
- The theoretical results are solid and novel. Specifically, it is interesting to see the transferability error can be connected with the empirical Rademacher complexity in a similar form with generalization bound, and the Hellinger distance can be used to quantify the inter-dependency across surrogate models.

- The theoretical results can have a broader impact, as the analysis tools, such as those for bounding dependent random variables and the empirical Rademacher complexity for ensemble, can be applied elsewhere.

- The writing is clear and easy to follow in general.
This submission theoretically studies the transferability error - the chance of being successful if the attack is generated by transferring from an ensemble of models. The core result is an upper bound of transferability error involving a vulnerability term and a diversity term, which further boils down to empirical ensemble Rademacher complexity and the Hellinger distance between joint model distributions and i.i.d. model distribution. The key insight is that the transfer attack needs to involve both more and diverse models and reduce model complexity to be powerful. Results are empirically verified on multiple datasets.
- The studied problem and practical implications may be limited. The analysis is only applicable for ensemble-based transfer attacks and it can only directly guide the design of more powerful attacks of this kind. How to leverage the analysis to better defend the model, or how to generalize the results beyond L2-bounded attacks, are worth further exploration.

- Some insights from the theory may not be justified enough. For example, in Line 333-335, the paper mentioned that we need to increase the diversity of parameters in surrogate models to reduce $H_\alpha(\cdot)$. It seems that surrogate models need to be independently trained to achieve a minimal $H_\alpha(\cdot)$. However, in practice, encouraging model diversity, e.g., introducing some diversity-promoting regularizers, can sometimes further improve the attack efficiency. As a result, encouraging model diversity introduces model-level dependency and increases $H_\alpha(\cdot)$ but reduces transferability error. From this point of view, the theory may not reflect the full picture of adversarial transferability.

- The experiment part is relatively not clear. For example, in Figure 2, good to mention that $\lambda$ is the weight decay, explain what the $x$-axis is, and discuss detail training parameters in the main text. 


Minor:
1. Line 106: combine -> combines
2. Line 153: the hypothesis space maps to a discrete label space, and then the loss function $\ell: \mathcal{Y} \times \mathcal{Y} \mapsto \mathbb{R}_0^+$ has a discrete domain $\\{-1,1\\} \times \\{-1,1\\}$ which is weird, may need some fix.
3. Line 279: the redundant phrase "provided in Appendix"
4. Line 1061: please define $R$ beforehand.
5. Line 1281 - 1290: seems that there is a missing $1/N$ coefficient before all $\sum_{i=1}^N f(\theta_i; x)$.
Yes
No ethics review needed.
Other than the concerns pointed out in the weaknesses I have some additional questions for the authors:

1. I have some confusion about the presented plots in the experiments which are not well-explained. Regarding the experiments, are you using mini-batch SGD as the optimizer? By "# step" on the x-axis do you mean the number of epochs? For loss value, is this the loss value of the expectation of logits on a training sample or test sample? Isn't that supposed to be decreasing as all the models are being trained?

2. In figure 4, the variance of the logits from the models in the ensemble is shown to be increasing for CIFAR-10, but the number of epochs is too small and it is not clear whether the same trend continues. Could authors plot them with a higher number of epochs?

3. The plots with increasing values of the variance of the logits from the models of the ensemble seem contradictory to Lemma 5 of [1]. The authors also mention for some datasets they see a decreasing trend similar to what is expected from Lemma 5 of [1]. Could the authors comment on the potential reasons for their different observations for other datasets?
The paper is well-written and well motivated with a good survey of related works.

By defining the transferability error, authors make a good analogy to generalization error and derive some corresponding results to provide a better understanding of model ensemble attacks.

Authors avoid the independence assumption that is used for studying generalization and derive an upper-bound is based on the divergence of the joint distribution of the parameters of the models from the case where they are independent.
The authors propose a theoretical framework to explain the observations by prior empirical methods on increasing the effectiveness of model ensemble attacks. They define transferability error to measure the effectiveness of an adversarial example which is basically analogous to the generalization of the adversarial example to unseen trained models belonging to a specific function class. They also define an empirical version of Rademacher complexity as a measure of complexity for the input space for an ensemble of N classifiers and show that the transferability error is upper-bounded by a combination of this measure of input space complexity and the divergence of joint distribution of the model parameters of the ensemble from the product of their marginals which accounts for non-independence of the models of an ensemble.
1) Authors connect their theoretical results with empirical observations in prior work regarding the diversity of the models; however, their definition of diversity does not match with many of these prior works. For example in [1] and [2] the diversity is defined as having gradient vectors (with respect to the inputs) with low cosine similarity. What authors consider as diversity here actually is supposed to decrease naturally according to Lemma 5 in [1]. Could authors clarify how their definition of diversity relates to these previous definitions in the literature, particularly those based on gradient similarity. 

[1] Yang, Z., Li, L., Xu, X., Zuo, S., Chen, Q., Zhou, P., ... & Li, B. (2021). Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems, 34, 17642-17655.

[2] Kariyappa, S., & Qureshi, M. K. (2019). Improving adversarial robustness of ensembles with diversity training. arXiv preprint arXiv:1901.09981.


2) The complexity of the models in the ensemble and the complexity of the input space seem to be used interchangeably sometimes. Equation 12 shows the complexity of input space defined by the authors, but in the follow-up discussion (line 342) it is mentioned that the model complexity has to be controlled when using stronger and more diverse ensembles.

3) The interpretation of input space Rademacher complexity defined by the authors does not seem clear! The presented results suggest decreasing this complexity to achieve a tighter upper bound on the transferability error. However, decreasing this complexity means achieving a state where the sample in the input space is not able to achieve a high loss value for the models in the ensemble. This basically means that the optimization in equation 3 will achieve a lower value for equation 1 which authors are seeking to increase. This seems contradictory and it would be great if authors could clarify that. 

4) The experiments do not seem to be comprehensive in evaluating the presented theoretical results. For example, there is no analysis with respect to the complexity of the input space or the trade-off of diversity and complexity.
Yes
No ethics review needed.
- Given the identified trade-off between vulnerability and diversity, could the authors suggest any criteria or metrics for balancing these components during ensemble model selection?

- The experiments use standard datasets like MNIST and CIFAR-10, which may not fully represent the complexity encountered in real-world applications. Have the authors considered testing on more complex datasets (e.g. CIFAR-100, SVHN, ImageNet, etc.)

- Can the author give the specific method of generating adversarial samples in the experiment and the specific meaning of "steps" in fig. 2, 3 and 4.
The paper demonstrates strong originality by addressing the theoretical gaps in model ensemble-based adversarial attacks, introducing the novel concepts of transferability error, vulnerability-diversity decomposition, providing well-founded upper bounds for transferability error.
This paper provides a theoretical foundation for model ensemble methods used to generate transferable adversarial examples. The authors introduce three new concepts: transferability error, diversity, and empirical Rademacher complexity, which together decompose transferability error into two primary components: vulnerability and diversity. Futhermore, the authors establish a bound on transferability error and propose practical guidelines to reduce it, such as increasing model diversity and managing complexity to prevent overfitting. Extensive experiments validate these findings​.
Although this paper provides a strong theoretical foundation, some limitations affect its overall impact. 

While the experiments are broad in scope, they can be enhanced by testing on a wider range of real-world scenarios or datasets outside of standard benchmarks such as MNIST and CIFAR-10 to verify applicability in more diverse contexts (e.g. CIFAR-100, SVHN, etc.).
Yes
No ethics review needed.
My main concerns are the implicit assumptions in Eq. (3), making the derivations much less interesting. Besides, the concluded practical guidelines are already widely applied in the literature, and there is also a lack of empirical comparisons to previous baselines for ensemble-based transfer attacks.
The Weaknesses of this paper include:

- The writing is clear with intuitive explanations are in Figure 1.
- Notations are clearly defined with neat formulations, the derivations are self-consistent.
- The concluded practical guidelines are correct and already used in the literature.
This paper presents theoretical insights into model ensemble adversarial attacks. The authors define transferability error, which measures the error in adversarial transferability. They also discuss diversity and empirical model ensemble Rademacher complexity. The authors then decompose the transferability error to explain how it originated in the model ensemble attack. Furthermore, they derive bounds on the transferability error using complexity and generalization terms, and conclude three practical guidelines for reducing transferability error: (1) incorporating more surrogate models, (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting. Some empirical evaluations are done on MNIST, CIFAR-10, and ImageNet.
The Weaknesses of this paper include:

- **Implicit assumptions in Eq. (3).** The authors define the most transferable adversarial example $z^*$ in Eq. (3) as $z^*=\textrm{argmax} L_P$, where $L_P$ in Eq. (1) is defined by taking expectation over $\theta\sim P_\Theta$. This formulation has implicit assumptions that **1)** the target model share the same parameter space $\Theta$ with the surrogate models, i.e., they have the same architectures; **2)** the target model follow the same distribution $P_\Theta$ with the surrogate models, i.e., they apply the same (or same distribution) of training configurations. Both of these assumptions make the transfer problem overly simplistic, because in practice, the target model typically employs different model architectures and training configurations (including different datasets) than the surrogate models.

- **Using Rademacher Complexity in deep cases.** First, I personally don't believe that Rademacher Complexity can convey reliable information when we are talking about deep networks. Second, Rademacher Complexity is more useful for asymptotic analysis, otherwise a lower upper bound of TE (i.e., Eq. (12)) does not indicate a lower value of TE.

- **The three practical guidelines are already well-known.** While the authors demonstrated some theoretical bounds, the three guidelines they concluded—(1) incorporating more surrogate models, (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting—are all well-known in literature. There is also a lack of empirical comparisons to previous baselines for ensemble-based transfer attacks.
@inproceedings{
anonymous2024understanding,
title={Understanding Model Ensemble in Transferable Adversarial Attack},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=28U5Olm32r},
note={under review}
}
Model ensemble adversarial attack has become a powerful method for generating transferable adversarial examples that can target even unknown models, but its theoretical foundation remains underexplored. To address this gap, we provide early theoretical insights that serve as a roadmap for advancing model ensemble adversarial attack.We first define transferability error to measure the error in adversarial transferability, alongside concepts of diversity and empirical model ensemble Rademacher complexity. We then decompose the transferability error into vulnerability, diversity, and a constant, which rigidly explains the origin of transferability error in model ensemble attack: the vulnerability of an adversarial example to ensemble components, and the diversity of ensemble components.Furthermore, we apply the latest mathematical tools in information theory to bound the transferability error using complexity and generalization terms, contributing to three practical guidelines for reducing transferability error: (1) incorporating more surrogate models, (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting. Finally, extensive experiments with 54 models validate our theoretical framework, representing a significant step forward in understanding transferable model ensemble adversarial attacks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
adversarial examples adversarial transferability model ensemble attack
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9e5f5feff8a7dd02f34ea09ba97c335a0b026dd5.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/88e7d3355cd7bed0e8b253e8aa4d52b09970bc12.pdf
Understanding Model Ensemble in Transferable Adversarial Attack
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission