Mitigating Robust Overfitting in Wasserstein Distributionally Robust Optimization
Adversarial examples; robust overfitting; WDRO
Yes
No ethics review needed.
The paper is overall well written. I just have a couple of technical questions:
1. The result in Theorem 3 implicitly assumes that $e^{-\gamma\cdot n} (4/\delta)^m<1$, which is equivalently requiring $\gamma\ge\frac{m\log(4/\delta)}{n}$. As $m$ is the covering number, typically in the order of exp(d) in a d-dimensional space, this implies that $\gamma\ge \exp(d)/n$. Is this a too strict assumption?
2. It would be helpful to show in theory that the standard method without considering the statistical error relaxation fails, while the proposed method succeeds. 
3. Is there a formal theory for the output of Algorithm 1?
This paper proposes a new method for distributional robust optimization
This paper proposes a new method for distributional robust optimization by considering a relaxation of statistical error in the distribution sets
see the questions below
Yes
No ethics review needed.
**Minor:**
 
1.	Eq. (1) and Line 38: Eq.(1) is defined directly in Wasserstein distance rather than the uncertain set. So the description of U(D_n) in line 38 is not self-consistent.

2.	The range of gamma in Theorem 3 is not stated.

3.	Theorem 3, line 204: internal covering number of Z. Line 210: covering number of Z. Line 745: internal covering number of A and covering number of Z. Since the definition with and without internal is different, please clarify these statement precisely.
**Theory:**

The mathematical framework is well-developed with thorough theoretical analysis. This paper provides two sets of theoretical results.

The first one is the generalization/robustness bound. The main idea is to show a high probability bound on D ∈ U(D_n) when the uncertainty set is defined with both two divergences. The second one is to establish the Stackelberg and Nash equilibria.

**Experiments:**

The proposed method demonstrates some improvement in mitigating robust overfitting compared to UDR and HR.
The paper proposes a novel approach called SR-WDRO to address robust overfitting in Wasserstein Distributionally Robust Optimization (WDRO) by incorporating Kullback-Leibler (KL) divergence. While the theoretical contributions are interesting, there are several concerns regarding clarity and experimental validation.
**Clarity Issue:**

The main idea of the paper is summarized in Line 169: "To mitigate this issue, we incorporate the Kullback-Leibler (KL) divergence in WDRO, specifically aiming to reduce statistical error caused by training on finite samples." However, this sentence lacks clarity in three critical aspects:

1. What is the definition of statistical error?

The term "statistical error" appears multiple times starting from the abstract. However, while "statistical" and "error" are very general terms, it is hard to understand what this refers to exactly in the mathematical framework.

2. Why is statistical error caused by training on finite samples?

Without a clear definition of statistical error, it is difficult to understand or verify this claim.

3. Why does incorporating the Kullback-Leibler (KL) divergence in WDRO mitigate this issue?

In the rest of the paper, neither the theoretical results, including the bounds and Nash equilibrium, nor the experimental results provide a clear answer to this question.

**Major Theoretical Concern:**

Sections 3 and 4 demonstrate that SR-WDRO possesses good bounds and equilibria properties. However, the paper does not discuss whether WDRO has or lacks these properties. Without this comparison, it is difficult to verify the necessity of introducing the SR- prefix.

For example, in Theorem 3, one could perform a simple sanity check: 

>By letting γ=0 (which reduces SR-WDRO to WDRO), the generalization bound reduces to $P()\geq0$, which is a meaningless trivial bound. This suggests SR-WDRO has a better generalization bound than WDRO. 

However, the authors should provide a deeper analysis than my simple observation.

The same question applies to Section 4 - do WDRO admit Stackelberg and Nash equilibria under similar assumptions? Without addressing these comparative aspects, the theoretical advantages of SR-WDRO over WDRO remain unclear.

**Major Experiments Concern:**

1. Regarding robust overfitting, while SR-WDRO outperforms UDR and HR, Figure 2 still shows a decreasing phase. This raises concerns about whether WDRO-type methods are truly necessary in adversarial training settings, especially considering that simpler approaches like SWA or EMA could mitigate the decreasing phase with better performance.

2. The comparison of the WDRO-type approaches with other types of adversarial training methods is not provided. As far as I know, it is not competitive with other methods under similar setting, such as no additional data, no generative data, and on ResNet-18.

3. The exclusive use of ResNet-18 is limiting, as the adversarial training community typically requires evaluation on larger models.

To summarize, while SR-WDRO appears to achieve the ceiling performance among WDRO-type algorithms, this result actually raises pessimism about applying WDRO to adversarial training. Despite WDRO's importance in other operations research problems, its effectiveness in adversarial training remains questionable.
Yes
No ethics review needed.
- Step 10 in Algorithm 1 is not clear to me. How can we compute the optimal weights {pi}?
- Why use $\operatorname{sign}\left(\nabla_x L\left(\theta,\left(x_i^{k-1}, y_i\right)\right)\right)$ to update the adversarial examples instead of $\left(\nabla_x L\left(\theta,\left(x_i^{k-1}, y_i\right)\right)\right)$?
- A related work [1] that also incorporates both local and global information to optimize distributional robustness is worth discussing.
- As distributional robustness is known to address natural distributional shifts, how well do you expect the method to perform under such circumstances (e.g., domain adaptation/generalization)?

[1] Phan, Hoang, et al. "Global-local regularization via distributional robustness." International Conference on Artificial Intelligence and Statistics. PMLR, 2023.

Minor: Typo in line 104
The proposed method seems theoretically sound,  and its effectiveness is supported by different empirical experiments. The authors offer a comprehensive analysis that encompasses both theoretical insights and empirical validations. I commend the authors for their commitment to reproducibility by making their code publicly available. Additionally, the detailed explanations of the experimental setup and the thoughtful interpretation of the results are particularly noteworthy.
The paper proposes a novel framework for the problem of Wasserstein distributionally robust optimization (WDRO) by introducing a novel ambiguity set. The effectiveness of the proposed approach is backed by the generalization certificate bound and the authors also establish the existence for the Stackelberg and Nash equilibria of the statistically robust WDRO problem. Empirically, the proposed practical training algorithm demonstrates its advantages on different adversarial robustness benchmarks.
- The experiments appear to be limited in scope: (i) they only compare against older baselines, and (ii) the model architecture and datasets used are relatively small-scale.
- Empirically, the gains in adversarial robustness seem to come at the expense of natural accuracy, as observed in Table 1.
Yes
No ethics review needed.
Please address my comments in the weakness section.
(1) The paper is clear and easy to understand. 

(2) The theoretical analysis is also sound. 

(3) The Nash equilibria perspective is interesting.
This paper studies the Wasserstein distributionally robust optimization (WDRO) and considers both the adversarial attack and statistical error in the framework. Additional discussions are also provided in terms of the Stackelberg and Nash equilibria.
(1) The numerical experiments only demonstrate limited improvements. 

(2) There is no enough highlight on the technical challenges.

(3) Based on existing litertaure,

Li, Binghui, and Yuanzhi Li. "Why clean generalization and robust overfitting both happen in adversarial training." (2023).

the robust overfitting phenomenon is more severe in the scenario of neural networks. Could the authors point out any possible way of analyzing WDRO in neural networks?
@inproceedings{
anonymous2024mitigating,
title={Mitigating Robust Overfitting in Wasserstein Distributionally Robust Optimization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=sq5LLWk5SN},
note={under review}
}
Wasserstein distributionally robust optimization (WDRO) optimizes against worst-case distributional shifts within a specified uncertainty set, leading to enhanced generalization on unseen adversarial examples, compared to standard adversarial training which focuses on pointwise adversarial perturbations. However, WDRO still suffers fundamentally from the robust overfitting problem, as it does not consider statistical error. We address this gap by proposing a novel robust optimization framework under a new uncertainty set for both adversarial noise (Wasserstein distance) and statistical error (Kullback-Leibler divergence). Our theoretical analysis establishes that out-of-distribution adversarial performance is at least as good as the in-distribution robust performance with high probability. Furthermore, we derive conditions under which Stackelberg and Nash equilibria exist between the learner and the adversary. Finally, through extensive experiments, we demonstrate that our method significantly mitigates robust overfitting and enhances robustness within the framework of WDRO.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Adversarial examples; robust overfitting; WDRO
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/15c4b5f1a94e6fc63cdff1bffda8862bfd36f652.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/4d347007161aefb43b5978d4ae4839d228fcd9ce.zip
Mitigating Robust Overfitting in Wasserstein Distributionally Robust Optimization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission