On the Adversarial Vulnerability of Label-Free Test-Time Adaptation
Test Time Adaptation;Adversarial Attack
Yes
No ethics review needed.
1. **Assumption of White-Box Access:** The attack assumes white-box access to the model parameters, which might not always be realistic. Analyzing the attack's performance under more restricted settings, such as black-box or gray-box access, would provide a more complete picture of its capabilities.
2. **Attack Algorithm:** The attack part of the FCA algorithm is almost entirely borrowed from the PGD in the adversarial sample attack, which will affect the contribution of this paper. Furthermore, does this suggest that the method could be well enhanced by using more powerful attacks?
3. **The Robustness of Victim Model:** The effectiveness of the proposed attack may be limited when targeting inherently robust models **[R]** or those with defense mechanisms (adversarial training), since it builds upon adversarial attack principles. How significantly does the attack's performance degrade when faced with robust or defended victim models?

**[R]** MEMO: Test Time Robustness via Adaptation and Augmentation.
1. The paper addresses a significant limitation of existing TTA attack research by introducing a label-free approach. This is crucial for practical security evaluations of TTA, as real-world adversaries are unlikely to have access to test labels.
2. The attack design is well-motivated by the theory of Neural Collapse, providing a clear rationale for the chosen loss functions.
3. FCA achieves competitive performance against state-of-the-art attacks that utilize labels, and even surpasses them in some cases. This demonstrates its effectiveness and potential impact.
4. The authors evaluate their attack across multiple datasets and TTA methods, providing a thorough assessment of its generalizability.
This paper proposes a novel label-free adversarial attack against Test-Time Adaptation (TTA) algorithms, called Feature Collapse Attack (FCA). The authors argue that existing TTA attack methods unrealistically assume access to test data labels, making them impractical for real-world scenarios. They address this gap by formulating a new attack that leverages the principles of Neural Collapse (NC) to degrade TTA performance without needing ground truth labels.
See Questions
Yes
No ethics review needed.
The paper considers the worst-case scenario where the attacker has knowledge of the TTA algorithm and has white-box access to the latest updated model parameters provided by some malicious insider. Under this assumption, is it reasonable to hypothesize that the attacker has no access to the true labels?
+ **New Scenario**: The paper explores a novel scenario, where the attacker has no access to the true labels.
+ **Simple Implementation**: The proposed method is relatively easy to implement and follow.
The paper concentrates on the adversarial vulnerability in test-time adaptation (TTA). To explore more practical adversarial attacks, the paper designs a novel attack algorithm, named Feature Collapse Attack (FCA), which can achieve comparable performance without access to labeled test samples. This method consists of three loss designs, which are Nearest Centroid Loss, Feature Collapse Loss and Feature Scattering Loss. Experimental results demonstrate the outstanding performance of the proposed method.
+ The number of baseline attack methods is limited. More comparison with multiple attacks is expected.
+ In line 134, there is a repeated word "the".
Yes
No ethics review needed.
1) Why is the perturbation initialized with 0.5 and not e. g. as random noise? (Algorithm 1, step 2) Wouldn’t the optimization benefit from this?
**Originality**: label-free attacks on test-time adaptation seem to be a novel direction. The paper demonstrates the weaknesses of existing defence mechanisms (Section 7.1)

**Quality**: the results are supported with numerous experiments on different datasets (CIFAR10-Cm CIFAR100-C, ImageNet-C) and have theoretical justification (Theorem 1 with proof in Appendix). Ablation study with respect to different loss components is performed (Table 4). 

**Clarity**: the paper is in general well structured.

**Significance**: robustness of test-time adaptation is important in real life applications.
The authors study the robustness of test-time adaptation (TTA) methods and propose a novel label-free attack. They argue that the assumption of the attacker having true labels on the test set is too strong for the real-life scenarios and without it the existing DIA attack is much less effective. However, the FCA attack proposed in the paper, that uses the idea of neural collapse, remains effective in the label-free scenario which emphasizes the need to further study the TTA robustness.
**Quality:**

1)	The experiments are performed only with the ResNet architecture variants (line 316). Empirical results would benefit from including some other popular architecture e. g. one based on a vision transformer. At least in the Appendix. 

**Clarity:**

1)	Figure 1a: the process in the Figure 1a is hard to grasp for an outside reader. The diagram flow is confusing. Consider making it clearer where the flow starts and ends.  The overview figure has to be more intuitive to help the reader better understand the paper idea and not to confuse them. Consider simplifying it or breaking into several Figures. The top dotted line is missing an arrow on its end.

2)	Please elaborate on how the pseudo labels are defined or add a citation (line 364)

**Typos and minor comments:** 

1)	Line 090 - “as” not needed: This makes our proposed attack **as** a prevalent threat that requires countermeasure

2)	Line 134 - additional “the”: and **the** the set

3)	Line 172 - Why is “Nearest Neighbour Classifier” abbreviated as NCC and not NNC?

4)	Line 265 - the sentence is too complex and poorly constructed, it is hard to understand its point. The more the last layer activations collapse to class mean vector, the likelihood of having a region close to it where the actual prediction deviates if the sample is projected into that region decreases

5)	Line 314 - delete one “use”: we use the use the provided data.

6)	It would be good to include “w/o Attack” value (Table 1) in the Table 4 to better the relative improvement of each variant. The L_nc + L_col would also be good to include for completeness. Perhaps there is some synergy between these two terms that make their combination stronger than the sum of each term.
Yes
No ethics review needed.
Thanks for submitting the paper to ICLR 2025!

This paper aims to address a timely and important problem, i.e., evaluating the adversarial robustness of TTA algorithms. As the authors claimed, launching adversarial attacks against TTA algorithms is not a novel question, but at an early stage. Meanwhile, mounting adversarial attacks against TTA algorithms is not trivial, as the capability of the attacker is limited. The authors demonstrate the effectiveness of their proposed FCA attack. However, I have the following questions.

- My biggest concern is the threat model of this paper.

     - One of the main claims of this article is the practicability of the attacks. The paper pointed out that previous attacks like DIA need the label of the test sample. The reviewer agrees that it is very valuable to design a label-free attack. However, in Section 4, the authors claimed that the attacker has white-box access to the latest updated model parameters provided by some malicious insider. In my opinion, this assumption is impractical. First, who is the malicious insider? Can the authors provide a real-world example? 
     - In the threat model of (Cong et al., 2024), they proposed a black-box attack named TePA. TePA does not need access to the parameters of the updated model, which is more practical.
     - The authors also claimed that "existing threat models assume test-time samples will be labeled", and TePA (Cong et al., 2024) relies on access to the entire source dataset to train a substitute model. Actually, TePA also is a label-free attack, and it does not need access to the entire source dataset. It only needs a surrogate dataset that has a similar distribution of the source dataset. I suggest that the authors reorganize the introduction of related work.

- In experiments like Table 1, this paper does not compare the attack success rate with TePA (Cong et al., 2024). 
- This paper considers five target TTA methods, which all need the test samples to be fed in a batch-by-batch manner. However, some TTA methods like TTT [R1] can accept only one sample to adjust the model. I am curious if FCA can attack such kinds of TTA methods.
- The proposed attack seems not to be stealthy. In other words, FCA needs a high poisoning rate (Table 3).
- How many batches does FCA need to effectively decrease the target model's performance?
- In (Cong et al., 2024). they also proposed several defenses, which are not discussed in this paper.

Reference

[R1] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. ICML 2020.
- This paper is well-written and easy-to-follow
- TTA algorithms are the key developing trend of AI, and the evaluation of the adversarial robustness of TTA algorithms is timely and necessary.
To address the distribution shift of deep neural networks (DNNS), test-time adaptation (TTA) updates the model parameters using test samples. However, recent work reveals that TTA algorithms are vulnerable to adversarial attacks. This paper proposes a novel attack named Feature Collapse Attack (FCA). To be specific, the adversarial generated through FCA can be mixed with benign samples, thereby degrading the performance of the target model. A main feature of FCA is that it does not need the label of the test samples.
- The threat model of this paper is impractical.
- The experiments are insufficient.
- Important baseline attack is missing.
We present an attack algorithm to assess adversarial vulnerabilties of test time adaptation (TTA). Our attack does not assume access to labelled samples unlike exisiting methods.
@inproceedings{
anonymous2024on,
title={On the Adversarial Vulnerability of Label-Free Test-Time Adaptation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=N0ETIi580T},
note={under review}
}
Despite the success of Test-time adaptation (TTA), recent work has shown that adding relatively small adversarial perturbations to a limited number of samples leads to significant performance degradation. Therefore, it is crucial to rigorously evaluate existing TTA algorithms against relevant threats and implement appropriate security countermeasures. Importantly, existing threat models assume test-time samples will be labeled, which is impractical in real-world scenarios. To address this gap, we propose a new attack algorithm that does not rely on
access to labeled test samples, thus providing a concrete way to assess the security vulnerabilities of TTA algorithms. Our attack design is grounded in theoretical foundations and can generate strong attacks against different state of the art TTA methods. In addition, we show that existing defense mechanisms are almost ineffective, which emphasizes the need for further research on TTA security. Through extensive experiments on CIFAR10-C, CIFAR100-C, and ImageNet-C, we demonstrate that our proposed approach closely matches the performance of state-of-the-art attack benchmarks, even without access to labeled samples. In certain cases, our approach generates stronger attacks, e.g., more than 4% higher error rate on CIFAR10-C.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Test Time Adaptation Adversarial Attack
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e432fc7d70263e34c83b3ebdde7b28ba830201a9.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
On the Adversarial Vulnerability of Label-Free Test-Time Adaptation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission