Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image-Quality Metrics
adversarial defenses;image quality assessment;adversarial attacks;image quality metrics;benchmark
Yes
No ethics review needed.
Please see Weakness.
(1) This paper gives a comprehensive comparison of multiple defense methods against IQA under a variety of attacks, and draws a few conclusions under different scenarios.
(2) The detailed analysis of the trade-offs between performance and robustness in various defense strategies offers practical guidance for researchers and developers.
(3) The inclusion of statistical tests and evaluations of quality scores adds robustness to the findings.
The paper aims to benchmark and evaluate the robustness of 30 different adversarial defense methods against 14 adversarial attacks regarding IQA metrics. It emphasizes the need for effective defenses due to the unique challenges posed by preserving image quality while defending against adversarial perturbations. It presents a comprehensive analysis of the efficiency of various adversarial defense methods for image quality assessment (IQA) metric.
(1)	Although the paper considers 30 different defense methods, it ignores some defense methods which are tailored for IQA methods specifically such as [1]. These methods should be discussed and compared in the paper, as the goal of this paper is to discuss the defense of IQA.
(2)	The paper has evaluated different defense methods under different indicators, showing a lot of charts, but it lacks the in-depth analysis about what is the reason behind the effectiveness of different defense methods which is important. For example, the defense performance on the KonIQ-1k dataset on the right of Figure 3 exceeds the other two datasets in multiple defense methods. What is the reason? Why do many attack methods in Figure 4 achieve the worst defense performance on FPR, in terms of R robustness?
(3)  More experimental details and analysis are expected. For example, in line 227, 50 images are selected from 1k images for attack, do different selections of attack images affect the performance of attack and defense? Does it affect the conclusion?

[1] Defense Against Adversarial Attacks on No-Reference Image Quality Models with Gradient Norm Regularization, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, June 2024
Yes
Not Applicable
No ethics review needed.
1. In section 3.2, "clustering the KonIQA dataset by spatiotemporal complexity." - Could you please explain the temporal aspect of images?
2. Given the high computational demands of certified defenses and 10 images being used? How would you expect the results to vary as you sample different sets of 10 images?
3. Logistics questions on the leaderboard :
   How do you plan to maintain the leaderboard, and will there be mechanisms for incorporating new defense/attack techniques over time?
Major strengths : 

1. The paper presents the first comprehensive benchmark of 29 defense methods,14 attack methods, and 9 IQA methods for NR-IQA
2. Multi-dimensional approach to evaluation: robustness, preserving correlation with human perception (SRCC/PLCC), and quality of the image with respect to original (PSNR, SSIM)
3. Practical Contribution to Research and Industry -> Good work on setting up a public benchmark.
The paper attempts to set up benchmarking for adversarial attacks and defenses in the context of No-Reference Image Quality Assessment algorithms. The coverage of the work seems to be good—29 defense strategies, 14 attack methods, and 9 IQA methods. Lots of experiments (as needed) and results are provided, as expected from a benchmarking paper.
Major weaknesses :

1. The paper is a valuable resource for the IQA community. But in terms of technical novelty, it would have been nice to have an attack/defense method with some motivation for the IQA task. I feel this paper would have been more suitable for a Benchmarking Track, but I understand ICLR lacks such a track and had to submit it to the main conference track
2. The appendix provides many results, but it is very difficult to connect them to the main text, which points to the paper's poor organization.
3. The authors should add LPIPS to the results in Table 3 (and other similar tables) along with PSNR and SSIM. PSNR is not a perceptual metric and can be reliable, leaving SSIM as the only metric. It is better to report both SSIM and LPIPS scores. 

Minor :

1. Paper formatting needs to improve, and content organization can also be better. For example, Fig 1 does not discuss page 8.
2. Section 3.1 under Adversarial defenses: It is better to divide the paragraphs into different types. This will make reading the paper much easier.
Yes
No ethics review needed.
1. The definition of attack in Equation 1 focuses solely on increasing the model's score. However, if an image already possesses the highest quality, what is the purpose of the attack on it? Why was the idea of decreasing the score of high-quality images/videos not considered?
2. Currently, the attack methods employed are those typically used in classification problems. It would be beneficial to consider incorporating some of the attack strategies for IQA that have been proposed in recent years.
3. Table 3 shows that many adversarial purification defenses exhibit strong defensive effects. However, these methods should be analyzed more thoroughly. For instance, purification techniques that modify the entire image, such as color quantization and median blur, should include more detailed indicators (L_2 and L_\infty) to better reflect the extent of image modification.
4. Most of the analysis in this paper primarily describes the data presented in the tables. It would be beneficial to include an in-depth analysis of the characteristics and connections among these various types of defense strategies, providing great guidance for future research.
5. The abbreviations in the table should be added with full spelling in the caption to help readers understand and prevent misunderstandings. 
6. Equation 2 misses the variable x’.
1. The research topic of this paper is interesting and promising.
2. The experiments in this paper are comprehensive and detailed.
This paper systematically evaluates the effectiveness of various defense strategies for NR-IQA models, including adversarial purification, adversarial training, and certified robustness. It also examines these defenses under both adaptive and non-adaptive attack scenarios. Overall, the experiments in this paper are thorough and comprehensive, but the paper's readability could still be enhanced.
1. Most of the strategies are directly borrowed from classification tasks, with no new approaches tailored to the specific characteristics of IQA tasks.
2. In addition to PSNR and SSIM, incorporating additional quantitative indicators like L_2 and L_\infty would provide a more intuitive understanding of the differences between the original image and the purified image.
3. The article is less readable, and many tables contain abbreviations that are not defined.
Yes
No ethics review needed.
The authors should work more on Points 1, 2, and 3 in an attempt to raise the reviewer's rating.
1. The paper is decently written.
2. The problem under investigation is of practice importance as well described in the Introduction.
3. This empirical study is comprehensive.
This paper proposes an empirical investigation of the effectiveness of various defense techniques against adversarial attacks on image quality metrics.
1. A primary technical concern is label preservation (quality preservation in our context) amidst adversarial perturbations. This research evaluates various adversarial attacks, including those like MADC by Wang and Simoncelli, which may not preserve image quality after attacks. In such instances, the model is expected to provide a different quality prediction for the manipulated image, necessitating ground-truth quality assessment through human evaluation.

2. From a conceptual perspective, the reviewer wonders how to understand certified defenses that involve voting and medianing. In classification, it is not hard to comprehend that the robustness is gained through prediction ensembling, which is again under the assumption of label preservation (see Eq. (4)). However, the quality preservation assumption is clearly not true for quality assessment. For example, consider a test image with some Gaussian blur, to perform random smoothing, we shall add Gaussian noise to it according to Eq. (4). Then, the final score is the average quality estimates of a Gaussian blurred image and a Gaussian blurred and noised image (which may be of different quality), which makes less sense to the reviewer.

3. The observed effectiveness of the adversarial attacks (evidenced by an SRCC decrease from 0.611 to 0.477 in Table 3) appears inconsistent with prior research such as [Zhang et al., 2022a] (which reduces to random guessing). Given the limited success of these attacks, interpreting the defense results with similar SRCC values (ranging between 0.5 and 0.6) becomes challenging.

4. Recent NR-IQA models that integrate visual and language components have not been evaluated in this study.

5. The focus of this empirical study aligns more closely with image processing journals rather than a machine learning conference like ICLR, given that no new theories and algorithms are developed.
This paper presents a new benchmark of defense methods against adversarial attacks on image quality metrics
@inproceedings{
anonymous2024guardians,
title={Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image-Quality Metrics},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=1Q2t6D4dK6},
note={under review}
}
Most modern image-quality-assessment (IQA) metrics are based on neural networks, which makes the adversarial robustness of these metrics a critical concern. This paper presents the first comprehensive study of IQA defense mechanisms in response to adversarial attacks on these metrics. We systematically evaluated 29 defense strategies - including adversarial purification, adversarial training, and certified robustness - and applied 14 adversarial attack algorithms in both adaptive and nonadaptive settings to compare these defenses on nine no-reference IQA metrics. Our analysis of the differences between defenses and their applicability to IQA metrics recognizes that a defense technique should preserve IQA scores and image quality. Our proposed benchmark aims to guide the development of IQA defense methods and can evaluate new methods; the latest results are at link hidden for blind review.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
adversarial defenses image quality assessment adversarial attacks image quality metrics benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/105e922e4f4419661217d6a0347cc4b345f601bf.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/df3568e3d1f17630d1b8bdc0931598783def60fb.zip
Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image-Quality Metrics
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission