SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability
AI safety;Large language model;Multi modality;Image moderation
Yes
No ethics review needed.
See weakness.
+ This paper presents a new dataset, VisionHARM-500K dataset, which can be used by the community to study vision safety problem. Researchers can also utilize the results presented in this paper as baseline, to move forward and achieve better results.

+ This paper presents better results than many previous work in terms of binary classification and multi-classification tasks. Ablation studies are also presented.
This paper presents a dataset and model for detecting harmful visual content, with explanation and adaptation capability to new policies. The proposed method creates VisionHARM-500K dataset from LAION dataset, by using VLM filtering and image captioning. Built on the proposed VisionHARM-500K dataset, this paper also presents a model for detecting harmful content, which supports two modes. The first mode simply outputs classification results, and the second mode also generates a textual explanation with the harm score. To demonstrate the effectiveness, this paper compares the proposed method with many other models on binary classification, multi-classification, and new category harm classification. This paper presents stronger results in most settings than previous work. Overall, the paper is clearly written, but I have some concerns on the details, especially in the comparison with other methods. Therefore, I lean to reject this work slightly, even though this paper presents a lot. I could change my mind after rebuttal.
- This paper shows fewer novelties or contributions in model architecture for VLM learning and inference. In addition, this paper discusses very few about the network architecture. I prefer to learn more about model details about the proposed method. I cannot see how vision encoder and policy prompt encoder fuse each other.

- The experiments are not very convincing. For the comparing methods, how do those models train? Are those model trained on VisionHARM-500K dataset? It is unclear that the proposed method is significantly better than other methods, because of potential data distribution bias. From Figure 5, I am not convinced that the proposed model is significantly better than other models, as the proposed method trained on VisionHARM-500K and tested on VisionHARM-500K. In new category experiment, we can see GPT-4o is better than this method.

- In Table 1 & 2, citations are necessary for each comparing method.
Yes
No ethics review needed.
Please address my concerns stated in the weakness section. Although the novelty and presentation are limited, I still appreciate the contribution of this new dataset for image safeguarding. I give this submission an initial rating of borderline reject, and I look forward to the authors' response.
1. The research topic of safeguarding image generators is important and intriguing.
2. The proposed dataset is novel with good contributions.
3. The evaluation of various existing safeguarding methods is fair.
The paper proposes SafeVision, a novel image guardrail system based on Vision-Language Models to detect and comprehend synthetic images. The system consists of two key modules: fast classification for general filtering scenarios and multimodal comprehension for policy-specified guardrails. The framework is evaluated with multiple safeguard scenarios and compared with diverse models. The proposed dataset is sufficiently scalable compared to existing datasets.
1. The proposed self-refinement training involves a testing procedure where the new version of the model is evaluated on the test set and misclassified instances are extracted & analyzed to curate new policies (Line 267 - Line 269). The paper should argue why and how such a procedure avoids **test set leakage**. If the test set information is encoded in the renewed policy, the procedure could lead to inflated performance on the test set because the model has already captured the exact test information during training. Also, the paper should compare how existing works handle misclassified instances in the test set.

2. The paper presentation needs significant improvement. For example, Appendix C.5 refers readers to a null table (Line 1407) and the prompt description on page 20 exceeds the page boundary (Line 1026 - Line 1079). The quotation marks in the prompt description are often monotonously right quotations (page 16, page 19, page 22). The reference list is also not well-curated. For example, the referenced website addresses often far exceed the page boundary and are obscured (Line 665, Line 778). Note that the Llama-Guard Team's paper is referenced as "Team (2024)" in the paper (Line 144, Line 215, ...), which reads strange. Also, there are many lines of unspecified space on page 18 (Line 922 - Line 958). **While a few minor errors in the paper presentation will not affect the rating, **too frequent observation** of them will harm the rating since they are not aligned with the proceeding guidelines and are not beneficial for future readers.**
Yes
No ethics review needed.
See Weaknesses.
1. This paper constructs VISIONHARM-500K, which is conducive to the further development of image moderation.

2. The proposed SAFEVISION can better distinguish harmful content and has extremely high efficiency.
This work explores multimodal large models for detecting harmful content. Existing work suffers from low generalization and difficulty in handling new categories of hazards. Therefore, the authors built VISIONHARM-500K, a high-quality unsafe image benchmark comprising over 500k images to cover a wide array of risky categories. Based on this benchmark, the authors proposed SAFEVISION, which supports multiple modes and provides precise explanations. Experiments show the effectiveness and efficiency of the proposed method.
1. The categories defined by the dataset should be reflected in the main body.
2. How does SAFEVISION judge the content of new categories? Is it controlled only by prompts?
3. In Section 4.2, is the improvement to the tokenizer to add category nouns to the vocabulary library?
4. This work declares the contribution of the dataset, but there is no related open-source plan in the text. Will the code and dataset of this paper be released?
@inproceedings{
anonymous2024safevision,
title={SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=E1SaL8aK7k},
note={under review}
}
As image generation models become increasingly prevalent, the need for efficient and transparent guardrails against unsafe content is more critical than ever. Traditional unsafe image classifiers, limited to predefined categories, often misclassify content due to the pure feature-based learning rather than semantic-based reasoning and struggle to adapt to emerging threats. The time and resources required for retraining on new harmful categories further hinder their ability to respond to evolving threats. To address these challenges, we propose SafeVision, a novel image guardrail system that integrates human-like understanding and reasoning with scalability. Within SafeVision, we propose an effective data collection and generation, policy-following training pipeline, and a customized loss function. In particular, we propose an efficient diverse QA generation and training strategy to enhance the effectiveness of the training process.
SafeVision is able to follow given safety policies during inference time to guardrail against new risk categories and thus avoid expensive retraining, provide accurate risky content predictions, and provide precise explanations. SafeVision operates in two modes: 1) rapid classification mode, and 2) comprehension mode that provides both classification and human-readable explanations.  In addition, considering the limitations of existing unsafe image benchmarks, which contain either only binary or limited categories, we provide VisionHARM-500K, a high-quality unsafe image benchmark comprising over 500k images to cover a wide array of risky categories. This dataset significantly broadens the scope and depth of unsafe image benchmarks. Through comprehensive experiments, we show that SafeVision achieves state-of-the-art performance in both efficiency and accuracy, with an accuracy of 91.77% on the VisionHARM-500K test set (17.77% higher than GPT-4O) and an inference time of 0.0979 seconds per image (over 50 times faster than GPT-4O). SafeVision sets a new standard for comprehensive, policy-following, and explainable image guardrail models, delivering state-of-the-art performance while aligning with human reasoning and enabling scalable adaptation to emerging threats.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
AI safety Large language model Multi modality Image moderation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/267ba1f1ddfb5b9d216c2372fcbd031a84edb4e1.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission