Multi-domain Analysis and Generalization of Image manipulation loCalization
Domain generalization;Diffusion-Based Inpainting;Misinformation Detection;Computer Vision;Benchmark Dataset;Visual Forensics
Yes
No ethics review needed.
See the Weakness Section. Overall, this is a solid piece of work. I will consider raising my rating if improve the presentation of details.
- The motivation is sound, as cross-dataset or cross-domain performance has consistently posed challenges in the field of image manipulation localization. A dataset focused on cross-domain performance analysis would serve as a valuable benchmark.
- Experiments are comprehensive in demonstrating the utilization of each protocol.
The paper primarily builds a Diffusion-based model and constructs an image manipulation localization dataset with 192k images focused on inpainting tampering types. This dataset is divided into three major categories based on source, content topic, and specific types of Diffusion models used, allowing for evaluation of the model's cross-domain generalization performance. The authors evaluated the performance of several SoTA models on this dataset. A survey on human feedback on the quality of this dataset is also reported.
## Main issue
- Although the authors claim to have used clustering for categorizing topics, some topics displayed in Figure 2 seem to vary significantly and do not appear entirely reasonable. For instance, it’s unclear how a bicycle image relates to the "ARTS" category, and both "People" and "Ruins" are grouped under "Media." Additionally, the results across many topic classes in Table 4 are quite similar, suggesting that the distribution between classes may not be as distinct as initially anticipated.
- A paper [A] proposed in Nov. 2023 on ArXiv and accepted by ACM MM 2024 also uses Visual News and COCO to create a fine-grained diffusion and GAN-generated dataset. I understand that ACM MM was held after ICLR submission. However, the two articles have considerable similarities in the background, purpose, and subject matter. While the two papers represent distinct works, it is recommended to discuss [A] in the Related Work section and reconsider the claim in line 149 regarding being the "first diffusion-based manipulation dataset." Since [A] retains the text prompts associated with images, which, with proper handling, could serve as a more accurate basis for topic categorization.
- For dataset-focused papers, it’s common to include tests with standard vision backbones, such as ResNet or Swin, to provide more straightforward benchmarks. Including these could serve as helpful references for comparison.

## Minor issue
- Many AUC metrics in the tables are too close, showing little distinction and some values are excessively low. For instance, Table 3 includes numerous metrics below 0.5, indicating that the model has not effectively learned the corresponding distributions. More distinctive metrics, such as F1 or IoU, may be needed better to assess the model’s performance on each protocol.

- The statement between lines 522–524 appears unconvincing. The current explanation does little to clarify why the performance of PSCC is nearly the opposite of the other two models.


## Reference
[A] Zhihao Sun, Haipeng Fang, Juan Cao, Xinying Zhao, and Danding Wang. 2024. Rethinking Image Editing Detection in the Era of Generative AI Revolution. In Proceedings of the 32nd ACM International Conference on Multimedia (MM '24). Association for Computing Machinery, New York, NY, USA, 3538–3547. https://doi.org/10.1145/3664647.3681445
Yes
No ethics review needed.
See weakness.
- MAGIC addresses a pressing issue in image manipulation detection by offering a large-scale dataset with a focus on domain generalization across multiple dimensions. This effort is commendable and fills a gap in manipulation detection research.
This paper introduces “MAGIC,” a large-scale dataset designed to evaluate the robustness and generalization of image manipulation detection models across multiple domains. MAGIC aims to assess model performance under various domain shifts, including different image sources, manipulation types, semantic topics, and manipulation scales. Results indicate that while the models perform well in distribution (ID), their OOD performance is limited, highlighting the challenges of domain generalization in image manipulation detection.
- The proposed dataset, while diverse, does not introduce fundamentally new manipulation detection methods or models. The dataset’s construction (e.g., sourcing from MS COCO and VisualNews, manipulation types) is novel but does not demonstrate significant methodological innovation beyond combining existing datasets and manipulation techniques. Thus, the contribution is more incremental than groundbreaking.

- The paper lacks a detailed comparison with other recent datasets or techniques, and the experiments primarily rely on existing architectures without substantial modifications or improvements. The work’s dependence on pre-existing models for analysis and lack of new methodological contributions weaken its overall technical impact.

I am not an expert in this field. But I think that a dataset for image manipulation localization is not sufficient for publication at ICLR.
Yes
No ethics review needed.
Please see the weaknesses.
1. The proposed datasets seem good for image manipulation location tasks.
2. The writing and experiments are pretty good.
This paper proposed a new image manipulation location benchmark for diffusion-based generation methods. It contains two image sources and seven manipulation techniques. The experiments under several settings, also provide some interesting insights.
1. The quality of the manipulated images in Figure 2 is worrying, especially for the removing class. Although the authors mentioned that they apply human evaluation for the generated images, I'm worried about the data balance of the three categories (removal, replacement, and insertion) under high-quality annotations.
2. The image manipulation methods used in the paper are not very new. Using SD series, like SD2, or even SDXL is better.
3. In Table 3, it's interesting that the OOD score is higher than the ID score when trained on MAGIC-News and tested on MAGIC-COCO. It's better to provide a more depth analysis.
Yes
N/A
No ethics review needed.
+ How to make sure the quality of the generated masks for MAGIC-News (since they are generated automatically from Mask2Former)
+ For the replacement operation, have you tested different-class replacements instead of same-class replacements
+ How to ensure the quality of GLIGEN since it is not a perfect method, any mechanism to ensure its quality?
+ How many images are used for training, val, test, and out-of-domain subsets?
+ Propose new datasets that cover many situations: news (MAGIC-News) or daily lives (MAGIC-COCO)
+ Cover many editing operations: insertion, removal, and replacement
+ Include many editing techniques: Stable Diffusion, Blended Diffusion, Glide Diffusion, and Adobe Firefly.
This paper introduces two novel datasets for image forensics specifically curated from diffusion-based editing methods: MAGIC-News and MAGIC-COCO. These datasets encompass various topics and object classes, with manipulations including object insertion, replacement, and removal, applied through various editing techniques such as Stable Diffusion, Blended Diffusion, Glide Diffusion, and Adobe Firefly. Experiments demonstrate the performance of several image forensic techniques on these new datasets.
Since this paper mainly focuses on new datasets, the presentation of the datasets should be prepared more carefully. Specifically:
+ Lack of high-quality examples of manipulated images and corresponding GT segmentation mask (Fig. 2 presents low-resolution images)
+ Instead of just listing some numbers only, the visual charts should be used to summarize the statistics of datasets (e.g., editing areas statistics
+ A table of Editing technique summarization would also help, including the number of images, and examples. 
+ For the Dataset Quality Survey, using a flowchart to visualize the process would be better. 
+ Lack of reporting IoU (along with AUC)
+ Lack of classification performance (decide whether an image is a manipulated image or genuine one)
+ The quality of the proposed dataset is a concern (Tab. 6) since some methods are just around 50%)
+ Demonstration of using the proposed datasets would help the performance of detection techniques on other datasets such as MagicBrush [a] and CocoGLIDE [b].

[a] Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems, 36, 2024. 2, 5, 6, 7278

[b] Fabrizio Guillaro, D Cozzolino, Avneesh Sud, Nick Dufour, and L Verdoliva. TruFor: Leveraging all-round clues for trustworthy image forgery detection and localization. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pages 20606–20615, December 2022
@inproceedings{
anonymous2024multidomain,
title={Multi-domain Analysis and Generalization of Image manipulation loCalization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=9YZKbSoDr6},
note={under review}
}
Advanced image editing software enables easy creation of highly convincing image manipulations, which has been made even more accessible in recent years due to advances in generative AI. Manipulated images, while often harmless, could spread misinformation, create false narratives, and influence people’s opinions on important issues. Despite this growing threat, current research on detecting advanced manipulations across different visual domains, remains limited. Thus, we introduce Multi-domain Analysis and Generalization of Image manipulation loCalization (MAGIC), a comprehensive benchmark designed for studying generalization across several axes in image manipulation detection. MAGIC comprises over 192K images from two distinct sources (user and news photos), spanning a diverse range of topics and manipulation sizes. We focus on images manipulated using recent diffusion-based inpainting methods, which are largely absent in existing datasets. We conduct experiments under different types of domain shift to evaluate robustness of existing image manipulation detection methods. Our goal is to drive further research in this area by offering new insights that would help develop more reliable and generalizable image manipulation detection methods. We will release the dataset after this work is published.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Domain generalization Diffusion-Based Inpainting Misinformation Detection Computer Vision Benchmark Dataset Visual Forensics
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/3abe0174f1b9ae59b6d691f9d0f0760575ab381a.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Multi-domain Analysis and Generalization of Image manipulation loCalization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission