MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models
diffusion models;MCMC sampling;image memorization;safety ai
Yes
No ethics review needed.
Besides providing a larger size of dataset, what is the contribution of the new dataset?
1. The paper reformulates the search for memorized image prompts as an optimization problem, which allows for more efficient prompt discovery.
2. It employs a Markov Chain Monte Carlo (MCMC) method to solve the optimization problem, resulting in a significantly larger set of memorized prompts compared to previous methods.
3. MemBench provides evaluation of existing memorization mitigation methods in diffusion models.
This paper proposes a benchmark to evaluate memorization mitigation methods in text-to-image diffusion models. MemBench includes a large set of trigger prompts that replicate training images, potentially causing privacy or copyright issues. It assesses mitigation methods by examining their effectiveness in preventing memorized images without degrading general prompt performance.
1. This paper does not consider the detection problem. Wen et al and Ren et al both use detection before mitigation. So they do not always mitigate on general prompts. It does not make sense to compare the generation performance on general prompts.

2. The evaluation metrics are not new. In existing paper like Wen et al and Ren et al, they use all the metrics mentioned in this paper.

3. The dataset construction requires white-box access, which is ineffective to API-based models. In addtion, although it does not use training set, it requires to use an API to search images on the web, which is not efficient.

4. The major problem is the benchmark, which didn't provide a reasonable perspective and new metrics.
Yes
No ethics review needed.
Please check out the weaknesses section, where the limited contribution and motivation are the major concerns that make me inclined to reject this paper.
- This paper is good in presentation, with good clarity, well-structured, and easy to follow.
- This paper addresses an important and practical task: the reverse process of the diffusion model always invokes almost or exactly the same memorized images from training data. This can lead to privacy issues, which are well-discussed and motivated in the introduction section.
- This paper proposes 3,000, 1,500, 309, and 1,352 memorized image trigger prompts for different generative models, compared to a previous benchmark that provides 325, 210, 162, and 354 prompts. Adding additional trigger prompts as a dataset is beneficial for a more comprehensive evaluation of mitigation strategies.
This paper studies the memorization issue in text-to-image diffusion models, where generated images is almost or exactly the same as the training image, which leads to privacy concerns. This paper propose a new triggering prompt dataset that contains prompts that will lead to memorized generations. This paper evaluates several baseline works’ mitigation strategies on the proposed datasets and also non-triggering prompts and have concluded that existing strategies still have room for improvement as they degrade overall performance of diffusion models.
1. **Limited contribution**. 
- While the additional trigger prompts introduced in this paper can indeed aid in evaluating existing mitigation strategies and provide further assessment of their performance, such contribution remains somewhat incremental. Rather than addressing a critical need in the field, it adds to existing resources that are already robust. For example, prior methods [1, 2] have utilized 500 trigger prompts, which a recent work [3] systematically organized by extracting and categorizing them into three types (Matching Verbatims, Retrieval Verbatims, and Template Verbatims) based on their distinct characteristics. This categorization already offers a comprehensive framework, making the additional prompts in the present work more of an enhancement than a necessary innovation. I understand that this work’s main contribution is proposing a new benchmark trigger prompt dataset; unlike others that focus on proposing strategies to detect and mitigate the memorization problem, such core contribution (the proposed prompt dataset) is still of quite limited contribution.
- I have also carefully read all the claimed contributions in the paper and appreciate the authors’ effort in presenting them in a well-structured way that is easy to follow. However, such concern remains as the other claimed contributions are also trivial. For example, the idea of using general (non-memorized) prompts in addition to those trigger prompts for evaluation is a good point. However, it is more of a good practice for the related work, instead of a significant contribution. Also, the finding of all mitigation methods can result in a reduction of text alignment, reflecting the trade-off nature of all mitigation strategies, which is already well-documented in the related works and is not a new finding.

[1] Detecting, Explaining, and Mitigating Memorization in Diffusion Models.

[2] Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention.

[3] A Reproducible Extraction of Training Images from Diffusion Models.

2. **Concerns regarding motivation**. I find the descriptions (motivations) in lines 46-50 to be misleading, where the paper states: “The current studies (Wen et al., 2024b; Somepalli et al., 2023b) have adopted the following workaround: 1) simulating memorization by fine-tuning T2I diffusion models for overfitting on a separate small and specific dataset of {image, prompt} pairs, and 2) assessing whether the images used in the fine-tuning are reproduced from the query prompts after applying mitigation methods …” In fact, these papers actually fine-tune on constructed non-triggering pairs of image and prompts to mitigate the memorization (overfitting) issue as per the training-time strategies, also they propose inference-time mitigation strategies that does not rely on the construction of prompt datasets. Both are quite practical, contrary to the paper's points.
Yes
No ethics review needed.
My main problem with the paper is the organization of the writing, I like the idea and I think it is novel, however writing does not focus on main contribution and lack to properly refrence some of the things I mentioned in weakness section

1. Language Model Limitations:
   - How do you address the temporal limitation of using BERT (trained pre-2018) for prompt generation? 
   - What percentage of potential memorization cases might be missed due to this limitation?
   - Have you considered using more recent language models or alternative approaches?

2. Methodological Clarity:
   - Could you provide specific details about the computational constraints that prevented certain comparison methods from being included in the main body of the paper not just in the appendix.
   - Can you clarify the source and context of the reported AUC values from Wen et al. 2024 in the main body of the paper?

3. Metric Choices and Evaluation:
   - What empirical evidence supports the choice of aesthetic score over FID? please also include FID in the paper
1. Novel MCMC-based approach for finding memorization triggers
2. Comprehensive evaluation framework for mitigation methods
3. Large-scale trigger prompt discovery
4. Practical insights into mitigation limitations
5. Not requiring training data
This paper introduces MemBench, a benchmark for evaluating memorization mitigation methods in diffusion models. The core technical contribution is an MCMC-based sampling approach that efficiently finds prompts triggering generation of training data, building upon the memorization detection metric (Dθ) from Wen et al. (2024). The benchmark includes both memorized image trigger prompts and evaluation metrics to assess mitigation methods' effectiveness. The authors demonstrate their method finds substantially more trigger prompts than previous work and reveal issues with existing mitigation approaches. 

However, the paper's organization and presentation significantly obscure its contributions. The main technical innovation - using MCMC for efficient prompt sampling - is not clearly highlighted and is buried under various other aspects. The benchmark itself, while valuable, would benefit from clearer exposition of its components and limitations.
1. Critical Methodological Limitations:
   - The paper relies heavily on pre-trained language models (specifically BERT) for prompt sampling but fails to acknowledge this as a fundamental limitation. This is particularly problematic as such models have a training cutoff date (pre-2018 for BERT), making it impossible to find triggers containing newer terms or concepts.
   - The evaluation framework potentially misses a significant portion of memorization cases due to this temporal limitation.
   - The dependency on pre-trained language models restricts the generalizability of the method.

2. Unclear and Redundant Writing:
   - The paper's findings section contains three nearly identical statements that could be consolidated:
     Quote: "MemBench reveals several key findings:
     1. All image memorization mitigation methods result in a reduction of Text-Image alignment between generated images and prompts...
     2. The mitigation methods affect the image generation capabilities of diffusion models, which can lead to lower image quality...
     3. The mitigation methods can cause performance degradation in the general prompt scenario..."
     These are essentially making the same point about performance degradation.

   - The paper makes claims about metric superiority without proper justification:
     Quote: "while Ren et al. (2024) measured FID, the Aesthetic Score offers a more straightforward way to evaluate individual image quality and better highlight these issues."
     No evidence or explanation is provided for why Aesthetic Score is "more straightforward" or "better."

   - Key metrics are used without proper introduction:
     The paper extensively uses SSCD scores without ever explaining what they measure or their significance.

3. Poor Communication of Technical Contributions:
   - The paper's main technical innovation (efficient prompt sampling) is not clearly highlighted.
   - The findings section contains redundant statements about mitigation methods that could be consolidated.
   - Claims about metric choices (e.g., aesthetic score vs FID) lack proper justification.
Yes
No ethics review needed.
In Appendix D, the authors mention they cannot do reverse image search on SD3 generated images since its training set is unknown. Have you tried to reverse search on the LAION 5B dataset despite SD3 had no confirmation of using LAION 5B. Given the size and popularity of LAION, I would assume it (partially) overlaps with the training set of many modern text to image diffusion models.
The proposed MCMC-based search is a novel idea for quickly searching for problematic prompts.

The design of MemBench is well thought out and is comprehensive in its metrics for wide adoption.
MemBench is a novel benchmark designed to evaluate image memorization issues in text to image diffusion models. This work includes a benchmark dataset that evaluates image memorization in text to image generation models (authors claim to be first). the benchmark itself contains comprehensive metrics on memorization. 

In addition to the benchmark itself, authors provided analysis of the effectiveness of various mitigation approaches.
The work uses Stable Diffusion 2 to show empirical evidence of memorization and how well their benchmark works. However, their proposal seems unable to apply to models like Stable Diffusion 3 (or other newer models than SD2) where training data is unknown. In Appendix D, the authors did not show may evidence of memorization of SD3. This makes me wonder how widely applicable this work is
We introduce a benchmark containing prompts that induce memorized images in text-to-image diffusion models. Using this benchmark, we evaluate various memorization mitigation methods.
@inproceedings{
anonymous2024membench,
title={MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Qg0gtNkXIb},
note={under review}
}
Diffusion models have achieved remarkable success in Text-to-Image generation tasks, leading to the development of many commercial models. However, recent studies have reported that diffusion models often repeatedly generate memorized images in train data when triggered by specific prompts, potentially raising social issues ranging from copyright to privacy concerns. To sidestep the memorization, there have been recent studies for developing memorization mitigation methods for diffusion models. Nevertheless, the lack of benchmarks hinders the assessment of the true effectiveness of these methods. In this work, we present MemBench, the first benchmark for evaluating image memorization mitigation methods. Our benchmark includes a large number of memorized image trigger prompts in various Text-to-Image diffusion models. Furthermore, in contrast to the prior work evaluating mitigation performance only on trigger prompts, we present metrics evaluating on both trigger prompts and general prompts, so that we can see whether mitigation methods address the memorization issue while maintaining performance for general prompts. Through our MemBench evaluation, we revealed that existing memorization mitigation methods notably degrade overall performance of diffusion models and need to be further developed.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
diffusion models MCMC sampling image memorization safety ai
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6f52976aac57dce9f7d2f00d937f456131200e63.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/ce62dc03d4ae2aad526e1e7cf2edec249ea814ab.zip
MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission