GRADE: Quantifying Sample Diversity in Text-to-Image Models
sample diversity;text-to-image;diffusion;evaluation
Yes
No ethics review needed.
1. The distinction between common and uncommon prompts lacks sufficient justification. For instance, in Figure 1, cookies are categorized based on familiar and unfamiliar backgrounds. However, do you think the shape of a cookie should change just because the background is uncommon? Isn’t it more concerning if the object changes rather than the background?

2. How do the authors recognize and address hallucinations when using LLMs and VQA?
The strength of this paper is introducing the fine-grained and interpretable metric that overcomes the limitations of traditional diversity metrics by quantifying concept-specific attribute diversity without relying on reference images. It provides deeper insights into text-to-image model behavior and highlights the impact of training data biases, offering a clear path for advancing generative model evaluation and diversity.
The paper presents GRADE (Granular Attribute Diversity Evaluation), a novel method for quantifying the diversity of images generated by text-to-image (T2I) models, which often produce limited variations due to underspecified prompts. Unlike traditional metrics like FID, GRADE uses large language models and visual question-answering systems to identify and measure concept-specific attributes, calculating diversity through normalized entropy. The study finds that even the most advanced T2I models frequently exhibit default behaviors, such as generating the same attributes regardless of context, and highlights that biases in training data contribute to these limitations. The authors call for improvements in training data and propose GRADE as a more fine-grained, interpretable diversity metric to advance generative modeling.
1. Generally, there is a common perception that "cookies" are round, and I share this view. If a square cookie were requested but a round cookie was generated, that would indeed be an issue. However, the problem highlighted by the authors is not about such cases but rather challenges the generalization itself. I find it difficult to relate why generating results that align with common sense is problematic. In other words, it seems the authors are raising an issue with what is an expected outcome.

2. The method uses an LLM to generate questions based on the given prompt, relying heavily on the performance of the LLM. For example, if the prompt involves a cookie, the model generates questions about the shape of the cookie. However, just as many VQA papers specify categories such as shape, attribute, and color, shouldn't there also be a strict specification of categories in this case? If a prompt where shape is crucial fails to address this aspect due to limitations or hallucinations of the LLM, wouldn't that pose a problem?
Yes
No ethics review needed.
- Q1: GRADE assigns maximum diversity to cases where all values of an attribute are equally prevalent in the evaluated images. However, roughly, text-to-image models are usually trained to maximize the likelihood of the "real-world" distribution of natural images. In this case, shouldn't one expect that generated images will reflect the distribution of natural images? If so, does it still make sense to expect that the probability of generating a start-shaped cookie should be equal to the probability of generating a round cookie?

&NewLine;

- Q2: GRADE depends on measuring diversity with respect to specific concepts and attributes, but how many concepts and attributes are "enough" to make sure GRADE is an accurate estimate of diversity?

&NewLine;

- Q3: I am confused by  the equation in lines 156 and 157. Does $p_c^i$ correspond to a single prompt (as indicated in the definition of $\mathcal{P}_c$) or a set of prompts (as it seems to be the case given the sum is over $i$)?   

&NewLine;

- Q4: Can the authors kindly provide examples of prompt and concept distributions in Section 3-d? I couldn't understand what exactly they corresponded to even after reading the text a few times. 

&NewLine;

- Q5: (Related to W1) Please perform statistical significance tests between model scores and report confidence intervals.

&NewLine;

- Q6: (Related to W3.1) A suggestion to elucidate this point would be assessing GRADE's performance on misaligned or unrealistic images, and checking how/if the metric's reliability was affected.

 
- Minor comments:
  - Typo: 
    - Line: 309: this step **a** second time
  - Equations are missing their respective numbers.
- S1: This work tackles the extremely relevant open research question of evaluating diversity of text-to-images.

&NewLine;

- S2: The proposed approach, GRADE, proposes to measure diversity in terms of images of specific concepts with respect to relevant factors of variation.

&NewLine;

- S3: GRADE is a reference-free metric and doesn’t rely on training any new models to be computed.
This work aims at tackling the open problem of evaluating diversity of text-to-image generative models. The authors propose GRADE, an approach that has two main components: measuring diversity of specific concepts with respect to attributes selected by a language model, and using as a metric the normalized entropy of the distribution of answers given by visual-question answering (VQA) model as metric for diversity. GRADE’s main components were evaluated by human inspection and the efficacy of GPT-4 as a VQA model was assessed via human evaluation. 12 different text-to-image models were compared in light of the introduced metric which showed that most of them perform similarly in terms of diversity (as measured with GRADE).
- W1: The proposed metric does not seem to be effective at discriminating between models. It is not clear what is the statistical significance in the difference of scores reported in Table 4. Given the average and standard deviation of GRADE, it seems that confidence intervals for all models would overlap. 

&NewLine;


  - W1.1: There is no rigorous validation of the proposed metric and given that it seems GRADE is not able to properly distinguish 12 generative models, it is unclear to me whether the metric is actually capturing diversity as claimed in the manuscript. Experiments considering images with known ground-truth for diversity should be included in order to show that the proposed metric is in fact performing as expected.

&NewLine;


- W2: The authors claim in the conclusion that “Our experiments demonstrate that GRADE is in agreement with human judgements of diversity”, however there is no evidence in the manuscript to support such a strong claim. The human evaluation performed in this work only takes into account the accuracy of GPT-4 on VQA tasks, and it is not even remotely evaluating diversity of generated images. In fact, a major weakness of this submission is that there is no clear evidence that the proposed metric is capturing diversity as perceived by humans.

&NewLine;


- W3: The metric relies on how well the VQA model is able to capture nuances in the differences across different values an attribute can assume. For example, if the diversity of sky images is being measured with respect to the color attribute, it is not clear whether dark blue and navy would be considered as different colors by the VQA model.

&NewLine;


  - W3.1: It is unclear what is the interplay between GRADE and other aspects such as alignment between prompt and generated image, and realism. What happens if the main concept is not actually present in the generated image and/or the generated objects do not look realistic? Would the VQA model be able to answer the question correctly?
&NewLine;


- W4: It is not clear how the approach validity as evaluated in Sections 4-a,b,c  would generalise if other concepts/aspects were taken into account, therefore making it unclear how GRADE performs in case diversity with respect to concepts and attributes of different nature such as people,  ethnicity, skin tone, gender, etc.

&NewLine;



- W5: The manuscript lacks clarity in many parts. Please refer to the "Questions" section of this review for more details. 

&NewLine;

 
- W6: There are several missing references in the text and very relevant related work has not been cited by the authors:
  - Diversity metrics:
     - Friedman, Dan, and Adji Bousso Dieng. "The vendi score: A diversity evaluation metric for machine learning." arXiv preprint arXiv:2210.02410 (2022). 
     - Pasarkar, Amey P., and Adji Bousso Dieng. "Cousins of the vendi score: A family of similarity-based diversity metrics for science and machine learning." arXiv preprint arXiv:2310.12952 (2023).
     - Alaa, Ahmed, et al. "How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models." International Conference on Machine Learning. PMLR, 2022.
   - VQA-based approaches to evaluate generative models:
     - Wiles, Olivia, et al. "Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings." arXiv preprint arXiv:2404.16820 (2024).
     - Cho, Jaemin, et al. "Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation." arXiv preprint arXiv:2310.18235 (2023).
     - Lin, Zhiqiu, et al. "Evaluating text-to-visual generation with image-to-text generation." European Conference on Computer Vision. Springer, Cham, 2025.
Yes
No ethics review needed.
For the concept-specific attributes, such as shape, color, and texture, is there a predefined set of attributes (other than shape/color/texture), or is it flexible? What range of different attributes is considered, and how consistent is this range across various concepts? Additionally, if you rely on LLMs to automatically generate these attributes, could this introduce new biases into the evaluation process?
- This paper is well-written, well-motivated, and well-organized. 
- A key strength is that they provide a clear and structured definition of the diversity metric and the systematic approach in Sec. 3. The method estimates distributions with entropy, and allows for a granular and concept-specific evaluation of the diversity. 
- The paper links low diversity in generated images to underspecified captions in training data, suggesting that this lack of diversity might come from biases in the data itself. This takeaway on data quality could help improve diversity by addressing such limitations in the training datasets.
This paper introduces GRADE (Granular Attribute Diversity Evaluation), a new evaluation approach for quantifying sample diversity in T2I models. GRADE addresses the limitations of existing diversity metrics by focusing more on concept-specific attributes such as shape, color or texture, without replying on reference images. The authors perform comprehensive experiments on 12 T2I models, showing consistent default behaviors and limited diversity across models. They attribute this lack of diversity to underspecified captions in training data, which drive models to favor common attributes by default.
- Since GPT-4o is used to generate prompts, attributes, and attribute values, its lack of version control could affect the consistency of the GRADE framework as GPT-4o evolves. This may alter diversity analysis results, making reproducibility difficult. Changes in GPT-4o’s outputs could lead to inconsistencies when comparing diversity evaluations of T2I models evaluated at different points in time.
- The definition of diversity in this work is focused on specific attribute variations (like shape, color, etc.) and may miss other important aspects of image diversity. The approach relies heavily on the capability LLMs, which could have their own limitations and biases in terms of diversity. 
- There's no limitation discussion for example how GRADE might handle more complex or abstract concepts where attributes are less clearly defined. Another part that would be good to include in the future version of the paper is the compositional diversity or relationships between multiple concepts in the same image.
We propose a method to evaluate sample diversity in T2I models
@inproceedings{
anonymous2024grade,
title={{GRADE}: Quantifying Sample Diversity in Text-to-Image Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=JddNOaw66n},
note={under review}
}
Text-to-image (T2I) models are remarkable at generating realistic images based on textual descriptions. However, textual prompts are inherently *underspecified*: they do not specify all possible attributes of the required image. This raises key questions: do T2I models generate diverse outputs on typical underspecified prompts? How can we automatically measure diversity? We propose **GRADE**: **Gr**anular **A**ttribute **D**iversity **E**valuation, an automatic method for quantifying sample diversity. GRADE leverages the world knowledge embedded in large language models and visual question-answering systems to identify relevant concept-specific axes of diversity (e.g., ''shape'' and ''color'' for the concept ''cookie''). It then estimates attribute distributions and quantifies diversity using (normalized) entropy. GRADE achieves over 90\% human agreement while exhibiting weak correlation to commonly used diversity metrics. We use GRADE to measure the overall diversity of 12 T2I models using 400 concept-attribute pairs, revealing that even the most diverse models display limited variation. Further, we find these models often exhibit *default behaviors*, a situation where the model consistently generates concepts with the same attributes (e.g., 98\% of the cookies are round). Finally, we demonstrate that a key reason for low diversity is due to underspecified captions in training data.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
sample diversity text-to-image diffusion evaluation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b05ca6e4215b84acd0d700591fbb80471b5992d0.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/4ab91b4eddb5ccfd0e0117f73e6b9a89f0b2180e.zip
GRADE: Quantifying Sample Diversity in Text-to-Image Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission