An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models
Information measures;diversity evaluation;conditional generative models
Yes
No ethics review needed.
My opinion of the work is already high so I do not have many questions. However, in light of Figure 6, it seems that it would be possible to achieve a high conditional Vendi score by a model that has poor alignment with the prompt. Is there any way that the proposed conditional/information Vendi decomposition can handle this case?
Strengths of this paper include:
- The proposed metrics occupy a clearly defined niche in the literature, providing a rich and informative way to evaluate prompt-based generative models. This is useful for better comparisons of generative models (even applicable across different modalities) and for better model development by elucidating the shortcomings of current models.
- Solid theoretical justification and empirical evidence are provided to support the proposed metrics.
- The paper is exceedingly well-written. Prose is clear and mathematical statements are presented in a precise yet intuitive way.
This paper proposes a metric for evaluating the diversity of generative models. Unlike previous approaches which focus on unconditional generations, this paper uses an information-theoretic decomposition of the entropy into conditional entropy (measuring variation in generation when conditioned on a prompt) and mutual information (measuring the remaining variation not explained by the conditional entropy). The first term is expressed as a conditional Vendi score and the second as an information Vendi score. It is shown that the Vendi score is the product of these two, and so generative models with similar Vendi scores can be compared based on their conditional diversity given a prompt. Theoretical justfication for the proposed metrics is provided, and it is also shown how the metric can be estimated from finite samples. The proposed metrics are explored in different contexts, including text-to-image, text-to-video, and image-captioning models. The results correspond with the intuitive interpretations of the scores.
Weaknesses of this paper include:
- The proposed approach can be viewed as incremental, since it is the combination of two ideas: the Vendi score and matrix-based conditional entropy. However, the execution of the paper is very good and is targeted at a gap in the current literature, so I do not count this as a major weakness.
- The contextualization and interpretation of Theorem 1 could be improved.
- Qualitative results could be improved in some places. For example, showing some of the generations from the models in Figure 4 would help to better clarify the tradeoffs of the models in terms of conditional Vendi vs. information Vendi score.
Yes
No ethics review needed.
No additional question other than the major weakness above.
- The motivation for decomposing the diversity of model generation is clear, despite that a more fine-grained structural analysis would be preferred.
- The theoretical interpretation of the scores provides sufficient insight for the decomposition. 
- The paper is well-organized and logically fluent.
The paper introduces an information-theoretic approach to evaluate the diversity of prompt-based generative models, focusing on decomposing the diversity into prompt-induced and model-induced components. The authors achieve this by decomposing the Vendi score into the Conditional-Vendi score, which measures internal diversity not related to prompt variations, and the Information-Vendi score, which assesses the statistical correlation between generated outputs and their corresponding prompts. The authors validate their methodology through various experiments on text-to-image, text-to-video, and image-captioning models.
While this paper tackles an important problem, the following weaknesses prevent it from being accepted.

- First of all, I think the paper lacks novelty in terms of the proposed method. The concept of conditional entropy $H_\alpha$ is nothing new, and the proposed conditional-Vendi score is simply its exponential version. If the paper is not contributing through new concepts or definitions, it should at least demonstrate some (new) important properties or use cases. Unfortunately, the only property discussed in the paper is its relevant stability when more diverse prompts are used, which, theoretically speaking, is expected. In such a case, the only novelty I can admit is that the paper demonstrates in experiments that the conditional entropy behaves normally in generative models. The authors are expected to provide a more comprehensive analysis of the given entropy and demonstrate a wider range of use cases if the concept itself is inherited from existing literature.
- Even if we consider only the stability property, the range of experiments in the paper is still limited. All experiments are simply measuring the scores with varying amounts of input clusters (groups), and the way that the authors make prompts diverse is also rigid. It is entirely unclear, based on existing experiments, that the proposed score can serve as an effective diversity measurement across a wide range of text prompts types. What's more, the authors do not show any support that the measure is accurate, except for this extremely intuitive "increasing curve". No golden metric, or even any proxy metric, has been introduced for comparison. This means that the "diversity measurement" claim is not convincing in the first place.
- Theoretically speaking, a large portion of the discussion about the selection of kernels is missing. From my understanding, since the authors only measure diversity from the generated data (rather than the underlying softmax distribution), the similarity metric used for text and images is very important and can largely change the metric. It is, therefore, necessary for the authors to carefully analyze the choice of kernels and make a more well-studied claim w.r.t. the proposed metric. However, notice that addressing this bullet is insufficient for making the paper reach the acceptance bar unless the first two weaknesses can be addressed.
Yes
No ethics review needed.
(Contribution) As mentioned in Weakness, could the authors explain their contributions to prior works such as Giraldo et al (2014)?

 (Applications) For prompts that can be easily clustered, why don't we just average unconditional entropy for each cluster? For more general distribution, what's the practical meaning of the conditional entropy when the distribution $\mathcal{T}$ changes?
 (Numerical) Why can current numerical results illustrate the advantages of proposed metric?
Originality: the problem of diversity quantification is indeed interesting and useful in generative models. 

Clarity: the paper is well-organized and straightforward to follow, with supportive numerical results and clearly stated details.

Overall the paper is a nice reading.
This paper focuses on the diversity in prompt-based generative models. Specifically, the authors want to quantify the prompt-induced and model-induced diversity in samples separately. An information-theoretic approach for internal diversity quantification based on kernel-based conditional entropy of the generated data is discussed, where the Conditional-Vendi score and the Information-Vendi score are applied to measure the statistical relevance between the generated data and text prompts.
My major concern is that the contribution of this work is incremental. As stated by the paper in Section 3.2, the conditional entropy $H_\alpha(X|T)$ is already developed by Giraldo et al (2014), while the Conditional-Vendi score proposed by the authors is $\exp\left(H_\alpha(X|T)\right)$, which is essentially the same. In addition, existing literature has considered Renyi entropy of the probability model as the model's unconditional diversity score, and the idea of constructing conditional diversity score based on these results does not compose of sufficient contribution. Thus, I feel the contribution is incremental.

Another concern I have is the potential applications of the proposed diversity measure. In most scenarios mentioned by the paper, say statistical interpretation in Section 5 and some motivating examples, prompts can be easily clustered, so what about just calculate the unconditional entropy for each cluster of prompts, and then use some weighted average to combine them? I personally feel this is a naive but reasonable way to measure the conditional diversity of the model and should serve as a baseline. If prompts cannot be easily clustered and come from some complex distributions, then mathematically the conditional entropy would depend heavily on the distribution of the prompt distribution, the the concrete meaning of this measure is unclear to me. 

For numerical results, though they are clear to me, after reading it I can still not tell if conditional entropy is a good measure, as there is no baseline compared (which is reasonable as this is a relatively new problem, but the authors should at least consider some naive methods such as the one mentioned above, to see if there are some scenarios that their proposed method works while naive methods don't) and the meaning of values reported (say, figure 5-7) remains unclear to me. Why can these figures and values justify that the conditional entropy is a good measure of model's diversity?
Yes
No ethics review needed.
1. What's new of your conditional-Vendi compared with the conditional entropy proposed by Sanchez Giraldo et al., besides the exponential?
2. Follow which equation can the conditional-Vendi score be calculated? It's better to write an explicit algorithm, perhaps in the appendix.
3. What do you want to convey by Section 5 (the statistical interpretations)? Is it discussing the implementations, or just broader discussions? A better leading sentence may be necessary in the beginning.
* The idea is sound to separate different sources of diversity, i.e., prompt-induced and model-induced.
* The proposed diversity metric is reasonable for model-induced diversity, and have the advantage that do not need repeated sampling from a single prompt.
This paper focuses on distinguishing different sources for diversity evaluation of prompt-based generative models, i.e. prompt-induced diversity and model-induced diversity. Specifically, this paper proposed a new diversity evaluation metric called *condtional-Vendi* score, which is an extension of *Vendi*, but focuses on model-induced diversity rather than overall diversity. Combining with another proposed *information-Vendi* score, the *Vendi* score can be recovered by a multiplication. Experiments on image and text generation tasks show that condtional-Vendi is correlated with model-induced diversity.
The main weakness of this paper is its contribution. The proposed condtional-Vendi and information-Vendi score are just exponentials of the conditional entropy and mutual information defined by Sanchez Giraldo et al. (line 241, 246), and the calculation method seems to directly inherit from Vendi proposed by Friedman & Dieng (line 317). The combination is straightforward and I did not find any new challenges that is addressed by this paper. As a result, I view this paper as just an incremental work, and not enough for acceptance.
@inproceedings{
anonymous2024an,
title={An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=IJiTI0fB0e},
note={under review}
}
Text-guided sample generation schemes are commonly evaluated based on the quality of generated data and their alignment with the input text prompt. On the other hand, several applications of prompt-based generative models require adequate diversity in the generated data to ensure the models' capability of generating samples possessing a variety of features. However, the existing diversity metrics are designed for unconditional generative models, and thus cannot distinguish the diversity of created data because of the variety of text prompts and the diversity contributed by the generation model. In this work, our goal is to quantify the prompt-caused and model-caused diversity of samples produced by a prompt-based generative model. We propose an information-theoretic approach to the model's internal diversity quantification, where we decompose the kernel-based entropy $H(X)$ of generated data $X$, to the sum of conditional entropy $H(X|T)$ given text variable $T$ and the mutual information $I(X;T)$ between the text and data variables. We utilize the conditional entropy $H(X|T)$ to define the *Conditional-Vendi* score for the quantification of the internal diversity of the model and prove theoretical results interpreting the application of these scores to mixture distributions. We perform several numerical experiments to show the correlation between the Conditional-Vendi score and standard text-based generative models. We also demonstrate the application of the proposed framework to detect inequities in text-based sample generation schemes.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Information measures diversity evaluation conditional generative models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/faf080b5b68c23db714c1438460c0477a15d9926.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/0a84d5d01b30676395a7a04d47087b3193bfed88.zip
An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission