Kitten: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities
Text-to-image generation
Yes
No ethics review needed.
see weakness
- This paper tackles a unique and underexplored aspect in the text-to-image generation community --evaluating the fidelity of specific entities in generated images. The motivation and evaluation framework are well-grounded.
- Utilizing Wikipedia as a knowledge base is a clever choice, ensuring broad coverage and relevance of entities. The idea makes the entity coverage more scalable for future extension.
- The proposed evaluation approach, incorporating both human and automatic metrics, strengthens the robustness of the study’s findings.
- The paper offers valuable insights and discussions that can guide future work aimed at improving fidelity in text-to-image models.
- The paper is well-written and easy to understand.
The paper introduces a benchmark dataset, Kitten, designed to evaluate the capability of text-to-image models in generating visuals with precise entity representations across eight distinct visual domains. For each chosen entity, the authors crafted four unique input prompt types, alongside corresponding support and evaluation image sets for each entity.

The paper incorporates human evaluations focusing on entity fidelity, with two primary criteria: (1) the model’s ability to follow instructions accurately, and (2) the faithfulness of generated visuals to the specified entity. Furthermore, the authors employ automatic metrics, including the CLIP-T score and cosine similarity within DINO's feature space, to assess alignment between generated and target images.

State-of-the-art models, such as Flux, Stable Diffusion (SD), Imagen-3, Dreambooth, and Instruct-Imagen, were tested on the dataset. Experimental results highlight the main challengings of backbone models as well as retrieval-augmented customized models. The results suggest future topics, such as enhancing entity fidelity without compromising the instruction-following capabilities of these models.
- While the study suggests a direction for future work aimed at balancing entity fidelity with creative flexibility, the notion remains somewhat ambiguous and challenging to envision in practice. Achieving an optimal outcome where both precise entity representation and creative interpretation coexist is complex, and the paper could benefit from a clearer exploration or concrete examples of what such a balance might look like in generated images.
- The paper suggests that the optimal image generation model should balance entity fidelity with creative flexibility, but it remains unclear whether such coexistence is achievable. If it proves difficult or even impossible to represent both precise entity fidelity and creative flexibility simultaneously, this limitation could reduce the paper’s impact. The study would benefit from a deeper investigation into whether these two qualities can genuinely coexist, as this would provide clearer guidance for future research directions and enhance the paper’s practical relevance.
Yes
No ethics review needed.
see weakness.
the focus of the paper is interesting, which is to evaluate the text-to-image model in terms of generating real-world entities with some modifications. the benchmark would also be beneficial to the community. the evaluation is based on two aspects. one is the faithfulness to the entity, and the other is the prompt following as the prompt may contain some changes to the entity. the metric is sound and reasonable. beyond, the authors perform a very comprehensive study on how the existing approaches work on these new benchmarks.
The paper introduces KITTEN, a benchmark specifically designed to evaluate the accuracy of text-to-image generation models in terms of real-world entities, such as landmarks, plants, and animals. on this benchmark, the authors evaluate multiple existing approaches, including both standard text-to-image models and retrieval-augmented models that leverage reference images.  the findings include 1) that even the most advanced text-to-image models often fail to generate entities with accurate visual details, 2) retrieval-based method often over-rely on these references and struggle to produce novel configurations of the entity as requested in creative text prompts.
as the main contribution is the benchmark, it may be better to provide more details on how the benchmark is collected. throughout the paper, i only find the information that the benchmark is collected from 8 categories and Table 7 in supl shows the number of prompts in each. it might be recommended to share more details, e.g. how these categories are collected, why it is these 8 categories, is it diverse enough, how the prompt is collected, who collected the prompts, what the prompt length and variance is. 

the paper performs lots of evaluation on the existing text-to-image approaches and the retrieval-based approaches. one important metric is the human evaluation, but it might be better to share more details on the human eval, e.g. how many annotators are there for each task, how the variance is, how reliable the evaluation is. 

[minor] in Fig 1, the results are all from proprietary company, e.g. adobe firefly, dalle. but in the experiments, all results are based on other approaches, e.g. flux model. there is no overlap here. it might be better to make it consistent. 
[minor] the Flux has multiple versions (https://github.com/black-forest-labs/flux), is it dev or pro?
Yes
No ethics review needed.
1.  Is there a theoretical basis for choosing the specific four types of prompts? How were these types determined, and do they align with the objectives of your study?
2. Is there a rationale or empirical evidence supporting this selection as being representative of real-world scenarios?
3. Have the authors discussed the trade-offs between manual and automated evaluation methods in terms of accuracy and cost? Are there any strategies for expanding the evaluation dataset in the future while reducing costs and maintaining quality?
4. Could you please explain in detail how you selected these four types of prompts and the rationale behind your choices? Can you argue for the extent to which these prompt types cover important scenarios involving knowledge entities? Have you considered adding other types of prompts？
5. How does your benchmark compare with existing benchmarks in terms of design goals and evaluation metrics? Are there particular aspects where you believe your benchmark offers distinct advantages?
6. What steps have been taken to mitigate potential biases resulting from the limited diversity of entities and prompts?
1. The paper presents a novel problem in the field of image generation, focusing on the factuality of real-world knowledge in image generation, which is rarely explored in the existing literature.
2. The results are clearly presented, with sufficient tables and visual images to illustrate the findings.
3. The paper is well-written, with a clear and logical organization.
This paper proposed a novel and significant issue: real-world knowledge in image generation. Focusing on the factual accuracy with which models generate real-world entities, the authors have developed a benchmark based on a diverse set of knowledge entities to systematically evaluate model performance in the domain of real-world knowledge. To more accurately assess the factuality of the images generated by these models, the authors have meticulously designed a set of human evaluation criteria. They conducted a comparative analysis with widely-used automatic metrics, highlighting the deficiencies of these automatic metrics and demonstrating the superior accuracy of human evaluations. The authors evaluated a variety of text-to-image models and retrieval-augmented customization models, uncovering significant deficiencies in the visual representation of world knowledge within these models.
1. Insufficient Experimental Setup: The current experiments are limited to real-world entities in only 8 domains and 322 knowledge entities, which lacks diversity. This number is significantly less than the variety of entities present in the real world and does not comprehensively reflect the model's performance, potentially leading to data bias. The OntoNotes and WordNet datasets include a more diverse range of categories and entities. If you only consider your 8 domains and limited number of entities, might this affect the generalizability of the results?
2. High Costs: The paper proposes a manual evaluation method, but conducting a single evaluation using this benchmark requires more than 6,000 human assessments, resulting in significant costs.
3. Limited Prompt Types: The paper includes only four types of prompts. It's unclear whether there is a theoretical basis for selecting these four types, and whether high scores in these prompts can fully reflect the model's true understanding of the knowledge entities.
4. Lack of Comparison with Other Benchmarks: The paper does not provide a clear comparison with recent benchmarks in the field, making it difficult to understand the similarities and differences between this benchmark and others (eg. HELM, GenAI-Bench and ConceptMix)
Yes
No ethics review needed.
see weaknesses
1. The concept of evaluating text-to-image (t2i) models' ability to generate real-world visual entities is intriguing and crucial for progress in this area.
2. The proposed dataset and evaluation criteria offer a valuable resource for the community, enabling more detailed and nuanced evaluations of t2i models.
This paper introduces a novel evaluation method to assess image generation models' capability to generate real-world visual entities. The authors propose a benchmark called Kitten and develop both human and automatic evaluation methods to test various image generation models. Through extensive analysis, the author gives a key insight that sota t2i models liake SD, flux exhibit lower faithfulness to the entity, while  retrieval-augmented models show  great advantage.
1. The paper's novelty is somewhat limited. The insights that retrieval-augmented models show advantage on faithfulness to the entity is understandable, however the finding that they lack instruction-following capabilities is more interesting, have the authors considered exploring methods to improve entity fidelity while maintaining instruction-following capabilities？
2. The dataset covers 8 different domains, Did the authors observe any significant performance variations across different entity domains (e.g., landmarks vs. insects)? If so, what might explain these differences?
@inproceedings{
anonymous2024kitten,
title={Kitten: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=MY1pJ1RDMv},
note={under review}
}
Recent advancements in text-to-image generation have significantly enhanced the quality of synthesized images. Despite this progress, evaluations predominantly focus on aesthetic appeal or alignment with text prompts. Consequently, there is limited understanding of whether these models can accurately represent a wide variety of realistic visual entities — a task requiring real-world knowledge. To address this gap, we propose a benchmark focused on evaluating Knowledge-InTensive image generaTion on real-world ENtities (i.e., KITTEN). Using KITTEN, we conduct a systematic study on the fidelity of entities in text-to-image generation models, focusing on their ability to generate a wide range of real-world visual entities, such as landmark buildings, aircraft, plants, and animals. We evaluate the latest text-to-image models and retrieval-augmented customization models using both automatic metrics and carefully-designed human evaluations, with an emphasis on the fidelity of entities in the generated images. Our findings reveal that even the most advanced text-to-image models often fail to generate entities with accurate visual details. Although retrieval-augmented models can enhance the fidelity of entity by incorporating reference images during testing, they often over-rely on these references and struggle to produce novel configurations of the entity as requested in creative text prompts.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Text-to-image generation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/0709585e151d492632634f0b7e7a9c063ad2355e.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Kitten: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission