Measuring And Improving Engagement of Text-to-Image Generation Models
image generation models;text to image models;engagement;stable diffusion;dalle
Yes
No ethics review needed.
What are the further direction in this research topic? Given the situation that many labs may lack computational resources, how to make it possible for these researchers to follow your work? Will the EngageNet also be appliable to other research domains, such as multimodal item recommendation?
- The idea of the engagement-optimized image generation is novel, as images are often created to drive user engagement beyond just aesthetic appeal.
- The paper introduces a large dataset of 168 million tweets with images and associated user engagement metrics like likes, paving the way for such research direction.
- Multiple methods are explored to enhance the engagement of text-to-image models, including prompt conditioning, supervised fine-tuning, and reinforcement learning.
This paper introduces EngageNet, a text-to-image generation model optimization method to improve the audience engagement of the generated images, rather than merely the aesthetic quality and realism. Based on this, the first automated arena, i.e., EngageNet Arena, is proposed to benchmark the engagement of text-to-image models.
Null
Yes
No ethics review needed.
-  Can you provide further detail on how negative samples were generated and their effects on training EngageNet? How does the negative sampling impact model robustness and alignment with actual user engagement?
- Can EngageNet be adapted to predict other engagement metrics (e.g., CTR, shares) in addition to likes?
- The paper presents an interesting application to optimize image generation for viewer engagement, a metric commonly aligned with recommendation and commercial objectives. 
- The dataset curation and model development appear thorough. EngagingImageNet is large, high-quality, and potentially valuable for further research on engagement in visual content. The paper includes rigorous validation of EngageNet’s correlation with actual engagement metrics, offering a credible alternative to existing models.
This paper investigates how current text-to-image generation models fall short of predicting viewer engagement. To address this gap, the authors propose several contributions including (1) EngagingImageNet: A dataset of high-quality enterprise images alongside engagement metrics like user likes, which serve as real-world indicators of viewer engagement, (2) EngageNet: A Vision Language Model (VLM) trained on EngagingImageNet to predict engagement by analyzing contextual factors around each image, such as tweet content, enterprise details, and posting time, and (3) Engagement Arena: A benchmarking platform where EngageNet scores various image generation models based on their capacity to generate engaging images. The paper explores approaches to enhance engagement in generated images, including prompt-based conditioning, fine-tuning, and reinforcement learning with EngageNet-generated reward signals.
- As EngagingImageNet is sourced from enterprise accounts on Twitter, the dataset may have an inherent bias toward corporate content. For a model aiming to set a standard for engagement-optimized image generation, more diverse data sources would provide a broader foundation and improve applicability across various domains. 
- The effectiveness of engagement-enhancing techniques is primarily measured using EngageNet scores and correlations. Incorporating additional real-world engagement experiments, for example, a small-scale human evaluation study, would strengthen the claims regarding the improvements in viewer engagement and provide additional validation.
- The negative sampling method for training EngageNet is vaguely described. The paper mentions that negative samples are randomly generated by pairing tweets with unrelated images, yet does not explain how the random sampling affects EngageNet's robustness or alignment with true engagement.
Yes
The paper does not clarify whether any collected data contain identifiable information. If they do, obtaining informed consent from individuals and getting the human ethics approval would be essential.
Yes, Responsible research practice (e.g., human subjects, data release)
1.	A more precise definition and scope of viewer engagement would be helpful, as the current metrics appear to include clicks, likes, and shares. Additionally, detailed statistics of the dataset should be provided to help the community better understand this dataset.
2.	Given the dataset spans over a decade, how is the temporal information being utilized?
3.	While the introduction of the engagement dataset and benchmarking metrics is a valuable contribution, the proposed methods—conditioning image generation, supervised fine-tuning, and reinforcement learning for diffusion alignment—are relatively straightforward adaptations of existing techniques. A more innovative approach to these tasks could elevate the paper’s impact.
4.	The paper could discuss a broader range of use cases and scenarios, such as recommender systems or advertising, beyond standard text-to-image applications, especially given the marketing focus of this work.
5.	An in-depth discussion on potential use cases would be beneficial.
6.	The paper does not clarify whether any collected data contain identifiable information. If they do, obtaining informed consent from individuals would be essential.
1.	The proposed method is robust, producing a large-scale, high-quality dataset that addresses the gap in image engagement and utility analysis.
2.	The paper is clearly written and well-structured.
3.	The authors evaluate various models, offering valuable insights into the dataset’s application and benchmark characteristics.
This study constructs a large-scale dataset of images paired with user engagement metrics to investigate image engagement and utility in real-world marketing contexts. An engagement-aware visual language model (VLM) is developed to predict viewer engagement. Contemporary techniques are explored to enhance engagement in text-to-image models, including conditioning image generation on optimized prompts, supervised fine-tuning of Stable Diffusion on high-performing images, and reinforcement learning to align Stable Diffusion outputs. The authors also introduce a benchmarking method for text-to-image engagement models, offering the research community a standard framework for evaluating advancements in engagement modelling.
Please see questions.
Yes
No ethics review needed.
1. Why did the authors not consider including retweets as part of the engagement indicator?

2. Why was SD 1.4 chosen for train-time optimization, as it is one of the weakest models?

3. Does KPI refer to the normalized likes of an image? So the regression value of KPI is just used in training phase?

4. I would like to see an analysis of the complexity or latency of the proposed components, such as the Retrieval module, since an excessively long generation process would be impractical for real-world applications.

5. All concerns in Weaknesses section.
1. The collected dataset EngagingImageNet is a strong foundation, as authors provide the detailed introduction of obtaining and filtering this datasets. This dataset encompasses multimodal information for each image, supporting not only engagement assessment but also various other applications.
2. The author trained the EngageNet with several tricks, obtaining a superior performance. The release of such model would offer both academia and industry a new perspective for evaluating generated images.
3. This paper introduces multiple strategies for enhancing the engagement of generated images, during run-time and train-time. They achieve notable improvement using these approaches.
This paper introduces a novel consideration in the text-to-image generation task: human engagement. The authors first collect a large dataset, EngagingImageNet, from an online platform. They then train EngageNet to assess the engagement level of generated images based on auxiliary information embedded in the prompts. Additionally, they propose methods to leverage EngageNet to enhance the performance of text-to-image models. Finally, the authors introduce Engagement Arena to assess the engagement levels of different text-to-image models. This work presents a comprehensive framework for evaluating and improving engagement of text-to-image models.
1. The paper would benefit from a more detailed explanation of the DDPO method utilized, including the hyper-parameters and loss terms.

2. The paper lacks a user study or online experiment to evaluate the engagement of images generated by the fine-tuned models. It is unclear whether the fine-tuned models genuinely improve engagement or merely optimize to satisfy the reward model.

3. The paper lacks discussion of some relevant works:

[1] Rich Human Feedback for Text-to-Image Generation, CVPR 2024.
[2] Towards Reliable Advertising Image Generation Using Human Feedback, ECCV 2024.
[3] Directly Fine-Tuning Diffusion Models on Differentiable Rewards, ICLR 2024.

Moreover, other works on using human feedback to improve image generation models should be considered.
We develop a framework to measure and benchmark the capability of text-to-image models in generating engaging images, and propose methods to optimize the image generation process to enhance engagement.
@inproceedings{
anonymous2024measuring,
title={Measuring And Improving Engagement of Text-to-Image Generation Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=TmCcNuo03f},
note={under review}
}
Recent advances in text-to-image generation have achieved impressive aesthetic quality, making these models usable for both personal and commercial purposes. However, in the fields of marketing and advertising, images are often created to be more engaging, as reflected in user behaviors such as increasing clicks, likes, and purchases, in addition to being aesthetically pleasing. Further, we find that existing image generation metrics like aesthetics, CLIPScore, PickScore, ImageReward, etc. fail to capture viewer engagement. To this end, we introduce the challenge of optimizing the image generation process for improved viewer engagement. In order to study image engagement and utility in real-world marketing scenarios, we collect EngagingImageNet, the first large-scale dataset of images, along with associated user engagement metrics. To address the lack of reliable metrics for assessing image utility, we use the EngagingImageNet dataset to train EngageNet, an engagement-aware Vision Language Model (VLM) that predicts viewer engagement of images by leveraging contextual information about the tweet content, enterprise details, and posting time. We then explore methods to enhance the engagement of text-to-image models, making initial strides in this direction. These include conditioning image generation on improved prompts, supervised fine-tuning of stable diffusion on high-performing images, and reinforcement learning to align stable diffusion with EngageNet-based reward signals, all of which lead to the generation of images with higher viewer engagement. Finally, we propose the Engagement Arena, to benchmark text-to-image models based on their ability to generate engaging images, using EngageNet as the evaluator, thereby encouraging the research community to measure further advances in the engagement of text-to-image modeling. These contributions provide a new pathway for advancing utility-driven image generation, with significant implications for the commercial application of image generation.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
image generation models text to image models engagement stable diffusion dalle
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/0f000489a6408770cffbcea59c4cbf669d86a0e6.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/4a7eb3284042baca64e4696270efc6a571663825.zip
Measuring And Improving Engagement of Text-to-Image Generation Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission