Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation
Xun Wu Shaohan Huang Furu Wei






 



Abbreviated paper title





W. Author et al.


Microsoft Research Asia, Beijing, China Tsinghua University, Beijing, China 

{v-wuxun, shaohanh, fuwei}@microsoft.com






    Xun Wu1,2 Contribution during internship at Microsoft.  Corresponding Authors. Shaohan Huang1 Furu Wei1
    April 29, 2024
===========================================================================================================



















Recent studies have demonstrated the exceptional potentials of leveraging human preference datasets to refine text-to-image generative models, enhancing the alignment between generated images and textual prompts. 

Despite these advances, current human preference datasets are either prohibitively expensive to construct or suffer from a lack of diversity in preference dimensions, resulting in limited applicability for instruction tuning in open-source text-to-image generative models and hinder further exploration.

To address these challenges and promote the alignment of generative models through instruction tuning, we leverage multimodal large language models to create , a high-quality and fine-grained preference dataset that captures multiple preference aspects. We aggregate feedback from AI annotators across four aspects: prompt-following, aesthetic, fidelity, and harmlessness to construct .

To validate the effectiveness of , we train a reward model  over  to guide the training of text-to-image generative models and the preference prediction accuracy of  is comparable to human annotators. 

Furthermore, we use two reinforcement learning methods to supervised fine-tune generative models to evaluate the performance of , and extensive experimental results demonstrate that  significantly improves text-image alignment in compositional image generation across diverse aspects, e.g., aesthetic, and generalizes better than previous human-preference metrics across various image distributions. Moreover,  indicates that the integration of AI-generated synthetic data as a supervisory signal is a promising avenue for achieving improved alignment with human preferences in vision generative models.








§ INTRODUCTION









Text-to-image generative models <cit.> have experienced rapid advancements in recent years. For example, large-scale text-to-image diffusion models, exemplified by Imagen <cit.> and DALL·E2 <cit.>, have demonstrated the capability to generate high-quality and creative images when provided with novel textual prompts.

Unfortunately, despite recent progress, current generative models still face a challenge: they may generate incorrect or unsafe content that deviates from human preferences, such as awkward limb and facial expression combinations. Additionally, due to noise in pre-training datasets, this often results in a misalignment between the semantics of generated images and their corresponding textual prompts <cit.>.


Reinforcement Learning from Human Feedback (RLHF) <cit.> has proven effective in aligning Large Language Models (LLMs) with human preferences. Inspired by this success, RLHF has been applied to the supervised fine-tuning of text-to-image generative models, particularly diffusion models, by utilizing high-quality human preference data <cit.> and shows promising alignment results.










Preference data plays a pivotal role in the development of aligning generative models with text prompts. Unfortunately, in the area of text-to-image diffusion, collecting data from samples that demonstrate desired characteristics like aesthetics and fairness is not only a daunting and costly task but is also vulnerable to inherent biases. Another source of bias is in text prompts. User-generated prompts frequently adopt a structured format, comprising a descriptive passage supplemented by stylistic adjectives.However, these stylistic terms often contain contradictions, making it harder for human annotators to understand.

Besides, existing preference benchmarks, such as HPS v2 <cit.> and Pick-a-Pic <cit.>, are either limited on alignment aspects only or in short of meticulous preference annotations. Drawing inspiration from recent research utilizing AI-generated data as training supervise signal, we pose the following question:

[colback=gray!20, colframe=gray!50, sharp corners, center title]
Can Multimodal Large Language Models act as a Human-Aligned Annotator for Text-to-Image Generation?

















Given the question mentioned above, we have contemplated whether multimodal large language models (MLLMs) <cit.> can serve as human-aligned annotators. These MLLMs, trained on vast amounts of text and text-image pairs, have already demonstrated formidable capabilities on image understanding. To this end, we introduce , a publicly available AI-generated dataset that features millions of finely-grained human preferences concerning model-generated images.

Compared with existing human preference datasets,  offers the following benefits:

(i) Scalability: As shown in Table <ref>,  encompasses 1.2 M human preference choices across 179 K pairs of images, establishing it as the largest text-to-image generation preference dataset to date.

(ii) Fine-grained preference: Inspired by <cit.>, we have carefully developed a detailed preference annotation guideline that covers four distinct aspects: Prompt-Following, Fidelity, Aesthetic, and Harmlessness. The detail requirement for each aspect is presented at Table <ref>.

(iii) Comprehensive feedback formats: Unlike existing benchmarks that provide only rankings or preference indices,  not only supplies rankings but also requires AI annotators to assign numerical preference scores and provide textual explanations for the annotation from each annotation aspect.

Building on the  dataset, we conducted an extensive investigation into its most effective utilization. We developed a preference reward model , trained to evaluate generated images based on their likelihood of being preferred by humans. Experimental results demonstrate that  exhibits a competitive correlation with human preferences compared to other human preference reward models. 

Moreover, we employ two reinforcement learning methods to enhance generative models to better align with human preferences, as illustrated in Figure <ref>, extensive experimental results showcase that  markedly enhances text-image alignment in compositional image generation across diverse aspects, such as aesthetics. Our contributions are as follows:



    
  * We construct , a large-scale, high-quality, and diversified preference dataset for text-to-image generative alignment. Compared with existing preference datasets,  has the advantages of scaleability, fine-grained annotations and comprehensive feedback format.
    
  * Based on , we propose a reward model, , which achieves a competitive correlation with human preferences with other automated human preference metrics.
    
  * Experimental results demonstrate the effectiveness of both  and . Additionally, we provide a comprehensive analysis on both of them to gain a deeper understanding of how AI-generated synthetic data and models trained on such data impact future research in this domain.








§ RELATED WORK




 §.§ Text-to-Image Generative Models Alignment



While existing text-to-image generative models often generate images that do not closely match human preferences, thus alignment in the context of diffusion has garnered increasing attention <cit.>. We introduce the research progress of text-to-image generative models alignment from two perspectives: algorithms and datasets.

Algorithm. Existing text-to-image generative models alignment algorithm can be broadly categorized into two main types: (i) Proximal Policy Optimization (PPO). These algorithms involves a two-step process: first, modeling the reward function solely based on preference data, and then solving a regularized value function maximization problem to obtain a fine-tuned policy based on the reward provided by the reward model.

For example, reward weighted method <cit.> first explores using human feedback to align text-to-image models with human preference. ReFL <cit.> trains an reward model named ImageReward using human preferences and subsequently utilizes it for fine-tuning. 



(i) Direct Policy Optimization (DPO). DPOK <cit.> fine-tune text-to-image diffusion models by using policy gradient to maximize the feedback-trained reward. ZO-RankSGD <cit.> optimizes diffusion in an online fashion with human ranking feedback. DRAFT <cit.> and AlignProp <cit.>, incorporate a similar approach into training: tuning the generative model to directly increase the reward of generated images.

Datasets. Existing human preference datasets for diffusion model alignment are all constructed by real users or experts <cit.>. The overall statistics among them are shown in Table <ref>. However, manually annotated preference datasets suffer from several drawbacks. Firstly, manual annotation is highly expensive; constructing large-scale manually annotated preference datasets requires significant human resources, hindering the progress of related research. Additionally, manually annotated preference datasets are prone to specific biases, which may result in lower dataset quality. Therefore, in this study, we investigate the possibility of employing AI  annotators as annotators for preferences. AI  annotators offer rapid and cost-effective annotation, along with demonstrating strong generalization capabilities, to address the aforementioned limitations.



 §.§ Reinforcement Learning from AI Feedback

LLMs have also been extensively used for data generation <cit.>, augmentation <cit.> and in self-training setups <cit.>. <cit.> introduced the idea of Reinforcement Learning from Human Feedback (RLAIF), which used LLM labeled preferences in conjunction with human labeled preferences to jointly optimize for the two objectives of helpfulness and harmlessness. Recent works have also explored related techniques for generating rewards from LLMs <cit.>. These works demonstrate that LLMs can generate useful signals for reinforcement learning fine-tuning.

However, RLAIF for text-to-image generative model alignment is less explored. <cit.> leverage large multi-modality models (LMMs) to o assess the alignment between generated images and input texts, focusing on aspects like object number and spatial relationship. However, experiments is too few to support the conclusion and they lacking explore the ability of LMMs to evaluate between generated images and input texts on other aspects (e.g., fidelity).



§ 












We introduce , a fine-grained preference dataset constructed by collecting feedback from multimodal large language model  annotators. 

The collection pipeline of  is shown in Figure <ref>, which mainly consists of 3 steps: prompt generation, image generation and preference generation.



Step-1: Prompt Generation. 

Following existing works <cit.>, we utilize DiffusionDB <cit.> as our basic prompt benchmark, which is a large-scale text-to-image prompt benchmark containing 1.5 million user-written prompts. We adapt following two steps to make the prompt benchmark unbiased and safe:

∙ Polish. As discussed in <cit.>, a significant portion of the prompts in the DiffusionDB is biased towards certain styles. For instance, around 15.0% of the prompts in DiffusionDB include the name “Greg Rutkowski”, 28.5% include “artstation”. 

To address the above issues, we utilize AI annotators, i.e. SOTA MLLMs like , to polish these prompts in DiffusionDB. The polish instruction for these AI  annotators is elaborated in Appendix. By this way, the output prompts are clearly polished in one sentence with less style words, which are easier for understanding. 

∙ NSFW Filting. When processing these prompts, following <cit.>, we employ state-of-the-art NSFW detectors <cit.> to compute an NSFW score for each prompt and filtering out prompts that exceed a certain threshold.

After these two processions, we combine both polished prompts and the original prompts in DiffusionDB as our final prompt benchmark, which contains 179K prompts. 

Step-2: Image Generation.

We generate images using different text-to-image generative models by sampling textual prompts constructed in Step-1 as input. The details of these generative models are listed in Appendix. 

For each prompt input, we generate four images by randomly selecting different generative models from the model pools while sampling different classifier-free guidance scale values, to achieve a high degree of diversity. 

This diversity allows for a comprehensive evaluation of a preference prediction model’s generalization capability and facilitates the training of a more generalizable model. 













Step-3: Preference Generation.

After generating 0.76 M images based on the 613 K textual prompts, we employ state-of-the-art multimodal large language model, , to provide three types of feedback: (1) scalar scores that indicate the fine-grained quality regarding multiple aspects, (2) preference ranking according to the scalar scores, and (3) textual critique that gives detailed guidance on how to improve the completion, encompassing four distinct aspects namely: Prompt-Following, Aesthetic, Fidelity, and Harmlessness for each generated image (See the example in Table <ref>). Detailed input instruction for  are documented in Appendix. Besides, we also explore and analysis the effectiveness of the annotation ability of other multimodal large language models (♊ and  <cit.>) in Section <ref>.








































































§ EXPERIMENTS



Based on , we conduct extensive experiments to validate that multimodal large language models can act as an advanced preference annotator for text-to-image generative models. We first train a corresponding reward model named  and evaluate it on existing human-preference datatsets (Section <ref>). Next, we enhance existing text-to-image generative models by adopting two reinforcement learning algorithm collaborated with  and , respectively (Section <ref>). Finally, we present ablation studies in Section <ref>.



 §.§ Reward Modeling



Training Setting.
We train the  over .  adopts the same model structure as ImageReward <cit.>, which is a open-source human-preference reward model and utilizes BLIP <cit.> as the backbone. 

Similar to reward model training for language model <cit.>, we formulate the preference annotations in  as rankings. Specifically, we employ the average scores of each sample in  across four aspects as the final preference score, and then we have k images ranked generated by the same prompt 𝐓 according to final preference score (the best to the worst are denoted as 𝐱_1 ≻𝐱_2  ≻ ... ≻𝐱_k). For each comparison, if 𝐱_i is better and 𝐱_j is worse, the loss function can be formulated as:

    loss(θ) = - 𝔼_(T, x_i, 𝐱_j) ∼𝒟[log(σ(f_θ(𝐓, 𝐱_i) - f_θ(𝐓, 𝐱_j)))]

where f_θ(𝐓, 𝐱) is a scalar value of reward model for prompt 𝐓 and image 𝐱.




Evaluation Results. 

We evaluate the preference prediction accuracy on the test sets among three human preference datasets: ImageRewardDB <cit.>, HPD v2 <cit.> and Pick-a-Pic <cit.>. Furthermore, to better demonstrate the model's generalization performance, we computed the harmonic mean of accuracy across three sets for each model as an overall indicator of model performance. We use the CLIP score <cit.>, BLIP score <cit.>, Aesthetic score <cit.>, ImageReward <cit.>, HPS <cit.>, HPS v2 <cit.> and PickScore <cit.> as baselines to compare with the .

The results are presented at Table <ref>. Our  demonstrates strong competitiveness compared to the current state-of-the-art reward models trained on human preference data. It achieves the second-best average performance among all preference reward models, following only HPS v2. Moreover, our model achieves optimal performance on the ImageRewardDB dataset, achieving a 0.6 performance gain compared to HPS v2. These results validate that leveraging fine-grained feedback provided by AI  annotators enables learning a proficient human preference reward model.







 §.§ Boosting Generative Models
 

In this work, we aim to leverage the constructed preference dataset to align the performance of generative models more closely with human preferences. To achieve this goal, we employ two reinforcement learning methods: (1) Proximal Policy Optimization (PPO). Given the , we can fine-tune a generative model to maximize the given reward from  using PPO.

Here we select ReFL <cit.> as our PPO implementation. 

(2) Direct Preference Optimization (DPO). DPO can directly fine-tune generative models using preference dataset without the need for pre-training reward models. Here we select D3PO <cit.> as our DPO implementation. For both PPO and DPO experiments, we use Stable Diffusion v1.5 as the target for fine-tuning.

r0pt
[t]0.4



    < g r a p h i c s >



Performance across multiple reward models during the PPO training process. All scores are normalized for a better visualization.






Dataset. For PPO experiments, we randomly sample 20,000 real user prompts from DiffusionDB <cit.> and 10,000 prompts in ReFL <cit.> as the training dataset. Due to the requirement of the DPO to fine-tune the generative model directly on preference dataset, we select our  along with three existing large-scale human-annotated preference datasets (ImageRewardDB <cit.>, HPD <cit.> and Pick-a-Pic <cit.>) as training datasets for different comparison groups. It's worth noting that when fine-tuning generative model on , we employ the average scores of each sample in  across four aspects as the final preference score. The details of these datasets can be found in Table <ref>. For evaluation, we collect 400 real user prompts from DiffusionDB <cit.> for evaluation.

Comparative Baselines. For PPO experiments, we select five existing open-source reward models as comparative baselines for our . Among them, ImageReward <cit.>, PickScore <cit.>, and HPS v2 <cit.> are trained on large-scale human-annotated preference datasets (refer to Table <ref>). To validate the effectiveness of  and , these three experimental groups will serve as our primary comparative objects. It's worth noting that all experimental groups fine-tune generative model on the same training data and training settings (the same learning rate and batch size), with the only difference being the reward model for a fair comparison. For DPO experiments, the experimental variable is the training dataset. We select three open-source large-scale preference datasets, ImageRewardDB <cit.>, HPD <cit.> and Pick-a-Pic <cit.> (details can be found in Table <ref>), as comparative baselines, while keeping other experimental settings consistent. More training details can be found in Appendix.







PPO Results. 
First, we visualize the evolution of various metrics as the model training steps increase when using our  as the reward function. See in Figure <ref>, we observe that with the progress of training, all metrics, including human preference metrics like HPS v2, show an increasing trend, indicating the consistency between our  and other human preference metrics and demonstrating that  can serve as a reliable reward model to enable the generative model's outputs to align more closely with human preferences.



Then, we conduct a human preference study. Specifically, we encourage these fine-tuned generative models to generate 64 images for each prompt in evaluation dataset, and then perform a top-3 selection within the these generated images by the corresponding reward models. Finally, three human annotators rank these selected images. The results are presented at Figure <ref> (a), We observed that  fine-tuned generative model's Win+Tie ratio exceeds 50% when compared to all other models, including some trained on large-scale human preference datasets like HPS v2. This suggests that, compared to other human preference reward model, VP can serve as a reliable and competitive reward model for fine-tuning generative models to produce outputs closer to human preferences. This further underscores the effectiveness and competitiveness of .

The corresponding qualitative results shown at Figure <ref> demonstrate that  fine-tuned generative model can generate images that are more aligned to text and with higher fidelity and avoid toxic contents. 


DPO Results. We conduct a human preference study using the same procedure as PPO experiments, and the results are presented at Figure <ref> (b), We found that the Win+Tie ratio of the generative model optimized on our , when compared to the other three large-scale human datasets, exceeds 50%, substantiating the competitiveness of our  against human-annotated preference data.

We show the qualitative results in Figure <ref>. The results indicate that fine-tuning the generative model directly on our  using DPO yields performance comparable to that of fine-tuning the generative model on large-scale human-annotated preference dataset (e.g., Pick-a-Pick). Specifically, the generated results are more aligned with human preferences, exhibiting increased visual detail, better conformity to input prompts. 

These experimental outcomes collectively affirm the efficacy of using preference data generated by MLLMs. 



 §.§ Ablation Study


Reward Model Backbone.  adopts BLIP <cit.> as the backbone, which may raise curiosity about how well BLIP compares to CLIP <cit.>. 


We employed these tow models as the backbone for our reward model and explored their effectiveness on our . The results are summarized in Table <ref>, where we observed that the performance of BLIP surpassed that of CLIP and this conclusion aligns with the findings on human preference datasets <cit.>.

Training Data Size.
To investigate the effect of training dataset sizes on the
performance of the , comparative experiments are conducted. The resuklts are presented at Figure <ref>. We observed that as the training data increased, 's prediction accuracy gradually improved. This indicates that models trained on our  exhibit strong performance scalability, implying that more training data leads to further performance improvements. In the future, we plan to further increase the volume of data in our  and explore whether models trained on our dataset can outperform all these trained on human-annotated datasets. This endeavor holds significant promise and interest.






§ ANALYSIS






 §.§ Which MLLM is the Best Annotator?
 


The annotation of  heavily relies on . Although many researchers pointed out that  capable of providing meticulous judgments and feedback <cit.>, we still concern whether the  preferences are qualified.

We then conduct a probing experiment by utilizing different MLLMs, , ♊ and , to provide their preference on two existing human-preference datasets (HPD <cit.> and ImageRewardDB <cit.>). The corresponding pair-wise preference prediction accuracy is shown in Figure <ref> (a). We observed that the accuracy of  surpasses that of both  and ♊ on both datasets, achieving accuracy rates exceeding or approaching 70%, and  notably scoring significantly lower than the former two. According to previous research <cit.>, the agreement rate between qualified human annotators is also around 70% (65.3% for ImageRewardDB and 78.1% for HPD). Therefore, the probing experiment validates that  can be a well human-aligned annotator, thus ensure the quality and reliability of our .

To further validate the efficacy on preference annotation ability of , we utilize ♊ and  to collect similar amount of data (1.2 M pair-wise preference choices) following the same collection pipeline described in Section <ref>. Then we train the corresponding reward model on these two datasets and show the performance in Figure <ref> (b). As we can see, consistent with the aforementioned conclusion, the testing accuracy of the reward model trained on data annotated by  exhibits the highest performance, followed by ♊. This demonstrates that  is currently the most proficient annotator for tetx-to-image generation.





 §.§ Encouraging GPT-4 Vision for Enhanced Annotations.


Prompt Manner. As described in Section <ref>, during the construction of , we encourage  to directly output scores for various aspects (e.g., aesthetic) of each image (denoted as score feedback). Another straightforward prompting manner (denoted as rank feedback) is encourage  to directly provide a ranking of images (α and β) in a certain aspect (i.e., α≻β, β≻α, or α = β). 

R0.4


4.5[0.04](a)

    < g r a p h i c s >


5[0.04](b)


    < g r a p h i c s >




(a) Preference prediction accuracy for score feedback and ranking feedback. (b) Visualization depicting the variation of annotation accuracy and consistency with changes in temperature τ.





It is interesting to explore which prompting manner is best suit for AI Annotators. We randomly sampled 1,000 samples from ImageRewardDB and 500 samples from HPD, and utilized the two aforementioned prompt manners to ascertain 's annotations. The results are presented at Figure <ref> (a), we observe that in both datasets, the accuracy achieved using score feedback is higher than that achieved using rank feedback. 

Temperature τ. Temperature τ is a hyperparameter used in multimodal large language models (e.g., ) to control the randomness and creativity of the generated results. A lower value of the temperature parameter will lead to a more predictable and deterministic output, while a higher value will produce a more random and surprising output. 

We investigate the influence of the variation in τ on both the accuracy of annotation and annotation consistency (where the same input yields identical annotation results). The results are shown in Figure <ref> (b), we observe a decrease in accuracy as τ increases, indicating that lower values of τ should be set when conducting preference annotations. 


Furthermore, as τ increases, annotation consistency continues to decline, which is sensible because more randomness in the results leads to different preference outcomes for identical inputs over time.



 §.§ Fine-Grained Feedback Leads to Better Results.




















Better Reward Modeling. In our previous experiments, we used the average score of each sample across four evaluation aspects as the final preference score for modeling reward or optimizing generation models. Here, we explore the impact of separately modeling the four evaluation aspects. We first train four reward models on these four different aspect in , namely , ,  and , respectively. The corresponding preference accuracy are presented at Figure <ref>. We can observe that the accuracy of reward models individually trained using a single aspect preference data is consistently lower than , which validates the effectiveness of our approach in designing four evaluation aspects to model the preference level.











Better Prompt-Following. We found fine-grained preference data enables our fine-tuned model to generate images that better adhere to the input prompt. For instance, as shown in Figure <ref>, we find the top-3 sampled images generated from our fine-tuned model all fulfill the requirement of "holding" as specified in the prompt. In contrast, among the baseline models, only HPS v2 achieves this.







More Aesthetically Pleasing. Fine-grained data enhances the visual appeal and vividness of images generated by our model. As shown in Figure <ref>, our results exhibit enhanced luminosity, dynamic sensation, and increased detail, aligning more closely with human aesthetic preferences.

Reduce Image Distortion. One aspect measured within our fine-grained data is the fidelity score of images, which ensures images remain undistorted and accurately represent the intended subject. We demonstrate that scoring at this granularity level enables our model to generate more precise results. In Figure <ref>, we observe that our model achieves the highest accuracy in generating hand images, while other comparative models exhibit varying degrees of deformation.

Enhance Image Safety. We employ unsafe prompts provided in <cit.> to generate 1 K images and utilize the built-in NSFW detector in the Diffusion library to quantify the frequency of generating harmful content. We find that  fine-tuned generative model's NSFW ratio (4.4%) is significantly lower than that of other models, being 5× lower than HPS v2 fine-tuned generative model (21.1%) and 4.8× times lower than PickScore fine-tuned generative model (22.3%). This indicates that the harmlessness evaluation incorporated into our fine-grained preference scoring mechanism effectively reduces the generation of harmful outputs by the generative model. More related details can be found in Appendix.


















§ CONCLUSION

In this paper, we explore utilize AI  annotators to construct a large-scale high-quality feedback dataset, , for diffusion models alignment and refining. Costly experiments conducted across various experimental settings have validated the efficacy of our . This also represents a comprehensive and substantial endeavor by RLAIF in the realm of visual generative models, demonstrating the effectiveness of utilizing AI-synthesized data for aligning visual generative models. 

Limitations and Future Direction.  provides three types of feedback data for each item, but we have not yet utilized the textual explanations data, which will be a interesting direction for our future exploration.









splncs04



 


Wu et al.


—Appendix
    
    April 29, 2024
==================















We organize our appendix as follows:

    
  * In Section <ref>, we augment the primary outcomes of both the PPO and DPO experiments by incorporating two new validation benchmarks, alongside additional human preference studies and qualitative results.
    

    
  * In Section <ref>, we further validate the efficacy of our finely-grained preference annotation approach through comprehensive ablation studies, augmented human preference evaluations, and extended visualization results.
    

    
  * In Section <ref>, we present the statistical analysis of , revealing that preference labels annotated by MLLMs demonstrate characteristics akin to those found in human-annotated preference datasets.
    

    
  * In Section <ref>, we provide training details of the experiments discussed in the main text.
    

    
  * In Section <ref>, we analyze the annotation costs of constructing  and demonstrate that employing MLLMs as annotators not only achieves results aligned with human annotators but also significantly reduces labor resources and time costs.
    

    
  * In Section <ref>, we provide the corresponding prompt templates used in employing  to polish existing prompts and employing  to generate preference annotations.






§ ADDITIONAL MAIN RESULTS


In this section, we present additional quantitative and qualitative results to further validate the efficacy of  and . 

Beyond DiffusionDB <cit.> test-set, we broaden our validation efforts by incorporating two additional benchmarks ReFL test-set <cit.> (250 examples) and HPD v2 test-set <cit.> (500 examples), into our assessment framework.



 §.§ Additional PPO Results
 


Quantitative Results. We detail the win count and win rates of generative models optimized with different reward models against  baseline on Table <ref>.

These results reveals that  consistently outperforms ImageReward and demonstrates competitive performance against two another reward models (PickScore and HPS v2), which are trained on nearly a million human-annotated preference data. For instance,  achieves the highest win rate in the ReFL test-set, surpassing all other reward models. In DiffusionDB and HPD v2 test-sets,  secured the second-best and third-best performances, with marginal differences from HPS v2 and PickScore. 

Figure <ref> displays the win rates of the generative model optimized with  against to generative models optimized with alternative reward models across three test benchmarks. It is observed that the generative model optimized with  achieves a win + tie rate exceeding 50% when benchmarked against others, underscoring the effectiveness of 

Qualitative Results. Additional qualitative results are presented in Figure <ref>. When compared to the  baseline generative model (simplified as SD 1.5), the generative model optimized with the guidance of  is capable of producing images with richer details and more closely aligned with the given prompts. This showcases competitive performance with generative models optimized through reward models such as HPS v2 and PickScore, demonstrating the effectiveness of .











 §.§ Additional DPO Results



Quantitative Results. We detail the win count and win rates of generative models fine-tuned on different preference datasets against  baseline on Table <ref>. 

Through our experimental results, it is observed that within the DPO framework, our  demonstrates competitive effectiveness when compared to human-annotated preference datasets. For instance, as shown in the Table <ref>, generative models trained on  achieves the highest win rate in both ReFL and HPS v2 test-set, surpassing these two generative models trained on HPD and Pick-a-Pic. Moreover, within DiffusionDB test-set, generative models trained on  secures a close second-best performance, achieving results comparable to generative models trained on HPD and Pick-a-Pic. 

Figure <ref> showcases that generative models trained with  exhibit competitive performance relative to models trained on alternative human-annotated preference datasets, achieving a win + tie rate that exceeds the 50% threshold across three testing benchmarks. This evidence highlights the efficacy of .


Qualitative Results. We provide more qualitative results in Figure <ref>. The visualization results indicate that generative models fine-tuned on  are capable of producing outputs that closely match human preferences, generating images that are more appealing to human users.














§ EFFICACY OF FINE-GRAINED FEEDBACK


In this section, we present additional ablation studies and qualitative results to validate the efficacy of fine-grained feedback design.



 §.§ Better Prompt-Following.
 

To further substantiate that “Prompt-Following” rating labels in  can enhance the ability of fine-tuned models to generate images that more accurately align with the input prompts, we removed the “Prompt-Following” rating labels from the , preserving labels for the other three aspects, and train a reward model, which we named ^†. An additional human preference study was conducted on the DiffusionDB, specifically focusing on the aspect of Prompt-Following. The outcomes, presented in Table <ref>, indicate two key findings: Firstly,  exhibits competitive performance in comparison to HPS v2. Secondly, the efficacy of ^† experiences a notable decline with the omission of Prompt-Following rating labels. These results decisively confirm the critical role of “Prompt-Following” rating labels in enhancing the model's proficiency in adhering to prompts, thereby facilitating the generation of images that more precisely reflect the provided descriptions.

Besides, we provide additional visualization results in Figure <ref> to validate that generative models guided by  are capable of producing images that more closely adhere to the descriptions provided in the prompts.



 §.§ More Aesthetically Pleasing.
 Similar to the last section, we removed the “Aesthetic” rating labels from the  and trained a reward model named ^. Subsequent to this, we embarked on an additional human preference study utilizing the DiffusionDB, with a singular focus on the dimension of “Aesthetics”. The findings, elucidated in Table <ref>, revealed that  achieved the best performance, while the exclusion of aesthetic labels markedly diminished the operational efficiency of ^. This phenomenon starkly highlights the integral value of aesthetic rating labels.

Further, we showcase additional visual outcomes in Figure <ref>. Our observations indicate that generative models refined under the auspices of  manifest the capacity to engender imagery replete with more vibrant detail and sophisticated interplays of light and shadow.



 §.§ Reduce Image Distortion.
 To ascertain the impact of “Fidelity” rating labels, we excised these labels from the  and subsequently trained a reward model ^. A human preference study, concentrated solely on the “Fidelity” aspect, is documented in Table <ref>. This study utilized the “anything” prompts delineated in <cit.>, encompassing 442 prompts, as the evaluation benchmark. The outcomes illustrate that generative models guided by our  manifest competitive performance. In contrast, ^ exhibits a discernible performance decrement relative to .

Additionally, the visualization results showcased in Figure <ref> demonstrate that models optimized under the guidance of  excel in producing images with diminished distortion, e.g., less distortion of human hands. Note that image distortion, particularly the deformation of hands and limbs, is a common issue with diffusion generative models. Our “Fidelity” assessment can only mitigate, not eliminate, this phenomenon. Therefore, we look forward to the development of more robust techniques to address this drawback.




 §.§ Enhance Image Safety.
 Similarly, we removed the “Harmlessness” labels from the  and trained a corresponding reward model named ^.

Then we employ unsafe prompts provided in <cit.> to generate 1K images and utilize the built-in NSFW detector in the diffusion library[<https://github.com/huggingface/diffusers>] to quantify the frequency of generating harmful content. Detailed results is presented in Figure <ref>. We find that ^, trained without the “Harmlessness rating” labels, exhibited a significant increase in the NSFW ratio compared to the original  (4.4% to 20.2%). This further underscores the importance of “Harmlessness” labels.





















































§ STATISTICS OF 


In this section, we provide more details of . We encourage readers to delve into Section <ref>, where we analyze the characteristics of preferences generated by . This analysis reveals that the generated preferences exhibit properties remarkably similar to those of human-annotated preferences. Such findings serve to demonstrate the capability of MLLMs to closely align with human judgment and preferences in the context of text-to-image generation.



 §.§ Prompts.
 A key step in  contruction pipeline is utilizing  to polish the existing prompt benchmarks. This process is designed to reduce potential biases and inconsistencies in user-generated terminology. We present examples of original prompts alongside their polished counterparts at Table <ref>, and quantitatively illustrates the frequency distribution of certain stylistic words and conflicting prompts at Figure <ref>. Our analysis reveals that the post-polish prompts not only align more closely with conventional expression norms but also demonstrate a significant reduction in the use of stylistically charged and specific words, such as platform and artist names. Moreover, the occurrence of prompts with conflicting information witnessed a marked decrease post-cleanup. As a result, these polished prompts are better suited for use as training data, meticulously crafted to minimize bias and enhance the model's robustness and generalization abilities.







 §.§ Images.
 Within , images are generated by employing four state-of-the-art text-to-image generative models. These models are ranked as the top four on the Hugging Face leaderboard, specifically: [<https://huggingface.co/runwayml/stable-diffusion-v1-5>], [<https://huggingface.co/stabilityai/stable-diffusion-2-1>], [<https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0>], [<https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0>]. Detailed descriptions of these models and the distribution of images generated by each within our dataset are methodically outlined in Table <ref>. For illustrative purposes, Figure <ref> showcases representative images produced by each of these models. 












 §.§ Preferences.
 

All preferences in  were generated using  API. As stated in the main text, we designed four evaluation aspects to assess the quality of each data entry. For each aspect, we individually invoked the  API to generate the corresponding preferences, the corresponding prompts regarding to these four aspects can be found depicted in Section <ref>. Please refer to Table <ref> to see some annotation examples.

We illustrate the distribution of 's ratings across four distinct aspects within  in Figure <ref>. Our analysis discerns a relatively even distribution of ratings for the Prompt-Following aspect, where the allocation of ratings from 1 to 5 is almost uniform. Conversely, in the domains of Fidelity and Harmlessness,  exhibits a propensity towards assigning the highest rating of 5 to a predominant share of the samples. This pattern suggests that the majority of generated images are free from substantial distortions and objectionable content.

 offers a unique opportunity to leverage ’ preferences for unbiased analysis. A critical step in the construction of  is the random application of class-free guidance values (from 3 to 12), aiming to enhance the generalization of . We analyze the impact of changing the class-free guidance values of  on its performance. For each guidance value, we compute the win ratio, representing the percentage of judgments where its use led to a preferred image. We also calculate the corresponding tie and lose ratios for each scale, enabling a detailed analysis of which classifier-free guidance scales are more effective. The results are presented at Figure <ref> (a), and we find higher guidance value always lead to a higher win rate, e.g., a guidance value of 9 usually yields preferred images when compared to a guidance value of 3. This conclusion is in well agreement with the conclusions obtained in the human-annotated preference dataset <cit.>.

Additionally, by comparing preferences for images generated by different generative models in response to identical prompts, we are able to identify the model that is more favorably rated by . For instance, considering judgments in which one image was generated by  and the other by , we can evaluate which model is more performant. As shown in Figure <ref> (b), we find that  typically exhibits a preference for  over . This preference aligns with the characteristics of human-annotated preference data <cit.>, demonstrating a consistency between the MLLMs generated preferences and human judgment.










§ TRAINING DETAILS




 §.§ Reward Model Training

Following <cit.>, we load the pre-trained checkpoint of BLIP (ViT-L for image encoder, 12-layers transformer for text encoder) as the backbone of , and initialize MLP head according to 𝒩(0, 1/(d_model+1)) decaying the learning rate with a cosine schedule. To avoid the overfitting of reward model during training phase and reach up to the best preference accuracy,  is fixed 70% of transformer layers and is trained on 4 × 32 GB NVIDIA V100 GPUs, with a per-GPU batch size of 16.



 §.§ Boosting Generative Models



PPO. Following the setting in ReFL <cit.>, we fine-tuned all text-to-image generative models employing the PNDM noise scheduler and half-precision computation on an array of 8 × 32GB NVIDIA V100 GPUs. The process utilized a learning rate of 1 × 10^-5 and a total batch size of 64 (32 for pre-training and 32 for ReFL).

DPO. Following the setting in <cit.>, we conducted a total of 400 epochs during the training process, utilizing a learning rate of 3 × 10^-5 and the Adam optimizer, alongside half-precision computation. This was conducted on a configuration comprising 8 × 32GB NVIDIA V100 GPUs.































































































































§ COST OF  CONSTRUCTION


One of the primary motivations for utilizing MLLMs as annotators is their ability to significantly reduce the cost of data construction compared to human annotators. Take construction process of the two largest existing human preference datasets as an example, during the construction process of Pick-a-Pic <cit.>, approximately 6,394 web users participated in tagging images with their preferences. For the development of HPD v2 <cit.>, a total of 57 high-quality annotation experts were employed and trained to construct preference labels. Moreover, these annotators were required to meticulously adhere to the annotation standards provided by the system throughout the labeling process. Thus, it is evident that obtaining large-scale humans preference annotation is time-consuming, resource-intensive, and laborious, which hinders the progress of related research. 

In contrast, employing MLLMs for annotation can effectively overcome these limitations. 

Utilizing the construction process of  as an example, each invocation of the  API is capable of tagging four images with preference labels pertaining to a specific aspect (e.g., prompt-following aspect), meaning a single API call can generate C^2_4 preference ranking results in that aspect. Throughout the construction of , each  API can accommodate approximately 10,000 requests per day, thus generating around 60,000 preference ranking results in a given aspect per day. In the specific construction process, we employed two APIs for parallel annotation, with the total annotation process taking approximately 15 days.


This efficiency and cost-effectiveness are significantly superior to using human expert annotations. Moreover, despite the minimal cost and high efficiency, the reliability and quality of the preference labels provided by MLLMs are not compromised.































































































§ PROMPT INSTRUCTION TEMPLATES





 §.§ Prompt Polish Instruction









 §.§ Preference Instruction













