Iterative DPO with An Improvement Model for Fine-tuning Diffusion Models
DPO;diffusion models
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
Yes
No ethics review needed.
Is the diffusion model sensitive to the CFG weights of positive samples in preference optimization?
1. This paper proposes an online DPO method with preference improvement model to expand the size limitation of the preference dataset in optimization.
2. Muti-task learning objectives are proposed for the preference improvement model for ensured quality and diversity.
3. Experiments show the effectiveness of the proposed method.
This paper proposes an online DPO method to iteratively improve the preference of the diffusion model, and expand the limitation of fixed size preference dataset in optimization. The preference pairs are generated from the original diffusion model and the preference improvement model that trained on the offline preference dataset. Experiments show the effectiveness of the proposed method.
1. The main idea of this work is to provide online preference positive samples through constructing the preference improvement model that maps the outputs from the original diffusion model to high-quality ones. It is still under the static Diffusion-DPO framework without dynamic adaption, which is somewhat less convincing.
2. It is wondered whether the preference improvement model needs to be evolved in iteration, as the fixed improvement model can be hard to adapt to the shifted distribution of the evolved diffusion model, and the positive samples provided by consequent preference improvement are doubted to bias from the original user preference. Additionally, the fixed improvement model is still setting the upper bound of the optimized diffusion model, with only enlarged optimization samples.
3. It is recommended to provide the per-iteration result to monitor the diffusion model behavior.
4. The role of guidance weights in yielding positive samples is not well illustrated with corresponding qualitative or quantitative results. Basically, we have no idea about the behavior of the improvement model in terms of its multi-objective learning. The validation experiments should be provided.
Yes
This paper has no ethical concerns.
No ethics review needed.
You need to provide a detailed explanation addressing the issues raised in **Weakness 1**, specifically whether this situation was encountered during implementation.

Also, conduct experiments on the number of iterations to verify the reasonableness of the iterative training process.
The writing is relatively clear, and the investigation is fairly thorough.
This paper introduces an approach for online iterative optimization of the diffusion models without introducing extra annotation of the online data. It uses the generated images from the improved models as winning samples to continue improving the base model in multiple iterations.
**Idea is unreasonable:** Using the generated images from the improved models as winning samples is not a reasonable approach. Due to the instability of the generative model, even with the same seed, it is difficult to ensure that the images generated by the improved models are consistently better in quality than those from the base model. This introduces a significant amount of noisy data, which can severely affect the quality of model training. Furthermore, the method involves multiple iterations of training, which exacerbates this issue.

**Lack of novelty:** The paper merely proposes an iterative DPO method without providing a well-thought-out solution to the core problem, making its novelty and contributions quite limited.

**Unreasonable experiments:** Firstly, a considerable number of the visual examples presented in the paper do not demonstrate the advantages of the proposed algorithm. The generated images do not align well with the prompts, and the improvements over the baseline model seem quite limited. Does this suggest that the method’s effectiveness is rather underwhelming? To verify the validity of this approach, I think it is necessary to conduct experimental analysis on the number of iterations.
Yes
No ethics review needed.
Please refer to the weakness section. Moreover, authors should illustrate the detailed reasons to choose such a complicated alignment process instead of directly training a reward model and applying RL.
The method is easy to understand. The visualization is good. The method is reasonable.
In this paper, authors propose a diffusion model alignment method with a limited number of preference datasets. Specifically, an improvement model is trained for generating high-quality samples. Then DPO is achieved on an extended synthetic dataset.
1. The authors should provide precise definitions and annotations for each concept, such as $\epsilon^w$ and $\epsilon^l$.
2. The proposed method introduces a semi-supervised learning framework to utilize a preference dataset for training a "high-quality" model, followed by fine-tuning the diffusion model using generated high- and low-quality data pairs. However, this approach appears to complicate the alignment process for diffusion models. A more straightforward solution would be to train a reward model directly, enabling reinforcement learning without the need to construct a high-quality model and apply DPO in such a rudimentary way.
3. Additionally, the performance gains achieved by the proposed method appear limited. As shown in Table 1, the results are only comparable to those of SPIN.
4. The authors should refine the contributions section, as the current version lacks clarity and specificity.
5. The authors should consider including a comparison with alignment methods based explicitly on reward model [1].
[1] Zhang, Y., Tzeng, E., Du, Y., & Kislyuk, D. (2024). Large-scale reinforcement learning for diffusion models. arXiv preprint arXiv:2401.12244, 4.
Yes
No ethics review needed.
I suggest the authors validating their method on larger models like Stable Diffusion XL and Flux. Otherwise I am deeply suspicious of its effectiveness on larger models. They should show the potential of scaling up of their method for practical application.
1. Reasonable storytelling and clear figures.

2. The method is easy to understand and implement.
The paper introduces the drawback of standard DPO methods that they are limited by their training data as the motivation of the paper.

To be specific: It first use pre-collected paired data to train a model similar to instruct-p2p (improvement model), which can be used to enhance the quality of input images.

The proposed method iteratively trains the   diffusion model and generate new "bad" samples with the diffusion model. The improvement model is used to generate "good" data given the new model.
**Validation**

1. The performance improvement is very minor. From Table 4, we can see that the win-rate improvement is at most 0.9% compared to SPIN. I also think the presented paired visual comparisons in the paper (those figures) are not impressive.

2. The validation is not sufficient. It only validates performance on stable diffusion 1.5. Many works including Diffusion-DPO report their performance on both stable diffusion 1.5 and stable diffusion XL. The generation quality of stable diffusion 1.5 is not good and therefore improving generation quality from it is much easier than stable diffusion XL. It makes me suspicious about its effectiveness on better and larger models.

3. Validation of the effectiveness of the improvement model. I didn't find any validation on how much the improvement model can improve the given images. (I am sorry if I missed them.) The only available evaluation on the improvement seems to be the Table 3, which can not reflect the ability of the improvement model. The improvement model even performs worse than SPIN on test set. How can we expect it to improve the quality of generated images.

4. Many experimental details are not clearly stated. How is the win-rate calculated in Table 4? 

5. Lack of user study including comparison and  details. 


**Reasonability**

1. Since all good images are generated with the improvement model, I think the generation quality might be upper-bounded by the improvement model. I am suspicious about the reasonability of the proposed method.

2. Generating new samples during training takes a lot of computation and time. I noticed in the paper that the authors train the improvement model 200K steps with batch size 2048, which is a very costly training compared to standard diffusion-DPO (They train their model with only 2K iterations with a similar batch size 2048).   The very high training cost might also be the reason that they do not train their model on SDXL or even larger models. 

**Presentation**
1. Most presented images are not visually appealing and their prompts are generally very short. It can not reflect the text-image alignment.


Due to the above reason, I think the paper still has a relatively large gap for publication and might not meet the bar of ICLR.
@misc{
fan2024iterative,
title={Iterative {DPO} with An Improvement Model for Fine-tuning Diffusion Models},
author={Ying Fan and Fei Deng and Yang Zhao and Sahil Singla and Rahul Jain and Tingbo Hou and Kangwook Lee and Feng Yang and Deepak Ramachandran and Qifei Wang},
year={2024},
url={https://openreview.net/forum?id=KJF3h0OpQ7}
}
Direct Preference Optimization (DPO) has been proven as an effective solution in aligning generative models with human preferences. However, as shown in recent works, DPO could suffer from constraints from the offline preference dataset. This paper introduces a novel improvement approach for online iterative optimization of the diffusion models without introducing extra annotation of the online data. We propose to learn a preference improvement model to extract the implicit preference from the preference dataset. The learned improvement model is then used to generate winning images from the images generated by the current diffusion model. We can construct new pairs of preference data by using images generated by the current diffusion model as losing images, and its corresponding improved images as winning images. The diffusion model can therefore be optimized via iteratively applying online preference datasets. This method enables online improvement beyond offline DPO training without requiring additional human labeling or risking overfitting the reward model. Results demonstrate improvements in preference alignment with higher diversity compared with other fine-tuning methods. Our work bridges the gap between offline preference learning and online improvement, offering a promising direction for enhancing diffusion models in image generation tasks with limited preference data.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Ying_Fan2 ~Fei_Deng1 ~Yang_Zhao5 ~Sahil_Singla1 ~Rahul_Jain1 ~Tingbo_Hou2 ~Kangwook_Lee1 ~Feng_Yang2 ~Deepak_Ramachandran2 ~Qifei_Wang3
Ying Fan Fei Deng Yang Zhao Sahil Singla Rahul Jain Tingbo Hou Kangwook Lee Feng Yang Deepak Ramachandran Qifei Wang
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
DPO diffusion models
I certify that there is no acknowledgement section in this submission for double blind review.
fan|iterative_dpo_with_an_improvement_model_for_finetuning_diffusion_models
/pdf/4b644558ef2ca3478f977cce5afd6a8f788ee804.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Iterative DPO with An Improvement Model for Fine-tuning Diffusion Models
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission