Information Theoretic Text-to-Image Alignment
Diffusion model;Text-image alignment;Mutual information
Yes
No ethics review needed.
Referring Weakness
1. The proposed method does not require additional image datasets for training.
2. The idea is relatively novel.
This work proposes a self-supervised approach using Mutual Information (MI) for model alignment, requiring only the pre-trained T2I model and a simple fine-tuning process, outperforming current state-of-the-art methods.
1. Comparison methods, "Attend and Excite (A&E) (Chefer et al., 2023b), Structured Diffusion Guidance(SDG) (Feng et al., 2023b) and Semantic-aware Classifier-Free Guidance (SCG) (Shen et al., 2024)"  mentioned in line 329, is implemented on SD1.4, but the proposed work is implemented on SD 2.1, which is powerful than SD1.4 and may introducing evaluation bias.
2. The experiments, the train set, and the test set are split from the same dataset, but this may exist some correlations, what's the results on out-of-distribution prompts?
3. Although mutual information provides a theoretical explanation, it essentially uses classifier guidance as a loss for fine-tuning. Given that this approach is so simple, has there been any similar attempt in previous work?
Yes
No ethics review needed.
Please see the waeakness.
- The idea of introducing  self-supervised fine-tuning manner is interesting.

- Mutual Information in the pipeline is simple and effective.

- It seems is a plug-and-play module, which is useful for most T2I models.
This paper introduces Mutual Information (MI) to guide model alignment, which uses self-supervised fine-tuning manner. It relies on a point-wise MI estimation between prompts and images to create a synthetic fine-tuning set for improving model alignment.
- More detailed ablations are needed. The authors employ MI as the metric to select fine-tuning samples, which eliminates the extra usage of other models. However, what if we use SOTA VQA models as the metric? Intuitively, SOTA VQA models are more precise than the MI metric.

- An inherent drawback of MI is that it can measure how much help comes from the prompt but cannot guide in the right direction. For example, in cases of color misalignment, how should we deal with this issue?
Yes
No ethics review needed.
1. How were the 700 prompts selected for the MI quantitative analysis?
1. The main problem "Is mutual information meaningful for alignment?" is compelling and necessary, showing the potential of MI as a new direction for text-image alignment
2. The paper is well-organized and easy to follow.
3. The proposed fine-tuning approach is intersting and seems to have predictive power.
The proposed MI-TUNE introduces a self-supervised fine-tuning approach to enhance text-to-image alignment in diffusion models. At its core, the method leverages Mutual Information (MI) between text prompts and their corresponding generated images to improve model alignment, eliminating the need for human annotation. The method generates multiple images per prompt, selects the top-K aligned ones based on MI estimation, and uses these for fine-tuning.
1. MI scores are missing from comparison tables and images despite being central to the method.
2. Figure 1 only demonstrates MI effectiveness on simple category prompts (color, texture, shape), lacking validation on more challenging cases like spatial relationships or complex compositions
Yes
No ethics review needed.
Please see the weakness part. How about the performance of MI Tuning on DiT based models or Flow-based model, since the flow-based model reveals stronger generation capability comparing to traditional DDPM training. If MI tuning strategy fails to have good performance on flow-based training, the impact may be weaker.
1.	This paper claims to be the first to introduce the mutual information (MI) for T2I model training alignment. And the MI is proven to be effective on model alignment and strongly agrees with well-established metrics.
2.	MI-TUNE results on several measurements (BLIP-VQA, HPS and Human) on image features reveals state-of-the-art performance.
3.	The paper also provides the proof for MI index to have a valid point-wise manner. Introducing mutual information into T2I model tuning is interesting, it may benefit the generative model training.
4.	The paper is easy to follow with algorithm and pseudo code provided.
This paper proposes MI-TUNE, a finetuning strategy for T2I models. This strategy introduces mutual information calculation into the T2I model tuning process to guide model alignment. In brief, the tuning method uses self-supervised fine-tuning and relies on a point-wise MI estimation between prompts and images to create a synthetic fine-tuning set for improving model alignment. This paper claims to be the first to introduce mutual-information calculation into T2I model training, leading to effective guidance for model alignment. The MI index is also measured and compared with well-established metrics and user study to prove the rationality. The experiments on MI are performed and evaluated on T2I-CompBench and user studies.
1.	Note that the paper mainly focuses on SD-based (SD 2.1, SDXL) models. These models are mostly the same styles, e.g., similar network structures and traditional denoising training strategies. Is there any possibility that the MI tuning incorporated with flow-based models like DiT-based models (SD3, Pixart series or so). And it is interesting to see if the proposed MI tuning behaves different with different types of models.
2.	The evaluations on MI mainly focus on only simple semantic concepts like color, shape and texture. Is MI-tuning sensitive to object numbers or so?
3.	The paper fixes the denoising steps to 50 when inferencing an image, are there any differences in performance of MI-tuning when using different steps except 50?
4.	In quantitative analysis of Sect. 3.1, the paper mentions that the point-wise MI ranks images and select 1st, 25th and 50th as the representative images. Why the three images are representative? This needs more detailed explanations. Also, the reason of the selection needs quantitative analysis.
5.  Some of the ablations mentioned in previous sections are hard to locate in the following contents, the writing can be improved in this part.
A novel fine-tuning method for text-to-image generative diffusion models, that uses mutual information to align generated images to user intentions through natural prompts.
@inproceedings{
anonymous2024information,
title={Information Theoretic Text-to-Image Alignment},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Ugs2W5XFFo},
note={under review}
}
Diffusion models for Text-to-Image (T2I) conditional generation have recently achieved
tremendous success. Yet, aligning these models with user’s intentions still involves a
laborious trial-and-error process, and this challenging alignment problem has attracted
considerable attention from the research community. In this work, instead of relying on
fine-grained linguistic analyses of prompts, human annotation, or auxiliary vision-language
models, we use Mutual Information (MI) to guide model alignment. In brief, our method
uses self-supervised fine-tuning and relies on a point-wise MI estimation between prompts
and images to create a synthetic fine-tuning set for improving model alignment. Our
analysis indicates that our method is superior to the state-of-the-art, yet it only requires
the pre-trained denoising network of the T2I model itself to estimate MI, and a simple
fine-tuning strategy that improves alignment while maintaining image quality.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Diffusion model Text-image alignment Mutual information
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1f19f6da51432229e2851c32ff6e6ededaa58077.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/b01fcd8d0a20a9b9aa80d804021e17785e9cab9f.zip
Information Theoretic Text-to-Image Alignment
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission