PoGDiff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generation
Diffusion Model;Probabilistic Methods
Yes
use face images
Yes, Discrimination / bias / fairness concerns
see weakness
This paper addresses the issue of imbalanced training data in diffusion models and proposes a novel fine-tuning method. The core idea is to modify the ground-truth image supervision signal during training by incorporating neighboring text embeddings. 
1) The issue addressed in this paper is valuable and important. 
2) The proposed solution appears to be reasonable. 
3) The writing and analytical approach of the paper are clear. 
4) The experiments also demonstrate the effectiveness of the proposed method.
This paper argues that current Diffusion models are trained on imbalanced datasets.
To solve this problem, they propose a fine-tuning framework, PoGDiff.
PoGDiff replaces the ground-truth distribution with a Product of Gaussians (PoG), which is constructed by combining the original ground-truth targets with the predicted distribution conditioned on a neighboring text embedding.
Experiments show that PoGDiff effectively addresses the imbalance problem in diffusion models, improving both generations' accuracy and quality.
Although the approach of PoGDiff is reasonable and effective, and I understand that the addition of text embeddings can increase the diversity of the supervision signal, I still have the following concerns: 1) From the results shown in Figure 1, some of the images generated by PoGDiff exhibit noticeable deviations in color and other aspects from the ground truth (GT). Does this modification align with the expected outcomes? 2) There are already several custom techniques that can achieve diversity with just a single or a few new style images, and in some cases, without any training. The fine-tuning method proposed by the authors might degrade the performance of the original model. How should we evaluate this? 3) The proposed method essentially resembles data re-weighting, yet the experiments lack comparisons and detailed analyses with similar methods.
Yes
No ethics review needed.
see weakness.
1. PoGDiff introduces the use of Gaussian products for fine-tuning in imbalanced datasets, an original approach that improves minority class image generation.

2. The paper provides theoretical analysis, showing that PoGDiff retains diffusion model properties while better representing minority classes.
This paper presents Product-of-Gaussians Diffusion Models, a approach to fine-tuning diffusion models for imbalanced text-to-image generation. PoGDiff addresses the challenges of generating minority class images by using a product of Gaussians (PoG) to combine original target distributions with nearby text embeddings. Experimental results show that PoGDiff outperforms traditional methods like Stable Diffusion and Class Balancing Diffusion Model (CBDM) across various datasets, particularly enhancing generation for minority classes.
1. Experiments are primarily conducted on AgeDB-IT2I and DigiFace-IT2I, which may not fully represent real-world, large-scale imbalanced datasets. Additional testing on broader datasets is necessary.

2. PoGDiff relies on neighboring samples for minority class improvement, which may be less effective in sparse data settings. There is a lack of discussion on how the model handles extremely sparse data.
Yes
No ethics review needed.
see Weaknesses
1. The authors introduce the Product of Gaussians technique to sample mixed prompts, which alleviates the need for numerous small image-text pairs in the training dataset. This technique allows one image to interact with more prompts, thereby increasing generative potential.
   
2. The approach of using multiple texts to represent a single image is a reasonable consideration.

3. The authors provide theoretical support for their proposed method, which is intriguing.

4. Extensive experiments are conducted to support this idea.

5. The paper presents valuable insights into the proposed methodology.
This paper aims to enhance the performance of diffusion models when trained or fine-tuned on imbalanced datasets. Instead of relying on a single prompt, the authors align one image with multiple group prompts sampled from the training data. To achieve this, they employ a Product of Gaussians technique. The authors conduct various experiments to demonstrate that their method is effective.
1. The authors consider the use of non-target prompts for the current image, which may introduce noise and misalignment. This could result in generated images that do not align well with the prompts, potentially leading to lower CLIP scores, a metric that is not reported in the paper. Thus, there may be issues with text-image alignment.

2. The baseline comparisons are limited, focusing only on SD and CBDM, which may not be sufficient to fully validate the proposed idea.

3. As shown in Table 1, the proposed method does not demonstrate a significant advantage compared to the baselines, leaving me unconvinced about its effectiveness.

4. The visualizations do not clearly support the proposed method. For instance, Figure 5 reveals a lack of diversity in the generated outputs.
Yes
No ethics review needed.
1.	Artifacts from PoGDiff appear to be present in the images generated at low density (e.g., Figure 1, lower left corner, J. Willard Marriott), but not in those generated at high density. Is this a result of the model's limitations?
2.	The paper mentions that when training a diffusion model on an imbalanced dataset, existing models often struggle to generate accurate images for less frequent individuals. Personalized methods (e.g., CustomDiffusion, PhotoMaker) can use 3 to 5 images to learn an identity and generate accurate images for these less frequent individuals. What is the difference between PoGDiff and personalized methods that learn a specific identity?
CustomDiffusion: Multi-Concept Customization of Text-to-Image Diffusion
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
3.	How to obtain the ground-truth distribution $q(x_{t-1}|x_t, x_0,y)$, when given $x_t$, $x_0$, and $y$ ?
4.	The $y^{'}$ in line 167 and the $y^{'}$in line 169 should be the same symbol.
5.	Fig. 3 is interesting, but the type and amount of data used in Fig. 3 is quite confusing to me.
6.	Equation 7 neglects $y^{'}$.
7.	Does the distance between the current text embedding $y$ and the sampled $y^{'}$ significantly affect the final generated results?
1.	Selecting neighboring embeddings from other samples is an interesting approach, as it can help get a new density.
2.	Figure 3 is interesting, but it would benefit from a more detailed explanation.
The paper proposes a novel method PoGDiff to address the long-tailed data distributions cause by the imbalanced datasets. This paper proposes a general fine-tuning approach, replacing the ground-truth distribution with a Product of Gaussians conditioned on a neighboring text embedding. Experiments are conducted on AgeDB-IT2I and DigiFace-IT2I using FID, DINO, Human Score and GPT-4o Evaluation evaluation metrics.
1.	“Encouraging the model to generate the same image given similar text prompts” may result in a loss of diversity in the generated images. How can this drawback be overcome?
2.	The paper mentions that in diffusion models, a data point is affected only by its text embedding. However, even with the same text embedding, different latent codes can produce images of varying quality. Additionally, classifier-free guidance and negative prompts also influence image generation. 
3.	Why is directly smoothing the text embedding not feasible?
4.	What is the basis for hypothetically defining $\sigma_{y'}^2 = \frac{\sigma_{t}^2}{\psi[(x,y), (x',y')]}$ ?
5.	What does ‘Cat’ refer to in line 249? It doesn’t seem to be explained in the paper.  The author should define or explain this term when it's first introduced
6.	What does the superscript of s in Equation 9 represent? The previous definition of s did not include a superscript (e.g., Equation 8).
We propose a general fine-tuning approach to address the performance drops on the imbalanced text-to-image generation tasks.
@inproceedings{
anonymous2024pogdiff,
title={Po{GD}iff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ACEuJBhhbN},
note={under review}
}
Diffusion models have made significant advancements in recent years. However, their performance often deteriorates when trained or fine-tuned on imbalanced datasets. This degradation is largely due to the disproportionate representation of majority and minority data in image-text pairs. In this paper, we propose a general fine-tuning approach, dubbed PoGDiff, to address this challenge. Rather than directly minimizing the KL divergence between the predicted and ground-truth distributions, PoGDiff replaces the ground-truth distribution with a Product of Gaussians (PoG), which is constructed by combining the original ground-truth targets with the predicted distribution conditioned on a neighboring text embedding. Experiments on real-world datasets demonstrate that our method effectively addresses the imbalance problem in diffusion models, improving both generation accuracy and quality.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Diffusion Model Probabilistic Methods
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/aa9ecf5bc2823f7fe5fdbbaa40f55971f2fb8b20.pdf
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
PoGDiff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission