TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation
Diffusion Models;Custom Concepts;Text-to-Image;Video
Yes
No ethics review needed.
- Please see the weaknesses.
+ The proposed method is reasonable and intuitive.
+ The generated results of multi-concept fusion are impressive.
+ The model is properly extended from image generation to video generation.
+ From the quantitative results, the proposed method outperforms previous models.
This paper proposes a method to fuse multiple concepts for personalized T2I diffusion-based models that are generalizable to both image and video generation. The method mainly involves two parts: (1) concept-aware sampling that coarsely localizes the concepts in the latent space, and (2) multi-concept fusion that generates the personalized concepts based on the extracted regional masks. The qualitative and quantitative results both demonstrate the effectiveness of the proposed method.
- $t_con$ and $P$ are two important hyperparameters. Can the authors provide generated results with varying $t_con$ and $P$, while keeping all other settings fixed?
- DINO [1] can extract subtle visual features that are not specific to any particular category. The metric of DINO image similarity should also be reported.
- The evaluation is conducted on combinations of more than three concepts; however, the paper does not present any results of combinations involving more than three concepts.
- It is recommended to compare the sampling time of the proposed method with that of other models.
- Missing reference on unsupervised multi-concept extraction [2].
 
[1] Caron etal. Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021.
[2] Hao et al. ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extraction. ECCV 2024.
Yes
No ethics review needed.
Please refer to the weaknesses above.
1. The idea is easy to follow and the paper is well written.
2. The results on Custom Concept 101 dataset outperforms the previous baselines, demonstrating better generation qualities compared to the concept personalization methods. 
3. The author's code has been open-sourced, which is somewhat helpful to the community.
This paper introduces a new approach called TWEEDIEMIX for multi-concept text-to-image generation. The author introduce a tuning-free approach during the inference stage and divides the process into two main stages. The key contribution of this paper is using a resampling strategy and multi-concept fusion sampling enabling generate multiple personalized concepts with higher fidelity than existing methods.
1. My main concern lies in the technical contribution of this paper. It seems like an incremental work of ConceptWeaver[1] which is also a training-free method that combines multiple concepts during inference.
2. In terms of qualitative results, there doesn't seem to be a significant improvement compared to ConceptWeaver which is also can handle more than two concepts.


[1] Gihyun Kwon, Simon Jenni, Dingzeyu Li, Joon-Young Lee, Jong Chul Ye, and Fabian Caba Heilbron. Concept weaver: Enabling multi-concept fusion in text-to-image models. In Proceedings of the IEEE/CVF  (CVPR), pp. 8880–8889, June 2024.
Yes
No ethics review needed.
1、Making use of Text-SAM and Tweedie’s formula do not contribute to the novelty.
2、Each concept a model. I suspect the performance gain may derived from the overfitting of specific concept. And the hard mask composition seems like a naive post processing. The whole framework is more like a system which is designed under substantial ablation experiments and lack of novelty and essential understanding towards this task.
3、The extension to video, if I do not misunderstand, is a technical trick on a off-the-shell I2VGen-XL. I2VGen-XL is finetuned from ModelScopeT2V with a large dataset. While DreamVideo use ModelScopeT2V only for subject injection with a little amount of custom videos. Maybe this need further discussion and I'd like to see opinions from other reviews.
1、The multi-concept generation results looks good；
2、The authors illustrate the framework details and experimental setting well.
The paper introduces TweedieMix, a novel method designed to customize diffusion models during the inference phase, particularly for the challenge of integrating multiple personalized concepts in image and video generation. It operates by dividing the reverse diffusion sampling process into two distinct stages: an initial stage that uses multiple object-aware sampling to ensure the inclusion of desired target objects, and a later stage that employs Tweedie’s formula to blend the appearances of custom concepts in the de-noised image space. The results indicate that TweedieMix outperforms existing methods in generating images and videos with multiple personalized concepts with higher fidelity.
1、The whole pipeline seems naive and lack of novelty.
2、Some problems exist in experimental settings.
See questions for more details.
In this paper, we introduce Tweediemix, a method for composing customized diffusion models during the inference phase.
@inproceedings{
anonymous2024tweediemix,
title={TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ee2c4MEx9l},
note={under review}
}
Despite significant advancements in customizing text-to-image and video generation models, generating images and videos that effectively integrate multiple personalized concepts remains a challenging task. To address this, we present TweedieMix, a novel method for composing customized diffusion models during the inference phase. By analyzing the properties of reverse diffusion sampling, our approach divides the sampling process into two stages. During the initial steps, we apply a multiple object-aware sampling technique to ensure the inclusion of the desired target objects. In the later steps, we blend the appearances of the custom concepts in the de-noised image space using Tweedie's formula. Our results demonstrate that TweedieMix can generate multiple personalized concepts with higher fidelity than existing methods. Moreover, our framework can be effortlessly extended to image-to-video diffusion models, enabling the generation of videos that feature multiple personalized concepts.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Diffusion Models Custom Concepts Text-to-Image Video
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/30c42fe96fb4e873dd31653cf0225d1159e73eda.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission