MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance
Image Personalization;Multiple Subjects;Diffusion Models
Yes
No ethics review needed.
- Could the authors provide a bit more detail on how the M-DINO metric is calculated and how it measures multi-subject fidelity?

- I came across a recent paper [1] on a similar task and recommend testing its evaluation metric on a few cases to see if it could be helpful for assessing multi-subject fidelity.

- The qualitative results presented are impressive, particularly in handling prompts with complex interactions among multiple subjects. Could the authors provide additional examples demonstrating these capabilities, as well as discuss any current limitations?

[1] Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models (https://arxiv.org/pdf/2404.04243)
- The paper is well-written and well-organized, presenting a clear and compelling motivation for the study.
- This method is the first to introduce layout-guided zero-shot image personalization for multi-subject scenarios.
- The paper showcases impressive qualitative results, particularly in layout control capabilities across both single- and multi-subject personalization, as well as in handling prompts with complex interactions among multiple subjects.
The paper introduces the MS-Diffusion framework, a layout-guided zero-shot image personalization approach for multi-subject scenarios. The author features a grounding resampler to enhance subject fidelity with semantic and positional priors and a novel cross-attention mechanism to ensure that each subject is represented in specific areas and facilitating the integration of multi-subject data while mitigating conflicts between text and image subject control.
- In Section 2.2, second paragraph, the sentence "Though past research in this field has significantly enhanced the ability to reference single subjects, few zero-shot multi-subject personalized models" is a bit unclear. I suggest rephrasing this sentence for clarity.

- In Section 3.1, as well as in the rest of the paper, it appears that the transpose notation (T) is missing for K in the equations of calculating attention maps.
Yes
No ethics review needed.
1. Comparing to method like Subject Diffusion and GLIGEN, what makes proposed layout control novel?
    2. In figure 2, the author proposed several limitation cases identified from previous methods, however, they are not further discussed or addressed in the following paragraphs, for example, how is subject overcontrol defined? It usually better to have some specific discussion to recall and address the problems mentioned in the introduction.
    3. On the multi-subject generation benchmark, the performance boost compared to the baseline SSR-encoder remains minimal, despite the proposed method being trained on a larger and more advanced base model (SDXL versus SD1.5). This outcome appears to serve as a counterexample, challenging the effectiveness of the proposed design.
    4. The seems to produce a higher CLIP-T scores (better text following) than previous methods, however, this performance boost might also come from the usage of SDXL where multiple text encoder is designed specifically for increase text adherence. Please justify this boost comes from the proposed model design rather than the base model.
1. High-quality multi-subject personalized generation.
 2. The novel design of the grounding resampler provides extraordinary performance boost compared to previous baselines.
This paper introduces MS-Diffusion, a zero-shot personalized image generation method for multiple subject entities. Given one or more reference subjects and the text prompt, the model generates high-quality target image based on the inputs accordingly, containing the subject following the desired text descriptions.
The main contribution of the paper is the proposal of integrating grounding information for multi-subject personalized generation, and a newly designed grounding resampler for incorporating the given grounding information. 
By training a SDXL model with the encourage modules on a large-scale private dataset, the algorithm yield favorable results compared to previous designs.
• Lack of baseline comparisons and limited contributions
        ◦ The author claimed that they are the first to incorporate grounding information for multiple subject image personalization, however, plenty of existing works have proposed similar designs, for example Subject Diffusion, where the layout is also determined by bounding boxes as cross attention regularizations, while there are no related in-depth discussions.
        ◦ The comparison with previous design on the benchmark is also limited.

    • Limited performance boost
        ◦ In Table 2, the performance boost is rather trivial when comparing with SSR-Encoder on Multi-subject setting. Specifically, while SSR-encoder was trained based on SD1.5 and the proposed method is trained based on SDXL, the performance for is almost the same, except for a higher CLIP-T score.
Yes
No ethics review needed.
[1] L243-253: In the description of the cross-attention layers, in addition to the information injected from the text features, the authors show information that is injected from the reference image as well. In SD, the only features injected are those from the text. While injected image features through the cross-attention layers have been proposed (e.g, IP-Adapter), this wasn’t part of the original model. Please revise this section, and consider explicitly indicating the origin of the image features (f_i). 

[2] Grounding Resampler: the authors initialize the queries with word-embeddings and bounding-box Sinusoidal/Fourier features. But in another part of the text, the authors claim these embeddings are learnable. I am confused, is it learnable because of the query project-matrices ? Or do you pass the entity-word-embedding along with Bounding-Box features to another Projection matrix ? Can you please clarify this point.   Please provide a step-by-step explanation of how the queries are initialized and then updated during training, clearly stating which components are learnable and which are fixed.

[3] Do the results in Figure 4 correspond to the MS-Diffusion model before or after fine tuning on DreamBench ? Please clearly label all results in the paper to indicate whether they are from zero-shot or fine-tuned models.

[4] The Multi Subject Cross Attention module has been used (e.g, in ELITE). To my understanding, MSDiffusion only extends this to multi-subject use cases. Can you please highlight the differences ?

[5] It would be nice to provide dataset statistics. In particular, what percentage of the data there are multiple-subjects, excluding trivial cases like a person and clothes. At least from the qualitative samples, it looks like the model performs very well on clothing entities and a single subject. 

[6] Can you provide some information regarding the computational complexity of the method. In particular, it would be nice to report the additional overhead indeed by the method. Please report inference time and memory usage compared to baseline zero-shot models, for both single-subject and multi-subject generation tasks.
[1] The paper introduces a zero-shot multi-subject personalization method. 

[2] There are interesting components to the proposed method like the data-collection from video and the Grounding Resampler Unit. However I feel like more in-depth analysis and ablation should be carried out to validate that design choices considered are indeed important.
The paper proposes a zero-shot method for multi-subject personalization of text-to-image models. This is done through a pre-training phase where an adapter is trained to adapt a diffusion model on additional conditions derived from input images. In particular, the authors use 3.6M videos to extract two views of the same subject. Each training sample comprises segmented entities, their bounding boxes and a ground truth frame including all entities. 

The paper discusses mainly two design choices for their adapter, namely, a Grounded Resampler, and a Multi-Subject Cross Attention Layer. The grounded resampler is used to distill the subject-features through Preceiver like architecture. The authors propose to initialize the queries using entity word-embedding and bounding-box coordinates, to boost the localization of the sampling queries.

In addition, to avoid any information leakage between subjects, the authors use a Masked Cross attention where queries corresponding to a certain subject only attend the subject features.The proposed method also uses element-wise masking to avoid injected features from the reference images into background patches.
[1] The paper is hard to follow and some parts need further clarification. In particular, The “Multi-subject Cross Attention” section is not clear and needs major revision. 

Lack of ablation on critical parts of the methods: 

[2] Using videos for Data collection (e.g, in which ref. Image is different from gt. image) is claimed to be a key contribution. However, there’s a lack of ablation on how much this method helps the authors.

[3] In the Grounding Resampler; how important is the initialization (words+bounding box) to the module localization capability ?

[4] Lacking qualitative samples of interesting use-cases such as - spatial relation (e.g, On-top) and object-interactions (e.g., two subjects playing tennis). The only interesting case I find in the paper is fitted to “virtual try-on” which could be a by-product of the training data.
Yes
No ethics review needed.
1.	The grounding resampler proposed in the article also compresses image features. Why is it better than the image encoder used in existing methods? Can more qualitative analysis experiments be provided?

2.	The experiments in the article are based on Stable Diffusion XL. Now, newly emerging text-to-image diffusion models such as Flux inherently have better control over the generated content. Do these models not need an additionally trained structure to achieve satisfactory control?
1.	The paper introduced the first layout-guided zero-shot image personalization with multiple subjects framework, which consolidates the accommodation of multiple subjects, the incorporation of zero-shot learning capabilities.

2.	The idea of decoupling and controlling the generation of the texture and position of the subject in the image is very reasonable.

3.	The paper is well written and easy to follow. The elaboration of the idea is very clear, and the framework of the structure diagram is also very easy to understand.

4.	The experiments are also very sufficient.
Text-to-image generation models have advanced but face challenges in multi-subject scenarios. The MS-Diffusion framework is introduced to address these. It uses grounding tokens and feature resampler for detail, and layout guided cross-attention for location control. Experiments show it surpasses existing models in fidelity, promoting personalized text-to-image generation.
1.	The practice of controlling image generation through local cross attention is not innovative enough. It has been widely adopted in many existing layout-guided text-to-image generation methods.

2.	Judging from the quantitative and qualitative experimental results provided in the article, the improvement of image generation effect compared to existing methods is relatively limited.

3.	The extraction method of image features used by the grounding resampler proposed in the article has also been used in past papers, and its innovation is limited.
Yes
No ethics review needed.
How does it perform in scenes where the theme is people and anime? Especially the problem of combining people with anime characters.
1.The authors alleviate the problem of combining natural objects in two thematic scenarios.
2.The framework is easy to think of and sensible.
3.The writing is clear and easy to understand.
This paper proposes multiple subjects (MS-Diffusion) framework, which consolidates the accommodation of multiple subjects, the incorporation of zero-shot learning capabilities, the provision of layout guidance, and the preservation of the foundational model's parameters. 
MS-diffusion explicitly utilizes the layout information of the reference images to extract the information of multiple subjects separately to inject into the base model.
1.Missing results for multiple topics (>2).
2.Missing results for comparison with paper [1].
3.The authors do not mention the Image encoder used.To the best of our knowledge, the Image encoder if it is a CLIP may lose the details of the themes, leading to the results in the last graph of Figure 4 and Figure 5.
4.From the CLIP-I scores in Table 2, it seems that text fidelity is not significantly improved.
5.How does it perform in scenes where the theme is people and anime?
[1]Cones 2: Customizable Image Synthesis with Multiple Subjects
An approach of zero-shot image personalization supporting multiple subjects and layout guidance.
@inproceedings{
anonymous2024msdiffusion,
title={{MS}-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=PJqP0wyQek},
note={under review}
}
Recent advancements in text-to-image generation models have dramatically enhanced the generation of photorealistic images from textual prompts, leading to an increased interest in personalized text-to-image applications, particularly in multi-subject scenarios. However, these advances are hindered by two main challenges: firstly, the need to accurately maintain the details of each referenced subject in accordance with the textual descriptions; and secondly, the difficulty in achieving a cohesive representation of multiple subjects in a single image without introducing inconsistencies. To address these concerns, our research introduces the MS-Diffusion framework for layout-guided zero-shot image personalization with multi-subjects. This innovative approach integrates grounding tokens with the feature resampler to maintain detail fidelity among subjects. With the layout guidance, MS-Diffusion further improves the cross-attention to adapt to the multi-subject inputs, ensuring that each subject condition acts on specific areas. The proposed multi-subject cross-attention orchestrates harmonious inter-subject compositions while preserving the control of texts. Comprehensive quantitative and qualitative experiments affirm that this method surpasses existing models in both image and text fidelity, promoting the development of personalized text-to-image generation.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Image Personalization Multiple Subjects Diffusion Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8f8127265a5e9899b63534b32c8dd2007ecea5d3.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/a7ead7abc7c2f61e7c26d8f4fb6c2eeab80c6fb5.zip
MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission