Incorporating Visual Correspondence into Diffusion Model for Visual Try-On
Virtual Try-On;Image Generation
Yes
No ethics review needed.
- Could you provide more details on the local flow warping process?

- In Figure 5, are all the points shown the correct correspondences? If not, would it be possible to use different colors to distinguish correct from incorrect point mappings? Additionally, how do you assess the accuracy or correctness of these point mappings?

- It would be better to include a figure to illustrate the feature injection process, specifically showing how point features are integrated into the Main-UNet?
- The paper leverages semantic point matching as a prior to enhance garment shape and texture preservation.

- Extensive testing on the VITON-HD and DressCode datasets demonstrates the model's robustness and superior performance.

- The authors have provided the code to ensure reproducibility of their results.
This paper introduces a novel approach for virtual try-on tasks, addressing the challenge of preserving garment shape and fine-grained details. Specifically, the authors propose Semantic Point Matching Diffusion (SPM-Diff), a model that leverages structured semantic points as visual correspondence cues between garments and target human models. By mapping 2D semantic points to 3D-aware cues through depth and normal maps, SPM-Diff aligns garment details closely with the target human body. Experimental results on VITON-HD and DressCode datasets demonstrate the effectiveness of SPM-Diff.
- In line 254, local flow warping is used as a method to associate semantic points with their counterparts on the target person. It would be better to provide more detail on the local flow warping process for better understanding.

- For Figure 5, it’s difficult to assess the accuracy of the matched points. Using different colors (e.g. red and green) to illustrate correct and incorrect mappings would improve clarity.

- Adding a figure to illustrate feature injection would help clarify how point features are incorporated into the Main-UNet.
Yes
No ethics review needed.
The core idea of this paper is to integrate visual correspondence into the diffusion model. However, visual correspondence is not a new thing that is widely used in traditional try-ons. In addition, some people used warping + diffusion, such as : WarpDiffusion: Efficient Diffusion Model for High-Fidelity Virtual Try-on.

Please clarify their differences and provide more details about the novelty, not just replace "warping" with "visual correspondence".
1. By identifying and aligning stable "semantic points" between garment and human images, SPM-Diff effectively reduces randomness in diffusion models, enabling precise and accurate garment reproduction.

2. The incorporation of 3D depth and normal maps enhances realism by accurately controlling garment fit over the body, reflecting considerable thought in modeling garment behavior in 3D space.
This paper presents SPM-Diff, a virtual try-on framework utilizing diffusion models that maintain garment details and shape leveraging visual correspondence. Due to warping variability, traditional VTON methods using diffusion models have difficulty preserving garment features. SPM-Diff addresses this issue by focusing on "semantic points" in garment images that capture detailed textures and shapes. Experiments on the VITON-HD and DressCode benchmarks show that SPM-Diff achieves state-of-the-art results in virtual try-on.
1. SPM-Diff's dependence on semantic points may lead to instability when many points are used due to interpolation and projection errors, as discussed regarding point count sensitivity.

2. SPM-Diff relies heavily on accurate depth and normal maps, which may limit its generalization to datasets or images without dependable 3D cues.

3. Although the model effectively preserves garment details, evaluations primarily concentrate on image quality metrics, neglecting user-centered metrics such as perceived realism in end-user studies.

4. Introducing "visual correspondence" into diffusion for virtual try-on is not new. Please clarify the "Semantic Point Matching" with "Warping parameters learning"
Yes
No ethics review needed.
There are a few questions apart from the weakness:
1.	As the local flow warping adopted in the paper relies on the coarse human body estimated with SMPL, I doubt if the warping method is capable of dealing with loose clothes like dresses or skirts. If not, will the proposed point prior still work to facilitate the generation process and get a good result?
2.	In lines 262-264, the author claims that pre-trained garment/geometry feature encoders are adopted to extract features for the following generation process, but I could not find any clarification on how these encoders were obtained. The author should clarify this.
1.	The proposed method is able to generate high-quality virtual try-on results. Experiments show that the proposed method outperforms existing pipelines.
2.	The idea of the paper that adopts pair of semantic points to facilitate the generation process and serve as an extra supervision signal is interesting and reasonable. If robust corresponding points could be found between the garment and the target body, they could be very good priors to boost the diffusion process. 
3.	The paper is well-written and easy to follow.
This paper presents a new method for virtual try-on by enforcing explicit correspondence through structured semantic points, which are first extracted from the garment image and then warped into the target body pose through local flow warping. The pair of points then serve as priors to guide the overall generation process of the try-on diffusion model, composed of a garment reference net and a main generation net. Experiments demonstrate this method generates high-quality try-on results.
The main weakness of the paper is on semantic point matching.
a)	How to acquire robust and accurate point matching should be the focus of the paper. However, this paper did not address this problem but rely on the local warping method proposed in GP-VTON[1], which is another virtual try-on method. This raises the concern about the contribution.
b)	The evaluation of the accuracy of the point-matching method is very limited. The point matching should achieve much more accurate results than the base diffusion model thus it could be used as a prior to guide the generation process. However, this is not demonstrated in the paper.

[1] GP-VTON: Towards General Purpose Virtual Try-on via Collaborative Local-Flow Global-Parsing Learning https://arxiv.org/abs/2303.13756
@inproceedings{
anonymous2024incorporating,
title={Incorporating Visual Correspondence into Diffusion Model for Visual Try-On},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=XXzOzJRyOZ},
note={under review}
}
Diffusion models have shown preliminary success in virtual try-on (VTON) task. The typical dual-branch architecture comprises two UNets for implicit garment deformation and synthesized image generation respectively, and has emerged as the recipe for VTON task. Nevertheless, the problem remains challenging to preserve the shape and every detail of the given garment due to the intrinsic stochasticity of diffusion model. To alleviate this issue, we novelly propose to explicitly capitalize on visual correspondence as the prior to tame diffusion process instead of simply feeding the whole garment into UNet as the appearance reference. Specifically, we interpret the fine-grained appearance and texture details as a set of structured semantic points, and match the semantic points rooted in garment to the ones over target person through local flow warping. Such 2D points are then augmented into 3D-aware cues with depth/normal map of target person. The correspondence mimics the way of putting clothing on human body and the 3D-aware cues act as semantic point matching to supervise diffusion model training. A point-focused diffusion loss is further devised to fully take the advantage of semantic point matching. Extensive experiments demonstrate strong garment detail preservation of our approach, evidenced by state-of-the-art VTON performances on both VITON-HD and DressCode datasets.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Virtual Try-On Image Generation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e84014e76132348cb745ec6654db2fc81127e53d.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Incorporating Visual Correspondence into Diffusion Model for Visual Try-On
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission