InstantPortrait: One-Step Portrait Editing via Diffusion Multi-Objective Distillation
Portrait Editing;Diffusion Multi-Objective Distillation;On Step Inference;Identity Preservation;Text Fidelity
Yes
No ethics review needed.
Please list the details of all the comparison methods, including which codebase you are using, which model checkpoint, etc.
1. The visual results are impressive. Many details are created, while the identity is well-preserved. 
2. The approach design is reasonable. It first generates the identity-preserved image, then distills the model for efficient inference. 
3. The experiments are convincing.
This paper tries to tackle the task of portrait editing with identity preservation and style consistency.  It first trains a full-step diffusion model to produce images with robust identity preservation. Then it trains a one-step model that is distilled from the full-step model to further enhance the style, and identity of the edited images.
Several limitations can be improved to make this paper better.
1. The dataset creation process is reasonable. However, it is still not very clear to me how this dataset is used. The authors mention that when creating the dataset, they use the AID loss to preserve identity. Then what data is used to train the first stage? Is the data from another model that also uses the AID loss? 
2. There are many losses included in the training. How are they tuned? Is there a guideline for the tuning of the hyper-parameters?
3. Using the distilled version of the model usually results in a reduced diversity of styles and texture details. If the diversity can be further evaluated, this work can be more convincing.  
4. For the facial expression control, any thoughts on why the current model cannot achieve it and any future plan on how to achieve it?
5. Could you list the source code link you use for IP-Control, which also achieves impressive results?
Yes
No ethics review needed.
1. This paper introduces a dataset for portrait editing. Will it be open-sourced? Making it publicly available would greatly benefit the development of this research area.
2. In constructing the dataset, the authors used IPAdapter combined with ControlNet. Similarly, Table 1 compares the performance of IPAdapter with ControlNet. Could you clarify whether the base models for IPAdapter and the choices for ControlNet in these two instances are the same? Different base model selections will undoubtedly lead to varying performance outcomes.
1. This paper constructs a paired dataset for portrait image editing, consisting of 10,000,000 pairs of images at a resolution of 1024x576. This extensive dataset is invaluable for both this paper and future research in the field of portrait editing. Additionally, the supplementary materials provide a detailed description of the dataset construction process.
2. This paper presents significant strengths in its approach to portrait image editing. It achieves one-step instruction-based editing with high precision in identity preservation, accurate execution of editing instructions, and fast inference.
This paper introduces the Instant-Portrait Network (IPNet), a one-step diffusion-based real-time portrait image editing model. The proposed IPNet can address challenges such as identity preservation and editing fidelity. The training process has two stages. The first stage is to train an Identity Enhancement Network for robust identity preservation. The second stage uses a novel Diffusion Multi-Objective Distillation approach to train the IPNet with multiple losses. Extensive comparisons demonstrate that IPNet outperforms existing models in identity consistency, instruction fidelity, and speed.
1. The description of the methodology in this paper lacks clarity and omits important details, which may confuse readers. For instance, in section 3.2.2 on Identity Distillation Loss, there is insufficient explanation regarding how the teacher and student models sample time steps during the training process, as well as how the IPNet is distilled to achieve one-step inference. Providing these details would significantly enhance the readers’ understanding of the proposed method.
2. The paper does not provide a comprehensive explanation of the model and experimental details, lacking important information such as the weights assigned to the different loss functions and specific training details for the two stages. Including this information would greatly enhance the clarity and completeness of the work.
3. This paper confines the task to portrait editing under fixed poses, only altering attributes such as clothing, makeup, and style without the capability to generate new poses or expressions. However, the comparison models, IPAdapter and InstantID, offer more versatile functionalities, including the ability to generate new poses and expressions. While IPNet demonstrates superior performance within the scope of this paper, it’s possible that IPAdapter and InstantID, when fine-tuned on the dataset presented in this paper, could achieve comparable results to IPNet.
Yes
No ethics review needed.
1. Is IDE-Net purely a reconstruction model? Does it take text as input, and could you provide some sample results?
2. If we directly train IDE-Net with the adversarial loss from 3.2.1, the triplet loss from 3.2.3, and the original IDENet losses, would the results surpass those of IPNet?
3. When performing distillation, since IPNet only involves a single step of sampling, how is the number of noise-adding and denoising steps for IDE-Net determined? 
4. Given that the proposed dataset is paired, why wasn’t diffusion loss, some photometric loss, or the losses used in IDENet, applied when training IPNet? Can using those losses provide a more accurate supervision signal than IDENet’s distillation loss?


I look forward to the author’s response, and I am willing to raise my score if some of my concerns are addressed.
1. They propose to distill IDENet into IPNet with the diffusion Multi-Objective Distillation process, achieving one-step instruction-based portrait image editing.
2. They propose an Annealing Identity Loss that balances identity preservation and text alignment.
3. They design a dataset generation pipeline to generate a large-scale paired dataset, where each pair contains: 1) an input image; 2) a text prompt; and 3) a target image generated by SDXL with IPAdapter and ControlNet.
4. Experiment results show that they can generate high-quality image editing results with better identity preservation.
This paper proposes a one-step text-based portrait image editing method, which can achieve the balance between identity preservation and text alignment. They firstly train an Identity Enhancement Network (IDE-Net) to ensure robust identity preservation. Then, they propose a diffusion Multi-Objective Distillation process to distill IDENet into IPNet, achieving one-step instruction-based portrait image editing.
1. This paper primarily allows editing only the style of portraits; it is not suitable for modifying expressions, poses, or the subject’s position within the image, which imposes certain limitations.
2. The necessity of IDENet remains unclear. Can you provide a more detailed justification for training IDENet rather than using existing methods like IPAdapter or InstantID, and discuss how this choice impacts the overall performance of IPNet?
We train IPNet via Diffusion Multi-Objective Distillation and achieve one-step instruction-based portrait image editing, with high-precision identity preservation, precise instruction-based image editing, and rapid model inference concurrently.
@inproceedings{
anonymous2024instantportrait,
title={InstantPortrait: One-Step Portrait Editing via Diffusion Multi-Objective Distillation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ZkFMe3OPfw},
note={under review}
}
Real-time instruction-based portrait image editing is crucial in various applications, including filters, augmented reality, and video communications, etc. However, real-time portrait editing presents three significant challenges: identity preservation, fidelity to editing instructions, and fast model inference. Given that these aspects often present a trade-off, concurrently addressing them poses an even greater challenge. While diffusion-based image editing methods have shown promising capabilities in personalized image editing in recent years, they lack a dedicated focus on portrait editing and thus suffer from the aforementioned problems as well. To address the gap, this paper introduces an Instant-Portrait Network (IPNet), the first one-step diffusion-based model for portrait editing. We train the network in two stages. We first employ an annealing identity loss to train an Identity Enhancement Network (IDE-Net), to ensure robust identity preservation. We then train the IPNet using a novel diffusion Multi-Objective Distillation approach that integrates adversarial loss, identity distillation loss, and a novel Facial-Style Enhancing loss. The Diffusion Multi-Objective Distillation approach efficiently reduces inference steps, ensures identity consistency, and enhances the precision of instruction-based editing. Extensive comparison with prior models demonstrates IPNet as a superior model in terms of identity preservation, text fidelity, and inference speed.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Portrait Editing Diffusion Multi-Objective Distillation On Step Inference Identity Preservation Text Fidelity
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/fdd06d1ea226bcaf7bb53da400204c8c5ae0fcdf.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/f856e71eb9bf6d4188040097e159bb36fde9e13f.zip
InstantPortrait: One-Step Portrait Editing via Diffusion Multi-Objective Distillation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission