Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model
transfer learning;diffusion models;fine-tuning;guidance
Yes
No ethics review needed.
Please refer to the weaknesses.
1. DoG’s design as a CFG variant is well-motivated, with clear delineation of the benefits over standard CFG. The illustrations of DoG’s guidance process effectively highlight its domain-alignment benefits.
2. Quantitative results show that DoG significantly outperforms CFG, particularly on datasets with substantial domain shifts. Qualitative results are also compelling, with DoG generating visually consistent images even with increased guidance weights.
3. The paper provides a theoretical foundation for DoG, showing how it maintains domain alignment and mitigates OOD sampling errors, supported by an illustrative analysis using Gaussian mixtures.
The paper introduces "Domain Guidance" (DoG), a new transfer approach for pre-trained diffusion models aimed at enhancing domain alignment and generating high-quality outputs. DoG builds on classifier-free guidance (CFG) but adapts it to transfer learning by incorporating pre-trained model knowledge to guide the generative process. Through empirical and theoretical analysis, the authors demonstrate DoG’s effectiveness in maintaining pre-trained knowledge, improving image fidelity, and reducing out-of-distribution errors. The results indicate substantial improvements in several metrics (FID and FDDINOv2) across various downstream tasks compared to standard fine-tuning and CFG.
1. The paper could benefit from more discussion on DoG’s adaptability, particularly in domains vastly different from the pre-trained model’s source domain.
2. The influence of guidance weights is explored, but further sensitivity analysis could strengthen the understanding of DoG’s stability across diverse settings.
3. While diverse datasets are used, further tests on varying resolutions and different architectures could better illustrate DoG’s flexibility.
Yes
No ethics review needed.
Please see Weaknesses 1.
- The motivation and idea of DoG is clearly presented, and the paper is overall easy to follow.
- Theoretical analyses and toy examples well depict the advantage of DoG.
- DoG outperformed CFG, a de facto standard method for diffusion guidance with a margin.
This submission proposed a new conditioning method to transfer a diffusion model
 to a new domain named Domain Guidance (DoG).
The idea behind DoG is to pull the generation process toward the target domain
from the pre-trained domain.
Given a pretrained model and fine-tuned version of it, domain guidance guides the process
to the direction that the fine-tuned one indicates and against the pretrained ones's direction.
Theoretical analyses show that DoG mitigates sampling from out-of-distribution area of the target domain.
Experiments show that DoG outperforms CFG in combination with diffusion models finetuned in relatively small
datasets.
- Range of applicable models is not discussed well:
The experiments are conducted with ImageNet-pretrained  DiT-XL/2 and 
but is it mandatory to use the same architecture and initialization weights to perform DoG?
Furthermore, transfering diffusion models to small datasets are often done with LoRA [a] rather than full finetuning.
Is DoG applicable to LoRA-based transfer?
[a] LoRA: Low-Rank Adaptation of Large Language Models 


- Relationship with Autoguidance:
A similar idea is presented in a preprint [b] which uses less-trained version of the model for guidance, 
instead of unfinetuned version in DoG.
Discussing this may be useful, while it it not mandatory due to its unpublished state and missing it should not be penalized.
[b] Guiding a Diffusion Model with a Bad Version of Itself, NeurIPS 2024 to appear.
Yes
No ethics review needed.
it is the same as Weaknesses.
Originality: The idea of using the pre-trained model as a domain guidance to guide transfer learning is innovative.

Quality and Clarity: The quality of this is satisfactory and the analysis is thoughtful. The paper is well-written, with a thorough and comprehensive ablation study.

Significance. This work provides a contribution to the transfer learning of diffusion learning. In particular, this work can be seamlessly integrated into the existing fine-tuning models without additional training.
This work leverages pre-trained knowledge as the domain guidance to guide the model toward the target domain. This work thinks that domain guidance facilitates better domain alignment and higher-quality generations, and provides correlation theoretical analyses.
1. Is there a domain gap between personalized scenarios and general scenarios?
2. During the transfer learning process, does all the knowledge from the pre-trained model need to be effectively utilized?
3. How can the impact of the knowledge in the pre-trained model be assessed in the target domain?
4. For DF-20M (which has no overlap with ImageNet) and ArtBench-10 (whose feature distribution is completely distinct from ImageNet), why can guidance from the model pre-trained on ImageNet lead to better generation?
5. Is the distribution of the downstream domain a subset of the distribution of the pre-trained domain?
This paper proposes a conditional generation perspective for the transfer of diffusion models and derives a simple approach named Domain Guidance to enhance transfer learning.
@inproceedings{
anonymous2024domain,
title={Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=PplM2kDrl3},
note={under review}
}
Recent advancements in diffusion models have revolutionized generative modeling. However, the impressive and vivid outputs they produce often come at the cost of significant model scaling and increased computational demands. Consequently, building personalized diffusion models based on off-the-shelf models has emerged as an appealing alternative. In this paper, we introduce a novel perspective on conditional generation for transferring a pre-trained model. From this viewpoint, we propose \textit{Domain Guidance}, a straightforward transfer approach that leverages pre-trained knowledge to guide the sampling process toward the target domain. Domain Guidance shares a formulation similar to advanced classifier-free guidance, facilitating better domain alignment and higher-quality generations. We provide both empirical and theoretical analyses of the mechanisms behind Domain Guidance. Our experimental results demonstrate its substantial effectiveness across various transfer benchmarks, achieving over a 19.6\% improvement in FID and a 20.6\% improvement in FD$_\text{DINOv2}$ compared to standard fine-tuning. Notably, existing fine-tuned models can seamlessly integrate Domain Guidance to leverage these benefits, without additional training.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
transfer learning diffusion models fine-tuning guidance
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/45d4c13fc2f69672ce1c461b98685efc246cc928.pdf
transfer learning, meta learning, and lifelong learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission