What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?
Transfer Learning;Diffusion Models;Visual Perception
Yes
No ethics review needed.
1. What training set was used for Table 1? Are you following the training data of DMP or Marigold? If so, why is the first row of Table 5 identical to the baseline? If not, why did you begin directly with a synthetic dataset?
2. For monocular depth estimation, why is DMP not included as a baseline?
3. The authors mention that using a customized head and loss could accelerate inference time. Could you provide a comparison to demonstrate this improvement?
1. The exploration of key factors influencing the transferability of text-to-image (T2I) diffusion models to dense visual perception tasks is intriguing and relevant.
2. The experiments are thorough and comprehensive, providing strong support for the findings.
3. The proposed deterministic one-step perception approach effectively integrates these findings and demonstrates comparable performance with minimal fine-tuning.
This paper investigates key factors affecting the transfer performance of pretrained diffusion models repurposed for dense visual perception tasks, emphasizing the importance of fine-tuning data quality, training strategies, and task-specific supervision. It introduces GenPercept, a one-step fine-tuning paradigm that enhances inference speed and improves fine-grained details in predictions across various perception tasks.
There are some confusing aspects in the experimental setup and comparisons for the downstream tasks. See questions.
Yes
N/A
No ethics review needed.
What would be the comparison of generalization for OOD data for models trained using synthetic and real data?
- The authors conduct extensive ablation studies from different aspects and finally figure out the key factors affecting transfer efficiency and performance when using pre-trained diffusion models.
- Starting from the inconsistency of the results of two existing methods, the authors dig deeper into the influencing factors and then propose their GenPercept. This approach can provide some new ideas for doing research.
- These findings are inspiring and can provide constructive insights into model design when adapting pre-trained models for downstream tasks.
This paper reveals key elements of diffusion models for downstream dense perception tasks. The authors did a full range of validation from the perspectives of model design and training data, unveiled some IMPORTANT factors, and proposed a new model named GenPercept,
The authors have conducted extensive experiments on five dense perception tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting. This extensive experimentation serves as a testament to the effectiveness and universality of their method.
- The biggest weakness is that all conclusions are based on experiments using large synthetic datasets trained on deep estimation tasks. I have two concerns. First, would these conclusions still hold if the datasets were small? Second, would these conclusions still hold if the dataset was completely real? 
- It is not appropriate to use the P3M 10K dataset to verify the robustness of GenPercept in image matting. This is because the human matting dataset is extremely similar to the dichotomous image segmentation task if only the boundary regions are considered and the deterministic foreground regions are ignored. Instead, if the authors wish to validate its robustness in the image matting task, more types of objects (e.g., semi-transparent objects) should be included and then its performance should be observed.
Yes
No ethics review needed.
The main reason that I currently give a slightly low rating is because of the incorrect numbers of existing models, the missing quantitative comparisons to some other existing models, the missing efficiency experiments, and some missing explanations (see the ‘weaknesses’ section). I would like to ask the authors to carefully address my concerns, answer the questions posed in the ‘weaknesses’ section, and revise the manuscript accordingly.

Additionally, I have some other questions/suggestions:

1.	From the text in L411-L423, it appears like the classes for semantic segmentation are always encoded into 3-channel colormaps, even when UPerNet is used. Is this really the case? If so, why isn’t the ‘regular’ semantic segmentation format used, with one channel for each individual class? If not, please clarify this in the text.
2.	In the related work section, it seems appropriate to also mention DINOv2, because it has shown to be very suitable for downstream visual perception tasks like depth estimation (e.g., with Depth Anything [g]) and semantic segmentation.

[f] Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” TMLR 2024.

[g] Yang et al., “Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data,” CVPR 2024.
1.	Most importantly, the analysis of the design choices that play a role when fine-tuning diffusion models for monocular depth estimation, described in Sec. 3, leads to various relevant and useful insights. 

    a)	The finding that fine-tuning can be done accurately with a one-step, deterministic approach (Tab. 1) is useful because it allows for significantly more efficient inference. 

    b)	Moreover, by formulating the task as one-step estimation, the model can now te trained with task-specific losses, such as the image angular loss for surface normal estimation. With experiments (Tab. 7), this is shown to improve the results.

    c)	The finding that most of the useful prior knowledge is contained in the U-Net of the pre-trained diffusion model, and not in the VAE decoder (Tab. 2), is useful because it means that the VAE decoder can be replaced with task-specific decoders. This is shown to be significantly boost the semantic segmentation performance in Tab. 9 (per L418-L423). 

    d)	The finding that the use of multiple timesteps during training is not necessary (Tab. 3) is useful because it allows the training procedure to be simplified to single-timestep training.

    e)	The finding that the U-Net denoiser should be fine-tuned fully, without low-rank adaptation (Tab. 4), is useful because it provides clear guidelines for future methods that aim to fine-tune diffusion models for depth estimation and other dense perception tasks.

2.	The presentation of the main findings of the paper in the text boxes (e.g., L283 and L296) is very helpful and useful. These text boxes clearly summarize the impact of the results that have been discussed thus far, and they help the reader to see the most interesting results of the paper at a glance.

3.	The presented GenPercept model is valuable because it takes advantage of the aforementioned findings of the design-choice analysis. The application and evaluation of this model on multiple tasks and datasets provides the reader with insights into the performance that can be obtained when combining the best individual design choices. Moreover, these results can serve as a baseline for future works that aim to fine-tune diffusion models for multiple downstream perception tasks.
This paper researches the process of fine-tuning pre-trained text-to-image diffusion models for dense perception tasks like monocular depth estimation and semantic segmentation, and analyzes several design choices in that process. In this analysis, it considers the model architecture, training procedure, model initialization, dataset selection, and fine-tuning protocol. Based on this analysis, five findings are made:

1.	Diffusion models can be fine-tuned accurately and efficiently for dense perception tasks with a one-step, deterministic approach.
2.	The most useful prior knowledge of the pre-trained diffusion model is contained in the U-Net denoiser. The VAE decoder can be replaced with other components without problems.
3.	With the one-step model, multi-timestep training is irrelevant, and additional text inputs do not significantly impact performance.
4.	Fine-tuning the U-Net denoiser leads to better results than freezing it or applying low-rank adaptation.
5.	Training data quality affects the prediction quality.

Based on these findings, the paper presents GenPercept, a paradigm for fine-tuning diffusion models for dense prediction tasks. Experiments show that GenPercept is an effective method to fine-tune Stable Diffusion for various dense prediction tasks. On several benchmarks, GenPercept achieves a competitive performance.
1.	The scores for DepthFM in Tab. 6 and GeoWizard in both Tab. 6 and Tab. 7 do not correspond to the scores originally reported in the respective papers. The numbers in these tables should be altered to reflect the originally reported numbers, or the text should explain why the numbers differ from these original numbers. Some examples:

    a)	GeoWizard: 9.7 AbsRel and 92.1 $\delta_{1}$ on KITTI in original paper [a], but 12.9 AbsRel and 85.1 $\delta_{1}$ in this submitted manuscript. 

    b)	DepthFM: 8.3 AbsRel and 93.4 $\delta_{1}$ on KITTI in original paper [b], but 17.4 AbsRel and 71.8 $\delta_{1}$ in this submitted manuscript.

2.	The results in Tab. 8 (and Tab. 2 of the supp) make it seem like GenPercept achieves state-of-the-art results in dichotomous image segmentation. However, this table does not contain the results of top-performing model MVANet [c]. This model achieves a max$F_{\beta}$ score of 0.916 on Overall DIS-TE (1-4), compared to 0.863 by GenPercept. Even with results that are inferior to MVANet, GenPercept has value, but it should be clear to the reader that GenPercept does not achieve state-of-the-art results, so these results should be added to the tables.
3.	For the image matting task on the P3M-500-NP dataset, GenPercept is only compared to the ResNet-34 variant of P3M [d], not the Swin-T or ViTAE-S variants which achieve much better results, e.g., 7.59 SAD for ViTAE-S compared to 11.23 SAD for ResNet-34. By only reporting the results for the ResNet-34 variant, it seems like GenPercept performs similarly to the state of the art, whereas this is not the case. The results of P3M for ViTAE-S and/or Swin-T should be added to the paper, or the text should clearly explain why such a comparison is not necessary.
4.	One of the main benefits of the one-step inference procedure of GenPercept is the improved efficiency compared to Marigold and other diffusion-based methods. The efficiency is also mentioned as a key characteristic of the method (L539). However, the paper does not explicitly evaluate the efficiency of the model. Therefore, it is not clear what the exact speedup is over existing diffusion-based models, or how the efficiency compares to the other methods reported in Tab. 6, 7, 8, and 10. The paper would be significantly stronger if the efficiency of GenPercept was shown quantitatively, by reporting the inference time of GenPercept and other methods.
5.	L187-L188 states that the results indicate that increasing the ‘training challenges’ leads to improved model performance. However, the paper does not provide any explanation for this. Why does the performance increase when the training task becomes more challenging? This seems counterintuitive. The value of the paper would improve if the authors provided an explanation (or hypothesis) for this phenomenon, as this could provide insights into the way these diffusion models learn dense perception tasks.
6.	The paper does not clearly define $\beta_{start}$ and $\beta_{end}$, even though first experiment and finding (Tab. 1 & L204) are fully focused around changing the values of these parameters. Fig. 2(b) illustrates that they impact the proportion of noise that is added to the latents in different time steps, but an exact definition is not provided. In L107, the paper refers to the supplementary material for more details, but the supplementary material only mentions $\beta_{s}$ and $\beta_{t}$. The clarity of the paper would improve if a clear definition of $\beta_{start}$ and $\beta_{end}$ was provided.
7.	The paper does not evaluate if the multi-class image segmentation performance of GenPercept is competitive with existing image segmentation methods. Currently, the paper only conducts an experiment where GenPercept is trained on HyperSim images with 40 semantic classes, and evaluated on ADE20K images with 40 classes. As this is a newly proposed setting, these results are not comparable with existing segmentation methods. To see if fine-tuning diffusion models is truly valuable for image segmentation, GenPercept should be fine-tuned and evaluated on standard segmentation benchmarks like ADE20K, and compared to existing segmentation models like Mask2Former [e]. 
8.	A minor weakness of the paper is that there is not a version of GenPercept that clearly works better than the other. As discussed in L357-L359 and shown in Tab. 6, the GenPercept model trained for depth estimation scores better on NYU, ScanNet and ETH3D, while the model trained for disparity estimation scores significantly better on KITTI and DIODE. As a result, two different models are necessary for different situations, and it is not always clear which of the models works best for an unknown application domain. 

Some minor weaknesses, which do not significantly impact my rating:

9.	L230: The acronym “DPT” is not defined, nor is there a reference to a work where it is presented. As a result, it’s not clear what a “DPT encoder” is.
10.	The meaning of the “fine-tune” column in Tab. 9 is not clear. From the text in L418-L423, it appears that it refers to the use of the UPerNet decoder. If so, the text “UPerNet” seems more appropriate than “fine-tune”. If it means something else than this, this should be better specified in the paper.
11.	The ordering of the labels in the legend of Fig. 2 (b) is confusing, as there appears to be no logical ordering. The legend, and the figure as whole, would be easier to interpret if the labels were ordered in descending or ascending manner, e.g. starting with (0.0002125, 0.003) and ending with (1.0, 1.0), or vice versa.

12.	There are a few errors in the text:

    a) L024: "tailed for" - Do the authors mean "tailored for"?

    b)	L187-L188: "increasing training challenges lead to" => "increasing training challenges leads to"

[a] Fu et al., “GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image,” ECCV 2024.

[b] Gui et al., “DepthFM: Fast Monocular Depth Estimation with Flow Matching,” arXiv:2403.13788, 2024.

[c] Yu et al., "Multi-view Aggregation Network for Dichotomous Image Segmentation," CVPR 2024.

[d] Ma et al., "Rethinking Portrait Matting with Privacy Preserving," IJCV 2023.

[e] Cheng et al., “Masked-attention Mask Transformer for Universal Image Segmentation,” CVPR 2022.
Yes
No ethics review needed.
1. Trained with synthetic data, why does GenPercept not suffer from the sim2real domain gap?
2. In the supplementary materials, line 22 should utilize the citation format 'citep'.
3. Can GenPercept be applied to general perception tasks, such as detection?
1. This paper provides systematical analysis of the design space of diffusion model for dense perception task, as shown in the 5 findings in paper. The core design of this paper is to employ a deterministic one step perception.
2. By using the proposed training protocol, the proposed GenPercept harnesses the power of the pre-trained UNet from diffusion models for dense visual perception tasks, including monocular depth estimation, surface normal estimation, dichotomous image segmentation and semantic segmentation. 
3. This paper presents a solid analysis accompanied by extensive experiments. The results are not only intuitive but also promising, demonstrating a strong potential for downstream application.
This paper presents a novel approach to harnessing pretrained diffusion models for general dense perception tasks, including monocular depth estimation, surface normal estimation, dichotomous image segmentation, semantic segmentation, and human pose estimation. The central idea of this work is to utilize the diffusion model as a robust pretrained backbone, subsequently fine-tuning it for a variety of downstream applications.
1. Given that GenPercept is currently trained on a relatively small dataset, it tends to lag behind models that benefit from extensive data training. To enhance its performance, it would be beneficial to scale GenPercept's training with a larger volume of data in the future.
2. Given the robust prior established by training on the extensive LAION dataset, the question arises: what would be the outcome of employing alternative self-supervised methods, such as MAE or CLIP, using the same LAION dataset? A comparative analysis of these approaches against the diffusion pretrain would provide valuable insights into their relative efficacy and potential advantages.
We comprehensively investigate the variours potential ways for reformulating diffusion models as general visual perception predictors, including monocular depth, surface normal, image segmentation, matting, and human pose estimation.
@inproceedings{
anonymous2024what,
title={What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=BgYbk6ZmeX},
note={under review}
}
Extensive pre-training with large data is indispensable for downstream geometry and semantic visual perception tasks. Thanks to large-scale text-to-image (T2I) pretraining, recent works show promising results by simply fine-tuning T2I diffusion models for dense perception tasks. However, several crucial design decisions in this process still lack comprehensive justification, encompassing the necessity of the multi-step stochastic diffusion mechanism, training strategy, inference ensemble strategy, and fine-tuning data quality. In this work, we conduct a thorough investigation into critical factors that affect transfer efficiency and performance when using diffusion priors. Our key findings are: 1) High-quality fine-tuning data is paramount for both semantic and geometry perception tasks. 2) The stochastic nature of diffusion models has a slightly negative impact on the deterministic perception tasks. 3) Apart from fine-tuning the diffusion model with only latent space supervision, task-specific image-level supervision is beneficial to enhance fine-grained details. These observations culminate in the development of GenPercept, an effective deterministic one-step fine-tuning paradigm tailed for dense visual perception tasks. Different from the previous multi-step methods, our paradigm has a much faster inference speed, and can be seamlessly integrated with customized perception decoders and loss functions for image-level supervision, which is critical to improving the fine-grained details of predictions. Comprehensive experiments on diverse dense visual perceptual tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting, are performed to demonstrate the remarkable adaptability and effectiveness of our proposed method.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Transfer Learning Diffusion Models Visual Perception
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6613bcb5cf85b85d98c35f4db8a84da35db1ed39.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/860530ba950f227f7dd8febf13cc9bb162a86944.pdf
What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission