G4Seg: Generation for Online Segmentation Refinement with Diffusion Models
Diffusion models;Inexact Segmentation;Semantic Correspondence
Yes
No ethics review needed.
How could this method extend to other UNet-based text-to-image diffusion models beyond SD2.1 (such as Wurstchen, SSD-1B, SDXL, SD1.5, etc.) and their distilled versions (such as single or reduced step variants, LCM versions)?
This paper provides a training-free method for open vocabulary single-class segmentation.
This paper proposes an image segmentation method using a pretrained text-to-image diffusion model, specifically Stable Diffusion 2.1. Given a text prompt with a class label, which conditions the image generation in UNet layers, it computes a semantic difference between the original image and a mask-conditional generated image after a one-step denoising step. This discrepancy, which is computed by Hausdorff distance, is employed to update the foreground probability of each pixel. Initial coarse segmentation follows the settings in the selected baselines.
I carefully reviewed the paper. Rather than an extensive list of less critical and debatable issues, I will concentrate on the most significant concerns that influence my rating. My focus will be on the following three limitations:

1) The reliance on a specific class label in the text prompt limits this method to only a single category for a given image. It is not clear how this method extends multiple classes. Running the proposed method multiple times with multiple yet separate class labels would face the problem of post-consolidating, most likely handling contrasting segmentation results outside the diffusion steps, which seems non-optimal.  

2) The main idea is that, given an imperfect segmentation mask, the generated image will have a discrepancy from the original image. This drives the intuition given in the paper that with a more accurate segmentation mask, the probability of generating the original image is more likely to be maximized. How this is implemented is described in 3.2.2 and Appendix E. Some explanation is given 317-323; however, the explanation also highlights the reliance on a good initial segmentation mask. The ablation study does not present how sensitive the proposed method is to the initial segmentation accuracy.    

3) The segmentation mIoU improvement is marginal. Besides, the comparisons with stronger SOTA are missing OVAM (CVPR 2024) and CoDe (CVPR 2024). DeOP (ICCV 2023), which could be another interesting baseline. 

I am open to reconsidering my rating based on a satisfactory rebuttal from the authors.
Yes
No ethics review needed.
Please see weaknesses
1. The proposal to use generative discrepancies from diffusion models for segmentation refinement is unique and well-explored, emphasizing the potential of generative models in traditionally discriminative tasks.

2. The method’s independence from training requirements is noteworthy, making it resource-efficient and broadly adaptable.

3. Comprehensive evaluation across standard datasets (PASCAL VOC12, PASCAL Context, MS COCO) demonstrates consistent performance improvements, achieving state-of-the-art results in certain scenarios.
The paper presents G4Seg, a novel approach leveraging large-scale pretrained diffusion models for refining inexact segmentation without additional training. The method capitalizes on discrepancies between an original image and its generated counterpart conditioned on a coarse segmentation mask to refine segmentation results. By applying a semantic correspondence alignment and employing pixel-wise probability updates, G4Seg proposes a training-free, plug-and-play solution applicable to weakly supervised and text-supervised semantic segmentation tasks.
1.  While results are promising, deeper comparisons with similar methods (e.g., training-free segmentation approaches) could be emphasized to highlight advantages and potential trade-offs more explicitly.

2. Although tested on popular benchmarks, it is not entirely clear how G4Seg would scale to other datasets.

3. While computational cost is discussed, there is limited exploration of scenarios where G4Seg may not perform optimally (e.g., highly cluttered or ambiguous images).
Yes
No ethics review needed.
1. How does the proposed method handle cases where the initial segmentation is significantly inaccurate or contains multiple errors? Is there a threshold of initial accuracy below which the refinement process becomes less effective or unreliable?

2. Given that the method leverages generative priors from Stable Diffusion, how sensitive is it to domain shifts or out-of-distribution images? Have the authors explored its performance on datasets or image types that differ significantly from those used in training Stable Diffusion?

3. Could this approach be extended or adapted to those fully-supervised methods that can already achieve accurate segmentation results, such as SegFormer and Mask2Former?
1.The proposed G4SEG introduces an innovative training-free framework for inexact segmentation refinement using generative models. This approach represents a significant departure from previous discriminative-based and diffusion model (DM)-based training methods, offering a novel perspective in the field.

2. The concept of explicit mask projection and semantic correspondence alignment is particularly noteworthy. By ingeniously decoupling the mask refinement process from target image reconstruction, the authors present a unique solution to the segmentation refinement problem.

3. The method demonstrates consistent performance gains when incorporated with current state-of-the-art approaches in both open-vocabulary and weakly supervised segmentation tasks. This broad applicability underscores the effectiveness and versatility of the proposed technique.
This paper presents an innovative diffusion-based approach to address the Inexact Segmentation (IS) task by leveraging the intrinsic generative priors inherent in Stable Diffusion. The authors propose a coarse-to-fine segmentation refinement strategy that exploits pattern discrepancies between original images and mask-conditional generated images. This is achieved through the establishment of semantic correspondence alignment and iterative updating of foreground probabilities. A key contribution of this work is the introduction of a semantic correspondence alignment methodology, which employs the pixel-wise Hausdorff distance as a discrepancy metric. The efficacy of the proposed method is demonstrated through extensive experimentation in both open-vocabulary and weakly supervised segmentation tasks. Notably, the approach consistently outperforms current state-of-the-art methods, showcasing substantial performance gains across various benchmarks.
1. The reliance on Stable Diffusion may potentially limit the model's efficiency compared to baseline methods, raising concerns about computational requirements and processing speed.

2. The improvements are most pronounced when applied to less accurate segmentation models, such as GroupVit, which is trained solely with text labels. When used with stronger base segmentation models (e.g., SCLIP and DiffSegmenter), the performance gains appear to be less substantial, potentially limiting the method's applicability across a broader range of models.
Yes
No ethics review needed.
In Table 6, how to inject boxes, points and scribbles into Stable Diffusion?
(1) The idea that using the difference between the generated and original image to help refine the mask is interesting.

(2) This paper is generally well-written and easy to follow.

(3) The introduction of G4Seg is comprehensive and detailed.

(4) G4Seg is training-free and easy-to-use.

(5) The framework is tested across multiple benchmarks, including TSSS and WSSS, showcasing its versatility and effectiveness under different settings.
This paper proposes an effective method G4Seg based on Stable Diffusion to tackle the Inexact Segmentation task. This method utilizes the difference between the generated and original image to help refine the mask. By establishing a semantic correspondence alignment, the foreground probability of the confusion area can be effectively updated. Experiments on various benchmarks validate the effectiveness and superiority of this training-free approach.
(1) G4Seg is based on Stable Diffusion and suffers from heavy time costs, resulting in poor practicality especially when inference at scale.  It would be better to report the specific inference cost for each dataset, not be confined to VOC but the complex COCO.

(2) As shown in the figures, the refinement of confusion areas is limited and imperfect. There are still a certain amount of pixels that G4Seg can not correct. What is the inherent reason for this phenomenon? The struggles to select confusion areas or failures in semantic correspondence alignment? Meanwhile, G4Seg should be compared to other mask refinement techniques such as training-free dense CRF and training-based CascadePSP [1], SegRefiner[2]. It seems that dense CRF is superior and more efficient than G4Seg. For training-based methods, I understand that these works may use additional pixel-level annotations for training, but the comparison can help readers understand the gap between G4Seg and SOTA mask refinement techniques.

(3) The feature maps in Stable Diffusion are heavily downsampled (e.g., 64x64). Does G4Seg work for small objects?

(4) VOC is not an ideal dataset to validate the boundary accuracy because areas near the boundary are labeled as “void” in GT masks. Datasets with more boundary-accurate annotations, like re-labeled VOC and BIG used in CascadePSP [1], are more appropriate. 

(5) Detailed parameter sensitivity analyses for the injection weight $\alpha$ and mixing coefficient $\beta$ are lost, which is important to assess the robustness of G4Seg.

---

[1] CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement.

[2] SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process.
Yes
No ethics review needed.
(1) The contribution and novelty could be improved

(2) More discussions about the discrepancies, and more experimental comparison
(1) The motivation of reducing discrepancies between original images and coarse mask-conditional generated images is easy for understanding.

(2) The proposed method can improve the segmentation quality of existing open-vocabulary and weakly-supervised methods.
In this paper, the authors  exploits the discrepancies between original images and coarse mask-conditional generated images to improve mask quality. The proposed method has performance gain on both open-vocabulary and weakly-supervised task.
(1) The contribution of this paper is limited, and the idea of reducing discrepancies is relatively incremental.

(2) The proposed method focuses on discrepancies. For irregular objects, it is difficult to give an accurate generation. Does the proposed method have negative results on these situations.

(3) The proposed method has limited improvement by performing a forward diffusion denoising process. 

(4) The proposed method is a refinement processing, which lacks of comparison with existing mask refinement method, such as CRF.

(5) It lacks the analysis about the hyper-parameters like injection weight and mixing coefficient. For implementation details, the settings on different tasks are different.

(6) The proposed method seems increasing the computational cost during inference according to Table 5.
@inproceedings{
anonymous2024gseg,
title={G4Seg: Generation for Online Segmentation Refinement with Diffusion Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=a7gOjgFswH},
note={under review}
}
This paper considers the problem of utilizing a large-scale text-to-image diffusion model to tackle the challenging Inexact Segmentation (IS) task. Unlike traditional approaches that rely heavily on discriminative-model-based paradigm or dense visual representations derived from internal attention mechanisms, our method focuses on the intrinsic generative priors in Stable Diffusion~(SD). Specifically, we exploit the pattern discrepancies between original images and mask-conditional generated images to facilitate a coarse-to-fine segmentation refinement by establishing a semantic correspondence alignment and updating the foreground probability. Comprehensive quantitative and qualitative experiments validate the effectiveness and superiority of our plug-and-play design, underscoring the potential of leveraging generation discrepancies to model dense representations and encouraging further exploration of generative approaches for solving discriminative tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Diffusion models Inexact Segmentation Semantic Correspondence
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1eb6947c0f8d37eb8bf4ce8f356aa51bfd2fc423.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
G4Seg: Generation for Online Segmentation Refinement with Diffusion Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission