SMITE: Segment Me In TimE
video segmentation;diffusion models;video diffusion;part segmentation
Yes
No ethics review needed.
- Is the proposed method also applicable to "Multi-object flexible granularity", which indicates that reference images annotated with multiple objects (or multiple classes) different granularity can be used as input to the proposed model?

- Could the author provide any failure cases?
Belows are the strong points that this paper has:

- It is well-written and well-organized, which help reader to understand new task and motivation of the proposed method.

- The proposed techniques including learning generalizable segments using alternative tuning of text embeddings and cross attention, temporal consistency via tracking voting mechanism, and spatial consistency with low-pass filter are in good harmony, which can outperform the other baselines with high margin.
This paper defines a task of flexible granularity, which aims to segment user-intended granularity temporally using one or a few reference images. To do so, it proposes several techniques to achieve temporal consistent multi granularity segmentation including alternative learning of text embeddings and cross attention module, tracking modules with voting mechanism to be aware of neighbor pixels and low frequency regularizer for spatial consistency. It shows its effectiveness on the proposed dataset, SMITE-50
Belows are points that the reviewer feel concerned about.

- In line #247, the authors mention that learning text embeddings and cross-attention layers is conducted in two phases to provide a better initialization for the next phase. However, this phased or alternating training approach could also increase training complexity. Could the authors provide an ablation study to compare the results of joint training versus alternating training to justify the chosen training setup? Specifically, analysis on training time with final performance of two training strategies can be provided.

- In the supplementary video, specifically at 1:46 in the "Qualitative_Comparison" section, it appears that the proposed method captures a relatively smaller region of the "eye" area compared to the baseline. Is this due to the proposed voting mechanism diluting the effective area by averaging nearby major segments? Would this issue be mitigated by replacing the current hard voting with a soft voting mechanism? Is it doable that quantitative analysis on different voting system can be measured?

- Is the proposed method shape-agnostic within the same class? To demonstrate that the method can segment other unseen images, the authors should provide test samples that differ in shape from the reference object. For example, as similar to what the author showed in "Cars" of Figure 7, if the reference image includes a detailed segmentation map of a "sedan" car, the test samples can be replaced with different shapes of cars, such as "SUVs?
Yes
No ethics review needed.
1. How sensitive is the method's performance to the choice of tracking algorithm? Have you experimented with other tracking methods besides CoTracker?
1. The low-pass regularization is mentioned as a key component. Can the authors elaborate on how this regularization term is formulated and how it balances preserving segment structures with smoothing transitions?
1. How does the proposed method handle videos with rapid movements, severe occlusions, or significant lighting changes? Are there strategies to mitigate potential performance degradation in such scenarios?
1. Originality : The proposed method innovatively combines pre-trained text-to-image diffusion models with a temporal tracking mechanism to achieve video segmentation with flexible granularity using only a few reference images.
1. Quality : The comprehensive experiments, including quantitative evaluations on the new SMITE-50 dataset, qualitative comparisons, ablation studies, and user studies, provide solid support for the method's effectiveness.
1. Clarity : The paper is well-structured, with clear explanations of the problem, methodology, and results. Figures and diagrams are used effectively to enhance understanding.
1. Significance: Addressing the challenge of few-shot video object part segmentation has practical implications for various applications, including VFX and video editing.
This paper presents SMITE (Segment Me In Time), a novel approach for one/few-shot video object part segmentation with flexible granularity. Building upon the image-based SLiMe method, SMITE leverages a pre-trained text-to-image diffusion model (Stable Diffusion) and introduces a tracking mechanism and low-pass filtering to maintain temporal consistency in video segmentations. Given one or a few annotated reference images, SMITE aims to segment unseen videos of objects from the same category, respecting the provided segmentation granularity. The authors introduce a new dataset, SMITE-50, to benchmark performance and compare against relevant baselines.
1. Methodological Clarity: While the overall explanation is detailed, some of the methodology is difficult to follow, especially for readers not familiar with diffusion models and the previous work SLiMe. The process of how tracking integrates into the attention map refinement could be clearer, with more intuitive descriptions of its working mechanism.
1. Dataset & Baselines: The new SMITE-50 dataset is an important contribution, but it appears somewhat small (with 50 videos). Expanding the dataset size or providing an analysis of its generalization for more categories could strengthen the paper’s claims. A comparison to more widespread benchmarks for video segmentation, such as DAVIS or other VOS datasets, would help position SMITE relative to not only a niche dataset but also larger, well-accepted benchmarks in the community.
1. Inference Efficiency: The paper does not deeply explore the computational cost or inference time of the method in practical settings, which would be relevant given the addition of several components (e.g., temporal voting, low-pass filters). Addressing this in the discussion or providing benchmarks on efficiency could help increase understanding of real-world applicability.
Yes
No ethics review needed.
Same as weakness.
1)This proposed method combines information from text, tracker and spatial temporal features for video segmentation at arbitrary level.
2)SMITE achieves state of the art performance on the SMITE-50 dataset.
This paper proposes a SMITE framework for segmenting object fragments across video frames. The method consists of denosing module, diffusion module and tracking module. Overall, it achieves satisfactory performance for video segmentation over various levels of granularity.
1)The proposed method is only compared with two methods in Table 1, while there are lots of few-shot and zero-shot segmentation methods. 
Besides, it seems that the proposed method can be directly applied on multi-target video segmentation datasets like DAVIS2017. So, there should be more comprehensive comparison, for both methods and datasets.
2)Speed is important for online tracking; so there should be some runtime analysis.
3)Limitations of the method is not discussed in experiments.
4)How does the proposed method work on random videos and random annotation references?
Yes
No ethics review needed.
1. The differences in Figure 3 could be better presented (the difference between (c) and (d) is hardly visible).

2. It was slightly challenging to follow all the technical details. 
Few small modifications might reduce the reading time:
For example: 
a. In section 4.2 3 loss terms were mentioned and in section 4.4 2 other loss terms were mentioned; It will be clearer if the authors stated clearly how and in what phase each loss term is used (all together? two-phase optimization?).
b. lines 267-268, sliding window -> temporal sliding window

3. Small technical note:
line 148: should the epsilon_theta after UNet be deleted?

4. Lines 303-305:
It is not clear how this scenario is different from using only cross attention maps. 
If I understand what the authors suggested correctly: average voting can be computed directly on the masks as a "tracking post-process step" without the need for an optimization for tracking.
If so, why can't you simply calculate the average vote on the S_WAS as well?
If not, what was your intention?

5. Line 105 - you mention that some parts cannot be described by text. 
Yet, you do find text embeddings that describe them, although it is reasonable to claim that these embeddings do not represent natural text descriptions. 
However, it will be interesting to acknowledge/discuss this distinction. 
Why does it make sense to find text embeddings for parts that cannot be described by texts?
Novel and interesting problem formulation. 
The motivations behind the design choices are presented clearly.
Qualitative results seem decent.
A new video part-level dataset and SOTA performance.
This work presents a framework for segmenting videos into parts at adjustable levels of granularity, using only a single or few annotated images as references. The framework comprises several main components:
First, it optimizes text embeddings and fine-tunes a stable diffusion model so that the WAS attention maps (a combination of self and cross-attention maps) align with the annotated reference images.
Next, temporal consistency is enforced through specific loss terms, enabling the framework to generate segmentation maps from the input video using the optimized text embeddings and fine-tuned diffusion model.
1. Limited competitors - The baseline (frame-by-frame) might be a bit too simplistic. 
Maybe a better baseline would be a simple video extension of SLIME (without fine-tuning the cross-attention layers). This video extension is pretty trivial and should have some consistency.
Another baseline could be using SLIME (single frame) with tracking.
2. Limited ablation study - 
I would have liked to know what is the contribution of each step.
In addition, I think more ablation would make the paper stronger. For example, how similar should be the input object to the one in the video? Would horse segmentation work on zebras? and on dogs?

In addition: Table 2 only states what happens if only the regularization step is dropped. What about dropping the tracking? And what about dropping both?
Yes
No ethics review needed.
What is the speed and memory requirement of the proposed method? How many frames can it process?

What if an inflated U-Net is not used during text embedding optimization (e.g., the segment prompts are just images, not videos) but only used during inference?
This paper tackles the hard yet useful task of few-shot, fine-granularity video segmentation. This task has potentially wide applications. I think the main contribution of this work is taking the initial crack at this problem with reasonable performance in segmentation quality and temporal stability when compared to applying the SLiME per frame. 

Technique-wise, segmentation tracking/voting and low-pass regularization are reasonable in handling temporal flickering unique to videos, and the results also supported this.

The proposed dataset SMITE-50 is also potentially useful for future work, as it contains densely labeled segmentation which is a rare type of data.
This paper presents SMITE, a few-shot video segmentation method with fine granularity tracking. SMITE is based on SLiMe, a few-shot image segmentation method that finetunes text tokens for a diffusion network and probes attention masks to output a mask.  SMITE additionally introduces cross-attention finetuning of the diffusion network, a segment tracking and voting module based on point tracking, and a low-pass regularization term. SMITE achieves better segmentation performance than applying baseline SLiME per frame in a newly proposed dataset SMITE-50.
I don’t see it mentioned in the paper, but I suspect that SMITE might be too slow for most practical applications. In addition to finetuning the text tokens and the cross-attention layers during a training stage, SMITE additionally performs dense, windowed point tracking – there seem to be a lot of windows (linear to video length) and a lot of repeated computation in the point tracker (linear to windows size). SMITE also minimizes an energy function which is also costly. Besides, given that SMITE expands the spatial self-attention to the spatiotemporal volume, it would be quite memory intensive for longer videos.

I think the authors can also compare SMITE to existing few-shot video object segmentation methods like [Multi-grained Temporal Prototype Learning for Few-shot Video Object Segmentation, ICCV 2023]. This can happen in two dimensions: 1) applying few-shot VOS to the fine-grained segmentation setting, and 2) applying SMITE to the few-shot VOS setting. This can highlight whether the fine-grained segmentation setting significantly differs from the object setting and supplements the existing sparse baselines.

In Table 2, should there be a row with neither of the loss and another row with only the low-pass loss? Also, can the ablations be run on the entire SMITE-50 dataset rather than just on the CARS subset? This would make the results more significant.
Segmenting any video by annotating few reference images
@inproceedings{
anonymous2024smite,
title={{SMITE}: Segment Me In TimE},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=KW6B6s1X82},
note={under review}
}
Segmenting an object in a video presents significant challenges. Each pixel must be accurately labeled, and these labels must remain consistent across frames. The difficulty increases when the segmentation is with arbitrary granularity, meaning the number of segments can vary arbitrarily, and masks are defined based on only one or a few sample images. In this paper, we address this issue by employing a pre-trained text to image diffusion model supplemented with an additional tracking mechanism. We demonstrate that our approach can effectively manage various segmentation scenarios and outperforms state-of-the-art alternatives.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
video segmentation diffusion models video diffusion part segmentation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8bf48ecce3bd4e57772c9de58e62214cf3650339.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/88f841cea6a6831d30f4432e46a049005a41e47c.zip
SMITE: Segment Me In TimE
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission