Emerging Tracking from Video Diffusion
Pixel-level object tracking;Temporal correspondence;Diffusion models
Yes
No ethics review needed.
Please see weaknesses section for all questions. It is critical to consider and address the issues with the presentation, and better explain the experimental details.
(1) The reported results show improvement over prior works

(2) The motivating toy example is a great way to show the lack of temporal information that the paper is investigating.
The paper investigates using features from a generative video diffusion model for the task of label propagation in videos. The findings appear to point to using a combination of the image diffusion model and video diffusion model to yield good results.
#### Presentation
The paper is lacking in key details, specifically:
 1. What is the method? The closest paper that comes to the actual description of how the label is propagated is probably on L312, "recurrently predict labels for subsequent frames". Appendix A.1 is not much more helpful in explaining how tracking is performed; however, it suggests that the actual method is DIFT with just appended video features.
  2. What are the evaluation datasets? The paper mentions DAVIS, Kubric-Similar and Youtube-Similar; however, these datasets are not detailed, especially Kubric-Similar and Youtube-Similar, which seem to be new. Please include details of these datasets and how they were made/curated, including examples. For Davis, please indicate clearly which year and which "task" flavour (semi-supervised, probably) is being used.

#### Contributions
 3. Some of the provided answers do not seem to be well supported. E.g. on L453 (Fig. 7 caption), it is stated that given the lack of appearance at a high noise level, Rv must capture temporal cues useful for tracking. However, temporal cues must be derived from appearance, which is obfuscated with noise. Similarly, it does not explain why it performs worse when there is no noise. Temporal ques are still available. It is also not clear how these experiments were performed, given the randomness involved; how many trials does each point in the graph represent? What are the variance bounds?
 4. It is also not clear what are the results in Fig. 9? Does each point correspond to the video or average over a dataset? Why does this positive correlation imply that training new video diffusion models will improve tracking? Does this hold for both SDV and I2V models?
 5. The methods reported in Table 2 for supervised are rather old and might leave a false impression of the difference in the performance of supervised vs unsupervised. Looking at the DAVIS 2020 challenge [1] (Davis 2017 is a subset) results, it is clear that supervised performance is in 80+ region. 
 6. Given that the contribution of the work is to append video diffusion features to the image of DIFT. There is arguably lack of significant learnings or exploration presented, as only 2 video diffusion models are tested. For example,  why is I2V better than SDV? Would time-tuned self-supervised features work instead [2]? Why only stop at 2 models? Are there better combinations for image features, as shown in [3]? Why are video features so much worse in isolation?
 7. The performance constraints/cost are not mentioned, but they are usually key considerations in tracking settings, as systems need to be real-time due to safety critical (e.g. self-driving) or UX (e.g. film editing) concerns. What are the performance characteristics, e.g. FPS, GPU memory, to run this method?

---

[1] DAVIS 2020 challenge. https://davischallenge.org/challenge2020/leaderboards.html 
[2] Salehi et al. Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations, ICCV 2023
[3] Zhang et al. A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence, NeurIPS 23
Yes
No ethics review needed.
How does this method proposed differ from DIFT (Tang 2023)?, this is not clear?
What is the computational cost to compute the video diffusion model if we wanted to use this method for real time tracking?, or can we do tracking in real time with this step included?  How and in what manner do we include windowing for the video diffusion and how does this blend in for a real time algorithm for visual tracking.
Table 2, it would be good if the datasets were the same for a fair comparison.  The results for your approach and DIFT appear the same and sometimes DIFT is better, comment please?  
It appears that the method is dependent on the dataset used?  I am curious on if the same dataset is used, how does DIFT differ from your algorithm in terms of results
The paper presents an interesting idea in leveraging visual diffusion for strong appearance correspondence and how it can be used for visual tracking with multiple objects.
Uses video diffusion models to extract exact motion representations for strong appearances for visual tracking.
I am not exactly sure how this proposed paper differs from the DIFT method?  They appear to be the same except for perhaps different training data.
I understand that you are leveraging the latent representations from visual diffusion however no where in the paper do you explicitly give the visual tracking algorithm in terms of a flow chart or pseudo code, this would be helpful.
The ablation study refers to t which is the tilmestep in diffusion however when it appears, you also use t for time, so this is confusing to the reader.
Yes
No ethics review needed.
Please see the weakness.
1. TED extracts the temporal information of tracking targets from the video diffusion models to assist in tracking.
2. TED introduce a new benchmark, Youtube-Similar, to evaluate the complex scenario of tracking multiple similar-looking objects in real-world videos. 
3. TED achieves good tracking results, especially on the Youtube-Similar benchmark, to show the better tracking ability to tackle the complex scenario of multiple similar-looking objects.
The authors leverage latent representations from video diffusion models to capture the temporal information for pixel-level tracking. Without additional training, the proposed method improves tracking performance in various video scenarios, even enabling tracking of similar-looking objects where previous methods struggle. The authors also introduce a new benchmark, Youtube-Similar,  to evaluate the complex scenario of tracking multiple similar-looking objects in real-world videos. The proposed method, TED, is evaluated on several benchmarks and achieves better tracking results.
1. The tracking results, which only train on ImageNet, are suggested to be provided for fare comparison.
2. The tracking speed and computation cost are suggested to be provided.
3. The authors are suggested to discuss the benefits of the temporal information extracted from diffusion compared with other models like transformers.
Yes
No ethics review needed.
- What about the time cost of the method? Considering the diffusion model often takes time to infer, how much time it will cost to track, for example, 100 frames? 
- And if the video sequence is very long, would the results decline? 
- as the method is built on standard Unet diffusion models, would some inference speed-up methods in the diffusion model help TED faster?
- The paper is well written. It is easy to follow the key idea of the paper.
- the experiments are comprehensive and the results are quite good. Especially, the demos, ie. fig 5, on identical appearance object matching.
- the findings along with understanding are interesting. how to deal with the similar appearance of objects in label propagation is very fundamental in the field. 
- helping understand diffusion models and latent representation from another perspective. It can benefit a board of readers in tracking and diffusion groups.
This paper introduces a method from video diffusion, TED, to extract motion representations for object-level tracking tasks. By leveraging diffusion representation and existing label propagation methods, the proposed method shows a very robust identification in similar appearance matching. The experiments demonstrate its effectiveness in tracking tasks and achieving SOTA results.
- Several improvements can be achieved. It would be better if more metrics were provided to understand the proposed method, i.e. the time-consuming. the details of the proposed framework can be provided in supplementary for readers to replicate the method.
Yes
No ethics review needed.
- I could not get from Fig.5 whether the feature visualization is a heatmap or something else. If it is a heatmap ranging from 0 to 1, how can one object be distinguished from another?
- The technical statements are quite clear and compelling.
- The PCA analysis seems interesting.
- The performances are quite improved on three input levels.
The paper introduces a simple method of extracting diffusion features from small clips to perform the object tracking task. It enables tracking similar objects, overcoming limitations of per-frame processing proposed in previous works. The performances are quite improved on segmentation-, pixel- and pose-level tracking.
- The progression from per-frame processing to batch processing seems natural, with no leap leading the methodology forward.
- The methodology of using generative models in simple understanding tasks was quite interesting when first DIFT was proposed. However, simply making incremental improvements to these models for marginal performance gains may not fully realize their power.
- Data statistics are missing, how is the number of samples/frames/objects? Why is the dataset claimed as a contribution while reusing from Youtube-VOS?
Our work achieves state-of-the-art performance in pixel-level object tracking by incorporating video diffusion model representations.
@inproceedings{
anonymous2024emerging,
title={Emerging Tracking from Video Diffusion},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=UDeARVACQi},
note={under review}
}
We find video diffusion models, renowned for their generative capabilities, surprisingly excel at pixel-level object tracking without any explicit training for this task. We introduce a simple and effective method to extract motion representations from video diffusion models, achieving state-of-the-art tracking results. Our approach enables the tracking of identical objects, overcoming limitations of previous methods reliant on intra-frame appearance correspondence. Visualizations and empirical results show that our approach outperforms recent supervised and self-supervised tracking methods, including the state-of-the-art, by up to 6 points. Our work demonstrates video generative models can learn intrinsic temporal dynamics of video, and excel in tracking tasks beyond original video synthesis.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Pixel-level object tracking Temporal correspondence Diffusion models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c67877ea300327b58752bbcf07bf04436e13b2e3.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Emerging Tracking from Video Diffusion
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission