FreeTraj: Tuning-Free Trajectory Control via Noise Guided Video Diffusion
Diffusion Model;Video Diffusion;Trajectory Control;Motion Control
Yes
No ethics review needed.
Please see the weakness.
- It is intuitive and reasonable to use noise and motion guidance.

- Noise resampling is an interesting idea.

- The analysis is comprehensive, and the results seem good to me.
This paper analyze the trajectory-controllable video generation and introduce Free_Traj, a training-free motion control method.
- Is mIoU a good metric for trajectory control? 


- Can this method unify trajectory control and motion control? For example, a man waves his hand. In terms of motion, it is a "wave" action; regarding trajectory, the hand follows its specific path. Can we unify them to achieve more realistic motion control?
Yes
No ethics review needed.
- The author mentions that this method can be used for LLM trajectory planners, but how this can be implemented? This lack of detail could leave readers uncertain about the practical application of the method in such contexts.
- Since the method does not require training, it could be applicable to any diffusion-based video generation method, such as Stable Video Diffusion or OpenSora. Could you provide more experiments with different video-diffusion baselines? This universality could potentially make it a versatile tool in the field of video synthesis, especially for applications requiring rapid deployment without the need for extensive training data.
- As in zero-shot setting, Motion-Zero[1] has some similar features, modifying initial noise, and guidance in attention. in video generation. What's the difference between the proposed method and [1]?

[1] C. Chen et al. Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation. Arxiv 2024.
- The method described in this paper does not require any training, which can significantly reduce computational overhead.
- This method analyzes the high and low frequencies of the initial noise and Attention Isolation, and proposes corresponding utilization methods and solutions.
- The provided generated video looks good in controlling.
This paper proposes a diffusion-based video generation method for controlling the trajectory of moving objects in a zero-shot manner and investigates the impact of initial noise on the trajectories of moving objects. By guiding the generated target in noise and attention, the proposed method can generate controlled trajectory of the target. Experimental results demonstrate the effectiveness of this method.
- In the paper, the author uses a box to control the motion of objects, but in the experiments, the objects are not actually inside the box; they merely maintain a trajectory consistent with that of the box.
- The method performs significantly worse than the Direct approach on the FVD and KVD metrics, and the author does not analyze this. This could potentially have negative effects on the clarity of the generated videos and on maintaining the identity of the objects.
- This method shows some weaknesses in generating complex and random trajectories. However, it seems feasible to achieve better results by increasing the control strength, according to the method. The author does not further analyze this aspect. The degree of control could potentially impact video quality. It would be beneficial for the author to include an ablation study on the intensity of control to better understand its effects.
Yes
N/A
No ethics review needed.
Please see the weaknesses.
1.	The exploration on the noise initialization for object motion control is interesting and demonstrates the importance of noise structure for content generation in video diffusion models. 
2.	Both of the qualitative and quantitative experimental results demonstrate the effectiveness of the proposed approach for tuning-free motion control.  
3.	The potential of complex motion control is verified by the model that is combined with FreeNoise under the setting of long video generation.
The paper presents a tuning-free video diffusion framework for object trajectory control. The authors in this paper first analyze the influence of noise construction for video motion generation and then introduce the FreeTraj. The proposed framework modifies the noise sampling and involves the local trajectory injection in the noise initialization. Besides, the object mask is also integrated into the attention to emphasize the content generation across frames. Experiments on both controllable short- and long-video generation tasks verify the proposed approach.
1.	My major concern is about the technical contribution which could be limited. The investigation of noise construction is more like an intuitive engineering work and there is no rationale behind. The influence of high-frequency noise part has been also explored in FreeNoise. The trajectory injection seems reasonable but more details should be included. Meanwhile, the authors argue that Peekaboo exploits the hard mask in attention. The motivation of the proposed soft Gaussian mask is not clear either. The hyper-parameter of such Gaussian kernel is not mentioned in the paper.
2.	In the experimental section, the comparison and discussion (line 465 to 470) between FreeTraj and MotionCtrl should be detailed. It is very difficult for readers to judge the technical differences or improvements from these descriptions. Why dose MotionCtrl only roughly control the object trajectory but not align the trajectory? The reason behind these results is not provided or discussed. 
3.	One suggestion is about ablation studies. There could be some quantitative analysis for different variants rather than only showing the visual cases. The quantitative evaluation results can reflect the performance from a global view.
4.	Some of the definitions should be aligned. For instance, the $F_{z_T}^{low}$ and $F_{z_T}^{L}$ in Eq. (7).
Yes
No ethics review needed.
Please refer to the weakness section.
1. The method is training free unlike most existing works.
2. Extensive quantitative evaluation.
3. Methods are intuitive and the paper is easy to follow.
4. Superiority over baselines compared.
5. Providing observation before they build their method sounds convincing and well-structured.
6. Applications of Longer / Larger video generation.
This paper introduces FreeTraj, a tuning-free framework for trajectory-controllable video generation using diffusion models, eliminating the need for additional training. It controls motion trajectories by guiding noise sampling and attention mechanisms. FreeTraj supports both manual and LLM-generated trajectories and enables longer video generation with precise trajectory control. Extensive experiments validate its effectiveness in enhancing trajectory controllability in video diffusion models.
1. By "LLM trajectory planner", do the authors mean like LLM-grounded video diffusion [1]. Plus, I think LLM-grounded video diffusion would serve as a good baseline to be compared (ignoring the llm planning part). Also, if the LLM trajectory planner is not implemented or showcased, the authors should not sell this point.

2. Does the method increase computation burden? Providing and comparing information on the memory / time consumption would increase value of the paper.

3. How long and big are the "long" and "large" video generation in the results of appendix?

4. I think the intuition behind High-Frequency Noise Resampling is well explored in both image/video diffusion works. Not just FreeInit, but also in Diffusion-Motion-Transfer [2] (their initial noise preparation stage). And the novelty is limited.

5. It's hard to grasp what attention isolation is easily. Providing visualizations in relevant sections would help reviewers understand the issue and how they overcome it.

6. Is Cross Attention Guidance method not explored in diffusion-based image generation/editing literature?

7. Isn't TrackDiffusion [3] a relevant work? If so, how does the model perform compared to the work? 

8. Quantitative ablations studies would add value of the paper.

[1] LLM-grounded Video Diffusion Models, ICLR 2024

[2] Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer, CVPR 2024

[3] TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models, Arxiv 2023
A tuning-free method for trajectory-controllable video generation with guidance of initial noise.
@inproceedings{
anonymous2024freetraj,
title={FreeTraj: Tuning-Free Trajectory Control via Noise Guided Video Diffusion},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=CU7QfWJ6nC},
note={under review}
}
Diffusion model has demonstrated remarkable capability in video generation, which further sparks interest in introducing trajectory control into the generation process. While existing works mainly focus on training-based methods (e.g., conditional adapter), we argue that diffusion model itself allows decent control over the generated content without requiring any training. In this study, we introduce a tuning-free framework to achieve trajectory-controllable video generation, by imposing guidance on both noise construction and attention computation. Specifically, 1) we first show several instructive phenomena and analyze how initial noises influence the motion trajectory of generated content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that enables trajectory control by modifying noise sampling and attention mechanisms. 3) Furthermore, we extend FreeTraj to facilitate longer and larger video generation with controllable trajectories. Equipped with these designs, users have the flexibility to provide trajectories manually or opt for trajectories automatically generated by the LLM trajectory planner. Extensive experiments validate the efficacy of our approach in enhancing the trajectory controllability of video diffusion models. Generated video samples are available at the anonymous website: https://FreeTraj.github.io.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Diffusion Model Video Diffusion Trajectory Control Motion Control
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/89b6dee7140e400fef965baf747ffd3c058dff64.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/28cc8e8e69ed89a69a04e5b9e9692ca42038817f.zip
FreeTraj: Tuning-Free Trajectory Control via Noise Guided Video Diffusion
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission