ST-Modulator: Modulating Space-Time Attention for Multi-Grained Video Editing
diffusion model;video editing
Yes
No ethics review needed.
- For part-level editing, is the method limited to adding objects, or can it also support modifications like changing the color of clothing or the color of an animal?
- The paper is well-written and organized, making complex concepts easy to understand.
- The Spatial-Temporal Layout-Guided Attention method in this paper effectively addresses the challenge of precise, multi-grained video editing by modulating cross- and self-attention. This approach enables accurate text-to-region control and clear feature separation, allowing independent edits to specific subjects while avoiding unintended changes in other areas, especially in complex multi-subject scenes.
- Compared to other approaches, ST-Modulator achieves high computational efficiency, showing lower memory usage and faster processing times.
This paper introduces a new task called multi-grained video editing, which includes class-level, instance-level, and part-level editing, and proposes a zero-shot approach, ST-Modulator, to address the challenge of distinguishing distinct instances (e.g., "left man" and "right man") by modulating space-time attention mechanisms for precise, fine-grained control over video content—all without additional training.
- Limited focus on background preservation. 
- In Figures 5 and 6, most examples demonstrate edits that include modifications to the background along with the main subjects. Could the authors provide additional examples where the editing focuses solely on specific subjects, allowing the background to remain unchanged? This would help illustrate the method’s capability for selective edits in multi-subject scenes.
Yes
No ethics review needed.
See weakness.
1. This is the first attempt at multi-grained video editing.
2. The results show the SOTA performance on existing benchmarks and real-world videos.
3. The method is intuitive and is easy to follow.
This paper presents a zero-shot approach that modulates attention mechanisms to achieve fine-grained control over video content. We enhance text-to-region control by amplifying each local prompt’s attention to its corresponding spatial disentangled region while minimizing interactions with irrelevant areas.  Extensive experiments demonstrate the method achieves state-of-the-art performance in real-world scenarios.
1. The readability and presentation of the manuscript need to be enhanced. For example, in Figure 4:
The "I" in "M_i^pos" denotes the i-th frame, whereas the "i" in "cross-attention modulation" appears to refer to the i-th token (e.g., "polar" is the 3rd token?). Readers should not have to guess what these symbols mean; it would be beneficial to clarify these points explicitly.
The "E" in "L × E" is undefined within the context of this paper, and Figure 2 does not utilize this information. Consequently, the shape information in Figure 4 could be omitted to avoid confusion.
It is unclear whether "p" in Figure 4 represents the original attention score. This should be clearly stated to avoid ambiguity.
2. The paper employs an additional model, SAM-Track, to perform instance segmentation, which might be unfair. If this is the case, one could directly use Grounding-DINO in conjunction with SAM-Track to accurately identify the edited area (mask) based on the provided text. Subsequently, methods like prompt-to-prompt or Video-P2P could be employed to control the editing process through attention maps, potentially achieving similar results.
3. A key concept of the paper is to enhance attention scores in relevant areas while suppressing them in irrelevant areas. However, it's worth noting that methods such as prompt-to-prompt already implement mechanisms to either reduce or increase attention scores for editing purposes. This aspect should be discussed more thoroughly to highlight the novelty and added value of the proposed approach.
Yes
No ethics review needed.
Please kindly address the questions in the weakness section.
1. This paper addresses a highly significant problem of editing granularity in video editing. It proposes a new and interesting task called multi-grained editing, which includes class, instance, and part-level editing. It allows flexible editing at any granularity and breaks the inherent feature coupling within diffusion models.
2. The empirical study is insightful, explaining why diffusion models are limited to class-level video editing.
3. The proposed method is novel and consistent with its motivation. It adjusts cross-attention for accurate attention weight distribution, enables text-to-region control, and modulates self-attention for feature separation. The modulation process is operated in a unified manner, and the overall framework is training-free.
4. The qualitative experiments are impressive, indicating that ST-Modulator surpasses previous approaches in multi-grained editing.
This paper presents a new task called multi-grained video editing,  which encompasses class, instance and fine-grained video editing. The empirical study shows that the key obstacles hindering multi-grained editing in diffusion models are text-to-region misalignment and feature coupling.  This work utilizes a pretrained T2I diffusion model to address these problems without any training. More Specifically ,the method modulates cross-attention for text-to-region control and self-attention for feature separation. Effectiveness has been proven by extensive experiments and convincing qualitative results.
1. Since this work proposes a new and interesting task focused on multi-grained editing, a holistic evaluation would be beneficial, such as establishing a benchmark and developing new metrics to evaluate the performance of current methods on multi-grained video editing.
2. ST-Modulator focuses on editing videos at different spatial granularities, but the temporal length is limited to 16-32 frames. I wonder whether ST-Modulator can be extended to handle longer video sequences, such as those with hundreds of frames.
3. What is the difference between multi-grained editing and multi-attribute editing?
Yes
No ethics review needed.
See weaknesses.
- The paper clearly defines the novel task of multi-grained video editing, motivated by practical needs in video editing. By enabling edits at class-level, instance-level, and part-level, it broadens the range of possible video modifications to better meet real-world demands.
- The paper effectively uses visualizations to illustrate key points, such as in the *Analysis of why the diffusion model failed in instance-level video editing.* These visual aids provide intuitive insights into challenges with existing diffusion models and help clarify the motivation behind the proposed approach.
- The proposed Spatial-Temporal Layout-Guided Attention focuses on self-attention and cross-attention mechanisms to address key issues in text-to-region control and inter-region feature separation. This method effectively targets the core issues in multi-grained editing within diffusion models.
- The proposed method does not require parameter tuning, which makes it resource-efficient—a significant advantage for video editing applications where computational cost and tuning complexity are critical factors.
This paper introduces ST-Modulator, a zero-shot framework for multi-grained video editing, enabling class-level, instance-level, and part-level modifications in diffusion models. To address semantic misalignment and feature coupling issues, ST-Modulator modulates space-time cross- and self-attention, allowing fine-grained text-to-region control and improved feature separation. Experimental results suggest that this approach performs competitively across benchmarks without additional parameter tuning, presenting a promising solution for multi-grained video editing.
1. **Focus on Image Editing over Video-Specific Needs:** The approach seems more aligned with image editing, as it emphasizes spatial layout control. However, a key distinction in video editing is maintaining inter-frame consistency, which is crucial for coherent video results. While the Spatial-Temporal Layout-Guided Attention is introduced, the temporal aspect appears to receive less emphasis or detailed explanation.

2. **Lack of Clarification on Additional Control Signals:** The authors mention that the method is compatible with ControlNet conditioning, but it remains unclear if the cases shown in the paper require extra control inputs. There is no ablation study on control conditions to assess the necessity or influence of these signals. This raises questions about fairness in comparisons with other models like ControlVideo, where discrepancies in control conditions might lead to potentially biased results.

3. **Dependence on SAM-Track for Segmentation:** The method relies on SAM-Track for instance segmentation, yet no ablation study examines its impact. This dependency raises concerns: it is unclear if the observed performance gain primarily stems from using SAM-Track. For example, using SAM-Track segmentations as conditional input to other video editing models might yield similar improvements, suggesting a need for direct comparisons to validate the unique contributions of the proposed approach.

4. **Lack of Code or Algorithm Details:** The paper does not provide code or a detailed algorithm description. Open-sourcing the code could help address concerns such as those in Weaknesses 2 and 3 by allowing for transparent validation of the method’s assumptions, control conditions, and segmentation impacts.
zero-shot method for class-level, instance-level and part-level video editing
@inproceedings{
anonymous2024stmodulator,
title={{ST}-Modulator: Modulating Space-Time Attention for Multi-Grained Video Editing},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=SSslAtcPB6},
note={under review}
}
Recent advancements in diffusion models have significantly improved video generation and editing capabilities. However, multi-grained video editing, which encompasses class-level, instance-level, and part-level modifications, remains a formidable challenge. The major difficulties in multi-grained editing include semantic misalignment of text-to-region control and feature coupling within the diffusion model. To address these difficulties, we present ST-Modulator, a zero-shot approach that modulates space-time (cross- and self-) attention mechanisms to achieve fine-grained control over video content. We enhance text-to-region control by amplifying each local prompt's attention to its corresponding spatial-disentangled region while minimizing interactions with irrelevant areas in cross-attention. Additionally, we improve feature separation by increasing intra-region awareness and reducing inter-region interference in self-attention. Extensive experiments demonstrate our method achieves state-of-the-art performance in real-world scenarios. More details are available on the project page.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
diffusion model video editing
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e0824280d2562b8195eae08552a6bb8aa4b494b6.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
ST-Modulator: Modulating Space-Time Attention for Multi-Grained Video Editing
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission