SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation
video generation;complimentary learning system;slow-fast learning;diffusion
Yes
No ethics review needed.
1. How are the parameters optimized across the slow and fast learning phases to ensure smooth interaction without redundancy? Could alternative approaches, such as dynamically adjusting memory encoding based on the video’s complexity, further streamline the integration?
2. For tasks that require nuanced, multi-step interactions (e.g., tasks involving conditional actions or dependencies between actions), how does SLOWFAST-VGEN manage action dependencies across time steps? Are there specific mechanisms to ensure that complex sequences remain consistent and contextually appropriate?
3. What do you think about the hierarchical learning paradigm? Such as the slow/fast learning v.s. long-horizon key-frame generation + show-horizon dense-frame generation. This is just a question, not need to conduct experiments.
1. **Novelty in Dual-Speed Learning**: By combining slow learning of world models and fast episodic learning, SLOWFAST-VGEN represents an innovative approach, mimicking cognitive memory systems effectively.
2. **Comprehensive Benchmarking**: Evaluations include diverse tasks and metrics, reinforcing the model’s robustness across applications.
The paper introduces SLOWFAST-VGEN, a dual-speed learning framework for action-driven, long video generation inspired by the complementary learning system in human cognition. This model integrates slow learning of world dynamics with fast learning for episodic memory storage, aiming to address challenges in generating consistent, coherent long videos. The slow learning component utilizes a masked conditional video diffusion model, while the fast learning component leverages LoRA to store episodic memory. The authors evaluate the model on various datasets and metrics, achieving superior FVD scores and scene consistency compared to baseline models.
1. **Limited Generalization**: SLOWFAST-VGEN’s performance may drop in scenarios outside its training domains, such as novel environments or complex, unforeseen actions, limiting its adaptability in dynamic real-world applications.
2. **Memory Inconsistency Over Long Sequences**: TEMP-LORA effectively stores episodic memory within single episodes but struggles with consistency across multi-episode or long-term tasks, potentially leading to fragmented recall in extended sequences. Since the authors mention that the slowing learning for world modeling can capture general dynamics, some long-horizon tasks need to be evaluated, such as moving to the right and then to the left.
3. **High Computational Cost**: The reliance on a diffusion model for video generation demands significant computational resources. Although using LoRA, the two-stage training. What about latent diffusion models for the slow learning, while video diffusion models for the fast learning?
Yes
No ethics review needed.
1. The approach is complex. Are there any possible ways to streamline the model while still leveraging the Slow/Fast concept?

2. How would this approach deal with a complex "real world" dataset of long videos beyond just constrained domains such as Kitchens or Car Driving?
1. The paper addresses a serious failing in contemporary approaches to video modeling. Current video models cannot scale well to long time horizons, and this paper shows promise in advancing this capability.

2. The paper is well written and easy to understand.

3. Experimental evaluation is thorough, and the datasets used is reasonably diverse. The quantitative results are convincing.
In this paper, the authors focus on the problem of long term video generation. To this end, they propose SlowFast-VGen, a model which leverages episodic memory (using a LoRA module from long text generation) to improve context utilization over longer time horizons. The authors also argue that the approach has connections to Cognitive Science. They then evaluate their approach on multiple datasets including RLBench and Minecraft. They demonstrate convincing quantitative results that beat baselines on metrics measuring long term consistency.
1. The proposed method is quite complex, requiring multiple working parts (Slow vs Fast modules).

2. I think the paper might benefit from more complex datasets. Datasets from video games, robotic simulation, and car driving are used, but these are not particularly diverse. Training/testing on a dataset such as Walking Tours (Venkataramanan et al 2024) would strengthen the paper.
Yes
No ethics review needed.
1. Is there a limit of the temp-lora in terms of the capacity of trajectories of individual objects in the scene if their motion is not the same as the global motion?
2. I did not find the number of diffusion steps T used in the experiments, please provide these numbers and any other hyperparameters used if different than the base model.
3. How is the performance of the model on dynamic objects in a scene if you re-visit an area, e.g. moving objects like cars or persons in an early episode that you re-visit later?
1. The paper is well structured, and additional details in the appendix help to clarify some aspects.
2. The authors introduce a bio-inspired interpretation of a slow-and fast learning loop to improve long video generation by maintining an episodic memory, and link the inspiration to the hippocampus-cortex learning hypothesis. I like the independence of the lora loop from the action label embedding and treating the previous and next chunk as a single continuum, making it easier to assess the actual impact of this architecture change.
3. A new dataset for benchmarking action-driven video generation models is introduced, leveraging existing datasets and adding new (mainly video game engine) data and labels. The authors compare their model with several s.o.t.a. models and introduce a new metric for scene consistency.
4. The model is benchmarked using several metrics, including one that requires multi-sampling with human labeling.
In this paper, the authors tackle the challenging action-driven video generation task. They approach it by dividing the model and training into a slow and fast learning loop, reflecting the changes in the  dynamics in the world model and in individual episodes. This is achieved by adding a temporal LoRa to the base architecture, which is also used during inference. While minimally increasing the inference time, this temporal-LoRa leads to an improved generative performance and temporal consistency.
To train the model the authors create a new large scale video-action-annotation dataset. The model performance is evaluated on video generation tasks and long-horizon planning.
1. The interpretation of  fast and slow memory or training loops is a staple e.g. in generative replay in the continual learning domain (including often citing the hippocampus inspiration), as well as several RNN-like architectures with slow and fast memory. A link to these previous works (even if just in the appendix) could further improve the paper.
2. Novelty: the authors cite some previous work on the LLM domain as the main source of inspiration for this usage of temp-lora. The term temp-lora and its application in the video generation domain was used in previous works such as [1] (which is only a preprint, but given it is from feb 2024 still relevant for this review), a statement how their lora application differs would be appreciated. Given these works are only preprints it does not mean they take away from the novelty of this submission. It is just to get a better understanding of the differences in how lora is applied. Even if it is the same technique it would be interesting to know if the authors think the chosen task and datasets are better suited for the technique compared to the taks & datasets in the previous works.
3. I would like to see more architectural and (hyper) parameter details of the model used in the experiments (esp. The Unet + Lora), even if it is based on existing pretrained models, and how the hyperparmeters were chosen (if different from the base models).

Other inputs (not weaknesses) that may be used to improve the paper: 
There seems to be repeating text in appendix A.2 Preliminaries On Low-Rank Adaptation

[1] Ren, Yixuan, et al. "Customize-a-video: One-shot motion customization of text-to-video diffusion models." arXiv preprint arXiv:2402.14780 (2024).
Yes
No ethics review needed.
* It would be useful if the authors could include some way to view the original video samples (not just snapshots of frames in the paper). Perhaps with an anonymous site?
* How long are the generated videos in seconds? Most figures show timestamps, but framerate is not mentioned in the paper.
* The paper is generally easy to follow, and well written
* The proposed algorithm is interesting, and could be a promising direction for incorporating memory through test-time training in neural networks
* Experimental results on long videos show better consistency than baseline methods, and without using their temporal LoRA module.
* The authors introduce a collection of action-conditioned video that could be useful for future related research
This paper proposes a slow-fast structure for retaining episodic memory in long video generation. The “slow” video model follows similarly to prior works in masked conditional video diffusion models, where x prior frames are conditioned on to generate y future frames. To incorporate longer-context memory when generating multiple chunks in a row, the authors introduce temporal LoRA parameters that are updated during inference time to store episodic memory. For more general long-context generation tasks, the paper additionally proposes a slow-fast training algorithm.
* The purpose of the “slow” video baseline comparisons are a little confusing to me. Is this not just showing that training on specifically curated action data results in a model that can better generate conditioned on actions?
* The authors have shown some benefits in retaining long-term memory over the course of multi-chunk video generation. However, I am still a little doubtful at the long-context abilities of this algorithm if used at scale. It seems difficult to propagate / learn abstract representations in long term memory that may be useful for long-term video generation, especially since the temporal LoRA is updated using a pixel-wise diffusion loss and thus biased to memorizing a specific instance of a scene. How would this model perform in more difficult long context tasks (e.g. remembering something seen 10-30s ago, or trying to localize position / orientation in 3D space), such as the datasets introduced in [1]? Or, in a more simpler environment, making one full rotation, and then randomly turning left and right (the first rotation should newly generate the scene, and afterwards only requires accurate scene retrieval from memory). 


[1] Yan, Wilson, et al. "Temporally consistent transformers for video generation." International Conference on Machine Learning. PMLR, 2023.
@inproceedings{
anonymous2024slowfastvgen,
title={SlowFast-{VG}en: Slow-Fast Learning for Action-Driven Long Video Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=UL8b54P96G},
note={under review}
}
Human beings are endowed with a complementary learning system, which bridges the slow learning of general world dynamics with fast storage of episodic memory from a new experience. Previous video generation models, however, primarily focus on slow learning by pre-training on vast amounts of data, overlooking the fast learning phase crucial for episodic memory storage. This oversight leads to inconsistencies across temporally distant frames when generating longer videos, as these frames fall beyond the model's context window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning system for action-driven long video generation. Our approach incorporates a masked conditional video diffusion model for the slow learning of world dynamics, alongside an inference-time fast learning strategy based on a temporal LoRA module. Specifically, the fast learning process updates its temporal LoRA parameters based on local inputs and outputs, thereby efficiently storing episodic memory in its parameters. We further propose a slow-fast learning loop algorithm that seamlessly integrates the inner fast learning loop into the outer slow learning loop, enabling the recall of prior multi-episode experiences for context-aware skill learning. To facilitate the slow learning of an approximate world model, we collect a large-scale dataset of 200k videos with language action annotations, covering a wide range of scenarios. Extensive experiments show that SlowFast-VGen outperforms baselines across various metrics for action-driven video generation, achieving an FVD score of 514 compared to 782, and maintaining consistency in longer videos, with an average of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm significantly enhances performances on long-horizon planning tasks as well.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
video generation complimentary learning system slow-fast learning diffusion
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/23b1d32ab297e2e5d954b07a76ce6e5c43a9c0c7.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/6e8362c23197fd7ec8205fc7db0709c68ff56394.pdf
SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission