MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences
Generative Model;Vison Language Model;Long Video Generation;Long Story Generation
**W7. Gaps that sometimes occur and solution.**

- First, the target ID in Fig. 8 is not meant to generate an exact match with the target ID, but rather to ensure that the generated character can be recognized as the same person as the target ID. We believe we have successfully achieved this goal.

- In our opinion, the gap that sometimes occurs is due to our data not reaching the model's capacity. We are confident that scaling up the model with more data will address this issue.

**W8. Training and inference cost.**

- Training cost has been discussed in Appendix A. For DAE, it takes 3 weeks with 6 NVIDIA A800 GPUs. For Autoregressive model, it takes 3 days with 4 NVIDIA H100 GPUs.
- For inference, both the 24GB and 80GB GPUs are capable of running the model. With the 80GB GPU, generating 128 keyframes takes approximately 3 minutes. This is because, during inference, all keyframe tokens are first generated, and then they are batched together for rendering into images. On the 24GB GPU, inference takes about 6 to 7 minutes, as it requires some off-the-shelf memory-efficient libraries, and during image rendering, smaller batch sizes are needed due to memory constraints.

**W9. Metrics on spatial-temporal quality.**

We have considered using metrics such as KVD, FVD to evaluate the spatio-temporal quality of videos. However, as discussed earlier regarding I2V in W5, we believe that video-related metrics primarily assess the image-to-video model's ability to generate short videos, rather than evaluating the core contribution of our work: ultra-long video generation. By taking both Table 1 and Table 2 into account, we believe that they already provide a comprehensive evaluation on long video level. Therefore, we respectfully believe that spatio-temporal metrics do not provide significant help in evaluating our core contribution.


We sincerely thank you for your time and thoughtful feedback. We are happy to provide any further clarifications or engage in additional discussions to address any remaining concerns!
Official Comment by Authors (2/2)
Thank you for your valuable feedback and suggestions. To clarify our approach, we first provide a brief summary of our video generation process:
1. The autoregressive model predicts keyframe tokens.
2. The DAE decoder decodes the keyframe tokens into keyframe images.
3. Leveraging existing image-to-video methods to generate video clips for each keyframe, resulting in a long video.

**W1&W2. Questrions about DAE.**

Thank you for pointing out the confusion. DAE refers to the Image Renderer Training in Fig. 2. Our DAE training is conducted in three stages:
1. In the first stage, we train on natural images (768x768) using Eq. 1 as the loss function.
2. In the second stage, we train on movie images (896x512) using Eq. 1 as the loss function.
3. In the third stage, we further incorporate face embedding and description embedding and train the model on movie images (896x512). The training loss for this stage becomes :
$
\mathcal{L}\_{DAE} = \mathbb{E}\_{\mathbf{x\_0}, \epsilon \sim \mathcal{N}(0, \mathbf{I})} \left\| \epsilon - \mathcal{D} \left( \mathbf{z\_t}, t, \mathcal{E}(\mathbf{x\_0}), \mathbf{f}, \mathbf{d} \right) \right\|\_2^2,
$ where $\mathbf{f}$ and $\mathbf{d}$ is the face embedding and description embedding, respectively.

We apologize for not clarifying this point. We will refine the figure and the explanation in the revised paper. Further details can be found in Appendix A.

**W3. Random masking.**

The random masking strategy is used only during training. It is applied in both the DAE and autoregressive model training. In the third stage of DAE training, we input concatenated image tokens, face embeddings, and description embeddings. With a probability of 0.15, we randomly replace each token with a zero token. During autoregressive model training, we randomly prevent the model from attending to masked tokens with a probability of 0.15 by applying attention mask. This strategy helps the model learn to infer missing content from the available information, improving its performance and reducing overfitting. During inference, random masking is not used.

**W4. High dropout rate.**

All parameters in the VLM are updated. We didn't try only tuning part of the model.  We conducted detailed experiments and found that a high dropout rate is indeed beneficial in our setting.

| Validation Loss/step  | 2500 | 5000 | 7500 | 10000 | 12500 | 15000 |
|:-------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|
| Dropout | 1.142  | 1.018 | 0.959 | 0.937 | 0.921 | 0.898 |
| No Dropout | 1.187 | 1.084 | 1.036 | 0.976 | 0.982 | 0.973 |

**W5. Questions about image-to-video model.**

- **i2v (image-to-video) is not our primary focus; the emphasis is on ultra-long generation. i2v is just a tool we use.** The core of i2v lies in generating high-quality short videos from images, whereas the core of our work is to generate coherent and consistent keyframes, and then use i2v to produce long videos. We choose to use SVD as our i2v method by default, as it is widely adopted in many existing works. However, it tends to produce results with limited motion. Therefore, this is an issue with SVD itself, rather than a limitation of our approach.

- Our main contribution is in exploring ultra-long video generation. We have emphasized in the paper that our method can be combined with any image-to-video model, and we demonstrate in the supplementary video that using other, more advanced i2v models can produce higher-quality, action-rich results. Therefore, we respectfully believe that the performance of our method would not be affected by any particular i2v approach. On the contrary, the performance of our method is determined by **the upper bound of the entire i2v field.**

**W6. The frame numbers of full-length video.**

- Currently, our model supports generating up to 128 keyframes at once. The total number of frames in the full-length video depends on the specific image-to-video method used. When employing the SVD-based approach outlined in the paper, the maximum number of video frames is 6400.

- However, if the storylines can be seamlessly connected between each long video, the long videos we generate can also be combined to form even longer coherent videos. Therefore, to some extent, the number of frames in our full-length video could be unlimited.
Official Comment by Authors (1/2)
Thank you for your valuable feedback and suggestions!

**ID-consistency Evaluation**

We have already provided ID-related evaluation in the paper, as seen in Table 1, with the ST and LT metrics. These are used to evaluate consistency in short-term and long-term, respectively. Detailed explanation for these two metrics is provided in Appendix B. Please let us know if you have any further questions.

**Boundaries of Our Method**

In Appendix F, we analyze the boundaries of the proposed method. Here, we summarize them as follows:

- When too many characters appear simultaneously, the model may sometimes confuse character IDs. (Figure 22)
- Since we perform sentence-level tokenization to ensure the resulting length, the model struggles to understand highly fine-grained and complex text prompts.
- Our method does not control object-level IDs. This can be addressed by further constructing object-level IDs.

**Technical Novelty and Contributions**

Our method is not a trivial combination of existing approaches. Our motivation and intuition behind the design are fundamentally different from those of existing methods. We believe our technical novelty and contributions are as follows:

- The problem we address is new. We solve an under-explored problem: unified long video generation and long story generation. To the best of our knowledge, few efforts have tackled this. Existing approaches are unable to generate videos of the length we achieve, and most are constrained by single scenes and lack controllability. Additionally, their results demonstrate significant drift over extended periods.
- The motivation behind our autoregression + diffusion approach is completely different from prior works using autoregressive + diffusion models. 
  - Previous video generation methods mostly rely on diffusion models. While these methods achieve good results on short videos of a few seconds, they face fundamental challenges when generating long videos, especially those in the tens of minutes. Scaling up video diffusion model requires enormous computational resources and data, which is nearly impossible for ultra-long video tasks. 
  - In the image domain, previous works about autoregression + diffusion mainly focus on the instruction-following ability. However, our focus is on the autoregressive model's ability to model long sequences. Our innovative framework inherits the advantages of both diffusion and autoregressive models, achieving state-of-the-art long video generation for videos of tens of minutes.
- Our method is orthogonal to existing video generation works. It can be integrated with various strong image-to-video models to produce higher-quality results.
- Ensuring Length: We design compact yet representative compressed tokens to ensure both quality and context length.
Preserving ID: We implement an ID-preserving strategy to ensure the consistency of multiple characters.
- Ensuring Controllability: We propose a multimodal script and innovatively implement sentence-level compression for text descriptions, ensuring both quality and extended context length.
- Supervision: Traditional autoregressive models are supervised with cross-entropy loss, but this is not feasible for our task, as we need to predict continuous tokens rather than discrete ones. Additionally, we observe poor performance when supervised solely with L1 and L2 loss. To address this, we supervise continuous real-valued token prediction, which effectively improves the model's performance.
- Furthermore, we demonstrate that our model is able to implicitly learn different shot types and transitions through multimodal scripts.

In conclusion, our approach is grounded in a thorough and careful analysis of the strengths and weaknesses of autoregression and diffusion in video generation, rather than simply combining existing methods. The task we address is both novel and of significant practical relevance. Furthermore, we have explored various aspects of our method, such as shot types, which further demonstrates its value and potential.

Once again, we are grateful for your thoughtful feedback! We hope these clarifications effectively address your concerns and we welcome any further discussion. Thank you for your time and attention.
Thanks for your kind review on our paper!

**Non-human Character Results**

Thank you for your valuable feedback. As discussed in our Future Work, our ability to maintain ID consistency for non-human characters is limited. For example, when generating a dog, the results often differ from the specific target dog we aimed for. Therefore, our method currently excels when the main character is human. 

However, we have included this limitation in our future work. We believe that constructing and leveraging non-human IDs will effectively address this issue.

**Adaptation Cost and Cherry-pick**

During implementation, we observed that the LLM adapts well to sentence CLIP embedding inputs with minimal adaptation cost. We believe this is because:
1. Both the CLIP text encoder and LLaMA handle rich semantic features. CLIP sentence embeddings are highly semantic, and understanding these semantic embeddings is easy for LLaMA because of its strong language capabilities.
2. The structured multi-modal scripts further assist the model in effectively understanding the sentence embeddings.

Our approach achieves a high success rate with minimal need for cherry-picking. Most of results are generated directly without selection.

We hope these responses address your concerns. Please feel free to let us know if you have any further questions!
Yes
No ethics review needed.
Since authors are changeing LLaMA input to CLIP embedding (for sentence), does it cause much for the LLM to adapt to such input?
How much cherry-pick do you need for a single long video?
The quality is GREAT!
Engineering wise, this paper produces a product-level solution to long-video (movie) generation; Scientificly, this paper also explores training a MLLM to output visual features given the input scripts, instead of simply using an agent framework to compose scripts, and use subject-driven models to preserve character identity.
The ID Preserving is also very effective
This paper presents MovieDreamer to make long video. This is challenging because current video diffusion models can only deal with short clips and do not have character consistency. The model trains a MLLM that can predicts the visual tokens with movie script conditioning. Then using the diffusion decoder to decode the condition into key frames, and using I2V to render it into video clips.
Lack of non-human results, don't know if such methods can generalize well to general domain long-vedio generation.
Yes
No ethics review needed.
Please provide the evaluation of the ID consistency of the model, clarify the technical contributions, and discuss the limitations of the framework.
+ The model provides an effective integration of autoregressive model and diffusion models, for the long visual sequence generation task. The combination is reasonable by successfully leveraging the arbitrary-length generation capability or AR model and the powerful rendering ability of diffusion models. 
+ The utilization of the gaussian mixture model for the continuous visual token prediction is reasonable and inspiring.
This paper presents a hierarchical generation framework for long video sequence generation, which can be potentially applied to movie video generation. The key idea is to leverage the autoregressive models to generate key frame tokens, and then use diffusion models to decode the tokens into RGB frames. This approach leverages the advantage of any-length future prediction ability of AR model and the powerful rendering ability of diffusion models. Through some other techniques such as multimodal script, face embedding condition, etc., the framework can generate ultra-long video sequences with better quality.
- The paper proposes a lot of techniques to improve the identity consistency among different clips, however, I cannot find any ID-related evaluation on the generation results. 
- The paper did not analyze the boundaries of the proposed method. For example, 
    - when will the model fail at ID consistency since the model only provides one face embedding as the condition. 
- The technical novelty is relatively limited. There exist many multimodal generative models can do the similar key frame generation task, though they may not be directly used for movie key generation.
Yes
No ethics review needed.
I have outlined some concerns and suggestions in the weaknesses section, and there are additional questions that need clarification.

1. Is the covariance matrix in Eq. 7 of the supplementary material a diagonal matrix?

2. In the few-shot training section, what is the “episode’s visual tokens” (line 302)?

3. In Fig 8, what are the reference images for generating the few-shot results?  If I understand correctly, face embedding is also used as part of the multimodal script to generate zero-shot results. For the few-shot generation, what images are used as references, and how do these reference images help improve identity consistency?
1.	The proposed framework seems promising for generating long video. 
2.	The proposed method can effectively generate character-consistent story keyframes, and the experimental results show the superiority of the proposed method to other state-of-the-art methods.
Existing diffusion model-based video generation models can only support short video generation and lack understanding of narrative structures and plot progressions. Therefore, this paper explores long-story video generation by leveraging the inference capability of autoregressive generation and video diffusion models’ rendering capability. Specifically, this paper proposes a hierarchical framework and a structured multimodal story input representation. The framework fine-tunes the multimodal large model LLAVA to generate compressed tokens of keyframes and fine-tunes SDXL as a decoder to generate keyframe images from the compressed tokens. Finally, SVD is used to generate the video. Additionally, the paper proposes a few-shot training method, enabling the model to achieve customized generation. Extensive experiments compare the proposed method with existing story keyframe generation methods, demonstrating superior performance.
The overall writing of the paper is relatively clear, and the experimental results are promising. However, some of the descriptions appear to be incorrect or misleading, and the experiments are also not entirely comprehensive.

1. In line 242, why are the parameters of the GMM stated as 2kd means and 2kd variances? According to the reference work, each GMM parameter should contain kd means, kd variances, and k coefficients for each compressed token.

2. What would the results look like if Eq. 2 were used as the sole objective function? There doesn't appear to be a corresponding analysis of this in the experiments. Is the inclusion of L2 and l2 necessary, or have previous works discussed the necessity of incorporating these terms?

3. In Fig 2, if I understand correctly, the multimodal script in the top-left corner is used to generate a single keyframe. However, the three keyframes shown on the right under the VLM section could easily give the impression that they were all generated by the same script. This may lead to some confusion for readers. Additionally, according to Equation 1, the input to the decoder should also include a noise latent. However, the inference process shown at the bottom of Figure 2 does not fully display the complete inputs and outputs, which could lead to some misunderstandings.
Yes
No ethics review needed.
See weaknesses
1. The proposed framework propose a hierarchical way to combine autoregressive modeling and diffusion image generator together for long-term complex video generation.
2. The paper proposed several novel techniques, e.g., continuous token supervision, anti-overfitting strategies, id-preserving, to improve the performance of the framework.
3. The proposed method achieved state-of-the-art performance on quantitative evaluation.
The paper proposed a novel hierarchical video generation framework MovieDreamer, which is able to generate long-duration videos with high visual fidelity. The framework consists of a SDXL-based visual autoencoder and a multimodality VLM. The autoencoder focuses on obtaining the visual tokens of input images, as well as rendering targeted keyframes, while VLM autoregressively predicts visual tokens. Quantitative evaluation shows that the proposed method outperforms sota in long video generation.
1. L212: Eq1, the DAE has not been clearly defined in the paper. Also, in Fig2, I could not find DAE in the pipeline. I hope the authors could clarify this.
2. It is a bit confusing whether Eq1 is the real training loss or not? According to Sec3.3 and Fig2, the input of SDXL should be image token, face embedding and description embedding, which does not align with Eq1. It is better to clarify the final training loss.
3. It is not clear how random masking strategies work during training and inference stages.
4. The high dropout rate (50%) seems to be unusual. Are all the parameters updated in VLM? Have the authors tried only tuning part of the network?
5. L323: if the feature of the first frame is always used, will generated character be limited to small motion, and limited content? For example, for large rotation and movement, the performance of the proposed method may drop?
6. What are the frame numbers of full-length video?
7. From Fig8, while compared to zero-shot setting, the few-shot setting indeed improves the ID-preserving ability. However, there is still an obvious gap between target ID and generated image. What could be the potential reason for this? Any solution?
8. What is the training cost? And what are the inference time and GPU memory requirements?  
9. For quantitative evaluation, it seems that the reported metrics all focus on image level. They can not evaluate how good the performance of spatio-temporal modeling. It is better the authors could evaluate the proposed method using other metrics for video generation.
@inproceedings{
anonymous2024moviedreamer,
title={MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=QHj2LL958o},
note={under review}
}
Recent advancements in video generation have primarily leveraged diffusion models for short-duration content. However, these approaches often fall short in modeling complex narratives and maintaining character consistency over extended periods, which is essential for long-form video production like movies. We propose MovieDreamer, a novel hierarchical framework that integrates the strengths of autoregressive models with diffusion-based rendering to pioneer long-duration video generation with intricate plot progressions and high visual fidelity. Our approach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering. This method is akin to traditional movie production processes, where complex stories are factorized down into manageable scene capturing. Further, we employ a multimodal script that enriches scene descriptions with detailed character information and visual style, enhancing continuity and character identity across scenes. We present extensive experiments across various movie genres, demonstrating that our approach not only achieves superior visual and narrative quality but also effectively extends the duration of generated content significantly beyond current capabilities.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Generative Model Vison Language Model Long Video Generation Long Story Generation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/3fc6ee6ee74471fd1d1f3ed29c6df3271e2c77f4.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/70d2f26220d208414f94f94c1eabae622076e470.zip
MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission