Mora: Enabling Generalist Video Generation via A Multi-Agent Framework
Video Generation;Multi-agent;Adaption Training
Yes
No ethics review needed.
（1）Optimization Challenges with the Self-Modulated Fine-Tuning Algorithm:

（1.1）The self-modulated fine-tuning algorithm in Section 3.2 is unconventional and lacks supporting citations. Could the authors provide more references or evidence to justify this approach?

（1.2）Given the complexity of optimizing such a system, training with only 96 samples seems inadequate. Can the authors elaborate on how this sample size is sufficient, or provide results from larger-scale experiments?

（1.3) The visual results show significant artifacts like blurriness and object deformation. How do the authors plan to address these issues to improve visual quality?


（2）Effectiveness of Data-Free Training Strategy:

（2.1）The data-free training strategy resembles a distillation approach and may result in overfitting to large models. Can the authors show comparative experiments to demonstrate that the proposed multi-agent approach offers clear advantages over simply distilling a smaller model?

（2.2）Additionally, can the authors provide a quantitative analysis of agent success rates and inference efficiency to better justify the benefits of using the multi-agent system?
(1) Comprehensive system: The proposed Mora is capable of handling a wide range of visual generation and editing tasks, including Text-to-Image Generation, Image-to-Image Generation, Image-to-Video Generation, and Video Connection. 

(2) Clarity of Writing: The writing is clear and easy to follow, making the paper accessible to a broader audience.

(3) Code Availability: Although the training code is not provided, the authors have shared part of their code in the supplementary materials, promoting transparency and potential reproducibility of their results.
The paper introduces Mora, a novel multi-agent  video generation framework, which leverages existing open-source modules to replicate Sora’s functionalities. Mora enhances inter-agent coordination through multi-agent fine-tuning with a self-modulation factor and uses a data-free training strategy to synthesize high-quality training data. It also incorporates a human-in-the-loop mechanism for data filtering. Evaluation shows Mora achieves comparable performance to Sora.
(1) Limited Novelty: The paper's primary contribution lies in utilizing a language model as an agent to call various pre-existing  visual generation models for video generation. However, neither the agent system nor the video generation models are novel contributions from the authors. Although the authors propose some additional modules to optimize the system, these present several issues, as outlined below.

(2) Concerns with Self-Modulated Fine-Tuning Algorithm: In Section 3.2, the authors introduce a self-modulated fine-tuning  algorithm that optimizes both the language model (A1) and visual generation models (A2-A5) end-to-end  using a visual generation loss. This concept is unconventional and lacks sufficient supporting research. The authors do not provide adequate citations to justify this algorithm. Additionally:

(2.1) The process is inherently difficult to optimize. The authors mention using only 96 samples for training in Section 4.2, which is unconvincing for such a complex task.

(2.2) Without releasing the training code, it is hard to assess the effectiveness of their methodology based solely on theoretical descriptions and hyperparameter choices.

(2.3) The visual results (Figures 9-12)  show significant artifacts, such as blurriness and object deformation, which raise doubts about the efficacy of the self-modulated fine-tuning algorithm.

(3) Data-Free Training Strategy and Distillation Concerns: The authors propose a data-free training strategy using large models to synthesize training data, which resembles a distillation approach. However, this strategy may cause the generated results to overfit to the large models. This calls into question the added value of using a multi-agent  approach, which typically comes with higher inference costs. The authors should provide a fair comparison to demonstrate that training Mora with this strategy yields better performance  than directly distilling a smaller model. Additionally, quantitative analysis of agent success rates and inference efficiency would strengthen the claims made about the multi-agent system’s benefits.

(4) Problem Definition in Section 3.1: The authors claim that their objective is to maximize quality metrics while ensuring diversity in the generated videos. However, quality metrics alone are not the ultimate goal for video generation. Beating existing benchmarks such as SORA does not inherently demonstrate the model's superiority. Furthermore, in an agent-based system, other factors such as module collaboration speed, accuracy, and robustness must be considered.
Yes
No ethics review needed.
Please refer to Weaknesses for more details.
- A new generalist Agent-based framework named Mora is proposed.
- The proposed Mora can perform a lot of video-related tasks and shows remarkable results.
- Experimental results indicate the effectiveness of the proposed method, and Mora can surpass the capabilities of existing leading models in several respects.
This paper introduces a new generalist framework named Mora that synergies Standard Operating Procedures for video generation. The proposed Mora can be used to tackle a lot  of video-related tasks. Specifically,  Mora consists of five Agents: Prompt Selection and Generation Agent, Text-to-Image Generation Agent, Image-to-Image Generation Agent, Image-to-Video Generation Agen, and Video Connection Agent. Experimental results show the effectiveness of the proposed method, and Mora can surpass the capabilities of existing leading models in several respects.
- This paper contains a lot of existing techniques involving Prompt enhancement, Image generation, Image editing, Video generation, and Video connection. The main technical innovations should be clarified. Explaining how the system's architecture or the interaction between components differs from or improves upon existing approaches may help readers to understand.
- Since many components are involved, Mora's inference speed and computational requirements to existing state-of-the-art models for each of the video-related tasks mentioned should be discussed. Additionally, is there any optimizations implemented to improve efficiency, given the multi-component nature of the system.
- Will there be accumulated errors between multiple components? Providing empirical results showing how errors may or may not accumulate across different components and tasks is necessary. Or are there any techniques implemented to mitigate potential error accumulation between components?
Yes
No ethics review needed.
How to explain that a multi-step generation can perform better than an end-to-end generation? For example, why a combination of open-sourced text-generation model, text-to-image model, image-to-image model and image-to-video model can outperform an end-to-end open-source text-to-video generation model and replicate a close-source model's functionalities.
1. The authors break down the process of video generation into several steps and utilize off-the-shelf models to implement the generation process.
2. The authors clearly describe the whole pipeline.
3. Data systhesis and filtering strategy are used to improve the quality of training.
This paper proposes a multi-agent framework named Mora to enable text-to-video generation by leveraging open-source models such as Llama and Stable Diffusion. The main contributions include a self-modulation factor for inter-agent coordination, a data-free training strategy and a human-in-the-loop data filtering. The authors compare the performances of Mora with Sora on six video generation tasks.
1. There are no demo videos provided by the authors, no gif files or no project website. It is hard to evaluate the quality of generated video with some sampled frames. 
2. The key component of the self-modulated training is the modulation factor. Is it able to visualized this factor?
3. How would the performances be if the agents A1 to A5 were replaced with different models?
Mora, a multi-agent framework, outperforms open-source text-to-video methods by using self-modulation, data-free training, and human-in-the-loop filtering, achieving results comparable to OpenAI's Sora on video generation tasks.
@inproceedings{
anonymous2024mora,
title={Mora: Enabling Generalist Video Generation via A Multi-Agent Framework},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=S7F7IMGX4O},
note={under review}
}
Text-to-video generation has made significant strides, but replicating the capabilities of advanced systems like OpenAI’s Sora remains challenging due to their closed-source nature. Existing open-source methods struggle to achieve comparable performance, often hindered by ineffective agent collaboration and inadequate training data quality. In this paper, we introduce Mora, a novel multi-agent framework that leverages existing open-source modules to replicate Sora’s functionalities. We address these fundamental limitations by proposing three key techniques: (1) multi-agent fine-tuning with a self-modulation factor to enhance inter-agent coordination, (2) a data-free training strategy that uses large models to synthesize training data, and (3) a human-in-the-loop mechanism combined with multimodal large language models for data filtering to ensure high-quality training datasets. Our comprehensive experiments on six video generation tasks demonstrate that Mora achieves performance comparable to Sora on VBench \cite{huang2024vbench}, outperforming existing open-source methods across various tasks. Specifically, in the text-to-video generation task, Mora achieved a Video Quality score of 0.800, surpassing Sora’s 0.797 and outperforming all other baseline models across six key metrics. Additionally, in the image-to-video generation task, Mora achieved a perfect Dynamic Degree score of 1.00, demonstrating exceptional capability in enhancing motion realism and achieving higher Imaging Quality than Sora. These results highlight the potential of collaborative multi-agent systems and human-in-the-loop mechanisms in advancing text-to-video generation.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Video Generation Multi-agent Adaption Training
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/3df0b626facd1677acecfb7cb923702639a77ee7.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/eca9282465a99754c37a0a98328933ba2a5a12e7.zip
Mora: Enabling Generalist Video Generation via A Multi-Agent Framework
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission