Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation
Diffusion models;Multi-modal data;Audio-visual generative models
Yes
No ethics review needed.
1. How is the low computational cost demonstrated in this work? Are there specific experiments or data that quantify this claim?
2. While the quality of the generated 2-second samples is good, how does the model perform on longer video samples?
1. This paper tackles an important problem in multi-modal generative modeling by introducing a cost-effective approach to synchronize audio and video generation, offering practical value for applications constrained by limited computational resources.
2. The custom loss function helps to steady the gradient of the discriminator. Extensive evaluations on benchmark datasets highlight enhanced fidelity for individual modalities and better alignment between audio and video, all achieved with minimal additional parameters, underscoring the efficiency of the proposed method.
This paper presents a method for constructing an audio-video generative model with low computational cost by leveraging pre-trained single-modal diffusion models for audio and video. The authors propose a lightweight joint guidance module that aligns audio and video outputs by adjusting scores to approximate the joint distribution, computed through the gradient of an optimal discriminator that distinguishes real and fake audio-video pairs.
1. Although I understand that the experiments are based on MM-Diffusion, the AIST and Landscape datasets are relatively small. The authors should discuss the generalizability of their method on larger datasets.
2. The paper claims that the method is model-agnostic; however, only one baseline is tested, which makes it difficult to confirm its general applicability.
3. Besides quantitative metrics, a subjective evaluation of the generated samples, would be beneficial to better assess the quality from a perceptual standpoint.
Yes
No ethics review needed.
* As mentioned in the Weaknesses, could the authors compare the training cost for different training setups?
* There are many values omitted in the FVD columns of Table 3 and 4. Are they the same as the corresponding rows below or different? Could the authors either explain or provide new values?
* Can the proposed method generalize to other pairs of modalities beyond video and audio?
* Will the proposed method remain valuable if diffusion models jointly trained on both modalities become widely available? If so, what does the cost-performance trade-off look like?
* The proposed method is well motivated and clearly presented, with a focus on achieving joint audio-video generation with low computational cost.
* The proposed method is training-based but model-agnostic, which can be more broadly applicable than model-specific designs.
* Extensive results on multiple datasets demonstrate the effectiveness of the proposed method in improving modality alignment and generation quality of videos and audio.
The submission presents a novel approach to audio-video joint generation with frozen single-modal generative models using a discriminator-guided cooperative diffusion process. The authors leverage existing pre-trained single-modal models for audio and video, aiming to reduce computational costs while enhancing multi-modal alignment. This approach, distinct from traditional model-specific or architecture-dependent techniques, incorporates a lightweight guidance module trained as a discriminator. The model effectively distinguishes between real and fake audio-video pairs, providing stability through a denoising regularization function inspired by standard diffusion methods.
* While the single-modal diffusion models remain frozen, the computation cost for training the joint discriminator, including forward passes through the frozen modules, is not reported. This cost needs to be compared to the total pretraining cost and a reasonable baseline of jointly trained diffusion models on both modalities.
* When text condition is provided to both diffusion models, the captions act as a weak alignment signal. Better experimental designs would be desired to remove the impact of this signal.
Yes
No ethics review needed.
1. Did you adopt other evaluation metrics to evaluate the proposed method on the performance of temporal alignment between generated audio and video? Because it seems that the ImageBind score is just for semantic alignment.
2. Did you try different discriminator structures or larger trainable parameters to see if there will be future performance improvement? 
3. Why the metric scores of MM-Diffusion in Table 2 are different from the ones of the original paper? 
4. It is better to compare the proposed method with "Seeing and Hearing" discussed in the related works by using the same pre-trained base models.
1. It is interesting to solve the joint audio-video generative problems by adjusting the pre-trained modality-specific generators with minimal trainable parameters rather than fully training the joint audio-video generation models.
2. Sufficient mathematic equations and proofs are given to support that the discriminator-based guidance can adjust the estimated scores from audio diffusion and video diffusion to approximately match the score of joint audio-video distribution.
3. The experimental results shown in the table present the proposed method can improve the alignment performance of generated audio and video in both in-domain and out-of-domain settings.
4. The paper is generally well-written.
This paper aims to leverage the pre-trained single-modal diffusion models for generating aligned video and audio. To achieve this objective and involve minimal computing costs, the authors proposed a lightweight joint guidance module to adjust the independent distribution from base models to match the joint distribution over audio and video by designing and training a discriminator. Based on the experimental results on the benchmark datasets, it seems that the proposed method improves the alignment score of generated audio and video while involving reduced trainable parameters.
1. The description that training a discriminator can adjust the pre-trained modality-specific diffusion models towards a joint audio-video generator is not very clear. For example, how does the discriminator-based guidance achieve both semantic and temporal alignment?  It is better to make it more clear. 
2. The improvement of metric score is limited especially for the OOD setting. In addition, from Table 4, the FVD and IB-TV scores of AudioLDM/AnimateDiff with the proposed guidance module are even worse. 
3. Based on the provided video files, although the performance in the IND is improved by the proposed method, the OOD setting performs poorly in generating high-quality video with the audio track. 
4. It is meaningful to see how different base models affect the performance. Whether more powerful pre-trained video and audio generators further improve the performance when equipped with the same guidance module?
Yes
No ethics review needed.
1. The training of generative adversarial models is always easy to explode. Did the authors face similar problems when optimizing the discriminator?

2. Since the additional training module is lightweight, I think the training procedure can be efficient. Can the author provide detailed training consumption required for training the discriminator? How much inference time does it increase?
1. The introduced joint module is lightweight and can be adapted to different single-modal generative models.

2. Authors conducted thorough experiments, including toy datasets, and in-domain and out-domain settings, demonstrating the effectiveness of the proposed method.
This paper presents a novel method for creating an audio-video generative model using pre-trained generative models for audio and video separately, and then guiding these models to work together using a discriminator.

The key innovation is a lightweight joint guidance module (an audio-video discriminator), which adjusts the scores estimated by the base models to match the score of the joint distribution over audio and video. This guidance is computed using the gradient of the discriminator, which distinguishes between real and fake audio-video pairs. The authors also introduce a regulation loss function that stabilizes the gradient of the discriminator.
1. Statement of datasets and evaluation metrics are duplicated in Sec 4.2 and 4.3.

2. In the section of related work, the author referred to Xing et al. (2024) as their similar work. However, neither quantitative nor qualitative comparisons are conducted with this work.
We construct an audio-video joint generative model with a low computational cost using frozen pre-trained diffusion models for each modality, computing a joint guidance through the gradient of a discriminator for real video-audio pairs and fake ones.
@inproceedings{
anonymous2024discriminatorguided,
title={Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=agbiPPuSeQ},
note={under review}
}
In this study, we aim to construct an audio-video generative model with minimal computational cost by leveraging pre-trained single-modal generative models for audio and video.
To achieve this, we propose a novel method that guides single-modal models to cooperatively generate well-aligned samples across modalities. 
Specifically, given two pre-trained base diffusion models, we train a lightweight joint guidance module to adjust scores separately estimated by the base models to match the score of joint distribution over audio and video. 
We show that this guidance can be computed through the gradient of the optimal discriminator distinguishing real audio-video pairs from the fake ones independently generated by the base models. 
On the basis of this analysis, we construct a joint guidance module by training this discriminator.
Additionally, we adopt a loss function to make the gradient of the discriminator work as a noise estimator, as in standard diffusion models, stabilizing the gradient of the discriminator. 
Empirical evaluations on several benchmark datasets demonstrate that our method improves both single-modal fidelity and multi-modal alignment with a relatively small number of parameters.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Diffusion models Multi-modal data Audio-visual generative models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e51b96631d88009d922e42e3a7e8c16d82082386.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/002a0a08a630cdbec4ef8021c406d0e8b4216e96.zip
Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission