AudioScenic: Audio-Driven Video Scene Editing
Kaixin Shen Ruijie Quan Linchao Zhu Jun Xiao Yi Yang















































  Audio-driven visual scene editing endeavors to manipulate the visual background while leaving the foreground content unchanged, according to the given audio signals. Unlike current efforts focusing primarily on image editing, audio-driven video scene editing has not been extensively addressed.
 In this paper, we introduce , an audio-driven framework designed for video scene editing. 
 integrates audio semantics into the visual scene through a temporal-aware audio semantic injection process. As our focus is on background editing, we further introduce a SceneMasker module, which maintains the integrity of the foreground content during the editing process.
 exploits the inherent properties of audio, namely, audio magnitude and frequency, to guide the editing process, aiming to control the temporal dynamics and enhance the temporal consistency. 
First, we present an audio Magnitude Modulator module that adjusts the temporal dynamics of the scene in response to changes in audio magnitude, enhancing the visual dynamics. Second, the audio Frequency Fuser module is designed to ensure temporal consistency by aligning the frequency of the audio with the dynamics of the video scenes, thus improving the overall temporal coherence of the edited videos.
These integrated features enable  to not only enhance visual diversity but also maintain temporal consistency throughout the video. 
We present a new metric named temporal score for more comprehensive validation of temporal consistency. We demonstrate substantial advancements of  over competing methods on DAVIS <cit.> and Audioset <cit.> datasets.









































: Audio-Driven Video Scene Editing
    Kaixin Shen, Ruijie Quan, Linchao Zhu, Jun Xiao, Yi Yang

Zhejiang University
    April 29, 2024
=================================================================================





§ INTRODUCTION












Video scene editing aims to manipulate the scene (background) content of videos while keeping foreground objects unaltered. It is a critical task in the world of multimedia, serving as the backbone of storytelling across various platforms, from movies and television to online content. Recent methods focus on using diffusion models <cit.> to guide editing. Other modalities like text <cit.>, image <cit.>, audio <cit.>, instruction <cit.>, and motion <cit.> assist in editing to improve result quality. 


Previous methods conducting audio-conditioned visual editing mainly centered on image scene editing <cit.>.
Specifically, these methods accept a source image as input and facilitate scene editing within the audio semantic condition. When utilizing audio to guide video scene editing, three primary challenges need to be addressed: i) Foreground preservation: While editing the scene areas of videos, the foreground content should be unaltered. ii) Temporal dynamics: After accurately editing the scene content, it is essential to present temporal visual variations aligned with the audio condition (e.g., in “Rain” scenes, the intensity of the rain changes over time and varies with the fluctuations in rainy sound).  Neglecting temporal dynamics can result in static scenes in the edited frames. iii) Temporal consistency: Maintaining temporal consistency is crucial in video editing to ensure consistency across frames and prevent visual flickering. 
It is worth noting that temporal dynamics and consistency are distinct yet crucial aspects of effective video editing. 


Visual scene editing aims to manipulate the visual background content while keeping the foreground objects unaltered, which is crucial in real-world applications, such as TikTok videos.

Recent audio-driven works primarily focus on using diffusion models <cit.> to guide image editing <cit.>, utilizing a source image as input and conducting editing on a static image given the audio conditions. 

In this paper, we introduce a new setting, audio-driven video scene editing, aiming to dynamically adapt the visuals to the changing tones and semantics of the audio.

When utilizing audio to edit video scenes, three principal challenges need to be addressed: i) Foreground preservation: while editing the scene areas of videos, the foreground content should remain unaltered. ii) Audio-aligned temporal dynamics: after accurately editing the scene content, it is essential to present temporal visual variations aligned with the audio condition (e.g., in “Rain” scenes, the intensity of the rain changes over time and varies with the fluctuations in rainy sound). Neglecting temporal dynamics can result in static scenes in the edited frames. iii) Temporal consistency: maintaining temporal consistency is crucial in video editing to ensure consistency across frames and prevent visual flickering. 














To address these challenges, we introduce , a new framework that leverages audio as a condition to guide video scene editing. In particular,  includes the temporal-aware audio semantics guidance process and three specialized modules: SceneMasker, Magnitude Modulator, and Frequency Fuser. Inside the temporal-aware audio semantic injection process, we extract semantic embeddings from audio clips, element-wise adding them with timestep embedding derived from the sampling timestep. Then we utilize the timestep embedding fused with audio semantics to direct a latent denoising process within an adapted 3D U-Net diffusion model during both training and inference stage. 1) To preserve foreground content, we propose to utilize SceneMasker, a mask blending module, to restrict the audio embedding's influence exclusively to the scene areas of videos. In addition, relying solely on audio semantics is insufficient to create temporally dynamic and coherent video scenes. We observe that magnitude and frequency are pivotal audio properties in this regard. 2) To create temporal dynamics, we employ the audio magnitude as a controller. The audio magnitude plays a crucial role in modulating the fine-grained temporal variations during the scene editing <cit.>. For instance, a stronger audio magnitude can lead to more dramatic changes in the video scene, enabling a broader spectrum of visual dynamics. Consequently, we design a module named Magnitude Modulator to modulate the influence of audio semantic embeddings on the video based on the audio magnitude features. 3) Further, we employ audio frequency information to maintain temporal consistency. Audio and video are naturally aligned in the frequency domain along the time axis, thus frequency information contains essential domain conditions of the audio data, which provide valuable temporal references <cit.>. We introduce Frequency Fuser, which merges audio and video frequency information. It transforms spatial video features into frequency space, enabling their integration with audio frequency features through a weighted-multiplication mechanism. 

On the whole,  demonstrates several compelling features: First, it combines audio guidance and SceneMasker module for video scene editing, striking a delicate balance by editing the scene while keeping the foreground content unaltered. Audio clips with the same semantic label can be employed to generate varying visual scenes. We can also utilize emotional resonance like music mood to conduct scene style editing, as shown in Fig <ref>. Second,  leverages the Magnitude Modulator module to control the scene content based on the audio magnitude. Such magnitude control enhances the temporal dynamics of synthesized scenes. Third, the Frequency Fuser module inside  enables the model to focus on frequency elements that demonstrate a true cross-modal correlation between audio conditions and video scenes over time, making the results maintain temporal consistency. 


For more comprehensive validation of temporal consistency, we present a new metric named temporal score. The temporal score measures the temporal consistency of results on the premise of semantically accurate scene editing. By fully exploiting the attributes of audio such as semantics, magnitude, and frequency, the result of  is the generation of scenes that are not only coherent but also exhibit high diversity.









In a nutshell, our contributions are three-fold:

  
  * We introduce a new task termed audio-driven video scene editing and develop a comprehensive protocol with a new evaluation metric, the temporal score, to enhance the assessment of methodological performance. 
  
  * We propose an audio-driven framework named  to edit video scenes.  utilizes audio magnitude and frequency information to enhance temporal dynamics and ensure temporal consistency in video scene editing. 
  
  * We demonstrate promising applications in video editing, showing that our method produces audio-synchronized videos and outperforms previous text-driven and audio-driven methods in video scene editing, as demonstrated by video samples using the DAVIS <cit.> and Audioset <cit.> datasets.






§ RELATED WORK



Text-Driven Video Editing. Compared to image editing <cit.>, video editing <cit.> is more challenging due to its extra temporal dimension. Tune-a-Video <cit.> first inflates a text-to-image 2D diffusion model for video editing, it finetunes the model on a single video and generates new videos. Fatezero <cit.> takes inspiration from Prompt-to-Prompt <cit.> and edits the videos by changing the text-image cross-attention map, showing promising results in preserving foreground shape and motion. Dreamix <cit.> uses a text-to-video backbone for motion editing and preserves temporal consistency. Both Text2Live <cit.> and StableVideo <cit.> utilize Layered Neural Atlases. Text2Live divides the video into several layers and edits each layer separately through a text description. StableVideo designs an inter-frame propagation mechanism and aggregation network to generate the edited atlases from the keyframes, thus achieving temporal and spatial consistency. Ground-a-Video <cit.> extracts the layout of objects in the video and uses this location condition for editing. TokenFlow<cit.> centers on enhancing video latent feature smoothness to reduce the video visual flickering. Rerender-A-Video <cit.> proposes a hierarchical cross-frame constraint to preserve temporal consistency. UniEdit <cit.> aims to achieve zero-shot motion and texture editing by injecting conditions into self-attention and cross-attention layers. Solely using text for editing content is unsuitable for indescribable objects and scenes, prompting recent work to leverage auxiliary visual conditions. We experimentally observed that text-based methods may face challenges in editing video scenes. 

We extend the condition modality to audio and improve the video scene synthesis quality using audio magnitude and frequency information since both audio and video naturally have temporal information, which is lacking in other modalities like text.

Audio-Guided Visual Synthesis. Prior methods mainly leverage audio to conduct image generation or video generation. Several methods <cit.> focus on image synthesis using generative models.  <cit.> employs audio to edit images. Recently some works have explored audio-driven video generation. AAdiff <cit.> combines Prompt-to-Prompt <cit.> with audio control, generating videos by multiplying the audio magnitude and the corresponding text token embedding. TPoS <cit.> integrates audio with temporal semantics and magnitude, generating a specified image and then conducting image-to-video generation.  <cit.> realizes audio-to-video, video-to-audio, and audio-video joint generation using audio and text as conditions. However, they rely on text-relevant audio and utilize audio as a textual auxiliary condition to generate videos through 2D text-to-image diffusion models.
Unlike most existing audio-driven works conducting visual generation, we aim to employ audio to guide video scene editing.
Sound-G <cit.> utilizes StyleGAN <cit.> to edit images using audio semantics. Due to the absence of temporal information, it is incapable of editing videos. The utilization of StyleGAN also results in its poor generalizability. Within audio semantic information, 
We additionally incorporate the audio magnitude and frequency feature to present temporal consistent videos. We show our ability to generate diverse and dynamic scenes.






§ PRELIMINARIES






Stable Diffusion. In this paper, we use Stable Diffusion, a widely used text-to-image model, to achieve video editing. Stable Diffusion is based on the Latent Diffusion Model (LDM), which conducts the denoising process in the latent space of an autoencoder, namely ℰ(·) and 𝒟(·), implemented via VAE pre-trained on large image datasets. This design shows an advantage in reducing computational costs while keeping the visual quality. 

During the training stage, an input image x_0 ∈ℝ^H × W × 3 is initially compressed to the latent space by the frozen encoder, yielding z_0=ℰ(x_0), then diffusion forward process gradually adds Gaussian noise to z_0 to obtain z_t through Markov transition with the transition probability:

    q(z_t | z_t-1) = 𝒩(z_t; √(1-β_t)z_t-1, β_t𝐼),

for t = 1,…, T, with T denoting the number of diffusion
timestep. The sequence of hyper-parameters β_t determines the noise strength at each step. Then, the backward denoising process is given by the transition probability:

    p_θ(z_t-1|z_t) = 𝒩(z_t-1; μ_θ(z_t,t), σ^2_tI) ,

for t = T,…, 1. Here the mean μ_θ(z_t,t) can be represented using the noise predictor ϵ_θ which is learned by the minimization of the MSE loss of network parameter θ:

    ℒ = 𝔼_ℰ(x_0), y, ϵ∼𝒩(0, 𝐼), t‖ϵ - ϵ_θ(z_t, t, τ_θ(y)) ‖_2^2 ,

where y is the corresponding textual description, τ_θ(·) is a text encoder mapping the text y to an embedding.











Audio Preprocessing. We take audio clips A as conditional. We introduce audio preprocessing for extracting audio semantic embedding E_a, magnitude features M_a, and frequency features F_a. 

To obtain the audio semantic embedding E_a, we employ the ImageBind <cit.>, a multi-modality model to extract the embedding from audio, thereby eliminating the need for additional training. 

For acquiring the magnitude feature M_a, we first employ a sliding window to divide the input audio clip into N chunks. Note that N corresponds to the number of frames along the time axis. Then we calculate the average magnitude within each chunk. Subsequently, we normalize and smooth the averaged magnitude of each chunk using the Softmax function:

    Softmax(M_i) = exp(M_i/τ)/∑_n ∈ Nexp(M_n/τ),

where M_i denotes the ith chunk. τ is the temperature parameter, controlling the smoothness of magnitude value of the audio chunks. The smoothness prevents the magnitude changes of the audio temporally drastic. Now we obtain audio magnitude M_a ={M_i}_1^N.

To extract the audio frequency feature F_a, we utilize a Mel Spectrogram (Mel). This method applies a frequency-domain filter bank to time-windowed audio clips and outputs the audio frequency feature, expressed as F_a=Mel(A). 




§ METHOD

Problem Formulation. Given the source video V ∈ℝ^N × C × W × H and source audio clip A ∈ℝ^L, our goal is to predict the scene-edited video V^*∈ℝ^N × C × W × H with a new audio clip A^*∈ℝ^L, where N is the number of frames, C, W, H denote the channel, width, and height, respectively, L is the audio clip duration.

Overview. Our  focuses on editing video scenes (backgrounds) with the assistance of audio properties such as semantics, magnitude, and frequency. We utilize a modified version of Stable Diffusion as our foundational framework for scene editing (<ref>). We propose a temporal-aware audio semantic injection process to integrate audio semantics into the model for editing guidance (<ref>). To edit the scenes while keeping the foreground content unaltered, we employ SceneMasker to restrict the impact of audio conditions exclusively to the scene areas (<ref>). Moreover, we introduce two audio-specific modules: Magnitude Modulator (<ref>) and Frequency Fuser (<ref>), aimed at enhancing temporal dynamics and ensuring temporal consistency, respectively.



 §.§  Framework





We introduce the video scene editing pipeline, which is depicted in Fig. <ref>. 
We employ Stable Diffusion <cit.>, which is composed of a VAE autoencoder and a U-Net. First, a VAE Encoder ℰ(·) compresses the source video into latent feature z_0, which can be reconstructed back to video by a VAE Decoder 𝒟(·). Second, we modify the U-Net to incorporate audio conditions and train it to remove the noise using the objective:

    ℒ = 𝔼_ℰ(x_0), y, ϵ∼𝒩(0, 𝐼), t‖ϵ - ϵ_θ(z_t, t, E_a, M_a, F_a) ‖_2^2 .

During inference, we obtain a noisy video latent feature z_t through the DDIM inversion process ℐ(·) without auditory condition. Then we employ a new audio clip A^* to guide its DDIM sampling process 𝒮(·) through the fine-tuned model. That is,






    V^*   = 𝒟(𝒮(ℐ(ℰ(V)),A^*)).







The modified U-Net is mainly composed of stacking Residual blocks and Transformer blocks, as shown in Fig. <ref>. The Residual block incorporates the temporal convolution layers and the Magnitude Modulator module. The Magnitude Modulator module is integrated between temporal convolution layers. In the Transformer block, the Frequency Fuser module is added between the text-video cross-attention layer and the feedforward network. Note that though we are not using the textual condition as input, we retain the text-video cross-attention layer and input blank textual strings to maintain the ability of the pre-trained diffusion model. Temporal convolution layers are utilized to enforce the model to capture temporal information of video latent features. The Magnitude Modulator module achieves temporal visual control through audio magnitude features M_a. The Frequency Fuser facilitates the video latent features z_t in acquiring the frequency information encapsulated within the audio frequency feature F_a. 








 §.§ Temporal-Aware Audio Semantic Injection


Previous text-driven video editing works <cit.> often leveraged cross-attention layers to introduce conditional guidance. 
These methods rely on the ability of textual descriptions and video to generate semantically matched attention maps through pretraining. Some works <cit.> add CLIP image embedding with timestep embedding to introduce indescribable image conditions. This method enforces the image embedding to apply a greater impact compared to the textual conditions utilized in the cross-attention layers. We observe that the audio conditions and image conditions share numerous similarities. Audio encompasses a range of low-level information that is difficult to convey through textual descriptions. Furthermore, there is a necessity to devise a method whereby audio can exert a powerful impact on videos to generate scenes effectively. Motivated by these considerations, we follow <cit.> and propose a temporal-aware audio semantic injection (TASI) process. The injection of the audio semantic into the U-Net is that we add the audio semantic embedding E_a with timestep embedding temb, which is derived from the sampling timestep t. Then we take the timestep embedding fused with audio semantics as input to the U-Net for auditory guidance.








 §.§ SceneMasker for Foreground Texture
 


In this section, we propose SceneMasker to preserve foreground content along with ControlNet. During scene editing, integrating audio conditions impacts the foreground texture. The core of this issue lies in the generative process, where audio conditions are added with timestep embedding, influencing all regions of the video including the foreground. Therefore, the texture of the foreground is altered in response to audio information. Prior approaches like ControlNet <cit.> focus on preserving the spatial structure, specifically the shape of foreground elements, but do not address the nuanced impact of audio conditions on texture consistency. Refer to <ref> for visualization. Consequently, using ControlNet is insufficient to preserve foreground content.




To tackle this specific challenge, we employ SceneMasker, a mask blending module, to constrain the influence of audio conditions only on the scene areas. In concrete, we obtain a binary mask ℳ_k that splits the foreground and scene content apart. Then the timestep embeddings fused with and without audio embedding are blended with the binary mask. That is,

    temb = ℳ_k ⊙ temb_uf + (1-ℳ_k) ⊙ temb_f,

where temb_f and temb_uf denote the timestep embedding fused with and without audio embedding, respectively. temb is the blended timestep embedding used for the proceeded denoising process. SceneMasker complements ControlNet by safeguarding against unwanted texture changes in the foreground, ensuring that both shape and texture remain consistent. This solution offers a more holistic approach to maintaining the integrity of visual content.



 §.§ Magnitude Modulator for Audio-Aligned Temporal Dynamics


In this section, we introduce the Magnitude Modulator module that is designed to control the visual effects based on audio magnitude. Prompt-to-Prompt <cit.> showcases an efficient approach for image editing through the weight control of the cross-attention map between text and images. Building on this foundation, previous work <cit.> employs the audio magnitude to modulate the generated video effects by multiplying the smoothed magnitude value with the attention map between target text tokens and the video. However, they rely on an additional textual condition and a specific mapping algorithm to match the text token that mirrors the semantics of the input audio. Worse, this dependency poses an extra challenge in that the required textual token with exact semantics may not always be present in the prompt, thus constraining the adaptability of this approach in video editing scenarios.

To address these limitations, we integrate the Magnitude Modulator to modulate the timestep embedding in the Residual blocks. At each denoising step, we multiply the preprocessed audio magnitude feature M_a with the timestep embedding temb, expressed as: 

    temb^'=M_a ⊙ f(temb),

f(·) is MLP. We add this weighted timestep embedding temb^' with video latent features z_t to achieve modulation of the editing effect.

In the inference stage, we can facilitate audio magnitude to control the editing effect of the video. For example, the video scene will change more drastically when the magnitude signal is strong. Our proposed Magnitude Modulator extends the temporal visual diversity of video scenes as we can generate rich scenes within the variance of audio magnitude.




 §.§ Frequency Fuser for Temporal Consistency


We employ audio frequency information to preserve video temporal consistency. The integration of audio frequency information during video editing is crucial. Noteworthy information within audio conditions often resides in the frequency domain <cit.>.
In many real-world scenarios, changes in both visual and auditory elements often coincide. 
Utilizing the frequency characteristics of audio facilitates the visual elements in capturing audio's temporal aspects, thereby producing videos with greater temporal coherence. In addition, one modality may contain noise or irrelevant data absent in the other. Merging audio and video in the frequency domain enables the refinement or filtering of visual frequency components based on audio data. This aspect has not been extensively explored.

We introduce a novel approach through the Frequency Fuser module, which leverages audio frequency characteristics as controlling weights applied to video frequency features within the frequency domain. Initially, we obtain the audio frequency feature F_a in the audio preprocessing stage and transform it into a controlling weight using a Weight Encoder Enc(·). That is, F_a^w = Enc(F_a ).
Note that the Enc(·) includes two MLPs and a non-linear layer.

Drawing inspiration from  <cit.>, we convert video spatial features z_t to the frequency domain, preserving the low-frequency components to retain the video's global layout. Subsequently, we integrate the audio frequency weight F_a^w into the high-frequency components to enhance the content details and temporal consistency. The operations are defined as:

    F^L_z   = ℱℱ𝒯_3D(z_t) ⊙𝒫, 
    F^H_z   = ℱℱ𝒯_3D(z_t) ⊙ (1-𝒫) , 
    F^H'_z   = F^H_z * F_a^w, 
    z_t^'   = ℐℱℱ𝒯_3D(F^L_z + F^H'_z),

where ℱℱ𝒯_3D denotes the Fast Fourier Transformation operated on video spatial features, ℐℱℱ𝒯_3D is the Inverse Fast Fourier Transformation that maps back the video frequency features. 𝒫 is the spatial-temporal Low Pass Filter. Note that we reshape the F_a^w to align with the video frequency features for effective multiplication.

Through the Frequency Fuser module, we ensure that video features capture the frequency information from the audio, aligning the visual content with the auditory component over time. Such cross-modal frequency components significantly contribute to the temporal coherence of the edited video.





















§ EXPERIMENT









 §.§ Implementation Details

Our  is implemented based on Stable Diffusion. We initialize the diffusion model and inflate the 2D U-Net structure to process 3D video input. In the fine-tuning stage, we fine-tune the model on a source video at a resolution of 768 × 768 and then conduct inference to edit videos with different scenes. We fine-tune the model for 300 steps with a learning rate of 3 × 10^-5 and employ the DDIM sampler with 50 steps. We benefit from ImageBind and Mel Spectrogram for the feature extraction of audio. During inference, we employ ControlNet to preserve the shape consistency of foreground content. We use Grounded-SAM and XMem to obtain the binary mask for the SceneMasker module. Detailed descriptions of these models can be found in the Appendix. We experimented on a single NVIDIA RTX 3090 GPU. Our code will be released.





 §.§ Datasets

DAVIS <cit.>. The DAVIS dataset is designed for the task of video object segmentation. In this dataset, the main objects are segmented in the scene and divided by semantics. We select video samples from DAVIS dataset for training and inference.

Audioset <cit.>. The Audioset dataset contains videos from YouTube annotated into 527 classes. For each class, the dataset contains an unbalanced training set, a balanced training set, and an evaluation set. We select videos and audio clips from the Audioset balanced training sets and evaluation sets for training and inference.

ESC-50 <cit.>. The Environment Sound Classification (ESC) dataset contains 2000 5s audio clips. It has 50 audio classes, including object and natural sound. We choose audio clips from this dataset for generating video scenescapes in the inference stage.



 §.§ Evaluation Metrics
 

In this section, we introduce three metrics for quantitative measure. Following <cit.>, we use Semantic Accuracy(Sem-A) to measure the editing semantic accuracy and Structural Similarity Index Measure (SSIM) to validate the foreground content degradation. In addition, we propose a new Temporal Score (Temp-S) to measure the temporal consistency comprehensively.


Sem-A <cit.>. Sem-A is the percentage of frames where the edited image has a higher CLIP-T score (average similarities between the CLIP embedding of conditions and all frames) to the target condition than the source condition. It measures the performance of editing methods at the semantic level. Note that we use the textual semantic labels of audio clips to compute the CLIP-T Score and compare the result with the text-driven and audio-driven methods.


SSIM <cit.>. SSIM measures the similarity between two images. We focus on editing video scenes while keeping foreground content unaltered. Therefore, we employ a mask to isolate the foreground regions of the video and subsequently calculate SSIM for these regions before and after editing. Through this metric, we quantify video foreground quality degradation.


Temp-S. When measuring the temporal consistency of edited videos, using the CLIP-F score (average pairwise similarities of the CLIP embedding of images) alone does not provide a comprehensive evaluation, as the CLIP-F score can yield high scores even when the video is not edited at all. Consequently, we introduce Temporal Score (Temp-S) for a holistic assessment, that is, Temp-S = CLIP-F * CLIP-T, where CLIP-T calculates average similarities between the CLIP embedding of conditions and all frames. Temp-S evaluates the temporal consistency of results on the premise of semantically accurate scene editing.































 §.§ Comparisons


Competing Methods. We show our superiority in foreground preservation, temporal dynamics, and temporal consistency through qualitative and quantitative evaluation. To ensure a fair comparison, we select two sets of competing methods: i) Text-driven methods: 1) Tune-A-Video <cit.> fine-tunes an inflated diffusion model on a single video to produce similar content. 2) FateZero <cit.> utilizes Prompt-to-Prompt in the video editing task and uses textual descriptions to change the value of the text-image cross-attention map, facilitating video editing. 3) VideoP2P <cit.> designs a Text-to-set model to generate a set of semantically consistent videos and uses a decoupled-guidance strategy to improve the model's robustness. ii) Audio-driven methods: Sound-G <cit.> trains an audio encoder to produce audio features aligned with image features. It then manipulates images with audio features using StyleGAN <cit.>.  





Qualitative Comparisons. We first compare the results of our model and text-driven methods in semantic scene editing. The qualitative results are depicted in Fig. <ref>. The text-driven methods are given the prompt “A jeep car is moving beside the sea wave” and our audio-driven framework takes a “Sea wave” semantic audio clip as input. The edited results from Tune-A-Video flickers and VideoP2P are weak in producing a vivid sea wave scene. FateZero preserves temporal consistency but it is weak in expressing temporal dynamics of the scene. The scenes generated by FateZero remain static. All the competing methods fail to maintain the shape and texture of the car. Our method produces a more coherent and dynamic scene of sea waves. Besides, our results retain the shape and texture of the car.

We compare our results with the Audio-driven methods, Sound-G, in semantic scene editing. As shown in Fig. <ref>, the videos generated by Sound-guided fail to present “Sea wave” scenes, with a noticeable loss of detail in the car, and a lack of temporal dynamics across frames. In contrast, our approach not only preserves the texture of the car but also introduces dynamic and coherent scenes.



























We compare with previous work in temporal dynamic scene editing through magnitude control. We establish a baseline that combines AADiff <cit.> and FateZero <cit.>, controlling the editing effect by employing audio magnitude to control the value of text-video cross-attention maps. We qualitatively compare our results with this baseline in Fig. <ref>. We employ an audio clip of “Cracking fire” to guide the scene editing. It can be observed that while the baseline method is capable of generating visual effects of flames in certain frames, it fails to modulate these visual effects in accordance with the variations in the audio magnitude. The more noticeable visual dynamics aligned with audio magnitude variance shows the priority of our method.




















































































Quantitative Analysis. We first compare our method with the text-driven methods using ten sets of comparisons. The results are shown in Tab. <ref>. The Sem-A score of our model is 0.825, which is significantly higher than the next-best score by VideoP2P at 0.525. This vast gap of nearly 36.3% suggests that our model has a deeper understanding of the conditions, leading to editing results that better reflect the semantically matched scenes. In addition, our model stands out with an SSIM score of 0.9403, lifting the score 8.42%, 4.88%, and 12.12%, compared to Tune-A-Video, FateZero, and VideoP2P respectively. This significant lead demonstrates our model's remarkable ability to keep the texture and shape of foreground content during editing. It shows that our model can edit scenes without losing the video's foreground appeal. The Temp-S score of our model promotes 6.82%, 3.15%, and 3.17%, over Tune-A-Video, FateZero, and VideoP2P respectively. This superiority illustrates our model's capacity to ensure temporally smooth transitions and consistency over time. This is pivotal for maintaining realness and preventing visual flicking in edited videos. 






User Study. To provide a complete measure of the quality of our method and text-driven methods, we conduct a set of human evaluation experiments, based on 10 pair-wise comparisons. Concretely, 10 academics are asked to respectively rank (1 is best and 4 is worst) different results based on two aspects (1) Realness  Please select the more realistic results, and (2) Coherence  Which results are more temporally consistent. The results are shown in Fig. <ref>. the average rankings of our method are 1.3 and 1.4 in content realness and temporal coherence, respectively, receiving the highest preference ratings from users and showing the superiority of . 







 §.§ Ablation Study


To verify the necessity of each component in our method, we qualitatively conduct ablation studies on the i) temporal-aware audio semantic injection (TASI), ii) SceneMasker module, and iii) Frequency Fuser module. The semantic label of input audio is “Waterfall”. The qualitative result is in Fig.  <ref> and the corresponding quantitative result is in Tab. <ref>. 

The Effect of TASI. To test the function of TASI, we eschew its use and instead employ audio-video cross-attention layers to integrate audio semantics for video editing guidance. The quantitative results show a drop in 75% in Sem-A score. The visual changes in Fig. <ref> also showcase the semantical mismatch. The model fails to present waterfall scenes. These results show that our TASI is effective and significant in introducing audio semantics into the model.

The Effect of SceneMasker. Through qualitative visualization, without the SceneMasker module, the color of the car changes (from red to black) before and after editing. In the quantitative assessment, the 6.82% drop of SSIM score also showcases the change of the car. These results emphasize the crucial role of the SceneMasker module in foreground preservation during editing.  

The Effect of Frequency Fuser. As shown in Fig. <ref>, the waterfall scene shows visual flicking between the first and third frames. In Tab. <ref>, we observe the 4.83% drop of Temp-S score for editing results without Frequency Fuser. These results indicate that deactivating the Frequency Fuser module degrades the temporal consistency of edited scenes and video quality.







 §.§ Applications of 

Audio-Guided Semantic Editing. We show the editing results of our method in Fig. <ref>. Given source videos (first row),  supports diverse and coherent scene editing with the corresponding audio variation (second to fourth rows). We also achieve scene-style transitions using emotional audio clips, like happy music and sad music. We can employ audio clips belonging to the same semantic label to generate different visual scenes, as shown in Fig. <ref>.
Refer to the Appendix for more visualization.

Magnitude-Controlled Temporal Scene Dynamics. Magnitude is one of the crucial properties of audio.  controls the temporal scene dynamics based on the change in audio magnitude. As shown in Fig. <ref>, our method can produce visual changes in the scene aligned with the audio magnitude, improving the temporal visual dynamics. Given the audio clips with “Cracking fire” semantics, the change of scene content becomes noticeable as the value of the magnitude variances along the time axis. 








§ CONCLUSION

In this paper, we introduce , an audio-driven framework designed for video scene editing. 
 integrates audio semantics into the visual scene through a temporal-aware audio semantic injection process. We further introduce a SceneMasker module that maintains the integrity of the foreground content.
 exploits the audio magnitude and frequency to control the temporal dynamics and enhance the temporal consistency. 
We present a new metric named temporal score for more comprehensive validation of temporal consistency. Finally, We demonstrate the effectiveness of our method. 








ACM-Reference-Format











































































































































































































































































































































































































































































































































































