V2A-Mark: Versatile Deep Visual-Audio Watermarking for Manipulation Localization and Copyright Protection
Xuanyu Zhang Youmin Xu Runyi Li Jiwen Yu Weiqi Li Zhipei Xu Jian Zhang













































  AI-generated video has revolutionized short video production, filmmaking, and personalized media, making video local editing an essential tool. However, this progress also blurs the line between reality and fiction, posing challenges in multimedia forensics. To solve this urgent issue, V^2A-Mark is proposed to address the limitations of current video tampering forensics, such as poor generalizability, singular function, and single modality focus. Combining the fragility of video-into-video steganography with deep robust watermarking, our method can embed invisible visual-audio localization watermarks and copyright watermarks into the original video frames and audio, enabling precise manipulation localization and copyright protection. We also design a temporal alignment and fusion module and degradation prompt learning to enhance the localization accuracy and decoding robustness. Meanwhile, we introduce a sample-level audio localization method and a cross-modal copyright extraction mechanism to couple the information of audio and video frames. The effectiveness of V^2A-Mark has been verified on a visual-audio tampering dataset, emphasizing its superiority in localization precision and copyright accuracy, crucial for the sustainable development of video editing in the AIGC video era.







<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224</concept_id>
       <concept_desc>Computing methodologies Computer vision</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Computer vision







  
    < g r a p h i c s >

  Two application instances of V^2A-Mark. (a): Original video, (b): Watermarked video, (c) Tampered video, (d) Tampered visual areas and audio period. We propose a versatile deep visual-audio proactive forensics framework, dubbed V^2A-Mark. Our method can embed an invisible cross-modal watermark into the original video frames and audio (a), producing watermarked video frames and audio (b). If they are tampered by object removal, copy-and-paste, or any editing methods during network transmission (c), we can accurately get the predicted visual tampered areas, audio tampered periods, and the copyright (d).
  

  




printacmref=false 



V^2A-Mark: Versatile Deep Visual-Audio Watermarking for Manipulation Localization and Copyright Protection
    
Xuanyu Zhang1,2, Youmin Xu1, Runyi Li1, Jiwen Yu1, Weiqi Li1, Zhipei Xu1, Jian Zhang1,2 

1 School of Electronic and Computer Engineering, Peking University

2 Peking University Shenzhen Graduate School-Rabbitpre AIGC Joint Research Laboratory

    April 29, 2024
=========================================================================================================================================================================================================================================================





§ INTRODUCTION

2024 is regarded as a boom year of AI-generated video. Benefited from diffusion models and the influx of extensive video data, a large amount of video generation models and editing methods <cit.> have emerged, offering convenience in the production of short videos, film-making, advertising, and customized media. Specifically, local editing <cit.> has become a vital feature of AI video generation tools. For instance, AI dubbing software, capable of altering the facial expressions, lip movements, and voices of characters in a video, is extensively used in simultaneous interpretation and movie dubbing. However, this powerful editing capability is a double-edged sword. It not only facilitates video editors and creators but also blurs the boundaries between reality and forgery, posing new challenges for tamper forensics. Therefore, it is urgent to develop a method for visual-audio tamper localization and copyright protection, which can be widely used in court evidence, rumor verification, and beyond. 

Most visual-audio manipulation localization methods <cit.> are passive, which mainly rely on excavating the temporal and spatial anomalous traces from the suspect videos themselves to predict tampered regions. However, these methods often prove ineffective against AIGC-based video tampering, which exhibits fewer artifacts and more realistic texture details. Additionally, most passive black-box localization networks typically require the introduction of specific types of manipulation during training, rendering them ineffective against previously unseen editing methods. Therefore, these methods have obvious shortcomings in generalization ability and accuracy of manipulation localization. 

Given the inherent drawbacks of passive detection and localization, visual-audio watermarking has become a consensus technology for proactive forensics. However, existing video watermarking methods are fraught with some issues. 1) Poor Accuracy: Although traditional fragile watermarking methods <cit.> can achieve block-wise manipulation location via hash verification, their accuracy is unsatisfactory and difficult to reach the pixel-wise localization. 2) Singular Function: Video manipulation localization and copyright protection tend to be treated as two distinct and separate tasks. Tampering forensic methods lack the capability for copyright protection, limiting the applicative value of their prediction results. Simultaneously, robust deep video watermarking methods <cit.> can only provide copyright protection and are unable to precisely pinpoint the locations of tampering within videos. 3) Single Modality: Most current forensic methods often only focus on a single visual <cit.> or audio modality <cit.> and have not established effective cross-modal interaction mechanisms. How to effectively utilize cross-modal information for manipulation localization and cross-verification of copyrights is an urgent issue.

To address the above-mentioned issues, we propose an innovative multi-functional and multi-modal watermarking method, dubbed V^2A-Mark. In the visual section, integrating the fragility of video-into-video steganography and the robustness of bit-into-video watermarking, we simultaneously embed both localization and copyright watermarks into the video frames, enabling the decoding network to independently extract tampered areas and copyright information. In the audio section, we insert a versatile watermark into the host audio and use it to assist in the reconstruction of visual copyright information, while identifying the tampered periods in the audio. Thus, our contributions are as follows.



113 (1) We design an innovative deep versatile, cross-modal video watermarking framework, dubbed V^2A-Mark, for visual-audio manipulation localization and copyright protection. It can embed invisible localization and copyright watermarks into video frames and audio samples simultaneously, and then obtain visual tampered area, audio tampered period, and exact copyright information in the decoding end.



113 (2) In the visual section, we develop a temporal alignment and fusion module (TAFM) and a degradation prompt learning (DPL) mechanism, enabling the network to fully leverage temporal information for high-fidelity concealment and robust prediction of localization and copyright results.



113 (3) In the audio section, we embed sample-level versatile watermarks into the pristine audio to identify the tampered samples and extract the copyright information. Furthermore, a cross-modal extraction mechanism is proposed to obtain the final copyright from the information of audio and video frames.



113 (4) The effectiveness of our method has been verified on our constructed visual-audio tampering dataset. Compared to other approaches, our method has notable merits in localization accuracy, generalization abilities, and copyright precision without any labeled data or additional training required for specific tampering types.



§ RELATED WORKS





 §.§ Manipulation Localization

Prevalent image forensic techniques have focused on localizing specific types of manipulations via exploring artifacts and anomalies in tampered images <cit.>. Recently, HiFi-Net <cit.> used multi-branch feature extractor and localization modules for AIGC-synthesized and edited images. SAFL-Net <cit.> designed a feature extraction approach to learn semantic-agnostic features with specific modules and auxiliary tasks. IML-ViT <cit.> firstly introduced vision transformer for image manipulation localization and modified ViT components to address three unique challenges in high resolution, multi-scale, and edge supervision. MaLP <cit.> introduced a large number of forgery images to learn the matched template and localization network. Targeted at video tamper localization,  <cit.> exploited the spatial and temporal traces left by inpainting and guided the extraction of inter-frame residual with optical-flow-based frame alignment. UVL <cit.> designed a novel hybrid multi-stage architecture that combines CNNs and ViTs to effectively capture both local and global video features. However, the above-mentioned passive localization methods are often limited in terms of generalization and localization accuracy, which usually work on known tampering types that have been trained. 



 §.§ Video Watermarking and Steganography

Video watermarking is a widely accepted forensic method, which can be broadly utilized for the verification, authenticity, and traceability of images. In terms of robustness levels for extraction, video watermarking can be divided into fragile and robust watermarking <cit.>. Although classical fragile watermarking <cit.> can achieve block-wise tamper localization, their localization accuracy and flexibility are unsatisfactory. Therefore, how to realize joint pixel-level tamper localization and copyright protection has still a lot of room for research. 

Thanks to the development of deep learning, learning-based video watermarking has attracted increased attention. For deep robust video watermarking, an intuitive approach is to apply image watermarking methods <cit.> frame by frame. For instance, HiDDeN <cit.> firstly designed a deep encoder-decoder network to hide and recover bitstream. Moreover, many differentiable distortion layers such as JPEG compression, screen-shooting, and face swaping <cit.> were incorporated to enhance the robustness of the encoder-decoder watermarking framework. Meanwhile, CIN <cit.> and FIN <cit.> utilized flow-based models to further improve the fidelity of container images. However, these deep watermarking methods have a singular function and cannot accurately localize the tampered areas. Moreover, there are other explorations to address video degradation and temporal correlations. For instance, DVMark <cit.> used an end-to-end trainable multi-scale network for robust watermark embedding and extraction across various spatial-temporal clues. REVMark <cit.> focused on improving the robustness against H.264/AVC compression via the temporal alignment module and DiffH264 distortion layer. LF-VSN <cit.> utilized invertible blocks and the redundancy prediction module to realize large-capacity and flexible video steganography.



§ METHODS




 §.§ Overall Framework of V^2A-Mark

To achieve multimodal, versatile, and proactive manipulation localization and copyright protection, as shown in Fig. <ref>, the proposed V^2A-Mark consists of two key sections, namely the visual hiding and decoding (Sec. <ref>), and the audio hiding and decoding (Sec. <ref>). In the visual hiding section, we sequentially embed pre-defined visual localization watermarks 𝐖_loc∈ℝ^H × W × T × C and the copyright watermark 𝐰_cop∈{0, 1}^k into the original video sequences 𝐕_ori∈ℝ^H × W × T × C to get the container video 𝐕_con∈ℝ^H × W × T × C. In the audio hiding section, we add versatile watermark 𝐰^'_cop∈{0, 1}^n to the original audio 𝐀_ori∈ℝ^L in a sample-level manner to obtain the 𝐀_con∈ℝ^L. Note that T and L denote the number of video frames and length of the audio, respectively. “Versatile” means that this audio watermarking and decoding module can achieve audio manipulation localization and copyright protection at the same time. Moreover, the potential impacts on container videos during network transmission can be divided into two types, namely malicious tampering and common degradation. Thus, the network transmission pipeline of video frames and audio is modeled as follows.



    𝐕_rec   =𝒟_v(𝐕_con⊙ (1 - 𝐌) + 𝒯_v(𝐕_con) ⊙𝐌), 
    𝐀_rec   =𝒟_a(𝐀_con⊙ (1 - 𝐦) + 𝒯_a(𝐀_con) ⊙𝐦),

where 𝒯_v(·) and 𝒯_a(·) respectively denote the video and audio manipulation function. 𝒟_v(·) and 𝒟_a(·) respectively denote the video and audio degradation operation. 𝐌∈ℝ^H × W × T and 𝐦∈ℝ^L respectively denote the tempered visual masks and audio periods. 

Upon receiving the video 𝐕_rec and audio 𝐀_rec, we attempt to recover the previously embedded watermarks on different robustness levels and conduct corresponding forensics based on the extracted watermarks. In the visual decoding section, our framework precisely extracts the tampered video masks 𝐌̂_vis and copyright information 𝐰^v_cop. Concurrently, as shown in Fig. <ref>, the tampered periods 𝐦̂_aud and the copyright 𝐰^a_cop in the audio will be extracted from the audio versatile decoding module. The final restored copyright information of the video 𝐰̂_cop will be obtained via cross-modal combination of 𝐰^a_cop and 𝐰^v_cop (Sec. <ref>). Finally, the visual-audio tamper forensics process of V^2A-Mark can be categorized into several scenarios, where  and  respectively denote the “element-wise and” and “element-wise or”.





113 (1) 𝐰̂_cop ≉ 𝐰_cop: Suspicious 𝐕_rec was not processed via our V^2A-Mark, and we are also unable to ascertain the authenticity of the corresponding audio 𝐀_rec. They cannot be used as evidence.



113 (2) 𝐰̂_cop ≈ 𝐰_cop  (𝐌̂_vis ≉ 0  𝐦̂_aud ≉ 0): Suspicious 𝐕_rec or 𝐀_rec has undergone tampering, disqualifying it as valid evidence. 



113 (3) 𝐰̂_cop ≈ 𝐰_cop  𝐌̂_vis ≈ 0  𝐦̂_aud ≈ 0: Suspicious 𝐕_rec and 𝐀_rec are both credible and have not been tampered with. V^2A-Mark ensures the authenticity and integrity of this video.



 §.§ Preliminaries and Motivations
 Previous work EditGuard <cit.> has already validated the feasibility of using the fragility and locality of image-into-image steganography for proactive image tamper localization. Specifically, fragility means the damage to the container image results in corresponding damage to the revealed secret image. Locality indicates that damage to the container image and the revealed secret image is essentially pixel-level and directly correlated. These two properties can also be effectively applied in proactive video localization. Meanwhile, EditGuard <cit.> adopts a “sequential embedding and parallel decoding” structure to realize united tamper localization and copyright protection. Clearly, one direct approach is to watermark each video frame via EditGuard. However, this method overlooks the exploitation of temporal correlation, making it challenging to ensure the robustness of the reconstructed watermarks and the temporal consistency of the watermarked videos. Therefore, the key issues addressed in this paper are: 1) How to utilize the auxiliary information from supporting frames for watermark embedding and decoding in reference frames; 2) How to improve the robustness of existing frameworks to video degradation; 3) How to employ the watermarks embedded in video frames for audio tamper localization and copyright protection. To address the above issues, we design the visual hiding module (VHM), visual revealing module (VRM), bit hiding module (BHM), and bit recovery module (BRM). Meanwhile, we design an efficient cross-modal extraction mechanism and introduce the advanced audio versatile watermarking and decoding method <cit.> to achieve cross-modal tamper localization and copyright protection.



 §.§ Visual Hiding and Decoding







  §.§.§ Input and Output Design of Visual Section

To achieve memory-efficient hiding and decoding, our V^2A-Mark employs a multi-frame input, single-frame output structure. As shown in Fig. <ref>, the visual hiding is operated group-by-group via a sliding window, traversing each video frame from head to tail. We set the length of a sliding window to 3. Given the original video group {𝐈_ori^(i)}^k+1_k-1 and localization watermark group {𝐈_loc^(i)}^k+1_k-1, we firstly use the TAFM to preprocess {𝐈_ori^(i)}^k+1_k-1 and adopt N invertible blocks to generate 𝐈^(k)_med and its by-product 𝐙_v. The copyright watermark 𝐰_cop is then embedded into 𝐈^(k)_med via a U-Net <cit.>, producing the final container frame 𝐈^(k)_con. For all video frames, we embed the same copyright watermark. After network transmission, V^2A-Mark decodes each received video frame 𝐈^(k)_rec individually. On one hand, 𝐰̂_cop is extracted from 𝐈^(k)_rec via a U-Net and an MLP extractor. On the other hand, we replicate 𝐈^(k)_rec threefold and feed it into the residual prediction module (RPM) <cit.> to produce the missing component 𝐙̂_v. Then, N invertible blocks and the TAFM are used to reconstruct the video groups and only select the intermediate frames as the result, namely 𝐈̂^(k)_ori and 𝐈̂^(k)_loc. Note that we introduce learned degradation prompts 𝐏_v, 𝐏_b in video revealing and bit recovery modules and fuse them with intrinsic features to further enhance the robustness of our method against common video and audio degradations.





  §.§.§ Temporal Alignment and Fusion Module
 To further enhance the temporal consistency of the container videos, we design a temporal alignment and fusion module (TAFM) to align the supporting frames {𝐈^(i)_ori}_i≠ k to the reference frame 𝐈^(k)_ori. As shown in Fig. <ref>, we resort to bidirectional cross-attention mechanisms between the supporting frames and the reference frame. Specifically, we define the scaled dot production operation as follows.



    Attention(𝐐, 𝐊, 𝐕)=softmax(𝐐 𝐊^T / √(D)) 𝐕,
where 𝐐∈ℝ^H × W × D is the query matrix projected by the reference frame 𝐈^(k)_ori, and 𝐊, 𝐕∈ℝ^H × W × D are the key and value matrix produced from the supporting frames {𝐈^(i)_ori}_i≠ k. Given the reference feature 𝐅_r and the supporting feature 𝐅_s, they are firstly layer normalized into 𝐅_r=Norm(𝐅_r) and 𝐅_s=Norm(𝐅_s). Then, we use linear layers to project 𝐅_r, 𝐅_s into D-dimension embedding space and calculate the cross-attention maps between reference and supporting frames as follows.

    𝐅_r → s=Attention(𝐖_1^r 𝐅_r, 𝐖_1^s 𝐅_s, 𝐖_2^s 𝐅_s),


    𝐅_s → r=Attention(𝐖_1^s 𝐅_s, 𝐖_1^r 𝐅_r, 𝐖_2^r 𝐅_r),

where 𝐖_1^r, 𝐖_2^r, 𝐖_1^s and 𝐖_2^s respectively denote the projection matrices. Finally, we perform temporal fusion between the reference frame and supporting frames via the residual connection and concatenation operation.

    𝐅̂^(k) = Concat(γ_1𝐅_s → r+𝐅_r, γ_2𝐅_r → s+𝐅_s),

where γ_1 and γ_2 respectively denote the learnable parameters. With our TAFM, V^2A-Mark can better exploit temporal correlations, thus achieving more effective concealment and more robust decoding.





  §.§.§ Degradation Prompt Learning

To further improve the robustness of V^2A-Mark in decoding both visual localization and copyright watermarks, we embed learnable degradation prompts 𝐏_v∈ℝ^h_1 × w_1 × c_1 × e_1, 𝐏_b∈ℝ^h_2 × w_2 × c_2 × e_2 into features of the bit recovery and video revealing modules, where c_1, c_2 respectively denote the channels of prompt, e_1, e_2 respectively denote the number of degradation prompt. The degradation prompt pool comprises a series of learnable embeddings, with each corresponding to a type of potential degradation. Supposing that 𝐅_v and 𝐅_b are the outputs of the RPM in the video revealing module and the U-Net in the bit recovery module in Fig. <ref> respectively, we utilize a channel attention mechanism (as shown in Fig. <ref>) to better encourage the interaction between the input features 𝐅_v/𝐅_b and the degradation prompt 𝐏_v/𝐏_b. Specifically, the features 𝐅_v/𝐅_b are passed to a global average pooling (GAP) layer, a 1×1 convolution, and a softmax operator to produce a set of dynamic weight coefficients 𝐰_v/𝐰_b, which is inspired by  <cit.>. These coefficients are used to fuse each degradation prompt, resulting in degradation-enhanced features. Then, we utilize convolution and concatenation operations to fuse the degradation-enhanced features with the features extracted from RPM or the U-Net in BRM. Note that we learned two distinct sets of degradation prompts for visual and bit decoding, since we aim for the BRM to be absolutely robust against degradation, while the VRM should retain some fragility against tampering.





 §.§ Audio Hiding and Decoding


Considering that video tampering is often accompanied by corresponding changes in the audio, we try to simultaneously identify the tampered areas of the audio, and utilize the extracted audio copyright to cross-verify the copyright in the video frame. To ensure the correspondence between video and audio, we set the audio versatile copyright watermark 𝐰^'_cop as part of the copyright 𝐰_cop in the video frames. For instance, 𝐰_cop is a 32-bit watermark, and 𝐰^'_cop is the first 16 bits of 𝐰_cop. Inspired by the advanced proactive tamper localization tool Audioseal <cit.>, we introduce an audio watermark generator and detector to achieve audio versatile watermarking and decoding shown in Fig. <ref>. Specifically, we utilize the watermark generator to predict an additive watermark waveform from the audio input 𝐀_ori, and use a detector to output the probability 𝐦̂_aud of the presence of a watermark at each sample of the container audio 𝐀_con. The detector is trained with mask augmentation strategy to ensure its accuracy and robustness. Meanwhile, we add a message embedding layer <cit.> in the middle of the watermark generator to embed 𝐰^'_cop into 𝐀_ori. In the decoding end, the detector will robustly decrypt 𝐰^a_cop, which will be used to combine with 𝐰^v_cop to get the final copyright 𝐰̂_cop.




 §.§ Training and Inference Details


Training: The training process of the visual section of the proposed V^2A-Mark can be divided into two steps. Given an arbitrary original image 𝐈^(k)_med and watermark 𝐰_cop, we first train the bit hiding and recovery module via the ℓ_2 loss.



    ℓ_cop=𝐈^(k)_con-𝐈^(k)_med_2^2+λ𝐰̂_cop-𝐰_cop_2^2,
where λ is set to 10. Furthermore, we freeze the weights of BHM and BRM and jointly train the VHM and VRM. Given a video group {𝐈_ori^(i)}^k+1_k-1, localization watermark group {𝐈_loc^(i)}^k+1_k-1 and copyright watermark 𝐰_cop, the loss is:



    ℓ_loc = 𝐈̂^(k)_ori-𝐈^(k)_ori_1 + α𝐈^(k)_con-𝐈^(k)_ori_2^2+β𝐈̂^(k)_loc-𝐈^(k)_loc_1,
where α and β are respectively set to 100 and 1. In the audio section, we use a pre-trained audio watermarking tool <cit.> to realize audio hiding and decoding.



Inference: As shown in Fig. <ref> and  <ref>, we can conduct forensics via the pre-trained components. To extract tampered masks, we compare the pre-defined watermark 𝐖_loc with the decoded one 𝐖̂_loc to obtain a binary mask 𝐌̂_vis∈ℝ^H × W × T:



    𝐌̂_vis[i, j, t]=θ_τ(max (|𝐖̂_loc[i, j, t, :]-𝐖_loc[i, j, t, :]|)),
where i ∈ [0, H), j ∈ [0, W) and t ∈ [0, T). θ_τ(z) = 1 (z ≥τ). τ is set to 0.2. |·| is an absolute value operation. The audio tampered period 𝐦̂_aud is directly extracted via the audio versatile decoder. To extract precise visual copyright, we conduct bitwise voting on the copyright extracted from each frame and select the most frequently occurring 0 or 1 as the final result 𝐰^v_cop. Meanwhile, we extract audio copyright 𝐰^a_cop∈{0, 1}^n and use it to cross-verify with 𝐰^v_cop∈{0, 1}^k, getting the final result 𝐰̂_cop∈{0, 1}^k. Considering that the audio copyright watermark can often be extracted more robustly and is not easily destroyed, we directly use it as the first n bits in the final multimedia copyright 𝐰̂_cop, which typically represents the ownership of the entire multimedia asset. The remaining k-n bits are taken from the extracted visual watermark 𝐰^v_cop, which will be related to the information of video frames such as resolution, time length, and frame rate. The cross-modal extraction process is:



    ŵ_cop=Concat( 𝐰_cop^a,𝐰_cop^v[ n:]).





§ EXPERIMENTS



 §.§ Implementation Details

We trained our V^2A-Mark in the Vimeo-90K <cit.> without any tampered data. We test our method on 30 testing videos of Davis dataset <cit.>. All video frames have a resolution of 448×256 and consist of 50 to 100 frames. To synthesize audio, we manually extract the video captions and use them as prompts with the VALLE-E-X audio synthesis tool <cit.>. The Adam <cit.> is used for training 250K iterations with β_1=0.9 and β_2=0.5. The learning rate is initialized to 1×10^-4 and decreases by half for every 30K iterations, with the batch size set to 8. N in Video hiding and revealing module is set to 16. The shape of two degradation prompts 𝐏_v and 𝐏_b are 36×36×72×2 and 36×36×16×6. We embed 32-bit copyright watermarks 𝐰_cop and pure blue visual localization watermarks 𝐖_loc to original videos. We use replication padding to process the first and last frame of the original video. Meanwhile, we also embed a versatile watermark 𝐰^'_cop into the original audio.



 §.§ Comparison with Visual Tamper Localization Methods

To evaluate the visual localization and copyright recovery accuracy, we compared our method with some SOTA passive methods OSN <cit.>, PSCC-Net <cit.>, HiFi-Net <cit.>, IML-ViT <cit.> and a proactive forensics method EditGuard <cit.>. Despite previous research on video tamper localization <cit.>, we can not find methods with publicly available code. Therefore, our comparative methods primarily rely on image-based tamper localization methods on a frame-by-frame prediction. For visual tamper localization, F1-score, AUC, and IoU are used to evaluate localization accuracy. For copyright protection, bit accuracy (BA) is used to assess the copyright recovery performance. We use two SOTA deep video inpainting methods, ProPainter <cit.> and E^2FGVI <cit.>, and a naive slicing approach to simulate malicious tampering.



As reported in Tab. <ref>, our V^2A-Mark achieves impressive localization performance with an F1-Score of approximately 0.95, an AUC of 0.99, and an IoU close to 0.9. In contrast, other passive localization methods, which rely solely on tampered video clues, perform poorly in localizing unseen types of manipulation. Furthermore, when using EditGuard to watermark each video frame, although it achieves satisfactory localization results, it falls short in effectively utilizing temporal information. Consequently, the IoU of the predicted masks in various tampering methods is generally about 0.03 lower than that achieved by our V^2A-Mark. Additionally, our V^2A-Mark achieves an over 99.5% bit accuracy across various tampering methods, which is also marginally higher than that of EditGuard. Furthermore, as shown in Fig. <ref>, our method has very obvious advantages over SOTA passive localization method PSCC-Net <cit.>, which can be attributed to our proactive tamper localization mechanism. Meanwhile, since we adopted a more effective temporal alignment and fusion method, we found that in some scenes where EditGuard can only locate the rough outline of the tampering area, our V^2A-Mark can still clearly predict the tampered traces.



 §.§ Comparison with Watermarking Methods



To evaluate the visual quality of 𝐕_con, we compared our V^2A-Mark with other watermarking methods on 30 
testing videos from DAVIS <cit.>. For a fair comparison, we also retrained our EditGuard on 448×256 original videos and 32 bits. Our comparison methods include the SOTA bit-hiding watermarking method <cit.>, large-capacity steganography method <cit.>, and a versatile image watermarking method <cit.>. As shown in Tab. <ref>, the PSNR and SSIM of our container videos far outperform most watermarking methods such as MBRS, PIMoG, and SepMark, but is close or slightly inferior to CIN. Note that these methods only hide 30 bits in the videos, but our V^2A-Mark hides both an RGB image and 32 bits. Compared with high-capacity steganography methods EditGuard, LF-VSN, and HiNet, our method also has clear advantages in visual quality. Meanwhile, the perceptual quality (NIQE) of our watermarked videos surpasses all other methods. To verify the security of our method, we perform anti-steganography detection via StegExpose <cit.> on container videos of various steganography methods. All the methods concealed pure blue videos into the original videos. Note that the detection set is built by mixing container and original video frames with equal proportions. We vary the detection thresholds in a wide range in StegExpose <cit.> and draw the ROC curve in Fig. <ref>. The ideal case represents that the detector has a 50% probability of detecting container videos from an equally mixed detection test, the same as a random guess. Evidently, the security of our method exhibits a significant advantage compared to all competitive methods. 




 §.§ Audio Tamper Localization

To evaluate the accuracy of V^2A-Mark for audio tamper localization, we randomly insert 1s - 2s tampered audio into our constructed 30 original audio. SNR and PESQ are used to evaluate the quantitative and perceptual quality of watermarked audio. Bit accuracy is used to evaluate the bit error rate of the pre-defined 𝐰^'_cop and extracted 𝐰^a_cop. AUC is used to calculate the localization accuracy between the predicted audio tampered period 𝐦_aud and the ground truth of the tampered area 𝐦. We observed from Tab. <ref> that the watermarked audio maintains high SNR/PESQ on 28.29 dB/4.50 with the original audio, indicating that our V^2A-Mark has little impact on the semantic fidelity of the audio. Meanwhile, our method can accurately localize the tampered areas with 99.63% AUC and obtain 100% bit accuracy under “Clean” degradation, which shows that our audio localization watermark is sensitive enough to malicious tampering. Furthermore, we adopt two classical degradations on the container audio 𝐀_con, namely Resample and Lowpass. “Resample” denotes resampling the container audio at a 90% sampling rate (16000Hz→14400Hz). “Lowpass” means applying low-pass filter to container audio 𝐀_con, cutting frequencies above a cutoff frequency (1000Hz). As plotted in Tab. <ref>, although the container audio 𝐀_con has undergone different degradations, our V^2A-Mark still maintains over 98% localization accuracy and nearly 100% bit accuracy, proving its robustness against common audio corruptions. 





 §.§ Robustness Analysis

To analyze the robustness of our V^2A-Mark, we compare our method with EditGuard, the best comparative method in the clean case. We selected three types of video degradation, including Gaussian noise, H.264 video coding, and Poisson noise. As reported in Tab. <ref>, we found that our V^2A-Mark has only slight performance degradation under various degradations compared to the clean scene, and both surpass EditGuard in localization accuracy and copyright reconstruction. Specifically, since we use a multi-frame input, single-frame output structure, which better explores temporal information, our method performs better in handling inter-frame degradation (such as H.264 video coding) than EditGuard which adds watermarks frame by frame. As reported in Tab. <ref>, the recovered bit accuracy of our method far surpasses EditGuard by 4.18% and 6.95% in QP=5 and QP=10. Meanwhile, our V^2A-Mark also outperforms EditGuard by 0.028 and 0.022 in localization accuracy (IoU), which proves its superiority in decoding robustness.





 §.§ Ablation Studies

To evaluate the contribution of each component, we mainly conduct ablation studies on the temporal alignment and fusion module (TAFM) and degradation prompt learning (DPL). Our results are reported on Tab. <ref>, where “random degradation” denotes that we randomly select one degradation from Gaussian noise, H.264, and Poisson noise. Comparing case (a) and ours in the “clean” scene, it demonstrates that the proposed TAFM can enhance the localization accuracy and achieve 0.012 gains in IoU, which proves that the proposed TAFM can boost the temporal interaction and realize effective temporal alignment. Comparing case (b) and ours in the “random degradation” scene, due to the learned degradation representations, we find that our method achieves significant gains on localization accuracy and copyright precision. 



 §.§ Applications

Our V^2A-Mark can provide focused protection for videos based on user-defined areas. This allows our V^2A-Mark to apply to some global tampering such as visual-audio deepfake. Specifically, we use EfficientSAM <cit.> to segment the facial regions that need focused protection, and add localization watermarks only to these parts, while still embedding a global copyright watermark. As shown in Fig. <ref>, we manipulate the identity in the container video frames via SimSwap <cit.>, and alter the first 0.5s of this audio from "there are many jobs for American" to "there are few jobs for American." Subsequently, our V^2A-Mark is capable of effectively detecting tampered areas in the face region as well as alterations in the audio. For the audio portion, we determine whether each sample point has been tampered with by evaluating the probability of alteration.







§ CONCLUSION

To address the challenges of AI-generated visual-audio forensics, an innovative deep watermarking method with strong generalizability, versatile function, and cross-modal properties dubbed V^2A-Mark is proposed. It embeds invisible visual-audio localization and copyright watermarks into the original video frames and audio. If encountering malicious tampering on visual or audio information, we can get accurate tampered visual masks, video copyright, and tampered audio periods in the decoding end via our V^2A-Mark. Facing the imminent explosive growth of the AIGC video industry, our V^2A-Mark has the potential to safeguard the sustainable development of the AIGC industry, and also establish a clean and transparent information environment.



Limitations: Since there is a certain contradiction between the fidelity and robustness of video watermarking, we are still committed to designing advanced modules to achieve better tradeoff. Additionally, as there are few video diffusion-based editing methods available, we have not conducted experiments on larger video editing models. However, we believe our method is robust and effective against all forms of local visual-audio manipulation.


ACM-Reference-Format



