Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Attacks
Copyright Tracking;Large Vision-Language Models;Adversarial Attacks;Fine-tuning
Yes
No ethics review needed.
- Is the statement that "changing the weights in the attention layers has a greater impact on trigger images than altering the weights in the MLP layers" based on empirical observation? Could you provide further reasoning or experimental data to support this claim?
- Have you considered potential vulnerabilities where attackers might detect or circumvent PLA’s adversarial triggers? If so, what measures could be implemented to defend against such countermeasures?
- How does PLA’s approach of reverse parameter updates specifically differentiate itself from existing model watermarking and adversarial trigger methods?
The proposed approach PLA offers several valuable strengths:
- is a non-intrusive way of copyright tracking
- generates robust trigger images that are resistant to model finetuning
- outperforms traditional methods by reliably identifying copyright ownership

The paper is well-written, presenting concepts clearly and systematically.
This paper tackles the problem of unauthorized usage and copyright infringement of LVLMs due to unlicensed fine-tuning of publicly available models. It proposes a method called Parameter Learning Attack (PLA) to address this issue by enabling copyright tracking without modifying the original model. PLA employs an adversarial attack to generate trigger images that produce distinct outputs for copyright verification. To enhance robustness, it involves adjustment of model parameters, ensuring the trigger images remain effective even on fine-tuned versions. This approach is non-intrusive, as it can be applied post-release without impacting the model’s performance. Experiments simulating real-world fine-tuning scenarios, show that PLA outperforms baseline methods in accurately identifying original ownership. It offers a solution for detecting unauthorized LVLM usage.
- PLA’s core innovation involves updating parameters in reverse to increase trigger robustness; however, the paper lacks an in-depth explanation of why this specific approach effectively simulates fine-tuning.
- The experiments focus solely on fine-tuning for VQA tasks, which limits the assessment of PLA’s robustness across broader vision-language tasks. Testing on additional tasks, such as image captioning, visual grounding, or multi-modal classification, would provide a more comprehensive evaluation of PLA's effectiveness in diverse VL applications.
- Scalability and Computational Efficiency: Reverse parameter updates during adversarial training may be computationally intensive, raising questions about PLA’s scalability for large models. Adding discussion or experiments on PLA’s computational demands, along with potential optimizations, would improve its applicability for real-world use.
- Limited novelty:  While the paper proposes PLA for copyright tracking, the use of adversarial triggers to mark models has similarities to prior watermarking and adversarial attack research.
Yes
No ethics review needed.
1. Can you show more details of how IF (Xu et al., 2024) is implemented in this paper? Why is IF less effective than the proposed method that does not embed trigger information into the model during training?

1. How does the method handle model ensemble or knowledge distillation?

1. Is this approach still effective when the model stealer finetunes the model using adaptive methods to evade detection?
Problem significance. This paper addresses an important and timely issue in AI model protection.

Writing is generally clear and easy to read.
This paper proposes a method for tracking the copyright of LVLMs, i.e. detecting whether an LVLM is finetuned from another one. They use adversarial attacks to create trigger images that, when paired with specific questions, elicit predetermined responses from models derived from the original model. The trigger is created in an adversarial training fashion, where the model is fine tuned to resist the attack while the attack is created again on the robustified model. The authors evaluate their method on LLaVA-1.5 across various fine-tuning scenarios and datasets, demonstrating improved detection rates compared to baseline methods.
Methodological limitations.
1. The authors do not provide detailed description on which part of the LVLM is being fine tuned. It seems that they only consider language model fine-tuning and ignore vision encoder and adapter fine-tuning, which does not cover the full picture of fine-tuning. The authors should provide analyses on how finetuning different parts of the LVLM influences the detection rate.
1. In line 353, why do you need to enhance the concealment of trigger images? For copyright detection purposes, there is no inherent need for imperceptibility. Arbitrary patterns or even pure noise can work as triggers, since the goal is detection accuracy, not visual stealth. And the perturbation budget of ε=16/255 seems arbitrary, and the author should provide varying levels of perturbation budgets and reveal the trend of detection rate as the budget increases.

Insufficient experiments and analyses.
1. This paper does not provide analysis on how to determine the number of steps for adversarial attacks. The authors should provide the detection rate vs the number of attack steps.
1. There is no proper treatment of false positives and false negatives. And it is missing ROC curves and threshold analysis. Not clear how this method works in practice.
1. Why would model creators prefer post-release triggers than finger-prints implemented during training? This paper shows in table 1 that the proposed method is more effective than training time triggers. It should elaborate more on this aspect by showing more comparisons.
Yes
No ethics review needed.
See in Weaknesses.
1. The issue addressed in this paper—copyright tracking for open-source VLMs—is highly significant. The approach of constructing adversarial examples to detect outputs from downstream models provides an insightful solution for identifying unlicensed usage of pre-trained VLMs.
2. The paper is well-structured and easy to follow.
The paper focuses on identifying whether a model has been fine-tuned based on a publicly released pre-trained visual-language model.
It proposes the use of adversarial data, generated through targeted attacks on the original pre-trained model, to detect outputs from downstream models, thereby assessing potential copyright issues.
To enhance generalization, the paper introduces a Parameter Learning Attack (PLA), which incorporates an adversarial training process to simulate parameter changes in downstream tasks.
Experiments on six downstream VQA datasets show that adversarial data generated by PLA can effectively track the copyright of pre-trained VLMs, achieving satisfactory accuracy.
1. Although the paper presents a novel scenario, the techniques employed to address the problem are not particularly innovative. The core issue can be understood as constructing adversarial examples with cross-model transferability based on a given pre-trained model through targeted adversarial attacks. This is a widely studied topic in the field of adversarial attacks. While the application of targeted attacks for copyright tracking is indeed creative, the paper overlooks many comparative methods, such as [1]-[5]. I believe that most adversarial attacks focusing on cross-model transferability could be tailored for the task presented in this paper.

[1] Dong et al. Boosting Adversarial Attacks with Momentum.
[2] Xie et al. Improving Transferability of Adversarial Examples with Input Diversity.
[3] Lu et al. Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models.
[4] Luo et al. An Image Is Worth 1000 Lies: Transferability of Adversarial Images across Prompts on Vision-Language Models.
[5] Yin et al. VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models.

2. The paper should evaluate the performance of PLA under varying degrees of model parameter changes. The experiments conducted are limited to relatively simple VQA datasets, where the parameter changes are minimal. If fine-tuning occurs on more complex datasets (such as ScienceQA), the extent of parameter changes may be more substantial. How would PLA perform under such conditions?
Yes
No ethics review needed.
1) The experimental results include finetuning on Llava1.5, ST-VQA, etc. Have the authors tried some other advanced models such as miniGPT4, QWEN2-VL (2b or 7b version), and InternVL (2b version )? What are the results?

2) The paper finetunes the model on certain datasets using both full finetuning and lora. Can the authors provide the performances on relevant benchmarks of the VLMs before and after fine-tuning? This is to check if the fine-tuning is conducted correctly.

3) can the authors also provide the TMR for related and unrelated models? For example, if the image-text pairs are designed for llava1.5, what's the TMR for llava1.6 and for miniGPT4?

I'm willing to raise the score once the questions are addressed.
1) The topic is both significant and practical.

2) The writing is clear and easy to follow.

3) The method is novel and the experiments show a clear increase compared to previous ones.
The paper propose a method PLA to track copyrights and detect unlicensed usage of LVLMs. The method is to design rare question-answer pairs, and optimize corresponding adversarial images. These adv image-text pairs are used as triggers to detect if the model are copyright infringing. To increase generalization, the paper adds gradient-based updates to model parameters when optimizing adv images.
Please refer to the questions. 

minor: line 94: traqcking
By allowing model parameter updates during adversarial attacks, the adversarial images can more effectively trigger fine-tuned models to output specific texts, thereby achieving the copyright tracking.
@inproceedings{
anonymous2024tracking,
title={Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Attacks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=K7xpl3LZQp},
note={under review}
}
Large vision-language models (LVLMs) have demonstrated remarkable image understanding and dialogue capabilities, allowing them to handle a variety of visual question answering tasks. However, their widespread availability raises concerns about unauthorized usage and copyright infringement, where users or individuals can develop their own LVLMs by fine-tuning published models. In this paper, we propose a novel method called Parameter Learning Attack (PLA) for tracking the copyright of LVLMs without modifying the original model. Specifically, we construct adversarial images through targeted attacks against the original model, enabling it to generate specific outputs. To ensure these attacks remain effective on potential fine-tuned models to trigger copyright tracking, we allow the original model to learn the trigger images by updating parameters in the opposite direction during the adversarial attack process. Notably, the proposed method can be applied after the release of the original model, thus not affecting the model’s performance and behavior. To simulate real-world applications, we fine-tune the original model using various strategies across diverse datasets, creating a range of models for copyright verification. Extensive experiments demonstrate that our method can more effectively identify the original copyright of fine-tuned models compared to baseline methods. Therefore, this work provides a powerful tool for tracking copyrights and detecting unlicensed usage of LVLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Copyright Tracking Large Vision-Language Models Adversarial Attacks Fine-tuning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/addee7ba1da231f959f62cb7769106ab134c0b00.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Attacks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission