Adversarial Attacks on Data Attribution
data attribution;adversarial attack;data-centric AI
Yes
No ethics review needed.
See weaknesses.
1. The paper is well-organized and easy to follow. The authors clearly define the adversary's objective, capabilities, and assumptions, which helps readers understand the problem setting.
2. The proposed attacks are novel. The shadow attack and the outlier attack are based on certain scenarios and are effective in manipulating data attribution methods.
3. The experimental results are comprehensive. The authors evaluate the proposed attacks on multiple datasets and data attribution methods, demonstrating the effectiveness of the attacks in practice.
This paper introduces a novel adversarial attack on data attributions, where the adversary aims to maximize the compensation share received by the adversary. The authors set a few restrictions and assumptions on the adversary's capabilities and the target model. Two different attacks are proposed based on different choices of assumptions. The shadow attack is based on the assumption that the adversary has access to the distribution of the training data and the validation data, while the outlier attack is based on the assumption that the adversarial can query the model to get the data attribution values. The experimental results demonstrate that both the shadow attack and the outlier attack are highly effective in manipulating data attribution methods, leading to substantial increases in the attacker's compensation share.
1. The surrogate objective used to replace the discrete compensation share is not well justified. The authors should provide more insights into why the surrogate objective is a good approximation of the compensation share. Is the surrogate itself already a reasonable to represent the compensation share?
2. The paper emphasizes the validity of the assumptions using the examples of large language models, while the experiments are mostly conducted on relatively smaller size datasets (*e.g.*, MNIST and CIFAR-10). It would be more convincing if the authors could provide more scalability analysis on larger datasets.
Yes
No ethics review needed.
See Weaknesses for my primary concerns, which is in fact somewhat critical to make decisions regarding the impact of the observations.
The paper is overall clear and easy to follow. 

Since the adversarial robustness of data attribution is a reasonable subject, the significance is not considered an issue.
This paper studies adversarial attacks to data attribution based compensation mechanism.

Two empirical attacks are proposed.
1. Shadow attack, which assumes access to data distribution & training algorithm, creates a surrogate data attribution system and adversarially optimize the compensations with the surrogate.
2. Outlier attack, which assumes access to the black-box query of the model and directly optimizes to minimize the confidence of the model to generate outlier data, which could correspond to higher compensation.
The experimental objective function of the adversarial attack to data attribution requires much better justifications.

The paper motivates the study of the robustness of data attribution with data attribution based compensation.
However, it is natural that such compensation designs reward only for data contributing to good/desired behaviors: For example, they might allocate compensations only based on attributions for **correct** predictions (these could be from a prepared validation/test set or from user feedbacks). Alternatively, they might withdraw compensations or even incur penalties for **incorrect/undesired** predictions.

A concern is that in such, arguably more reasonable settings, neither of the proposed attacks will be in fact effective. For shadow attack, only data attributed greatly for correct predictions will be awarded, which is totally fine as the "attack" is in some sense actually making the data more valuable, with efforts. For outlier attacks, the outliers are less likely to be attributed if the prediction is correct and consistent with regular training data.
Yes
No ethics review needed.
Thanks to the authors for the submission to ICLR 2025. I enjoyed reading about the compensation function used for the adversarial objective in the threat model section. Most of my suggestions are already mentioned in the above Weakness section. Some questions (might be open-ended) are listed below. It would be great if more discussion or clarification on the following questions could be added to the paper:
- How is distortion budget measured for both image classification and text generation tasks? What norm is used for images and is the budget consistent with other white-box and black-box attack baselines?
- Analyzing how data attribution methods affect attack performance would strengthen the study of the attacks. For example, if one were to experiment with the Influence Function, Data Shapley, and TRAK methods for a particular {task, dataset, target model} configuration, how do the attacks perform?
- If we increase or decrease the size of Z_0 and Z_1 for each setting, would this make the attack more successful (i.e., how does performance change with respect to the amount of data)?
Thank you for submitting to ICLR 2025. I enjoyed reading about the compensation function used for the adversarial objective in the threat model section. I have listed several points that I find the paper do well below:
- The threat model is well defined for both attack methods. Assumptions made of target model and training/testing data access is clearly stated. The periodic data contribution attack setting between AI Developers and Data Providers is realistic. 
- The adversarial objective is very clear and the compensation function defined in the paper captures it in a novel way. Additionally, the Shadow Attack and Outlier Attack methodology is designed within the limits of the threat model.
- Baseline evaluation helps articulate the performance of the attack methods. Since the compensation share metric is new, the baseline evaluation puts into context how performant the attack is.
This paper introduces two data attribution attack methods, Shadow Attack and Outlier Attack, that aim to inject new training data that maximizes the influence of target model outputs. Both attacks optimize over a compensation share function, which quantifies the influence of a subset of the training data to the target models predictions on test data. Shadow Attack assumes the adversary has access to data distributions and the training algorithm, and uses gradient ascent on training data to generate data points. Outlier Attack assumes black-box query access to the target model, and uses black-box image attack algorithms on training data to generate data points. The attacks are evaluated on image classification and text generation tasks using metrics that measure the change in compensation share before and after adversarial data is introduced.
The problem discussed is very interesting and deserves more attention from researchers and developers from both academia and industry. It is worth noting the effort made by the authors in designing novel attack methodologies. However, I believe there is still room for improvement, especially in making claims (e.g., attack effectiveness and efficiency) more concrete and clear and including more baseline evaluations that shed light on the performance of proposed attack under the given threat model. More details can be found below.
- The paper lacks well-defined research questions in the evaluation. It can be inferred that sections on Shadow Attack and Outlier Attack measure effectiveness, but it would improve the paper if there were claims made that the result would prove/disprove.
- Another baseline metric would be great to understand the performance of the attack under the defined threat model. While the random perturbation baseline helps contextualize the compensation share ratio of the attacks, another baseline assuming a white-box threat model (i.e., the adversary has access to the target model and its parameters) would highlight the performance under the defined threat model with more limitations.
Yes
No ethics review needed.
See above
**Strengths:**

- **Addresses Critical Gap in Research**: A key contribution of this work is that it addresses a crucial research gap. The robustness of data attribution methods has remained largely unexplored. This study is the first to systematically investigate vulnerabilities in data attribution.

- **Significant Attribution Inflation**: The authors introduce two effective attacks that can substantially increase the contribution scores of input samples. The reviewer was impressed by the triple-digit inflation rates achieved, showing the potential impact of these adversarial techniques.
Data attribution quantifies the contribution of individual data points during model training. Higher contribution scores indicate that a data point has more value within the training dataset. This paper proposes two adversarial methods—Shadow Attack and Outlier Attack—designed to manipulate data attribution. The authors evaluate these methods against various models and datasets, demonstrating that they can significantly inflate the contribution scores of otherwise benign data points.
1. **Threat Model**: The authors describe a black-box threat model, which traditionally implies that the attacker has no information about the model other than query access. However, the authors assume access to the model’s architecture, indicating a grey-box rather than a black-box scenario. It would be beneficial for the authors to revise the text to accurately reflect this nuanced threat model.

2. **Alternative to Creating Outliers**: The paper presents two algorithms designed to create natural-looking outliers with a high impact on the model. Given the complexity of these algorithms, a simpler approach might involve manually mislabeling inputs to create outliers (e.g., labeling a "cat" as a "dog" in a classification task), which would likely produce a similar exaggerated impact on the model. The authors should consider addressing why this more straightforward and cost-effective method was not explored.

3. **Simplistic Pipeline**: The authors assume a straightforward threat model where the defender does not employ methods to detect out-of-distribution (OOD) samples or sanitize inputs. Adversarial samples, such as those generated by the Shadow Attack, are known to be vulnerable to detection and degradation [1]. Testing the attacks against state-of-the-art OOD detection and adversarial input detection methods would help establish their robustness.

4. **Simple Defense**: A basic yet effective defense against these attacks would be data augmentation, such as cropping or shifting images to generate additional samples. This method improves model generalization, a common practice in machine learning, and could reduce the impact of adversarial samples like those introduced by the authors. The authors should demonstrate that including data augmentation would not impact their attack, as it more accurately reflects real-world training environments.

[1] https://arxiv.org/abs/1902.02918
We provide a comprehensive study of the adversarial attacks on data attribution methods.
@inproceedings{
anonymous2024adversarial,
title={Adversarial Attacks on Data Attribution},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=oJgIRwkIUB},
note={under review}
}
Data attribution aims to quantify the contribution of individual training data points to the outputs of an AI model, which has been used to measure the value of training data and compensate data providers. Given the impact on financial decisions and compensation mechanisms, a critical question arises concerning the adversarial robustness of data attribution methods. However, there has been little to no systematic research addressing this issue. In this work, we aim to bridge this gap by detailing a threat model with clear assumptions about the adversary's goal and capabilities and proposing principled adversarial attack methods on data attribution. We present two methods, *Shadow Attack* and *Outlier Attack*, which generate manipulated datasets to inflate the compensation adversarially. The Shadow Attack leverages knowledge about the data distribution in the AI applications, and derives adversarial perturbations through "shadow training", a technique commonly used in membership inference attacks. In contrast, the Outlier Attack does not assume any knowledge about the data distribution and relies solely on black-box queries to the target model's predictions. It exploits an inductive bias present in many data attribution methods - outlier data points are more likely to be influential - and employs adversarial examples to generate manipulated datasets. Empirically, in image classification and text generation tasks, the Shadow Attack can inflate the data-attribution-based compensation by at least 200%, while the Outlier Attack achieves compensation inflation ranging from 185% to as much as 643%.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
data attribution adversarial attack data-centric AI
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/bed570fb5cae0e25127145bdce58fa6245982d15.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/a9aa22cf4e6b81fd58ef37f707aa15d1fb1f9b45.zip
Adversarial Attacks on Data Attribution
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission