The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text
Privacy;language models;synthetic data
Yes
No ethics review needed.
- It would be helpful to see the MIA scores when transitioning from the last words of the prefix to the first words of the suffix in generated canaries to demonstrate their effectiveness.
- It would benefit the paper if the authors could highlight their contributions more effectively based on the results.
- How costly is it to compute the n-gram model when dealing with a large volume of synthetic data generated by the target model? A cost analysis for different n-grams and data sizes would be informative.
- Given a large corpus of synthetic data, would it be feasible to train a transformer-based attack model to predict the next word probabilities as the MIA scores?
- The problem is intriguing, particularly as designing MIAs becomes challenging without access to the target model.
- The paper is well-written.
The paper investigates the use of MIAs to audit privacy in synthetically generated text. The authors have designed new canaries that begin with an in-distribution prefix (to maximize the chance of regeneration by the fine-tuned model) and an out-of-distribution suffix to maximize memorization.
- In Table 1, the authors use only 30 words for in-distribution canaries, compared to 50 words for synthetic canaries (as mentioned in a footnote). A larger canary size generally results in higher AUC-ROC numbers. Even for SST-2, the in-distribution canaries perform better than synthetic canaries, which diminishes the contribution of the authors regarding the efficiency of their canaries. This pattern is also visible in Tables 6 and 7.
- The AUC-ROC scores for model attacks in Table 1 suggest that the fine-tuned model is overfitted (likely due to a high n_rep=12). Knowing this, results of MIA scores for synthetic data are little bit low. The scores using SIM_jac and SIM_emb are nearly random, with only the 2-gram model scores slightly better. It also raises the question of the practicality of having high n_rep values like 12 or 16 in real-world synthetic data privacy auditing. These results do not strongly support the new MIA contribution. Additionally, Figure 1 (a and b) show near-random curves for n_rep=2 or 4.
Yes
No ethics review needed.
See weaknesses for points that I would like to see addressed in the rebuttal.
* Makes progress on an important problem by proposing new methods to perform MIA against synthetic data instead of directly with model access, and developing a sound methodology to design canaries for this task

* Thorough empirical evaluation providing quanlitative and quantitative analysis of the different aspects that affect MIA success
This paper investigates membership inference attacks (MIAs) on synthetic text generated by fine-tuned Large Language Models (LLMs). It highlights that traditional MIAs, which rely on access to the model or its logits, are not directly applicable to scenarios where only the synthetic data is released.  The authors find that standard "canary" examples, designed to be highly vulnerable to model-based MIAs, are less effective when auditing the privacy risks of synthetic data because their influence on the generated text diminishes.  To address this, they propose crafting specialized canaries with an in-distribution prefix and an out-of-distribution suffix to improve their impact on synthetic data generation while maintaining memorizability. Their experiments demonstrate that this approach significantly enhances the power of MIAs on synthetic text, enabling a more accurate assessment of the privacy risks associated with releasing LLM-generated data. They also analyze the membership inference signal and find that it relies on subtle shifts in the word probability distribution within the synthetic data, rather than explicit regurgitation of canary subsequences.
* Lacks a centralized presentation of the overall auditing strategy. In particular, Algorithm 1 is presented as adding a single canary one time to the dataset, while in practice most of the evaluations add several canaries with multiple repetitions. The authors should considering spelling out the full auditing method with all hyper-parameters and design choices in a single algorithm block.

* The description of the threat model (L128-138) only focuses on the access mode available to the adversary. This needs to be expanded to capture the knowledge of the training pipeline available to the adversary (base dataset, training method, sampling method, etc). This is important because in some cases (e.g. when using shadow models and when designing canaries for synthetic data auditing) the adversary leverages some of this information to perform the attack, while in other methods the attacker uses less information. A table summarizing these differences would make it easier to compare the different approaches evaluated in the paper.
Yes
No ethics review needed.
1. Are the membership scores using n-gram probability and similarity metric novel techniques that you are proposing?
2. The authors mention that memorization is more likely when there is an abrupt transition in entropy between sub-sequences. Could you discuss the intuition behind this?
3. Were experiments on low-perplexity prefix lengths of 40 and 50 performed? It would be great to provide those results, too.
1. The paper tackles an important problem of privacy auditing synthetic text generation from LLMs
2. The ideas in the paper are easy to follow
3. The result showing that increased canary perplexity leads to lower MIA AUC for synthetic data-based attacks is interesting
The paper investigates privacy auditing of synthetically generated text using MIAs. The authors demonstrate that standard canaries for model-based MIAs are ineffective for data-based MIAs. Hence, they design a new canary generation technique that improves data-based MIAs. The main challenge of generating effective canaries is ensuring that they align with the prompt, so they can at least be generated. But at the same time need to be unique enough so that they are memorized by the model. So they create a canary where the prefix contains a truncated original record, and the suffix is generated by a pre-trained LLM.
1. I find the main results from 5.1 that current canary designs aren’t effective for MIAs on synthetic data a bit underwhelming. I feel like this is already well-known and intuitive that MIAs based on just the generated labels/tokens is much harder than having logits because you loose the uncertainty quantities that are usually exploited in standard MIAs. Unless I am missing something that is the novelty of the results in 5.1.
2. It’s unclear to me how the author’s proposed canary generation method improves over the standard canary generation. For example, how much does the canaries with in-distribution suffix improve over the baseline canaries for MIAs on synthetic data in terms of AUC? This would help with connecting the main results from 5.1 as I only see improvements stated in terms of TPR which feels disconnected from 5.1. 
3. There seems to be a TPR improvement for increasing prefix length on SST-2 in Figure 1, but marginal improvement for AG News. This ambiguity in results makes it inconclusive about the exact effect the prefix length has improving MIAs (large vs small). Unfortunately, we’d need to see more results on at least one more dataset.  
4. I guess generalizng points 1. and 2., I’m uncertain about the exact contribution of the paper. It is hard to find in the introduction and there is no conclusion section to make the contribution more concrete.
Yes
No ethics review needed.
- How would the results change if the adversary is able to choose the prompts for the synthetic data generation?
- How much hyperparameter tuning was done when training the model?
- The authors propose a new method for generating canaries optimized for this application.
- They run a thorough experimental evaluation to understand how the performance varies with the different parameters.
This paper studies membership inference attacks (MIAs) that target data used to fine-tune pre-trained LLMs, which are used to generate synthetic data. The authors focus on the case where the adversary does not have access to the fine-tuned model, but only to the generated synthetic data. In this case, prior MIAs are not applicable because even though canaries may be memorized by the target model, they are less likely to be present in the synthetic data generated by in-distribution prompts. The authors address this by constructing canaries starting with an in-distribution prefix and transitioning into an out-of-distribution suffix, where the former increases the likelihood of influencing the synthetic data, and the latter increases the likelihood that the model memorizes them.
- It would be great if the authors could discuss concrete applications of this work or scenarios where it is useful.
- This work only gives a lower bound on the privacy risks, and it is not clear how the results generalize.
We design specialized canaries to audit the privacy of synthetic text generated by LLMs
@inproceedings{
anonymous2024the,
title={The Canary{\textquoteright}s Echo: Auditing Privacy Risks of {LLM}-Generated Synthetic Text},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=r7wMVdGFro},
note={under review}
}
How much information about training examples can be gleaned from synthetic data generated by Large Language Models (LLMs)? Overlooking the subtleties of information flow in synthetic data generation pipelines can lead to a false sense of privacy. In this paper, we investigate the design of membership inference attacks (MIAs) that target data used to fine-tune pre-trained LLMs that are then used to synthesize data, particularly when the adversary does not have access to the fine-tuned LLM but only to a synthetic data corpus. We demonstrate that using canaries crafted to maximize their vulnerability to attacks that have access to the model are sub-optimal for auditing privacy risks when only synthetic data is released. This is because such out-of-distribution canaries have limited influence on the model’s output when prompted to generate useful, in-distribution synthetic data, thus significantly limiting their vulnerability to MIAs. To tackle this problem, we leverage the mechanics of auto-regressive models to design canaries that leave detectable traces in synthetic data. Our approach significantly enhances the power of MIAs, providing a better assessment of the privacy risks of releasing synthetic data generated by LLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Privacy language models synthetic data
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6fec47fac3c7d3a314895c6a01af7d0dc9e46161.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/187488cb891a9bc88b29e99bf90be8e44c4accc6.zip
The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission