Privacy as a Free Lunch: Crafting Initial Distilled Datasets through the Kaleidoscope
Dataset Distillation
Yes
No ethics review needed.
Look at weaknesses section for my concerns.
1.  **Well-Motivated Problem:**  The paper validly raises significant concerns about the privacy implications of naive data distillation, showing deep leakage when high IPC conditions exist.
    
2.  **Effective Randomization with Kaleidoscopic Transformation:**  The proposed KT approach is intuitive and well-motivated, as introducing randomness during initialization adds another layer of privacy protection. This idea is supported by theoretical claims, and  **Theorem 2**  provides a solid foundation demonstrating that KT strengthens privacy in the resultant distilled datasets.
    
3.  **Introduction of Explicit Privacy Leakage as a Metric:**  Authors introduce  **explicit privacy leakage**—calculated as the average minimum LPIPS distance between the distilled and original datasets—for quantifying privacy risks.
    
4.  **Experimental Evaluation:**  The authors use multiple datasets and baselines during experimental evaluation of KT
The paper addresses significant privacy risks in data distillation, particularly noting that a high number of images per class (IPC) can cause distilled images to closely resemble the original data, leading to substantial privacy leakage. This concern is theoretically supported, demonstrating that higher IPC values in naive distillation increase the risk of data leakage. To mitigate these risks, the authors propose  **Kaleidoscopic Transformation (KT)**, a technique that introduces randomized transformations to real data during the initialization phase. This approach enhances privacy by adding an additional layer of randomness to the distillation process. The effectiveness of KT in strengthening privacy is theoretically validated by  **Theorem 2**. Additionally, the paper introduces a new metric,  **explicit privacy leakage**, defined as the average minimum LPIPS distance between the distilled and original datasets, to more accurately quantify privacy risks.
1.  **Initial Lack of Clarity in Presentation:**  While the paper writing is good , the first two pages were hard to follow. Technical terms such as IPC and explicit privacy leakage are introduced later but the figure is shown earlier and discussion is made in the first two pages without adequate explanation, making the reading the paper challenging. Additionally, Figure 1 presents percentage metrics for differential privacy and explicit privacy leakage but lacks explanations of what these "percentages" represent and how the metrics are calculated.
    
2.  **Undefined Terms in Theorems:**  Variables like  P  and  Q  in Theorems 1 and 2 are not clearly defined. Although looking at the proof helps, providing explicit definitions would enhance clarity and understanding.
    
3.  **Ambiguity in Figure 4:**  The meaning of the numbers presented in Figure 4 is unclear, necessitating further explanation to interpret the results effectively.
    
4.  **Claim :  "Free" Privacy Improvement is not substantiated:**  The claim that KT provides a “free improvement in privacy without significant computational overhead" seems inaccurate (Line 461). For instance, Figure 5 shows a trade-off between privacy and utility":  increased transformations lead to reduced utility. While it is possible to claim that KT may offer an improved balance under certain conditions, describing it as "free" implies no utility loss, which is misleading with Figure 5 results.
    
5.  **Claim: "Improved balanced performance with Baselines" needs a little more explanation:**  In Table 2, the paper shows certain trade-off points between data generator methods for a fixed privacy budget of 10. It is unclear whether these baseline methods depend on a fixed budget or if comparisons can be made with other epsilon values. It would be ideal if a tradeoff curve could be shown that shows KT offfers better tradeoff. Looking at Table 2 I cant be sure of this conclusion as I can only see KT has high utility. This ambiguity makes it difficult to substantiate the claim that "our approach demonstrates a balanced performance in privacy preservation and data utility"
Yes
n/a
No ethics review needed.
For W1, W2, and W3, if there are any misunderstandings in my interpretation, please clarify them.

Q1: Please clarify how the proposed approach differs from privacy amplification via sub-sampling. Also, please provide a more rigorous justification for why the random sampling method in data distillation provides differential privacy guarantees without prior privatization. 

Q2: How can the proposed method be practically implemented, given the constraints described in W2? In addition, if there are any relaxations or modifications that could make it more feasible for real-world data distillation scenarios, including these things in this paper might be helpful.
- S1: The paper is well-organized and easy to follow.
- S2: The paper identifies that distilled datasets produced by state-of-the-art distillation methods strongly resemble real data, indicating a significant risk of privacy leakage. The problem addressed is interesting and relevant.
- S3: Experimental results provide various comparisons with existing methods.
This paper studies privacy guarantees in data distillation methods. It introduces the concept of explicit privacy, which considers how similar the generated data is to the original data, and then examines the extent to which this privacy metric is preserved in data distillation methods. The paper also analyzes the relationship between privacy guarantees in data distillation and differential privacy.
- W1: The discussion of differential privacy achieved through random sampling in data distillation appears problematic. It closely resembles the concept of privacy amplification via subsampling. Privacy amplification through subsampling assumes that data has already been privatized to a certain extent, and this privacy level is then amplified by subsampling. However, Proposition 1 assumes that none of the data has been privatized to ensure differential privacy prior to subsampling. Consequently, the subsampled data do not satisfy any differential privacy guarantees. Please clarify how the proposed approach differs from privacy amplification via sub-sampling. Also please provide a more rigorous justification for why the random sampling method in data distillation provides differential privacy guarantees without prior privatization. 
- W2: Even if Proposition 1 were applicable, it seems impractical because $\delta$ would need to be less than or equal to $n^{-1}$, which implies $S = 1$. This would mean that the initialization process in data distillation is restricted to sampling only one record from the entire dataset of size T. Such a limitation may be undesirable for data distillation methods. This paper needs to clarify how the proposed method can be practically implemented given such constraints. If there are any relaxations or modifications that could make it more feasible for real-world data distillation scenarios, including these things in this paper might be helpful.
- W3: The comparison with differentially private generators appears to be unfair. While these DP generators are trained under $\epsilon = 10$, the proposed method does not assume the same privacy level. For a fair comparison, the authors should evaluate performance at an equivalent privacy level. A possible suggestion is that the authors conduct additional experiments where their method is constrained to provide equivalent privacy guarantees as the DP generators and to provide a more detailed discussion of how privacy levels can be meaningfully compared between the proposed approach and existing DP methods.

Overall, the analytical study on differential privacy in data distillation appears to have several significant issues. If there are any misunderstandings in my interpretation, please clarify them.
Yes
No ethics review needed.
I think the question above is critical.

Some of the citation is ?, please check the latex code.
# Originality
The Kaleidoscopic Transformation (KT) module is a creative approach to enhance privacy by introducing strong perturbations during data initialization, which is an original contribution to the field.
# Clarity
The paper clearly articulates the problem of explicit privacy leakage and its implications, making it accessible to a broad audience.
The description of the KT module and its integration into existing distillation methods is clearly presented, allowing others to replicate the work.
This paper investigates the issue of privacy leakage in the process of dataset distillation. The authors point out that existing distillation methods produce datasets that closely resemble real data, leading to significant privacy leakage, termed as explicit privacy leakage. The paper analyzes how a high IPC (images per class) can weaken the protection of both differential privacy and explicit privacy.

To address this issue, the authors propose a module called "Kaleidoscopic Transformation" (KT), which applies strong perturbations to real data during the initialization phase to protect privacy. Experimental results show that this method ensures both differential privacy and explicit privacy while maintaining the generalization performance of the distilled data.

In summary, the contributions of the paper include:

1. Identifying and analyzing the problem of explicit privacy leakage caused by high IPC.

2. Theoretically demonstrating that using real data during the initialization phase leads to privacy leakage.

3. Proposing and validating a module that enhances privacy protection without sacrificing performance.
I believe the critical flaw in this paper lies in its attempt to ensure differential privacy through random sampling of datasets, as described in Proposition 1. While sampling can introduce randomness, the problem is that the algorithm fully exposes the privacy of the selected samples without additional protection. This does not meet the requirements of differential privacy. Specifically, refer to the second paragraph on page 18 of Dwork's privacy book [1] . For convenience, I will quote that section: "Typically we are interested in values of (\delta) that are less than the inverse of any polynomial in the size of the database. In particular, values of (\delta) on the order of (1/||x||_1) are very dangerous: they permit 'preserving privacy' by publishing the complete records of a small number of database participants." ( ||x||_1 refers to the size of the dataset, in the is paper, it should be |T| ) In the authors' paper, Proposition 1 describes exactly this situation: completely sacrificing the privacy of the sampled participants. Therefore, I believe that the proof based on Proposition 1 provides meaningless differential privacy protection (\delta=|s|/|T|). In fact, differential privacy cannot be achieved solely through sampling; it still requires the addition of noise, such as Gaussian noise. When Gaussian noise is added, sampling can enhance privacy to some extent, as in the DPSGD algorithm, which still adds isotropic Gaussian noise to the gradients obtained from sampling. I consider this a fatal flaw in the paper.

[1] Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4), 211-407.
Yes
No ethics review needed.
1. What would happen if you used random initialization or added noise to the real image selection process for initialization?
2. In your case, does the adversary have access to the original dataset?
This paper is well-structured and presented, with a good flow.
This paper examines the privacy leakage in data distillation when real data is used for initialization. It proposes perturbing the initialization data as a method to enhance privacy protection.
I have following concerns:

1. The concept of explicit privacy leakage introduced here seems somewhat ill-defined. Visual resemblance does not necessarily imply privacy leakage and, in some cases, could even enhance privacy. For instance, if synthesized images resemble non-sensitive images not included in the training data, they may help protect privacy, particularly when the adversary lacks access to the original dataset. In contrast, DP offers stronger and more formal privacy guarantees than this notion of explicit privacy leakage. Therefore, the motivation for introducing it is unclear to me.

2. Data distillation commonly achieves DP through random initialization (e.g., Dong et al., 2022). Your claim of additional privacy leakage, however, stems from the specific selection of real images, which is unsurprising since prior information is embedded in this selection. While this approach may affect the privacy of the chosen image, it should not impact the privacy guarantees for other images in the dataset. DP is breached in your case because DP considers the worst-case scenario. Thus, the findings in this paper appear somewhat obvious.
@inproceedings{
anonymous2024privacy,
title={Privacy as a Free Lunch: Crafting Initial Distilled Datasets through the Kaleidoscope},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ckabXglfiT},
note={under review}
}
The advancement of deep learning necessitates stringent data privacy guarantees.
Dataset distillation has shown potential in preserving differential privacy while maintaining training efficiency.
This study first identifies that data generated by state-of-the-art dataset distillation methods strongly resembles to real data, indicating severe privacy leakage.
We define this phenomenon as explicit privacy leakage.
We theoretically analyze that although distilled datasets can ensure differential privacy to some extent, a high \IPC can weaken both differential privacy and explicit privacy.
Furthermore, we reveal that the primary source of privacy leakage in distilled data stems from the common approach of initializing distilled images as real data.
To address this, we propose a plug-and-play module, Kaleidoscopic Transformation (KT), designed to introduce enhanced strong perturbations to the selected real data during the initialization phase.
Extensive experiments demonstrate that our method ensures both differential privacy and explicit privacy, while preserving the generalization performance of the distilled data.
Our code will be publicly available.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Dataset Distillation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cb6f1ffcb3be161997deab48bcbd784a34b9a502.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Privacy as a Free Lunch: Crafting Initial Distilled Datasets through the Kaleidoscope
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission