FedGO : Federated Ensemble Distillation with GAN-based Optimality
Federated learning;ensemble distillation;data heterogeneity;generative adversarial network
Yes
No ethics review needed.
Please refer to the weaknesses.
1. The paper provides a theoretical foundation for the proposed approach, which validates its effectiveness and enhances its credibility.
2. The paper analyzes communication, privacy, and computational complexity within different scenarios, providing valuable insights for implementing the proposed approach.
This paper proposed a novel federated ensemble distillation approach that utilizes generative adversarial networks (GANs) to address the challenges posed by data diversity across clients. Specifically, the proposed approach employs GANs to optimize the weighting of client predictions, thereby improving the quality of pseudo-labels generated during the ensemble distillation process. The paper provides theoretical insights that establish the effectiveness of the proposed method. Comprehensive experiments demonstrate that the proposed approach outperforms existing methods in robustness against data heterogeneity.
1. This paper needs to demonstrate the effectiveness of the proposed approach on different model structures, such as VGG and MobileNet.
2. The effectiveness of the proposed method relies on the quality of the discriminator and generator. The paper needs to conduct related ablation studies.
3. This paper should conduct ablation studies to analyze the impact of hyperparameters (e.g. $E_s$ and $E_d$) on the effectiveness of the approach.
4. The experimental settings of the baselines are not clearly stated, and it is important to clarify the fairness of the experimental comparison.
5. The additional computational and communication overhead introduced by the GAN-based approach may not be suitable for FL scenarios, particularly those with strict resource constraints.
Yes
No ethics review needed.
1. In introduction on page 2, Our main contributions are summarized in the following: "Federated Ensemble Distillation" instead of "Ferated Ensemble Distillation".

2. In theoretical analysis, near-optimal performance is heavily affected on discriminator performance. I do not understand how to select the discriminator architectures? Can you give me some detailed description?
This paper demonstrates originality through its innovative integration of GAN-based techniques with federated ensemble distillation. The use of discriminators trained at the client side to optimize the weighting of client contributions during the distillation process is a novel approach that has not been extensively explored in previous federated learning research.

The method's originality is further enhanced by its theoretical grounding, which employs results from GAN literature to develop a provably near-optimal weighting method. 

The experimental setup is well thought out
The paper introduces a new approach to address the issue of data heterogeneity in federated learning. By applying Generative Adversarial Network (GAN) techniques to federated ensemble distillation, the paper proposes a near-optimal weighting method that enhances the training process of the server model. Extensive experimental validation demonstrates significant improvements in model performance and convergence speed across various image classification tasks. Moreover, the study provides an in-depth analysis of the potential additional communication costs, privacy leaks, and computational burdens introduced by this method, showcasing its practicality and flexibility in protecting data privacy and enhancing system efficiency.
The paper claims near-optimal performance based on theoretical justifications rooted in GAN literature. However, these claims might depend heavily on certain idealized assumptions about data distributions and discriminator performance. Real-world deviations from these assumptions could lead to suboptimal performance. The paper does not explain how to select discriminator architectures.
Yes
No ethics review needed.
Please see Weaknesses.
1. The paper is well-written and easy to follow.
2. The authors conducted extensive experiments to verify the effectiveness of the proposed method.
This paper proposes FedGO: Ferated Ensemble Distillation with GAN-based Optimality, for federated ensemble distillation. This algorithm incorporates a novel weighting method using the client discriminators that are trained at the clients based on the generator distributed from the server and their own datasets. The generator distributed from the server can be either off-the-shelf or trained with the unlabeled dataset on the server. The exchange of the generator and the client discriminators between the server and the clients occurs only once before the main FL algorithm starts, resulting in minimal additional overhead.
Extensive experiments demonstrate significant improvements of FedEDG over existing research both in final performance and convergence speed on multiple image datasets.
As far as I am concerned, distillation-based FL is data dependent and requires access to an auxiliary dataset derived from publicly available proxy data sources for knowledge transfer, whereas a desirable auxiliary dataset is not always available since its construction requires careful deliberation and even prior knowledge about clients’ private data to achieve satisfactory performance, which is inconsistent with the privacy-preserving nature of FL. In addition, I argue that FedEDG with Pretrained Generator proposed in this paper also has the above-mentioned issues. This is because pre-trained generator needs to be trained on public datasets. Therefore, I remain skeptical of this research direction, even if the paper contains theoretical evidence. Furthermore, if the author wants to convince me, please provide some feasible solutions to address the aforementioned issues.
I'll raise my score if author can address the above problems.
@inproceedings{
anonymous2024fedgo,
title={Fed{GO} : Federated Ensemble Distillation with {GAN}-based Optimality},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=4ftMNGeLsz},
note={under review}
}
For federated learning in practical settings, a significant challenge is the considerable diversity of data across clients. To tackle this data heterogeneity issue, it has been recognized that federated ensemble distillation is effective. Federated ensemble distillation requires an unlabeled dataset on the server, which could either be an extra dataset the server already possesses or a dataset generated by training a generator through a data-free approach. Then, it proceeds by generating pseudo-labels for the unlabeled data based on the predictions of client models and training the server model using this pseudo-labeled dataset. Consequently, the efficacy of ensemble distillation hinges on the quality  of these pseudo-labels, which, in turn, poses a challenge of appropriately assigning weights to client predictions for each data point, particularly in scenarios with data heterogeneity. In this work, we suggest a provably near-optimal weighting method for federated ensemble distillation, inspired by theoretical results in generative adversarial networks (GANs). Our weighting method utilizes client discriminators, trained at the clients based on a generator distributed from the server and their own datasets. 
Our comprehensive experiments on various image classification tasks illustrate that our method significantly improves the performance over baselines, under various scenarios with and without extra server dataset. Furthermore, we provide an extensive analysis of additional communication cost, privacy leakage, and computational burden caused by our weighting method.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Federated learning ensemble distillation data heterogeneity generative adversarial network
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6b2e8890f4adec7fd264be251ae9a25ed2cc14f5.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/e0904c57b7a83a113c52e17373c63900898442ce.zip
FedGO : Federated Ensemble Distillation with GAN-based Optimality
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission