Towards Realistic Mechanisms That Incentivize Federated Participation and Contribution
Federated Learning;Mechanisms;Realistic;Utility
Yes
NA
No ethics review needed.
The paper lacks a detailed discussion on the computational overhead of implementing REALFM, especially given the added accuracy-shaping and monetary reward calculations. Including complexity analysis would help readers understand potential trade-offs.

The paper could be enhanced if  it could further exploring the impact of varying device capabilities (e.g., computational power, storage) on REALFM’s effectiveness and scalability.

A discussion on privacy-preserving mechanisms to protect device-specific information in the proposed setups would enhance the paper’s relevance to practical federated applications.

Given that contract-based FL mechanisms (e.g., using registration fees to penalize free-riders) are common in the literature, adding a comparative analysis with these mechanisms would offer a more rounded assessment of REALFM’s advantages and limitations.
REALFM incentivizes edge devices to participate in federated learning by offering accuracy-based and monetary rewards proportional to each device's data contribution. This mechanism enables the central server to attain better model performance by motivating devices to contribute more data than they would on their own.
The paper introduces incentive mechanisms in federated learning (FL).
The paper lacks a detailed discussion on the computational overhead of implementing REALFM, especially given the added accuracy-shaping and monetary reward calculations. Including complexity analysis would help readers understand potential trade-offs.

The paper could be enhanced if  it could further exploring the impact of varying device capabilities (e.g., computational power, storage) on REALFM’s effectiveness and scalability.

A discussion on privacy-preserving mechanisms to protect device-specific information in the proposed setups would enhance the paper’s relevance to practical federated applications.

Given that contract-based FL mechanisms (e.g., using registration fees to penalize free-riders) are common in the literature, adding a comparative analysis with these mechanisms would offer a more rounded assessment of REALFM’s advantages and limitations.
Yes
No ethics review needed.
- Please explain why "for example, accuracy improvement from 48% to 49% should be rewarded much differently than 98% to 99%" 
- Please explain how the proposed approach can be extended to covariate and concept shift.
- How is the proposed approach handling clients with minority data? The approach aims to punish free-riders but it might mistakenly punish minority clients, in particular by offering a noisy model
- Please add the dataset name and dirichlet parameter to the figures to improve readability.
- What kind of "real" applications would require this incentive scheme?
-The paper provides a solid theoretical framework, supported by rigorous proofs and analyses
- The paper is well-organized, with clear definitions and explanations that make the methods and results easy to follow.
The paper proposes REALFM, a federated learning mechanism designed to incentivize edge device participation and data contribution by modeling device utility and removing the free-rider dilemma under non-i.i.d distributions. It introduces non-linear reward structures to enhance both device and server utility in federated settings with heterogeneous data. The proposed approach shows significant improvements in device contributions and model accuracy compared to existing mechanisms.
- The paper focuses on non-i.i.d case but uses "data heterogeneity" for so long before being precise, which can be misleading for the reader. Moreover, assumptions and mechanisms would struggle in highly variable data distributions: The accuracy shaping mechanism assumes that central server updates generally benefit all participants, which would not be the case for label or covariate shift. 
- REALFM introduces a complex reward and incentive mechanism that may be challenging to implement in "real-world" federated learning systems. The added layers of utility modeling and accuracy shaping might make it difficult to scale across diverse device types and operational environments. Additionally, the "real" part might be an overestimation. 
- It is unclear what kind of "realistic" scenarios can benefit from such incentive scheme. 
- The figures could have better readability by adding more information about the datasets
Yes
No ethics review needed.
1. Are some ablation studies missing, such as examining performance under different client numbers or aggregation algorithms? （A mini Question)
1. Well-written: The paper is clear and well-structured.
2. Solid Theoretical Analysis: The theoretical analysis and proofs are strong and well-supported.
The paper introduces an incentive mechanism for federated learning, addressing key issues like the free-rider problem and the lack of realistic incentives. The approach is well-motivated, theoretically solid, and supported by experiments demonstrating improvements over existing methods. However, some areas, such as more comprehensive literature reviews in related work, practical implementation details, and additional ablation studies, could further strengthen the work.
1.	Free-rider problem has been widely studied in FL settings, such as [1], and literature in its related works. In addition, previous studies of mechanisms for FL-related scenarios, such as crowdsourcing, can be suitable and easily adapted to FL settings.
2.	Achieving the goals of this paper seems to require knowledge of the data amount on each device. However, in the context of privacy-preserving machine learning, is it essential for FL clients to disclose their data sizes? This raises the question of whether the proposed approach truly aligns with the notion of “REALISTIC MECHANISMS” as suggested in the title.
3. Including well-organized source code for the experiments would enhance the paper's reproducibility and allow reviewers to verify specific details. 

[1] Meng J, et al. Federated Learning and Free-riding in a Competitive Market, 2024
Yes
No ethics review needed.
- The authors claim that the work does not involve data sharing, but $c_im_i$ formulation is the same as [1]. Does this imply that the analysis from [1] can readily be extended to cases without data sharing?

- In Figure 4, data contribution from non-linear RealFM in MNIST case goes to 10^8 magnitude, while it is 10^4 for Cifar10 case. I find this unreasonable, as Cifar10 is a harder task and more data will be helpful. This has something to do with the chosen phi function, where utility can go to infinity when accuracy is close to 1 (which is the case for easier tasks).

[1] Karimireddy et al. Mechanisms that Incentivize Data Sharing in Federated Learning
- The mapping from the model accuracy to utility is non-linear, which renders it more realistic.

- Server utility is explicitly modeled, which has not been widely explored in previous research.
This work proposes a federated learning mechanism to incentivize data contribution and device participation. Building on the work of [1], this paper further allows more realistic settings such as non-linear mapping between utility and model accuracy, non-iid local data distribution (to some extent), preventing data sharing, and modeling of server utility.
- In calculating server accuracy, it’s unclear why Assumption 3 holds, especially if local accuracies vary significantly across devices.

- The study aims to address a cross-device setup; however, the experiments involve only 16 devices, which somewhat limits the persuasiveness of the empirical results.

- The comparison of utilities in Figure 3 seems unfair. For the non-linear and linear methods, agents’ utilities are distinct functions of model accuracy, meaning that even with the same model accuracy, their utilities would differ.

- Typo line 412: incentives -> incentivizes
We propose a truly federated learning mechanism which realistically models device utility and incentivizes devices to participate and contribute more to federated training all while provably removing the free-rider dilemma.
@inproceedings{
anonymous2024towards,
title={Towards Realistic Mechanisms That Incentivize Federated Participation and Contribution},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=sgHnfLX9Lt},
note={under review}
}
Edge device participation in federating learning (FL) is typically studied through the lens of device-server communication (e.g., device dropout) and assumes an undying desire from edge devices to participate in FL. As a result, current FL frameworks are flawed when implemented in realistic settings, with many encountering the free-rider dilemma. In a step to push FL towards realistic settings, we propose RealFM: the first federated mechanism that (1) realistically models device utility, (2) incentivizes data contribution and device participation, (3) provably removes the free-rider dilemma, and (4) relaxes assumptions on data homogeneity and data sharing. Compared to previous FL mechanisms, RealFM allows for a non-linear relationship between model accuracy and utility, which improves the utility gained by the server and participating devices. On real-world data, RealFM improves device and server utility, as well as data contribution, by over 3 and 4 magnitudes respectively compared to baselines.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Federated Learning Mechanisms Realistic Utility
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f3d6d2da279e21f52a28b44829c61736f9e4e625.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/0d160ac69cff4e685d047a1b1a985949d80645d2.zip
Towards Realistic Mechanisms That Incentivize Federated Participation and Contribution
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission