Communication-Efficient Federated Learning via Model-Agnostic Projection Adaptation
Federated Learning;Low-Rank Adaptation;Communication Efficiency;Subspace Optimization
Yes
No ethics review needed.
NA
1. The Model-Agnostic Projection Adaptation (MAPA) method introduces a novel approach to reduce communication overhead in federated learning (FL) by factorizing the entire model parameter space. The method employs a fixed reconstruction matrix and a shared random seed, allowing for larger reconstruction matrices and smaller projection vector dimensions. The paper provides strong theoretical support.
2. By transmitting only the trainable projection vectors, MAPA addresses a major challenge in FL by cutting communication costs. 
3. Experimental results indicate that MAPA outperforms existing FL methods in both communication efficiency and model performance.
This paper proposes Model-Agnostic Projection Adaptation (MAPA) to address communication overhead in federated learning by factorizing the model parameter space as a single vector, independent of layer count or model architecture. MAPA splits the update into a fixed reconstruction matrix and a trainable projection vector, sending only the projection vector to the server. This reduces communication costs while allowing a flexible, high-capacity reconstruction matrix. Experimental results show MAPA outperforms existing methods in both communication efficiency and model performance. Yet, aligning the experimental setup with model-agnostic principles remains an area for improvement.
1. Although the method is described as 'model-agnostic,' it currently uses the same CNN model and identical initial weights across all clients in the first round. This setup could contradict the claim of model agnosticism by implying a reliance on a specific model structure and initialization. Therefore, it is essential to either revise the claims or provide further evidence that MAPA can operate with diverse initializations and model types across clients.
2. Additional experiments with different model architectures are recommended to showcase the method’s versatility and robustness.
3. Further testing on more complex datasets, such as ImageNet, would help demonstrate MAPA’s scalability.
Yes
No ethics review needed.
Refer to the weakness section.
1. The paper provides theoretical analysis of MAPA's convergence and communication efficiency by clear and precise mathematical notations. The experiments on three dataset demonstrates MAPA's superiority over other methods.
2. The paper is well-organized, with straightforward writing and clear illustrations, especially the overview diagram, which effectively conveys MAPA's idea. Results are intuitively displayed, showcasing MAPA's improvement in reducing communication costs.
To reduce the substantial communication overhead in FL, this work presents MAPA and its extension MAPAX, a novel matrix factorization operating independently of the model architecture. The proposed method sends only projection vectors that leverage a shared reconstruction matrix generated at each round. MAPA treats the entire model parameter space as a single vector and factories it. The authors performed numerical experiments that showed that MAPA improves communication efficiency while maintaining strong model performance.
1. The paper lacks comprehensive experimental results for MAPAX, specifically missing accuracy and minimum cost outcomes on MNIST and CIFAR-10, which limits the ability to fully evaluate its effectiveness.
2. I am curious whether the proposed method can be applied to the IID setting or the scenario where clients are sampled in each communication round. Evaluating and clarifying MAPA's performance under these conditions would strengthen the paper.
3. Given the author's claim that MAPA is model-agnostic, the paper could strengthen this claim by combining MAPA with more FL models to demonstrate its versatility beyond the current implementations.
4. There are several presentation issues that can be addressed: a) the text is too small to read in Fig. 4; b). misalignment between L. 517-524 and Fig. 6. Also, it would be better to provide more analyses and explanations for Fig. 6 in the main paper.
Yes
No ethics review needed.
See the "Weakness" section.
1. Compared to related approaches like FFA-LoRA, this work’s MAPA method applies LoRA directly in the parameter space, thereby improving flexibility for agnostic models.

2. The extended method, MAPAX, provides a valuable trade-off between computational constraints and communication overhead. This balance makes MAPAX an interesting direction for further exploration.
This paper introduces a method that factorizes the model within the parameter space, aiming to reduce communication overhead in the FL setting. By transmitting only the lower-dimensional projection vector rather than the entire parameter vector, the approach reduces the parameter need to be transmitted
**Algorithm design**

1. The method requires generating and storing a matrix $A$ with $k \times d$ entries, where $d$ represents the number of parameters in the entire model, which is typically very large. This raises significant concerns about the method's practicality due to the immense computational and memory requirements. Although the authors introduce MAPAX as a solution to mitigate these issues, there is a lack of experimental results validating its effectiveness. Without empirical evidence, it's challenging to assess whether MAPAX successfully addresses the computational and storage challenges posed by MAPA.

2. The matrix $A$ is generated randomly, similar to the approach in FAA-LoRA [1]. Recent studies [2] have indicated that random generation of such matrices may lead to suboptimal performance.

**Theoretical analysis**

1. The "representation capacity rate" introduction in Definition 2 lacks clarity and justification. The term "representation capacity" is not well-defined in the context of the paper, and it's unclear why the proposed quantity effectively measures it. There is no evident connection between the introduced definition and established metrics for the complexity of function classes, such as covering numbers.

2.  In Definition 2, the focus is solely on the rank $k$ of matrix $A$, neglecting the influence of the dimension $d$. In representation learning literature, the complexity typically depends on both $d$ and $k$.

3.  Due to the unclear definition of the representation capacity rate, the claim in Proposition 1 that the proposed method "...achieves a higher representation capacity..." lacks substantiation.

4.  The convergence analysis is built upon Assumption 4, which appears to be too restrictive and not reflective of practical scenarios. For instance, this assumption may not hold if $W_t$ reaches a saddle point in the non-convex optimization landscapes.

**Experiments**

1. The reported accuracy on the CIFAR-10 dataset is unexpectedly low. Previous work, such as FedRep [3], published in 2021, has demonstrated significantly higher accuracy levels, even though it is not SOTA.  Besides, the selection of baseline algorithms in the experimental evaluation is inadequate.

2. Although MAPAX is proposed to address the computational and memory burdens introduced by MAPA, the paper does not present any experimental results for it.

References:

[1] Sun, Youbang, et al. "Improving loRA in privacy-preserving federated learning." arXiv preprint arXiv:2403.12313 (2024).

[2] Guo, Pengxin, et al. "Selective Aggregation for Low-Rank Adaptation in Federated Learning." arXiv preprint arXiv:2410.01463 (2024).

[3] Collins, Liam, et al. "Exploiting shared representations for personalized federated learning." International conference on machine learning. PMLR, 2021.
@inproceedings{
anonymous2024communicationefficient,
title={Communication-Efficient Federated Learning via Model-Agnostic Projection Adaptation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=rhfOzJzsKN},
note={under review}
}
Federated learning (FL) enables collaborative model training across distributed clients without centralizing sensitive raw data while benefiting from diverse data sources. 
Despite recent advancements in FL, the communication overhead remains a significant challenge, especially for large-scale models.
Recent low-rank adaptation (LoRA) techniques have shown promise in reducing these burdens in FL, but they are typically applied to each layer individually and depend on the model architecture, which limits their performance.
To address these shortcomings, we propose  Model-Agnostic Projection Adaptation (MAPA), a novel approach that applies factorization to the entire model parameter space, which we view as a *single vector*, regardless of the number of layers and model architecture. 
MAPA factorizes the single-vector model update into a fixed *reconstruction matrix* and a trainable *projection vector*, with the reconstruction matrix being randomly initialized using a shared seed at each round. 
This ensures that *only* the projection vectors need to be communicated to the server, thereby reducing the communication cost.
Furthermore, MAPA's vector-based representation and relaxed rank constraints allow for a larger reconstruction matrix and smaller projection vector dimensions compared to LoRA, enhancing the expressiveness of model updates while significantly reducing communication overhead. 
Experimental results demonstrate that MAPA outperforms existing FL methods in both communication efficiency and model performance, effectively coupling optimization and communication efficiency in FL environments.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Federated Learning Low-Rank Adaptation Communication Efficiency Subspace Optimization
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/fcd07fc98c807bbddfa19b7ddd885be5564bfe3e.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/9b730477e109deaeab6b141fa4ef8abf828a067e.zip
Communication-Efficient Federated Learning via Model-Agnostic Projection Adaptation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission