Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization
Federated Learning;Communication-Efficient Federated Learning;Low-Rank Nature;Cross-Device Federated Learning
Yes
No ethics review needed.
Q1. The paper presents FedLoRU and its variants by applying low-rank updates in a federated learning setting. However, the novelty of this proposed method is limited. The idea of using low-rank updates in federated learning has been explored before, and the paper does not provide a compelling argument for why the proposed method outperforms existing approaches. 

Q2. While the paper utilizes the stable rank metric to analyze rank properties between local clients and the central server, the discussion around this metric is lacking. The claim that stable rank "serves as a continuous proxy for rank and is robust" is made without sufficient references or supporting literature. Additionally, more discussion is needed on how this concept is adapted from related fields, and why it is appropriate for the federated learning context.

Q3. Figure 2(a) is difficult to interpret. Both the datasets with 50 and 500 samples show a high stable rank at the 15th epoch, which is counterintuitive and requires further explanation. It would strengthen the paper if the authors could repeat the experiment multiple times and provide clearer insights to support the observed trends.

Q4. The experiment shown in Figure 2(b) does not convincingly support the authors' intuition without a more detailed description. A thorough explanation of the experimental setup and its relation to Theorem 3.2 would significantly improve the clarity and impact of the results.
FedLoRU uses successive low-rank updates for both pre-training and fine-tuning in federated learning and achieves good performance.
The paper applies FedLoRU and its variants to impose the local update in a low-rank subspace to achieve implicit regularization.
W1. The novelty is not justified sufficiently. 

W2. More discussions and justifications regarding the stable rank metric are needed.

W3. The experiment setup and results are not convincing.
Yes
No ethics review needed.
See in weaknesses.
1. The proposed method is well-motivated, the paper investigates the rank properties of client and server losses, analytically showing that under stochastic sampling, the rank of the Hessian of the loss function increases with smaller sample sizes.
2. The empirical results show empirical evidence of the higher rank structure of client losses and demonstrate that restricting the rank of local updates aids in implicit regularization.
To address the issue of communication efficiency and heterogeneity in Federated Learning, this paper proposes the FedLoRU method. This general low-rank update framework enforces low-rank client-side updates and accumulates these updates to form a higher-rank model. The authors provide empirical results to demonstrate that FedLoRU performs better than other algorithms.
1. In the theorems that are presented, summarizing the main insights of these theorems may be needed since currently they are just written as long paragraphs.
2. In experiments, the least partial client participation ratio is set as 0.5. In more realistic settings, the participation ratio is lower with more clients.
3. The author should consider more baselines, which apply low-rank factorized update models, such as [1].
[1] Nam Hyeon-Woo, Moon Ye-Bin, Tae-Hyun Oh. FedPara: Low-rank Hadamard Product for Communication-Efficient Federated Learning. ICLR 2022.
Yes
No ethics review needed.
Please refer to the limitation.
This paper reveals that client loss in federated learning has a higher rank structure (in gradients and Hessian subspaces) than the server's loss.
 Based on this, they propose that restricting client optimization to a low-rank subspace could provide implicit regularization. They then introduce FedLoRU, a framework that enforces low-rank updates on the client side and aggregates them into a higher-rank model. Finally, they add another low-rank module pair to adapt to environments with statistical and model heterogeneity.
This paper reveals that client loss in federated learning has a higher rank structure (in gradients and Hessian subspaces) than the server's loss. Based on this, they propose that restricting client optimization to a low-rank subspace could provide implicit regularization and then introduce FedLoRU, a framework that enforces low-rank updates on the client side and aggregates them into a higher-rank model. Finally, they add another low-rank module pair to adapt to environments with statistical and model heterogeneity.
The novelty is limited, there is no close connectiong between the analysis and the algorithm. I think this algorithm is a federated version of ReLoRA if we consider on the non-personalized version, aggregating low-rank modules for higher rank training. 

There is no theoretical analysis for the algorithm. It's fully heuristic. When we consider the personalized strategy this paper studied, I don't know what kind of solution will this algoritm converge to. Will the introduced L, U fully concel out the A, B modules and make this algorithm fully consider local loss? The author didn't provide the reasonability of their strategy. 

According to my understanding, this algorithm is still a full parameter training algorithm as it initializes W every $\tau$ step. So the comparison to LoRA is unfair. On the other hand, there are numerous algorithms for conventional federated learning. If you want to highlight your algorithm's advantage, you should compare your algorithm with the conventional algorithm, rather than just beatting LoRA. 

You can't accurately solve argmin_{A,B} f. This step is computation-heavy even if you use an \epsilon-approixition. This step is actually one step of full LoRA tuning.  Therefore, this algorithm is not suitable for LLM fine-tuning.
Yes
No ethics review needed.
* The title mentions "its connection to implicit regularization". To my knowledge, implicit regularization refers to the phenomenon that optimizers without explicit regularization, such as SGD, prefer regularized solutions [1]. However, FedLoRU explicitly works in a specific rank-$r$ space. Could the authors please explain in what sense is FedLoRU connected to implicit regularization?
* The theory part analyzes the rank structures of *loss Hessians* at server and client side. At the same time, FedLoRU proposes to perform low-rank updates on the model's *weight matrices*. Could the authors please explain the connection between the rank structure of loss Hessians and weight matrices?

[1] Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. arXiv preprint arXiv:1810.02032, 2018.
* The paper provides rigorous theoretical analysis on the Hessian rank structures, establishing interesting asymptotic results within a mathematically general framework.
* The proposed algorithm achieves performance comparable or superior to other known methods in experiments, while significantly reducing the communication overhead by low-rank updates.
* The presentation of this paper is well-organized and the motivation and methodology are clear to follow.
This paper studies communication-efficient low-rank update framework for federated learning. 

It provides theoretical asymptotic analysis for the rank structures of the Hessian at server side and client side, which motivates the design of FedLoRU algorithm. Generalizations of FedLoRU under statistical and model heterogeneity, namely pFedLoRU and mFedLoRU, are also presented. Finally, the authors conduct experiments on computer vision pre-training and language model fine-tuning tasks to demonstrate the performance of FedLoRU and its generalizations.
* Although the authors provide some Hessian rank structure analysis, the design of FedLoRU can be better supported from the theoretical side. For example, some convergence guarantees, since low-rank updates lead to loss of information compared to full-rank updates and may hurt the optimization.
* The title mentions "its connection to implicit regularization", but I was not able to spot sufficient discussion on implicit regularization of FedLoRU; also please see a conceptual question below.
* The design of FedLoRU seems a straightforward extension to federated setting of existing methods for low-rank matrix accumulation such as ReLoRA [1]. 
* This is not a major weakness but more evaluations on LLM fine-tuning could be done, as most of the experiment details are devoted to computer vision tasks on small datasets.

[1] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. Relora: High- rank training through low-rank updates. In The Twelfth International Conference on Learning Representations, 2023.
We examine the rank structure in federated learning and introduce a new FL algorithm that uses low-rank updates, improving communication efficiency and scalability with a large number of clients.
@inproceedings{
anonymous2024communicationefficient,
title={Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=DdPeCRVyCd},
note={under review}
}
Federated Learning (FL) faces significant challenges related to communication efficiency and heterogeneity. To address these issues, we explore the potential of using low-rank updates. Our theoretical analysis reveals that client's loss exhibits a higher rank structure (gradients span higher rank subspaces of Hessian) compared to the server's loss. Based on this insight, we hypothesize that constraining client-side optimization to a low-rank subspace could provide an implicit regularization effect. Consequently, we propose FedLoRU, a general low-rank update framework for FL. Our framework enforces low-rank client-side updates and accumulates these updates to form a higher-rank model. Additionally, variants of FedLoRU can adapt to environments with statistical and model heterogeneity by employing multiple or hierarchical low-rank updates. Experimental results demonstrate that FedLoRU performs comparably to full-rank algorithms and exhibits robustness to heterogeneous and large numbers of clients.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Federated Learning Communication-Efficient Federated Learning Low-Rank Nature Cross-Device Federated Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/54155b9ee984c5d9b8dac744f98fd97851706a27.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission