Encryption-Friendly LLM Architecture
Homomorphic Encryption;PPML;Encrypted Fine-tuning;Transformer Architecture
Yes
No ethics review needed.
For reproducibility, can the authors comment on the source code? Whether they intend to make it public for validation?

Also for reproducibility (and in lack of code), I am interested in understanding better the polynomial approximations used. In Section E the authors talk about penalizing the model in a "pre-training stage". I am not sure I understand how this would work, especially in the context of secure inference (no fine-tuning). What do the authors mean exactly? The model owner retrains the model using this new loss function? Most prior works take a pre-trained model, changing only its non-linearities by polynomial while keeping its weights. What are the authors proposing to do here? Does this require changing the model's weights? Re-training?
The problem statement is well motivated. Several works are currently exploring evaluation of LLMs under FHE, and the potential applications are also quite compelling. This is an extremely difficult task in terms of achieving viable efficiency, and any method that advances the state-of-the-art in this direction is welcome. The results achieved here show that several optimizations in other domains, that is, LoRA and the use of Gaussian Kernels, turn out to be useful for the evaluation of LLMs in FHE. I am not aware of this observation being made and explored in this depth in other papers (it is worth mentioning https://arxiv.org/pdf/2410.00433, which appeared after the submission deadline).
This paper presents optimizations to LLM architectures to make them more friendly towards Homomorphic Encryption evaluation. Concretely, first they propose to use LoRA for fine-tuning, where the data used for fine-tuning is given in encrypted form. This is useful to avoid re-training a lot of parameters, since LoRA focuses on updating only a few weights. This minimizes the amount of homomorphic operations w.r.t. plain fine-tuning. They also observe that LoRA is useful for reducing the dimension of homomorphic matrix multiplications. Secondly, they replace Softmax-based with Gaussian Kernel attention. They show that this is a simpler function to evaluate in HE, leading to significant savings with little impact in accuracy.
I am not particularly impressed by the novelty of this paper. It uses existing FHE tools with existing ML optimizations. This may not be a weakness on its own given the positive results of combining these techniques, but I still think the improvement factors may not be big enough for these techniques to become "enablers" of private LLM applications in practice. Put differently, I am not convinced that the gains here are a significant enough to overcome the blockers that prevent LLMs + FHE from becoming more widely spread.
Yes
No ethics review needed.
N/A
1. Compared to MPC approaches, n on-interactive property helps HE to be feasible to compute over large-scale LLMs without including considerable communication overhead among computing parties.
2. This work has a great focus on the fine-tuning stage to make LLMs secure for users, which also concentrates on the key components like attention layers in the transformer, and it is also combined with SoTA techniques like LoRA to make the process more efficient.
3. Writing with bottleneck-improvement pattern for LoRA and GK looks good for readers to figure out key ideas.
As LLMs surge, privacy issue becomes important considering government-level regulations. Among methods in privacy-preserving machine learning, homomorphic encryption (HE) can provide cryptographic security by computing over encrypted data directly without extra communication like MPC. However, HE is not efficient to compute matmul or non-poly operations in the transformer-level scale. This work focus on fine-tuning stage with LoRA and softmax variant to create HE-friendly transformer architecture.
1. When you mentioned SoTA LLMs, you should notice that decoder-based models have been proved very powerful in generative tasks. After iteration of the recent few years, BERT series is not as useful and prevalent as decoder models. Hence, the significance to protect BERT-based model is less essential in the current LLMs.
2. Although this work introduces how HE and CKKS work in the secure way, this work does not specify adversary model, such ability of adversaries, type of adversaries (e.g., semi-honest or malicious) and kind of attacks (e.g., member inference attack) this architecture counters with.
3. In the conclusion, this work is too vague on the future work. For example, how cryptographic community develops helps the improvement of this work (e.g., any change on HE). Also, how LLMs itself evolve may change security issue based on this work.
Yes
No ethics review needed.
1.	One core idea of this paper to solve the difficulty of evaluating Softmax under HE is to replace it with Gaussian kernel, however, this method is very similar to that in the literature “Chen, Yifan, et al. Skyformer: Remodel self-attention with gaussian kernel and nystr\" om method. Advances in Neural Information Processing Systems 34 (2021): 2122-2135”. However, this paper does not describe the difference with this paper or even cite this literature. Is there a difference in the performance of the two? Furthermore, the polynomial approximation method of this paper seems to be very common way.
2.	In Section 3, is the author's approach in solving Bottleneck 1 just an application of some existing methods? If not, please clarify the differences and improvements.
3.	Does the use of LoRA and Gaussian Kernel affect the interpretability of the model? How does the author balance efficiency and explainability?
This paper is oriented to the problem of inefficiency of transformer architecture under HE, although the existing research has produced richer results. The main contribution of this paper is to enhance the speed of transformer architecture under HE, and the authors have carried out many experiments to verify the rationality and advantages of the scheme. I think the experiments in this paper are full, and the advantages of this paper are elaborated in terms of speed and model performance, which speed is emphasized in this paper. The experimental results are thorough and well-analyzed. Overall, this paper seems to incorporate some of the SOTA approaches in the current field and apply them to a widely researched topic.
This paper starts from the poor performance problem of homomorphic encryption (HE) in transformer architecture and focuses on optimizing the speed of HE in it. Specifically, the scheme tries to avoid CCMM computations to solve the poor performance brought by full fine-tuning. In addition, this paper aims to address the difficulty of evaluating under HE with Softmax, the core idea of which is to replace Softmax with a Gaussian kernel. Finally, this paper carried out many experiments, and the results show that their scheme is comparable to existing schemes in terms of modeling performance, while at the same time, the computational speed has been significantly improved. The overall narrative of the paper is clear, and has good logic.
1. Insufficient innovation. First, the topic chosen for this paper is a more widely studied one. Second, the solutions in this paper seem to be a direct combination and application of existing advanced schemes, and it is not intuitively obvious in the paper that the authors have improved on existing methods.
2. The description in 2.1 does not seem to be consistent with Figure 1. Furthermore, why does the statement “LLM weights are protected in the strict cryptographic sense (line 149)” hold?
3. Although the paper proposes a privacy-protecting LLM architecture, the security considerations of the model, especially against attacks and model theft, is insufficient.
@inproceedings{
anonymous2024encryptionfriendly,
title={Encryption-Friendly {LLM} Architecture},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=pbre0HKsfE},
note={under review}
}
Large language models (LLMs) offer personalized responses based on user interactions, but this use case raises serious privacy concerns. Homomorphic encryption (HE) is a cryptographic protocol supporting arithmetic computations in encrypted states and provides a potential solution for privacy-preserving machine learning (PPML). However, the computational intensity of transformers poses challenges for applying HE to LLMs. In this work, we propose a modified HE-friendly transformer architecture with an emphasis on inference following personalized (private) fine-tuning. Utilizing LoRA fine-tuning and Gaussian kernels, we achieve significant computational speedups---6.94$\times$ for fine-tuning and 2.3$\times$ for inference---while maintaining performance comparable to plaintext models. Our findings provide a viable proof of concept for offering privacy-preserving LLM services in areas where data protection is crucial.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Homomorphic Encryption PPML Encrypted Fine-tuning Transformer Architecture
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9abcc2fe5a28d463c4076517df2ae8bd2cc3756e.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Encryption-Friendly LLM Architecture
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission