MPC-Minimized Secure LLM Inference
secure inference;secure multi-party computation (MPC);transformer;large language model (LLM);open-source foundational model;fine-tuning;LoRA;head-merging
Yes
No ethics review needed.
The use-case considered here somewhat differs with what has appeared in prior works. Considering public models first posses the question: what does it mean to "fine-tune on private data"? who owns this data? this makes the most sense only if the data is held by a collection of parties, since if it's held by only one party, what is preventing this party from performing the fine-tuning themselves? However, in this case, the fine-tuning would this an MPC protocol among who? Who runs the secure computation? I think these details are unclear and are important to understand the context, and the associated inefficiencies. In particular, at the end of the fine-tuning the model is "private" (well, part of it), so this probably means shared among at least two parties... who is the client then who performs the query at inference time? If it's an additional party than this has become a 3-party protocol. These details matter and they are left unspecified.

Why do you cite [73] as the SOTA for 2PC+Dealer? Earlier in the text you refer to Sigma as such SOTA (which makes sense).

NOTE: SIGMA appeared at PoPETs 2024. Check this and other preprint references.
The ideas considered are sound, and the authors provide an implementation for reproducibility (which, disclaimer, I didn't verify myself). The idea of head-merging is, to the best of my knowledge, novel. It is relevant as it seems to work well in terms of accuracy, but it boosts the efficiency of MPC noticeably. The experiments are comprehensive and thorough. They are well described and enable good reproducibility.

Overall the paper is well written.
This paper considers a setting where a public open-source model is fine-tuned on private data so that the resulting (private) model can be used for private inference. They key contributions is the design of a series of optimizations that make it so that the "private part" of the fine-tuned model is comparatively small w.r.t. the full model, and furthermore is located only towards the final layers of the architecture. This allows the client with the private query to "preprocess" most of the model computation in the clear, given that the first layers are all public. Then, the MPC protocol would only need to execute the final private layers.

The optimizations include (1) freezing many of the initial layers prior to fine-tuning, (2) merging some of the attention heads to minimize the impact of their evaluation under MPC, and (3) using LoRA to reduced the dimension of the involved matrices in MPC. They show that (different combinations of) these can lead to reasonably small degradation loss wr.t. full fine-tuning while leading to substantial efficiency gains in the MPC context.
I am not sure the problem is very well motivated. This is not about running inference obliviously, which is very well motivated. This is is what most (if not all) of the cited prior works consider: a model owner wants to keep their model private and a client wants to keep the query hidden, while still learning the inference result. This works starts from the premise that the model that the client may want to query is actually a fine-tuned version of an open-source model, the the data for fine-tuning is private. I am not sure about in which category this falls and what types of use-cases it solves. I will elaborate on this in my questions below. However, in general, I don't think the setting is clear and I am unconvinced whether it is really that relevant (in particular, since the paper cites as baselines works that study a fundamentally different problem).

The claimed proof in the appendix is very high level and feels more like a "checkbox" rather that anything that is supposed to shed any light. In particular, there are several details unspecified which an actual proof would need to deal with. For instance, the functionality is inherently defined over real values, but all of the protocols cited in the paper operate over fixed point values. There's no way then to instantiate the functionality as written. Furthermore, most of the cited protocols use probabilistic truncation for fixed-point arithmetic, which is not provably secure (https://www.usenix.org/conference/usenixsecurity23/presentation/li-yun).
Yes
No ethics review needed.
Please see Weaknesses.
The paper describes three techniques to minimize the use of MPC: Layer Freezing, Low-Rank Adaptation (LoRA), and Head Merging. These techniques reduce the cost of MPC in secure inference.
The paper presents a secure inference framework that combines MPC (Multi-Party Computation) and LLM, aiming to address privacy issues in LLM-based inference services by minimizing the use of MPC during secure inference. The proposed method reduces the number of costly operations required during MPC by introducing high-level architectural changes during the fine-tuning process.
1-The paper's innovation is limited. The proposed method does not include improvements to the MPC protocol itself but focuses on different partitioning and head merging methods in the fine-tuning network structure. Regarding the Layer Freezing method in the partitioning approach, is there any reference basis for selecting how many layers to freeze for different LLMs to achieve a balance between accuracy and timeliness?

2-Insufficient comparison of accuracy in experimental results. The paper does not compare the accuracy of other methods that combine MPC and LLM. Please supplement with more relevant experiments.
Yes
No ethics review needed.
1. In the real-world enterprise-level LLMs, the system usually takes millions of requests at a short time. So, instead of 2-PC or 3-PC setting, have you considered the extension of this work to n-PC setting or distributed setting?
2. According to Weakness 1, if the faction f is too large or too small, what will it happen?
3. Can you provide more information about the difference between head-pruning and proposed head-merging.
1. This work innovatively introduces layer freezing and LoRA techniques from plaintext inference to secure inference. Especially it reduces bottlenecked matmul dimensions in MPC-based inference.
2. This work well describes threat models in secure inference of LLMs and explains why each component can be optimized with considering potential attacks. Also, it explains why some parts cannot benefit from optimization by nature of MPC, such as mixture of weights.
Inference over LLMs  has risks to leak private information such as prompts while LLMs are prevalent in many downstream AI tasks nowaday. MPC-based approaches have been well studied in the literature to tackle the issue. However, due to expensive overheads incurred by MPC, this work focuses on architectural changes to reduce expensive MPC operations to make the inference of LLMs efficient while preserving privacy.
1. In Section 5.1, authors do not explain the criteria or threshold of the frozen fraction f.
2. Since only 2PC-Dealer supports GPU acceleration, the comparison to other MPC approaches with CPU only looks unfair due to the hardware difference. If we want to do apple-to-apple comparison, it is better to make 2PC-Dealer run with CPU only, too.
@inproceedings{
anonymous2024mpcminimized,
title={{MPC}-Minimized Secure {LLM} Inference},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=beAlX6RjsW},
note={under review}
}
Many inference services based on large language models (LLMs) pose a privacy concern, either revealing user prompts to the service or the proprietary weights to the user. Secure inference offers a solution to this problem through secure multi-party computation (MPC), however, it is still impractical for modern LLM workload due to the large overhead imposed by MPC. To address this overhead, we propose
MARILL, a framework that adapts LLM fine-tuning to minimize MPC usage during secure inference. MARILL introduces high-level architectural changes during fine-tuning that significantly reduce the number of expensive operations needed within MPC during inference, by removing some and relocating others outside MPC without compromising security. As a result, MARILL-generated models are more efficient across all secure inference protocols and our approach complements MPC-friendly approximations for such operations. Compared to standard fine-tuning, MARILL results in $2.2−11.3\times$ better runtime and $2.4−6.9\times$ better communication during secure inference across various MPC settings, while typically preserving over $90$% performance across downstream tasks. Anonymous code is available at
https://anonymous.4open.science/r/MPC-auto-B100.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
secure inference secure multi-party computation (MPC) transformer large language model (LLM) open-source foundational model fine-tuning LoRA head-merging
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7a6e63198862dbe72cfbd7f4be26fed272865920.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
MPC-Minimized Secure LLM Inference
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission