MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts
Mixture of Experts;Large Language Models;Efficient Foundation Models
Yes
No ethics review needed.
1. How do individual components, like zero experts or gating residuals, contribute to MoE++'s overall performance?
2. Could certain components be removed or modified for specific use cases?
With the proper hyper-parameters, MoE++ introduces zero-computation experts who can reduce the computational load by bypassing or simplifying processing for certain tokens, leading to efficient resource use.
These heterogeneous experts with suitable routing designs can improve the model's efficiency and effectiveness.
This paper introduces the MoE++, which adds zero-computation experts to enhance the efficiency of computation.
It utilizes zero-computation experts to minimize overhead by allocating fewer resources to basic tokens and concentrating on complex ones. MoE++ enhances expert selection stability by employing pathway-aware routing with gating residuals, resulting in higher performance and throughput than conventional MoE models at a reduced computational cost.
1. Parameters such as $\tau$, which regulate token allocation between zero-computation experts and original, may complicate model tuning, as the performance and burden distribution are sensitive to them.  
2. The pathway-aware routing with gating residuals adds complexity to the expert selection process, which may require careful tuning for optimal results.
3. The dynamic routing mechanism can still result in load imbalances, especially under diverse data distributions, which may lead to underutilized or overloaded experts that affect efficiency.
Yes
No ethics review needed.
Are the baseline MoE models also trained with residual gating? For each model, many experts are activated in each layer? I could not really tell.
The authors provide a fairly simple and effective addition to the standard MoE architecture which effectively allows more heterogeneous computing in different layers with minimal overhead. The paper is written clearly and a thorough set of ablations are performed.
This paper proposes the addition of a new class of experts in the MoE architecture called "zero-computation" experts. The zero-computation type experts include a zero expert, a copy expert, and constant experts. Each of these experts is extremely computationally lightweight relative to a standard FFN expert. The authors demonstrate that incorporating this expert type is both GPU friendly due to the low overhead and can improve performance and throughput.
The empirical gains don't seem to be too significant and can only start to be seen at large scale ~7B parameters.
Yes
No ethics review needed.
- How are these zero-computation experts specialized to input tokens? Specifically, it would be great to show some examples of tokens allocated to zero, copy, and constant experts, respectively.
- This work improves the existing MoE framework in terms of both efficiency (throughput) and effectiveness (performance), making it impactful for real-world applications.
- This paper is well-written and insightful, with clear motivation and illustrations.
- Figures and tables are clear and easy to read.
This paper introduces MoE++, a general and heterogeneous MoE framework that uses a fixed mixing mechanism for all tokens and optimizes computation allocation by assigning fewer FFN experts to simple tokens, allowing more FFN experts to be dedicated to challenging tokens. Extensive experimental results demonstrate the lower computational overhead and better performance of MoE++ than vanilla MoE.
- While Table 1 shows the “complexity between the proposed MoE++ and MoE” and zero-computation experts enjoy a complexity of 0, they still likely lead to some extra computation overhead. Therefore, this work lacks real-world wall-clock time demonstrations of these zero-computation expert operators, especially regarding a batch of tokens.
- This work introduces three types of zero-computation experts (i.e. zero, copy, and constant) but provides minimal justification for this specific set. From Table 5, we can see they only conducted basic combinatorial ablations.
We propose MoE++, a general and heterogeneous mixture-of-experts framework that achieves better performance while delivering 1.1$\sim$2.1$\times$ expert forward throughput compared to a vanilla MoE model of the same size.
@inproceedings{
anonymous2024moe,
title={MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=t7P5BUKcYv},
note={under review}
}
In this work, we aim to simultaneously enhance the effectiveness and efficiency of Mixture-of-Experts (MoE) methods. To achieve this, we propose MoE++, a general and heterogeneous MoE framework that integrates both Feed-Forward Network (FFN) and zero-computation experts. Specifically, we introduce three types of zero-computation experts: the zero expert, copy expert, and constant expert, which correspond to discard, skip, and replace operations, respectively. This design offers three key advantages: (i) **Low Computing Overhead**: Unlike the uniform mixing mechanism for all tokens within vanilla MoE, MoE++ allows each token to engage with a dynamic number of FFNs, be adjusted by constant vectors, or even skip the MoE layer entirely. (ii) **High Performance**: By enabling simple tokens to utilize fewer FFN experts, MoE++ allows more experts to focus on challenging tokens, thereby unlocking greater performance potential than vanilla MoE. (iii) **Deployment Friendly**: Given that zero-computation experts have negligible parameters, we can deploy all zero-computation experts on each GPU, eliminating the significant communication overhead and expert load imbalance associated with FFN experts distributed across different GPUs. Moreover, we leverage gating residuals, enabling each token to consider the pathway taken in the previous layer when selecting the appropriate experts. Extensive experimental results demonstrate that MoE++ achieves better performance while delivering 1.1$\sim$2.1$\times$ expert forward throughput compared to a vanilla MoE model of the same size, which lays a solid foundation for developing advanced and efficient MoE-related models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Mixture of Experts Large Language Models Efficient Foundation Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d2ca5c9a11b53151adf0a0099a74b80e379be9e1.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission