BLIMEY: Towards Better Routing Methods in Sparse Mixture of Experts
mixture-of-experts;neural networks;large-scale machine learning;routing algorithms;interpretability;scaling laws;computational efficiency;nlp
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
@misc{
soboleva2024blimey,
title={{BLIMEY}: Towards Better Routing Methods in Sparse Mixture of Experts},
author={Daria Soboleva and Esha Singh and Nolan Simran Dey and Joel Hestness},
year={2024},
url={https://openreview.net/forum?id=ZEC0oBtzhN}
}
Mixture of Experts (MoE) architectures offer a promising avenue for scaling neural networks by facilitating parameter-efficient model expansion while optimizing FLOP utilization. However, several challenges persist, including sub-optimal expert utilization, vanishing gradients, sub-par expert specialization, and inconsistent routing decisions. We introduce BLIMEY, a novel diagnostic framework that systematically quantifies these critical issues, enabling design and implementation of more robust and efficient expert routing algorithms. BLIMEY equips researchers with granular, interpretable metrics on MoE dynamics, explaining both the performance advantages over monolithic architectures and the persistent algorithmic bottlenecks, thereby offering a comprehensive diagnostic framework for MoE optimization. Leveraging BLIMEY, we establish new scaling laws, that surpass established benchmarks like Chinchilla, achieving 3x reductions in FLOPs and 5\% performance gains. Furthermore, our framework unveils significant optimization potential in routing algorithms, revealing sub-optimal expert specification and load imbalances in current methodologies. To accelerate innovation in MoE routing and computational efficiency, we have open-sourced BLIMEY framework, including its diagnostic tools and implementation libraries.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Daria_Soboleva1 ~Esha_Singh1 ~Nolan_Simran_Dey1 ~Joel_Hestness2
Daria Soboleva Esha Singh Nolan Simran Dey Joel Hestness
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
mixture-of-experts neural networks large-scale machine learning routing algorithms interpretability scaling laws computational efficiency nlp
I certify that there is no acknowledgement section in this submission for double blind review.
soboleva|blimey_towards_better_routing_methods_in_sparse_mixture_of_experts
/pdf/b36094f6f01b2bce4fcae72c35c71eec04f2fdb4.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
No
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
BLIMEY: Towards Better Routing Methods in Sparse Mixture of Experts
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission