Tight Clusters Make Specialized Experts
Mixture of Experts;robustness;clustering
Yes
No ethics review needed.
1.	How does the AC router handle scenarios with overlapping or dynamically evolving clusters where expert specialization may be less clear?

2.	What mechanisms could be integrated to enhance adaptability in routing without relying on previous layer assignments?
1.	The proposed router uses a unique, feature-weighted approach for adaptive clustering, enabling more efficient specialization in MoE models. It achieves improvements without extra learnable parameters or added computation cost, making it ideal for scalable applications.

2.	The paper provides robust theoretical proof supporting the method's faster convergence and increased robustness

3.	The experiments cover various large-scale tasks, such as WikiText-103 and ImageNet, and include backbones like Switch Transformer, GLaM, and Swin Transformers, evaluating performance under both clean and corrupted data conditions.
This paper proposes an Adaptive Clustering (AC) router for Sparse Mixture-of-Experts (MoE) architectures to improve routing efficiency. It focuses on optimizing the matching of tokens to experts by adaptively transforming the input feature space. This method promotes faster convergence, increased robustness against data contamination, and improved overall performance. The authors demonstrate the AC router's effectiveness in large-scale language and image tasks, outperforming baseline routers in robustness and efficiency without added learnable parameters.
1.	The approach seems to tailor to settings with well-structured clusters, which may limit performance on datasets or tasks where latent clusters are less distinct or not aligned with chosen features.

2.	The AC router relies on prior layers for initial expert assignments, potentially constraining flexibility and requiring consistent embedding sizes across layers.

3.	Although tested on large-scale datasets, further application to more dynamic datasets could strengthen claims of robustness and adaptability.
Yes
No ethics review needed.
1. The choice of mean absolute deviation (MAD) as the measure of dispersion is justified, but the paper does not deeply explore the impact of other potential measures, such as variance or interquartile range, on performance. Have you considered or tested other measures of dispersion besides MAD? How sensitive is the method to the choice of dispersion measure, and could alternative measures potentially improve performance or robustness?

2. How does the AC router handle situations where the assignments from previous layers are noisy or not well-defined? Is there a way to initialize or adjust the method to be effective in early layers or when prior assignments are unreliable?

3. Since the method requires consistent embedding dimensions between layers, how can it be adapted for architectures where the embedding size changes, such as in some convolutional neural networks or transformers with variable dimensions?

4. The theoretical analysis relies on Gaussian mixture models. How does the AC router perform when the data clusters have non-Gaussian distributions or are not well-separated?
1. The paper introduces a novel perspective by viewing the MoE routing mechanism through the lens of feature-weighted clustering optimization. This theoretical framing allows for the derivation of optimal feature weights that improve cluster separability.

2. The authors provide rigorous theoretical analysis that support their claims on the improved robustness and convergence speed of their method.

3. Extensive experiments on large-scale datasets for both language modeling and image classification demonstrate the effectiveness of the AC router.

4. The proposed method requires no additional learnable parameters and introduces negligible computational overhead. And it can be integrated into any existing MoE architecture
This paper proposes a novel routing mechanism called Adaptive Clustering (AC) router for Mixture-of-Experts (MoE) architectures. The key idea is to compute token-expert assignments in an adaptively transformed space that better reveals latent clusters in the data. The transformation is derived from a feature-weighted clustering optimization perspective, where features that promote tight clustering for each expert are upweighted. The authors demonstrate both theoretical and empirical advantages of their method, showing improved convergence speed, robustness to data contamination, and overall performance across language modeling and image classification tasks.
1. The AC router relies on expert assignments from the previous layer to compute the adaptive transformation. This dependence may limit its applicability in scenarios where embedding sizes change between layers or in networks without consistent layer structures.

2. The effectiveness of the AC router is closely tied to the quality of expert assignments in earlier layers. Since the method relies on routing decisions from prior layers, it may be less effective if the early layers have not yet learned meaningful or well-separated cluster structures. In practice, early layers in deep networks often learn basic features that may not be semantically distinct enough to form tight clusters. As a result, the AC router may struggle to make optimal routing decisions in later layers if the cluster assignments in the initial layers are poor.

3. The method assumes that the input data naturally clusters in the feature space. While this assumption holds in many cases (e.g., language modeling and image classification), it might not generalize to tasks where the input data is more uniformly distributed or lacks clear clustering patterns.
Yes
No ethics review needed.
no further questions
- the theoretical framework is solid, with clear explanations and remarks on the main body, and abundant material (only skimmed through) in the appendix concerning proofs  of the framework 

- the paper is clearly written and motivated. 

- the experimental section tackles two complementary domains and, whereas has less critical mass than the theoretical part, it is well executed

- extensive material in the appendix complement and completes the paper
the paper studies policies for selection of experts in sparse MoE models, and casts the problem as optimal clustering

the main contributions are 1) to derive closed form expression for the cluster weights. the formulation elegantly extends the classic topk MoE formulation using a diagonal matrix using inverse of cluster dispersion measure,  2) to theoretically bound the incorrect assignment probability and 3) prove faster convergence (though is not clear under which assumptions the prove holds

the paper presents experimental results on  text and image classification, mostly presenting results over baseline models
# necessity of the framework

while the theoretical framework provides and the analysis in Tab 7 shows the method to only provide a rather modest additional complexity,  and the theoretical proofs shows it to be (in a sense) sufficient to solve the routing task, an open question is whether it is also necessary for the task.

otherwise stated, in the case of top-1/top-2 expert selection, an open question is whether it would be possible to significantly simplify the operations to be carried on (same way you do not need a full sort if you only need the maximum value of a vector) to attain similar performance ?

similarly, a more systematic comparison with related competitor work would better allow to appreciate the practical impact.

# structural difference of the solution 

authors do a fair job in visually shown differences in the top experts (Fig 1), but that remains at an anecdotal level.  The load balancing analysis in sec 4.3 provides some complementary view, but is too aggregated (as it looks at measures of variation, but the baseline of perfect load balancing is clearly not a target as it likely would not be attainable).

as such, assessing and quantifying the /differences/ at individual expert level, or input level would provide a factual quantitative assessment of the number of times ACMoE expert selection differs from competing selections. (ie. instead of maps in Fig, providing a distribution of the times where top-k selection differed).  

As in many cases the end-to-end performance improvement is slim (few percentage points) this intermediate assessment could allow to perceive the amount of top-k choices that were changed wrt the baselines models.


# deviation measures

it is unclear to what extent the proofs (and the practical performance) of the framework depends on subtle hyperparameter choices that are fixed and  not considered in the paper: e.g., measure of dispersion used in sec 3 is the mean absolute deviation, which is easier to compute but less robust than median absolute deviation (admittedly, this would go in the opposite direction as for simplification; but is unclear to what extent practical details as this one can have a performance impact in practice).

to have no impact on the selection, the top-k rank would need to be maintained across a family of measures of dispersion -- the impact of which is far from being clear after reading the paper. If the proof holds for families with given properties, the paper could be reinforced by making that clear.
Yes
No ethics review needed.
None
The solution is theoretical based along with extensive evaluation.
The paper presents an Adaptive Clustering (AC) router for MoE models. The Ac router offers faster convergence, better robustness, and overall performance improvement.
- The paper targets MoE experts only, some of the notations and terms seem to be used specifically in the MoE domain.
We propose a novel Mixture-of-Experts routing method that computes token-expert assignments in a transformed space that promotes separation of latent clusters in the data and more easily identifies the best-matched expert for each token.
@inproceedings{
anonymous2024tight,
title={Tight Clusters Make Specialized Experts},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Pu3c0209cx},
note={under review}
}
Sparse Mixture-of-Experts (MoE) architectures have emerged as a promising approach to decoupling model capacity from computational cost. At the core of the MoE model is the router, which learns the underlying clustering structure of the input distribution in order to send input tokens to appropriate experts. However, latent clusters may be unidentifiable in high dimension, which causes slow convergence, susceptibility to data contamination, and overall degraded representations as the router is unable to perform appropriate token-expert matching. We examine the router through the lens of clustering optimization and derive optimal feature weights that maximally identify the latent clusters. We use these weights to compute the token-expert routing assignments in an adaptively transformed space that promotes well-separated clusters, which helps identify the best-matched expert for each token. In particular, for each expert cluster, we compute a set of weights that scales features according to whether that expert clusters tightly along that feature. We term this novel router the Adaptive Clustering (AC) router. Our AC router enables the MoE model to obtain three connected benefits: 1) faster convergence, 2) better robustness to data corruption, and 3) overall performance improvement, as experts are specialized in semantically distinct regions of the input space. We empirically demonstrate the advantages of our AC router over baseline routing methods when applied on a variety of MoE backbones for large-scale language modeling and object recognition tasks in both clean and corrupted settings.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Mixture of Experts robustness clustering
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/43668f51b068eccdeb00c091df2bc6a3badfa666.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/617a28f733b42534b5e88e287076a6abc5537280.zip
Tight Clusters Make Specialized Experts
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission