Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling
Model ensembling;LLM
Yes
No ethics review needed.
- Could you provide more theoretical support or intuition for why limiting alignment to the top-k tokens effectively enhances ensemble performance? How does this approach balance between accuracy and computational efficiency at a probabilistic level?
- How does UNITE handle models with markedly different response styles in practice? Would introducing a preprocessing step to standardize response formats (e.g., for tasks like QA or summarization) enhance compatibility?
- Since UNITE is partially motivated by efficiency, adding a comparative breakdown of memory and latency for each method would clarify the computational trade-offs involved. Including charts or tables that detail average and worst-case latency per token would help underscore UNITE’s operational benefits.
- The paper identifies compatibility challenges in LLM ensembling and focuses on top-k tokens, aligning this strategy with empirical evidence that vocabulary alignment often introduces computational inefficiencies. 
- The authors conduct extensive experiments on multiple models and benchmarks, analyzing factors like vocabulary size, response style, and latency. The results support UNITE’s superiority in maintaining high performance while minimizing latency and token manipulation.
-  The proposed determine-then-ensemble strategy offers a generalizable framework for selecting compatible models, making the findings applicable to real-world LLM applications that require efficient model collaboration.
This paper introduces a novel ensembling approach, UNITE (Union Top-k Ensembling), that efficiently integrates large language models (LLMs) by focusing on the union of top-k tokens from each model rather than aligning the full vocabulary. It seeks to improve the computational efficiency and effectiveness of LLM ensembles by addressing key issues of compatibility, vocabulary size, and response styles. The authors propose a model selection strategy to identify compatible models, limiting the influence of incompatible LLMs on the ensemble's performance. Experimental results across multiple benchmarks validate the benefits of UNITE in enhancing performance, reducing latency, and decreasing the computational burden.
- While the paper addresses task-specific challenges, it could benefit from deeper exploration into why certain tasks (like MMLU) see greater performance improvements than others. Further insight into how task characteristics impact ensembling effectiveness would add depth to the analysis.
- The model selection process relies on response style consistency and performance alignment within a 10% range, which may limit scalability when dealing with a large pool of candidate models. The method would benefit from a more automated or quantitative metric for determining compatibility.
- While UNITE is evaluated across standard datasets, some benchmarks like GSM8K and TriviaQA may not fully capture the diverse range of LLM applications. Including more varied tasks could strengthen the argument for UNITE’s general applicability.
Yes
No ethics review needed.
Does the optimal value of k for the top-k tokens vary across tasks and domains? Is there a principled way to determine k?
- The paper presents concrete empirical conclusions about what factors are actually important for ensembling (performance gap, response style) or not (vocabulary size difference.)
- It presents UNITE, a novel top-k union ensembling approach that drops the computational overhead tremendously by using the top-k tokens of each model instead of the full vocabulary, and that obviates the need for full vocabulary alignment.
- UNITE significantly outperforms the baselines (LLM-Blender, DeePen, GAC) in average accuracy across six popular benchmarks for question-answering, reasoning, and knowledge.  
- The paper is clear and well written.
The paper identifies and tests three hypotheses about importance factors that influence the performance of logit-based LLM ensembles: performance discrepancy, vocabulary size differences, and stylistic response differences, and provides guidelines for choosing models to ensemble with. The authors propose UNITE, an ensembling method that uses the top-k logits from each model, and show that it outperforms numerous ensembling baselines while being significantly cheaper.
- The analysis in Conclusion II on vocabulary size differences is restricted to English language tasks. It may benefit from a discussion of multilinguality, where tokenization might be less consistent between models.
- In Conclusion III the paper identifies differences in response style as a major problem for ensembling models. The proposed solution of limiting longer responses to 2x the length of shorter ones would benefit from theoretical justification. A robust solution to this problem which enables ensembling of models with different response styles would be ideal, as the current approach limits the practicality of the method.
Yes
No ethics review needed.
Like Weaknesses#2 above, can you provide reasons or analysis why the top-k approach is better in performance than the whole vocabulary on performance?
1. It includes good analysis and preliminary experiments for LLM ensembling.
2. Based on the preliminary analysis, they reduce the vocabulary for the previous method into the top-k selection for the model ensembling.
This paper proposed an LLM ensemble method, which is to ensemble among only top-k tokens probability.
1. I believe the experiments should include the efficiency compared to the previous approach.
2. It seems, in theory, that including the whole vocabulary should work better, although maybe marginally or at least the same compared to top-k, since it includes the whole picture of the token distribution. I am questioning your experimental results because top-k is better in every dataset, which logically cannot be the case.
Yes
No ethics review needed.
1. Do you have an insight about why increasing k beyond 10 does not improve perf? I think this result is counter-intuitive, and needs more analysis.
2. What is the impact of base model perf difference when using UNITE?
1. UNITE’s token-level aggregation without full vocabulary alignment is an innovative method which reduced computational needs and making ensembling more efficient.
2. Empirical results are shown over a concrete set of benchmarks.
3. Their focus on model selection strategy brings up insightful practical guidelines which are very useful in practice.
4. They present clear results comparing latency against existing methods.
Paper has two main contributions:

1. Presents a thorough analysis of features affecting ensemble performance: bade model performance gap, vocabulary size, and response style consistency.
2. A new ensembling method, UNITE, that focuses on combining only the top-k tokens instead of the entire vocabulary. The method is efficient in runtime and shows a high performance across multiple benchmarks.
1. Lack of analysis on possible limitations or settings or benchmarks where topk-k token alignment fails to improve base models perf.
2. Figures and tables could have more detailed captions with information to be self-contained. (ex figures 4,5 and 6)
3. Figures and plots' font size are very small. Consider increasing the font size to assists readers.
This study introduces a general model selection strategy for ensembling and proposes an efficient ensemble method that operates on the top-k candidate tokens.
@inproceedings{
anonymous2024determinethenensemble,
title={Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=FDnZFpHmU4},
note={under review}
}
Large language models (LLMs) exhibit varying strengths and weaknesses across different tasks, prompting recent studies to explore the benefits of ensembling models to leverage their complementary advantages. However, existing LLM ensembling methods often overlook model compatibility and struggle with inefficient alignment of probabilities across the entire vocabulary. In this study, we empirically investigate the factors influencing ensemble performance, identifying model performance, vocabulary size, and response style as key determinants, revealing that compatibility among models is essential for effective ensembling. This analysis leads to the development of a simple yet effective model selection strategy that identifies compatible models. Additionally, we introduce the \textsc{Uni}on \textsc{T}op-$k$ \textsc{E}nsembling (\textsc{UniTE}), a novel approach that efficiently combines models by focusing on the union of the top-k tokens from each model, thereby avoiding the need for full vocabulary alignment and reducing computational overhead. Extensive evaluations across multiple benchmarks demonstrate that \textsc{UniTE} significantly enhances performance compared to existing methods, offering a more efficient framework for LLM ensembling.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Model ensembling LLM
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/20a30109f8d15c2d515a097734ea18f297195ff6.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission