SpecFuse: Ensembling Large Language Models via Next-Segment Prediction
Ensemble LLMs;Next-Segment Prediction;Generative;open-domain instruction response
Yes
No ethics review needed.
- For the MMLU and ARC-C datasets, the answer is one of the provided options -- what is the sequence being decoded in these cases? How long are the generations?
- The main idea behind SpecFuse is intuitive and interesting -- and empirical results show its effectiveness.
- SpecFuse does not require training any additional modules and can be plugged into any set of existing models.
- Several datasets are used for evaluation and there seems to be a consistent gain across all of them.
This paper introduces SpecFuse, a novel technique for ensembling LLMs by generating multiple segments in parallel and reranking them using the average log probabilities assigned by each model. The paper also introduces an early exit mechanism which prunes out LLMs from the ensemble which do not contribute the maximum scoring segments.

Results show that this ensembling technique leads to notable improvements when combining base LLMs in the 7-9B parameter range, often outperforming larger models in the 70B range.
- A very similar technique called Mixture of Agents [1] was introduced recently, which the paper does not mention. The key difference is that this paper divides the response into chunks before aggregating, whereas MoA does it at the response level. Nevertheless, a comparison between the methods would be useful.
- The main evaluation is done on the Alpaca-GPT4 datasets -- where the ground truth references come from GPT-4 responses. This is a curious choice -- the standard way of evaluating instruction following in models is via AlpacaEval, where a separate LLM evaluator is used to grade the responses. Such an evaluation would help situate the proposed method with other techniques proposed in the literature (e.g., MoA above).
- The model exit mechanism seems rather ad-hoc. It would be useful to provide an ablation study showing that the different components (e.g., cumulative quality score, entropy) are indeed needed. A comparison to simpler baselines (e.g., remove all models which do not contribute in the first n segments) would also help.

[1] Wang, Junlin, et al. "Mixture-of-Agents Enhances Large Language Model Capabilities." arXiv preprint arXiv:2406.04692 (2024).
Yes
No ethics review needed.
1. Regarding Weakness 2: Could you elaborate on the computational trade-offs and whether the observed improvements are justified by the additional resource requirements?
2. Related to Weakness 3: Can you include comparisons with other existing ensemble inference enhancement techniques?
3. Have you considered testing SpecFuse on more representative or real-world datasets to better illustrate its practical effectiveness?
1. SpecFuse introduces a novel and practical approach for model ensembling by enabling cooperative generation among multiple LLMs, thereby mitigating the low-quality outputs often associated with single-model approaches.
2. Experimental results highlight significant performance improvements across various tasks, including instruction-response, reasoning, and commonsense benchmarks.
3. The methodology is clearly articulated, with diagrams effectively illustrating the operational mechanics of the framework.
This paper introduces an innovative framework called SpecFuse, designed to ensemble LLMs through next-segment prediction. SpecFuse operates by iteratively executing its inference and verification components. In each round, candidate segments generated by base LLMs are ranked, and the top-ranked segment is broadcast to all models to facilitate collaborative generation and enhance response quality. The framework also incorporates a “Model Exit Mechanism” that dynamically excludes underperforming models to optimize computational resources. Experimental results demonstrate that SpecFuse achieves superior performance compared to individual models and traditional ensemble methods across six benchmark datasets,.
1. SpecFuse relies on accessing logits, which limits its applicability to models that require API-based interactions, thus constraining its usage.
2. The framework demands substantial additional computational resources, but the paper lacks a comprehensive discussion on this aspect. Additionally, it is necessary to evaluate whether the performance gains justify the increased resource consumption.
3. The paper does not provide comparisons with other ensemble inference enhancement methods, which would contextualize the advantages of SpecFuse more effectively.
Yes
No ethics review needed.
In addition to those I mentioned in Summary

1) Can you compare the proposed methods with Best-of-N (an extra reward model might be needed)?
1. A new approach for LLM ensemble.
2. Extensive empirical studies across various benchmarks.
The paper proposes a new approach to leveraging multiple LLMs for better performance. The authors divide generation of a response into several rounds (each round corresponding to generation of  a segment). In each round, the output of a model is judged by all the LLMs, and the output with the highest score is selected as the segment for the round. To balance efficacy and efficiency, the authors also propose an exit mechanism, where models with scores lower than a threshold are removed from ensemble. Empirical studies are conducted on a variety of benchmarks, and evaluation results look good to indicate the effectiveness of the proposed approach. 

Here are my concerns:
1) As 1/3 contributions, the exit mechanism does not receive enough analysis in empirical studies. The authors only show that the mechanism leads to slight performance drop, and mention that it also significantly reduces the number of models invoked in Abstract and Introduction, but how does this mechanism work? Will awkward models be kicked out as expected for specific queries? How does the number of models vary with respect to different tasks and different queries? Given that there is a lot of heuristics in the mechanism, it is important to clearly reveal how the mechanism works and justify why it should be designed like that. ( Note that I just list a few questions here, it is better to have a comprehensive study regarding to the mechanism).
2) The authors categorize existing work into two groups: post-hoc and pre-selection, but only involve one post-hoc method in comparison. Why there are no pre-selection methods (e.g., based on a certain router) selected? It is also better to involve more post-hoc methods as baselines.
3) I also worry about latency, especially when responses become long (e.g., math with COT) and models become large (e.g., from 7B to 70B). Can you directly compare different methods in terms of time and resources required for a query, and discuss how the metrics vary according to responses in different lengths? I am also confused about the concept "first token latency", can you clearly define it?
Please refer to my concerns in Summary.
We introduce SpecFuse, a novel ensemble framework that generates fused outputs by iteratively producing the next segment through collaboration among LLMs, allowing base LLMs to be seamlessly integrated without any training or adaptation.
@inproceedings{
anonymous2024specfuse,
title={SpecFuse: Ensembling Large Language  Models via Next-Segment Prediction},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=lhLQpS33YL},
note={under review}
}
Ensembles of generative large language models (LLMs) can integrate the strengths of different LLMs to compensate for the limitations of individual models.
However, recent work has focused on training an additional fusion model to combine complete responses from multiple LLMs, failing to tap into their collaborative potential to generate higher-quality responses.
Moreover, as the additional fusion model is trained on a specialized dataset, these methods struggle with generalizing to open-domain queries from online users.
In this paper, we propose SpecFuse, a novel ensemble framework that outputs the fused result by iteratively producing the next segment through collaboration among LLMs.
This is achieved through cyclic execution of its inference and verification components.
In each round, the inference component invokes each base LLM to generate candidate segments in parallel, and the verify component calls these LLMs again to predict the ranking of the segments. 
The top-ranked segment is then broadcast to all LLMs, encouraging them to generate higher-quality segments in the next round. 
This approach also allows the base LLMs to be plug-and-play, without any training or adaptation, avoiding generalization limitations.
Furthermore, to conserve computational resources, we propose a model exit mechanism that dynamically excludes models exhibiting poor performance in previous rounds during each query response.
In this way, it effectively reduces the number of model calls while maintaining overall performance.
We conduct extensive experiments using ensembles of five LLMs with different architectures across six benchmarks, covering instruction-response, reasoning, commonsense, and instruction-following tasks. The experimental results demonstrate that SpecFuse consistently enhances performance across all benchmarks, with RougeL scores improving by $+3.1$ on the Chinese and $+3.0$ on the English human-computer interaction benchmarks. Furthermore, the model exit mechanism reduces the average models invoked per round from $5$ to $2.4$, with only a slight reduction in performance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Ensemble LLMs Next-Segment Prediction Generative open-domain instruction response
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/044e5385b477a7c7f068f15e74ce0f91900a7b06.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/40dce560a55825efefce42bb6792f381e3f57edd.zip
SpecFuse: Ensembling Large Language  Models via Next-Segment Prediction
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission