Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding
Large language model; Knowledge fusion; Speculative decoding
Yes
No ethics review needed.
- In 3.2 Verification, for Tree-Based Verification, you claim to use benchmark datasets such as GSM8K to train the classifier, but then in your test, you incorporate the GSM8K dataset as well. Is there any information leakage in terms of that you are training your verifier on the test set so that it gains advantage over other models?
- The authors have put effort in experimenting their proposed framework under different setups, including various draft and assistant models, different simulated scenarios, etc.
- The proposed framework gains some advantage over the existing framework of Co-LLM in certain scenarios.
This paper proposes Collaborative Speculative Decoding (CoSD) that fuses LLM knowledge at test time. The algorithm employs a draft model to generate initial response sequences and a rule-based or decision tree to decide when to leverage an assistant model to improve the drafts.
The authors have conducted experiments using different pairs of LLMs and under various experimental setups.
- In Table 2, I notice that in most cases, the fused model underperforms the draft model and the assistant model.
For instance, for Pair 1, none of the fusion methods outperform both draft and assistant model for GSM8K, HumanEval; for Pair 2, none of the fusion methods consistently outperform both draft and assistant model for GSM8K, and MMLU.
Then I wonder what is the point of fusing knowledge in these cases if we can simply adopt one model instead of the other?

- It seems that for Pair 3, CoSD-Rule performs exceptionally well on GSM8K, yielding 45.47 while the draft and assistant models yield 25.01 and 35.43, which is very different from the performance patterns for this same pair on other datasets such as MMLU and also other pairs. Could you give more insights into such a result? Could you present some examples that CoSD-Rule excel at under this situation that cannot be addressed by either the draft nor the assistant model?
Yes
No ethics review needed.
1. Is the proposed algorithm suitable for collaboration among multiple LLMs? What will be the potential challenges?
2. Can you explain more about the limitations of the current method? I'm curious when it doesn't work well.
1. This paper provides an interesting perspective to fuse knowledge between LLMs using speculative decoding, which leverages the strengths of different LLMs while still keeping the efficiency.
2. The experiment setting is interesting, which tries complementary knowledge fusion, catastrophic forgetting recovery, capacity imbalance and different tokenizers.
This paper introduces a novel collaborative speculative decoding algorithm which can efficiently fuse the knowledge from different LLMs during inference. The experiment setting is quite interesting and includes different types: complementary knowledge fusion, catastrophic forgetting recovery, capacity imbalance and different tokenizers. The results are better than different baselines.
1. The paper only does the experiment in each pair of the LLMs. It would be interesting to see more LLMs collaboratively fuse knowledge.
2. It would be better to show more details about the limitations of the proposed method and show some error analysis.
Yes
No ethics review needed.
- "the algorithm regenerate and re-verify iteratively until all tokens are accepted" How many iterations does it take on average? 
- during the training process of the decision tree, if neither the draft model's generation nor the assistant model's generation match the target, you drop the sample and continue the loop with i &larr; i+1. Any ideas of improvement other than simply dropping these samples?
- typos: line 287, "tree" to "three", "drat" to "draft"
- interesting and inspiring idea on fusing knowledge at the decoding time
- The algorithm is clearly presented in this paper through both a workflow diagram and mathematical expressions.
- Both Rule-Based verification and Tree-Based verification are well-designed and both make sense to me.
This paper introduces the Collaborative Speculative Decoding (CoSD) algorithm that enables efficient LLM knowledge fusion during decoding. Upon the idea of speculative decoding, the paper proposes two novel decision-making rules: Rule-based and Tree-Based. The method features 1) efficiency (parallel generation, no additional model training), 2) transferability across different domains and models with different tokenizers, and 3) interpretability. CoSD successfully improves baselines by up to 10%.
- I'm not sure if the goal of this algorithm is to A) achieve performance comparable to the assistant model but in a more efficient way, or if it's aimed at B) outperforming both the draft model and the assistant model individually (1+1>2). How do these objectives apply to the four scenarios of knowledge fusion discussed in section 4.1? If the goal is A, since the draft models in complementary knowledge fusion and catastrophic forgetting recovery scenarios are about the same size as the assistant model, and the algorithm involves autoregressive generation of the draft model, I doubt the algorithm improves efficiency.  If the goal is B, I can't see improvement based on Table 2.
@inproceedings{
anonymous2024speculate,
title={Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=2Y6xGE1K60},
note={under review}
}
Large Language Models (LLMs) often excel in specific domains but fall short in others due to the limitations of their training. Thus, enabling LLMs to solve problems collaboratively by integrating their complementary knowledge promises to improve their performance across domains. To realize this potential, we introduce a novel Collaborative Speculative Decoding (CoSD) algorithm that enables efficient LLM knowledge fusion at test time without requiring additional model training. CoSD employs a draft model to generate initial sequences and an easy-to-learn rule or decision tree to decide when to invoke an assistant model to improve these drafts. CoSD not only enhances knowledge fusion but also improves inference efficiency, is transferable across domains, and offers greater explainability. Experimental results demonstrate that CoSD improves accuracy by up to 10% across benchmarks compared to existing methods, providing a scalable and effective solution for LLM-based applications.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large language model; Knowledge fusion; Speculative decoding
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8c7bc724e9f1abc87d659e8c61bf52a690295352.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission