Optimized Multi-Token Joint Decoding With Auxiliary Model for LLM Inference
LLM Inference;Speculative Decoding
Yes
No ethics review needed.
target model (larger model) is fine-tuned on chat-GPT data for proposed method, is the same fine-tuned model used for other Speculative Decoding (SpD) methods as well? 

What about the draft model? is that fine-tuned as well or used as is? 

Can qualitative generations be shown in the paper? There are cases when lower PPL is observed during SpD as the draft and target model are both generating and accepting repeated tokens.

Did the author also check performances (PPL, speed-up) when using a larger draft model? 

From table 4., it seems like for past SpD methods there is no correlation between high speed (token/s) will lead to low energy (Joule/s), but that doesn't seem to make sense, as both high speed and low energy should be related to high acceptance rate which leads to small number of target model calls. Any comment/thoughts on this? 

Can the authors also add plots on acceptance threshold vs acceptance rate? Given the high token-rates shown in Table 4., I am curious to see how the acceptance rates look like.
The paper solves an interesting problem of sampling from a joint distribution which is challenging and proposes an alternate approximate method based on draft model from Speculative Decoding (SpD) 

It uses ideas from SpD to also boost the inference speed.

Paper uses an auxiliary model to predict joint distribution of multiple tokens w/ beam sampling to sample the tokens fast. 

The power analysis provides some insight into the importance of using SpD

Authors provide a robust set of experiments
The paper proposed multi-token assisted decoding by combining multi-token joint decoding and speculative decoding (SpD) to improve the quality of the tokens and inference speed.
the experiment results can be re-checked, my concern is that lower PPL not always results in better quality, it could also be that the draft model is generating repeated tokens which the target model is accepting, given that there's no guarantee that the draft tokens accepted will follow the target distribution. 

the reduction in energy consumption is a by-product of SpD and could have been mentioned in a sub-section rather a full section. 
The acceptance rate of draft model should affect both the token-rate and the energy consumption.

No mention of draft model or target model training and if they are different from the baseline algorithms draft and target model in the experiments section.
Yes
No ethics review needed.
1. The threshold for the verification phase of vanilla speculative decoding is sampled from U(0, 1) to ensure losslessness, and MTAD uses a fixed threshold (e.g. 0.1). I'm curious to see if the same speedup can be achieved with the vanilla speculative decoding method using a fixed threshold.

2. The impact from loose thresholds has little effect with PPL and LM-Judger metrics, some more rigorous evaluation (e.g., Humaneval's Pass@K) is needed to validate the methodology.
1. The proposed MTAD guides the generation and verification of draft tokens based on joint probabilities, providing a new idea for speculative decoding.

2. This paper analyzes the speculative decoding technique from an energy perspective for the first time and verifies the energy efficiency of speculative decoding. 

3. In addition to the experimental analysis, the authors also theoretically demonstrate the bounded error as well as the effectiveness of the MTAD algorithm.
This paper introduces a new framework named multi-token assisted decoding (MTAD) by combining speculative decoding with multi-token joint decoding (MTJD). MTAD generates high quality draft tokens by estimating the joint distribution of the target model using draft model, while improving both the speed and quality of inference.
1. Despite generating texts with lower perplexity, MTAD is still a lossy speculative decoding method. It is unfair to compare MTAD with lossless speculative decoding methods. It might be useful to add some comparisons with other lossy verification methods (e.g. Medusa's Typical Sampling) to verify the superiority of MTAD.

2. The main experiments in this paper use perplexity as an evaluation criterion for performance, and it is clear that MTAD based on PPL verification will have advantages. More experiments on downstream tasks are needed to verify whether MTAD can really improve the quality of text, since PPL and metrics are not strictly positively correlated.
Yes
N/A
No ethics review needed.
See above.
- The writing is good and easy to follow.
- The addressed problem is of great importance to the community.
- A new metric is provided to evaluate the effectiveness of speculative sampling, which is insightful.
- Strong experimental results:
  - Evaluated across a range of datasets
  - Shows significant improvements in terms of perplexity, speed, and energy efficiency.
This paper focuses on improving the inference speed of large language models (LLMs). The authors propose a variant of speculative sampling that enhances both inference speed and output quality. Specifically, they explore multi-token joint decoding (MTJD), which generates multiple tokens from their joint distribution at each iteration, theoretically reducing perplexity and improving task performance. Building on this, they incorporate speculative sampling to approximate multi-token joint decoding and design an efficient algorithm, multi-token assistant decoding (MTAD), which achieves bounded error with respect to MTJD. Empirical evaluations demonstrate the effectiveness of the proposed method in terms of speed and perplexity.
- In line 114, the authors claim, “As shown in Table 1, lower perplexity correlates with improved downstream performance, even in one of today’s largest models.” This assertion is questionable. The results in Table 1 are based exclusively on experiments with the Spider dataset, but in many cases, lower perplexity does not inherently lead to better downstream performance. For example, in tasks like machine translation and abstractive summarization, a lower perplexity score sometimes fails to correlate with improved downstream performance. This also can be validated by the right plot in Figure 1.
- Lack of guarantee on quality: In vanilla speculative sampling and multi-draft speculative sampling, one of their advantages is the property of being lossless. This ensures that the quality of output is maintained, which further accelerates LLM inference. However, the proposed method does not have such guarantee. 
- The verification process in this work is similar to that in [1], as both aim to verify a sub-sequence of tokens in a single step rather than one token at a time. However, this study does not include a comparison between MTAD and the method in [1]. The author should further compare their work with it to further improve the quality.
- The experiments also have some limitations:
  - This paper places too much emphasis on perplexity. Since a lower perplexity score does not always correlate with improved downstream performance, the authors should provide a more comprehensive evaluation of downstream performance. I noticed that Table 6 contains some comparisons on downstream performance, but this is insufficient. I encourage the authors to add a "downstream performance" metric to Table 4 so that readers can better understand the effectiveness of the proposed method.
  - I still do not fully understand why the proposed method significantly outperformed multi-draft speculative sampling methods such as Specter and SpecInfer. I suspect this is because (1) the selected baseline is weak, and (2) the configurations are not fair. For example, in Table 13, the number of sequences in Specter and SpecInfer does not match the number of beams in MTAD. I encourage the authors to do the following: (1) select more recent multi-draft speculative sampling methods for comparison, and (2) ensure consistent configurations.


[1] Sun, et al. Optimal Block-Level Draft Verification for Accelerating Speculative Decoding.
Yes
No ethics review needed.
Please refer to the weakness.
1. This paper theoretically proves that multi-token joint decoding (MTJD) can reduce perplexity and improve task performance compared to single token decoding.
2. This paper introduces multi-token assisted decoding (MTAD), a novel framework designed to accelerate multi-token joint decoding. Theoretically, this paper demonstrate that MTAD closely approximates exact MTJD with bounded error.
3. These results highlight MTAD’s ability to make multi-token joint decoding both effective and efficient.
Considering the inference efficiency, this paper proposes multi-token joint decoding (MTJD), which generates multiple tokens from their joint distribution at each iteration. More importantly, this paper theoretically proves that their method can reduce perplexity and improve task performance compared to single token decoding. Besides, to mitigate the high cost of sampling from the joint distribution of multiple tokens, it introduce a small model to assist the decoding of a larger model. Empirical evaluations reveal that the method reduces perplexity
by 21.2% and improves downstream performance compared to standard single-token sampling. It also achieves a 1.42× speed-up and consumes 1.54× less energy than conventional speculative decoding methods.
1. This paper primarily conducted experiments on the OPT series and Llama 2 series models. However, the performance of these large models has already fallen far behind the recent Llama 3.1 and Llama 3.2. Could you provide more results on the latest models to demonstrate the generality of the results?

2. The previous methods compared in this paper were all published last year. It would be beneficial to compare with some recently published methods, such as MEDUSA-2 and "Better & Faster Large Language Models via Multi-token Prediction", to prove the effectiveness of the results.

3. In this paper, a major point is that multi-token joint decoding can reduce perplexity and improve task performance. However, the experimental results do not reflect related analytical experiments. Adding relevant experiments could enhance the self-consistency of the method.
We propose a novel decoding that improves perplexity and downstream performance with 1.4 times faster and 1.5 times less energy cost compared to speculative decoding by considering joint probability of multiple tokens
@inproceedings{
anonymous2024optimized,
title={Optimized Multi-Token Joint Decoding With Auxiliary Model for {LLM} Inference},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ZHhBawo3k5},
note={under review}
}
Large language models (LLMs) have achieved remarkable success across diverse tasks, yet their inference processes are hindered by substantial time and energy demands due to single-token generation at each decoding step. While previous methods such as speculative decoding mitigate these inefficiencies by producing multiple tokens per step, each token is still generated by its single-token distribution,
thereby enhancing speed without improving effectiveness. In contrast, our work simultaneously enhances inference speed and improves the output effectiveness. We consider multi-token joint decoding (MTJD), which generates multiple tokens from their joint distribution at each iteration, theoretically reducing perplexity and enhancing task performance. However, MTJD suffers from the high cost of sampling from the joint distribution of multiple tokens. Inspired by speculative decoding, we introduce multi-token assisted decoding (MTAD), a novel framework designed to accelerate MTJD. MTAD leverages a smaller auxiliary model to approximate the joint distribution of a larger model, incorporating a verification mechanism that not only ensures the accuracy of this approximation, but also improves the
decoding efficiency over conventional speculative decoding. Theoretically, we demonstrate that MTAD closely approximates exact MTJD with bounded error. Empirical evaluations using Llama-2 and OPT models ranging from 13B to 70B parameters across various tasks reveal that MTAD reduces perplexity by 21.2% and improves downstream performance compared to standard single-token sampling.
Furthermore, MTAD achieves a 1.42× speed-up and consumes 1.54× less energy than conventional speculative decoding methods. These results highlight MTAD’s ability to make multi-token joint decoding both effective and efficient, promoting more sustainable and high-performance deployment of LLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM Inference Speculative Decoding
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/26b62223e89010b8ba83a44c73ea30bbec3616da.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/1df8716189d6e84c7a00ed6e97bcf42ce75fec3d.zip
Optimized Multi-Token Joint Decoding With Auxiliary Model for LLM Inference
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission