Reflection Window: Text Generation with Selective Refinement
Selective Refinement;Autogressive Text Generation;Reflection Window;Large Language Model
Yes
No ethics review needed.
1. See above
2. Any human alignment results for MT-Bench since you choose LLM to judge ?
3. Why \sigma can be defined as shown in Eq(4) ?
4. One concern is about section 2.1. If we consider the attention distribution of LLMs, it is possible to let (b) become (a), I do not see the logic here and the proposed method also is auto-regressive method.
1. The theoretical characterization of sub-optimality is reasonable.
2. The proposed method can be a potential solution for the gap between beam search and greedy decoding.
The paper proposes a new decoding strategy called reflection window by using beam search at fixed generation window once detected conditional probability drop at specific position. Furthermore, the paper shows the effectiveness and efficiency of the proposed method compared with two baselines: greedy decoding and beam search on MMLU and MT-Bench with selected subsets.
1. The experimental results are not convincing: 1) the selection of beam size and window size, there is no analysis about how to select them; 2) Only two baselines are used in the experiments while there are many speculative decoding methods; 3) the performance gap between greedy decoding and reflection window is too small in table 1, there is no significant test.
2. There are a lot of cherry-picking implementation details: 1) the gap in STEM of MMLU is relatively bigger then the author chooses three subsets of STEM to conduct later experiments without revealing the reason and the performance on whole set. This is very important since this makes all conclusions afterwards can not stand. 2) table 2 shows there are only 100 test examples for MT-Bench, and there is no greedy decoding in table 2.
3. The necessity of reflection window is not clear and lots of analysis are missing, including the effects of beam size and window size on the whole set of MMLU and MT-Bench, efficiency analysis and so on.
Yes
No ethics review needed.
Please refer to the weakness
1) Novelty in technique: Utilizing a fast-slow pointer for reflection and generation is quite a technically interesting idea for LLM decoding. 

2) Theoretical formalization is good to understand the problem of auto-regressive understanding.
This paper proposes a novel technique for generation and reflection of LLM models. It utilizes a fast-slow pointer to maintain a slide window, where reflection tokens are generated. Generally speaking, the proposed method is quite interesting, and could be an easy strategy to implement. The proposed strategy would be able to balance the generation and reflection, and experimental results demonstrate its superior performance compared to greedy decoding and beam search.
1) Insufficient baseline methods: I think author should at least compare the proposed method with:
decoding algorithms: top-k/p sampling
Prompting based ‘reflection’ method and automatic post-editing strategy for fair comparison. 
Only comparing with beam/search decoding is insufficient.

2) Lack of Clear Demonstration on Distinction. Though an interesting idea, this paper does not highlight the difference between the proposed strategy with other reflection thinking strategies, practically or principally. 

3) lack of logical necessity between the theoretical analysis and the proposed specific method. Despite the theoretical analysis provided in this paper, other methods are also applicable within this theoretical framework and can be viewed as specific cases under this analysis framework. Consequently, why propose a fast-slow pointer under such a framework instead of conventional approaches? Why is the proposed method superior under such an analysis framework? This paper does not answer those questions.
Yes
No ethics review needed.
* After the refinement, what if the entropy condition still holds i.e., the newly generate tokens are also uncertain?
* The theoretical treatment of autoregressive sub-optimality is valid and interesting. 
* The authors did a good job highlighting a prevalent issue with current LLMs: The inflexibility of autoregressive word-by-word style generation.
* The proposed approach shows improvement over vanilla greedy decoding.
The paper proposes a novel generation technique that allows the LLM to pause autoregressive generation at one point and “reflect” over a window of the generated context, before resuming autoregressive generation. The authors formally show that autoregressive generation is suboptimal even with a good base LLM. Empirically, their approach operates by observing the entropy of previously generated tokens up to a certain window size, and if the entropy is above a certain threshold (indicating uncertainty) then generation is paused and beam search is used instead of greedy decoding. They evaluate their approach over MMLU and MT-bench, comparing it to greedy and beam search.
* The authors refer to this approach as a "reflection/refinement" technique, but the refinement they refer to merely involves using beam search as opposed to greedy. In other words, the approach seems to me like a hybrid greedy/beam approach, rather than a refinement/reflection setup. While the idea of pause-and-reflect is very interesting, I find the execution to be very poor. Why not pause and run some refinement over the window (e.g., ask the LLM to revise the output).
* The whole approach is based on an assumption that the LLM is calibrated--and therefore an "oracle"---LLM, in the sense that highly likely sequences are correct/preferable. This motvates their assumption that beam search should serve as an approximation to globally optimal sequences. We know this is not the case, and LLMs are in many cases poorly calibrated. This can explain some results where beam search performs worse/on par with greedy decoding (such as in Table 3). In other words, the approach relies on a perfecly calibrated LLM, which may not be available
* Experimental results are not very convincing. Table 1 shows beam search to be better, and only 80 responses were used for evaluation on MT-bench. 
* The authors do not provide qualitative examples at all. I would be curious to see how this “refinement” process works, and how the rewritten parts eliminate mistakes and/or improves writing. 

Therefore, in its current state, I believe the paper is missing a lot, and I would ask the authors to revise their paper accordingly.
Yes
No ethics review needed.
.
The proposed method, reflection window, is novel to address auto-regressive limitations since it includes self-correction in generation steps without generating full token sequences.
This paper targets to address a limitation of auto-regressive text generation in LLMs, where the process generates text token by token without refinement. They propose a novel method, reflection window, which allows the generation process to pause and reflect on previously generated tokens to correct errors when needed based on an entropy-based criterion. They focus on two benchmarks, MMLU and MT-bench and showed effectiveness compared to greedy decoding, obtaining scores comparable to beam search.
The pausing criterion's dependency on entropy threshold and window size means performance may vary with task and domain shifts. Therefore, it is necessary to consider diverse datasets with various generation tasks.

To demonstrate the robustness and effectiveness of the proposed method, more recent baselines for generation methods need to be considered.

Relying solely on automatic evaluation does not guarantee improvements in fluency, coherence, or error correction.

The paper specifically discusses a few decoder-only LLMs. Different types of models need to be evaluated for robustness.
We propose selective refinement within a reflection window for text generation, to address the pitfall of the purely autoregressive approach.
@inproceedings{
anonymous2024reflection,
title={Reflection Window: Text Generation with Selective Refinement},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=aS1IhKdLPP},
note={under review}
}
The autoregressive approach to text generation in large language models (LLMs), while widely used, is inherently suboptimal due to the lack of a built-in mechanism to perform refinement and/or correction of the generated content. In this paper, we consider optimality in terms of the joint probability over the generated response, when jointly considering all tokens at the same time. We theoretically characterize the potential deviation of the autoregressively generated response from and its globally optimal counterpart that is of the same length. Our analysis suggests that we need to be cautious when noticeable uncertainty arises during text generation, which may signal the sub-optimality of the generation history. To address the pitfall of autoregressive text generation, we propose an approach that incorporates a sliding reflection window and a pausing criterion, such that reflection and generation can be carried out interchangeably as the generation proceeds. Our approach utilizes a selective refinement mechanism to strike the balance between efficiency and optimality, and the experimental results demonstrate the effectiveness of our method.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Selective Refinement Autogressive Text Generation Reflection Window Large Language Model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/de96b6896f3e6328f937fd0860e20176ba67e2e6.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reflection Window: Text Generation with Selective Refinement
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission