Integrating Geodesic Interpolation and Flow Matching for Non-Autoregressive Text Generation in Logit Space
Flow Matching;Non-autoregressive text generation
Yes
No ethics review needed.
Given the low quality of presentation, I have no further questions. I hope that the authors can make full preparations and improvements before the next submission.
1) Novel theoretical approach: The use of KL-divergence geodesics for flow matching in discrete sequence modeling is a novel concept. The theoretical justification provided for the likelihood function and its relation to the flow matching velocity adds to the rigor of the method.
This paper presents a novel approach for non-autoregressive text generation in logit space. It uses Kullback-Leibler (KL) divergence geodesics for flow matching between initial and target distributions of discrete sequences. A loss function is defined to maximize the conditional likelihood of discrete tokens, and its theoretical properties are explored. Despite initial poor results on the TinyStories dataset, an empirical sampling scheme based on a pretrained denoiser is proposed, which significantly improves performance. The method is also applied to image generation tasks for comparison.
1) Extremely low writing quality: The writing and presentation of this article are extremely poor and unreasonable.

2) Limited dataset evaluation: The evaluation is conducted on two uncommon datasets.
Yes
No ethics review needed.
See the Weaknesses part
- The paper introduces a novel application of KL-divergence geodesics for text generation, addressing limitations in linear interpolation commonly encountered in discrete sequence modeling.

- The use of a pretrained denoiser-based empirical sampling scheme demonstrates ingenuity, compensating for initial performance shortcomings and achieving improved generation results.
The paper introduces a novel method for non-autoregressive text generation using KL-divergence geodesics and flow matching in logit space. The authors propose a conditional flow matching approach to address the challenges of discrete sequence modeling, demonstrating theoretical alignment between the loss function and flow matching velocity. To enhance performance, they implement an empirical sampling scheme based on a pretrained denoiser. Experiments on both text and image datasets show that the method outperforms traditional autoregressive models. Despite promising results, the sampling technique lacks full theoretical justification.
- The paper seems to have been written in a hurry and lacks proper polish, with numerous missing references that make it difficult for me to follow. For example, references are missing at lines 32, 33, 39, 53, and 90, which disrupts the flow of the paper.
- I find the experimental section quite limited, as it only includes a single experiment for both text and image generation. A detailed ablation study is missing, making it hard to understand the impact of different components.
- I believe the evaluation metric for text generation is too restricted, relying almost exclusively on perplexity. While perplexity is useful for understanding how well the generated text fits the probable distribution, it can fail to capture semantic richness. I would recommend adding metrics like BLEU, ROUGE, or exploring newer evaluation methods for a more comprehensive assessment.
- After reading the introduction, I still do not fully understand why flow matching is necessary for generation models. The motivation for choosing this specific approach remains unclear to me.
Yes
No ethics review needed.
N/A
N/A
This work presents a flow matching approach for generating discrete sequences. This approach treats discrete tokens as one-hot vectors and constructs a flow by interpolation on the logit space. Randomized top-k sampling is proposed for inference.
This paper is only half-baked and needs substantial refinements before resubmission. For example, the presentation is poor (many variables are not explained, Figure 1/2 have the same caption, and some references are placeholders), experiments are only conducted on toy datasets (Tiny Stories, MNIST), and evaluation metrics are not sound (only use generative perplexity for language modeling).
@inproceedings{
anonymous2024integrating,
title={Integrating Geodesic Interpolation and Flow Matching for Non-Autoregressive Text Generation in Logit Space},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=44WiKy8THW},
note={under review}
}
Non-autoregressive language models are emerging as effective alternatives to autoregressive models in natural language processing, enabling simultaneous token generation. This study presents a novel flow matching approach using Kullback-Leibler (KL) divergence geodesics to interpolate between initial and target distributions for discrete sequences. We establish a loss function that maximizes the conditional likelihood of discrete tokens, demonstrating that its maximizer corresponds to the flow matching velocity under logit interpolation. While initial tests on the TinyStories dataset yielded unsatisfactory results, we introduce an empirical sampling scheme based on a pretrained denoiser, which significantly improves performance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Flow Matching Non-autoregressive text generation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/165278a7d107aa97cc706e1f207eb25ca1a21298.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Integrating Geodesic Interpolation and Flow Matching for Non-Autoregressive Text Generation in Logit Space
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission