Efficient Generative Modeling with Residual Vector Quantization-Based Tokens
generative models;residual vector quantization;discrete diffusion;transformer;text-to-speech
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
@misc{
kim2024efficient,
title={Efficient Generative Modeling with Residual Vector Quantization-Based Tokens},
author={Jaehyeon Kim and Taehong Moon and Keon Lee and Jaewoong Cho},
year={2024},
url={https://openreview.net/forum?id=GkGVNmjAwh}
}
We explore the use of Residual Vector Quantization (RVQ) for high-fidelity generation in vector-quantized generative models. This quantization technique maintains higher data fidelity by employing more in-depth tokens. However, increasing the token number in generative models leads to slower inference speeds. To this end, we introduce ResGEN, an efficient RVQ-based discrete diffusion model that generates high-fidelity samples without compromising sampling speed. Our key idea is a direct prediction of vector embedding of collective tokens rather than individual ones. Moreover, we demonstrate that our proposed token masking and multi-token prediction method can be formulated within a principled probabilistic framework using a discrete diffusion process and variational inference. We validate the efficacy and generalizability of the proposed method on two challenging tasks across different modalities: conditional image generation on ImageNet 256×256 and zero-shot text-to-speech synthesis. Experimental results demonstrate that ResGEN outperforms autoregressive counterparts in both tasks, delivering superior performance without compromising sampling speed. Furthermore, as we scale the depth of RVQ, our generative models exhibit enhanced generation fidelity or faster sampling speeds compared to similarly sized baseline models. The project page can be found at https://x8cg6mhs1qtf.github.io.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Jaehyeon_Kim1 ~Taehong_Moon1 ~Keon_Lee1 ~Jaewoong_Cho1
Jaehyeon Kim Taehong Moon Keon Lee Jaewoong Cho
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
generative models residual vector quantization discrete diffusion transformer text-to-speech
Yes, at the sentence level (e.g., fixing grammar, re-wording sentences)
I certify that there is no acknowledgement section in this submission for double blind review.
kim|efficient_generative_modeling_with_residual_vector_quantizationbased_tokens
/pdf/8f1f2771afe97eac4fbebb85f710e3d73615df1f.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
No
No
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Efficient Generative Modeling with Residual Vector Quantization-Based Tokens
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission