HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis
Text-to-speech synthesis;LLM-based TTS;neural audio codec;long-form generation
Yes
No ethics review needed.
* The description of the LRVQ modules appears to omit the use of commitment loss for residual vector quantization. If the LRVQ module does not require commitment loss, the authors would explain the reasoning behind this and how the training is stabilized without it.
* Providing a detailed description of the pre-trained LLM-based TTS model that HALL-E is post-trained on would greatly aid in understanding the training process and setup of the proposed model.
* The paper offers thorough quantitative and qualitative results, demonstrating that the proposed generative modeling outperforms baseline methods in long-form speech synthesis.
* The introduction of MinutesSpeech is a noteworthy addition to TTS research, providing high-quality, long-context speech data. The illustrated curation and preprocessing techniques further enhance its utility.
* The ablation studies on MRVQ provide insights into the necessity of each component, justifying their inclusion in the architecture.
The paper presents HALL-E, a neural codec language model aimed at addressing challenges in minute-long zero-shot text-to-speech (TTS) synthesis. It introduces Multi-Resolution Requantization (MReQ) to reduce frame rates in neural audio codec (NAC) models, and proposes HALL-E, which leverages MReQ for efficient and effective long-form speech synthesis. The work also introduces MinutesSpeech, a new speech dataset for long-context conversational TTS. The experimental results are extensive, showing that the proposed method can outperform baseline models both quantitatively and qualitatively.
* Although the MReQ module achieves low-resolution quantization while maintaining sample quality, its design is excessively complicated, posing challenges for future research to build upon. The LRVQ design incorporates multiple RVQ modules—pre-quantizer, main quantizer, and post-quantizer—alongside the additional HSR loss, making it significantly more complex than prior RVQ methods. Furthermore, the use of multiple RVQ quantizations within the NAR transformer of HALL-E adds another layer of complexity, reducing the accessibility of the proposed approach.
* The paper only compares the proposed method with VALL-E and does not include recent zero-shot TTS baselines. Considering that non-autoregressive modeling can be advantageous for long speech generation, incorporating comparisons with state-of-the-art non-autoregressive zero-shot TTS models would better demonstrate the effectiveness and relevance of the proposed method.
Yes
No ethics review needed.
* This technique seems well suited for LLM-based TTS solutions, similar to Valle with AR and NAR components. However, can you also say anything about how this kind of encoding might be used as a general purpose codec for speech in general? It would be interesting if this model could have lower bitrates or higher robustness. Furthermore, does this means that the PreQ and PostQ components of your LRVQ blocks would be unnecessary since they are primarily used to train the NAR components of the TTS model?

* With a VALLE model based on Encodec, once can remove RVQ code sequences from the NAR generation to get faster generation, in exchange for coarser generation. For this MReQ based paradigm, how does the audio quality here change?

* For the second ablation in section 7.4, how does this effect appear with the VALLE model that you trained?

* For the third ablation in section 7.4, it is very surprising that even w/o pretraining, it does not seem like this technique suffers significantly (compared to the performance drop in Table 11 for HALL-E). Why is that?

* The splits of the dataset in Table 2 and the description of why these splits were created in Section 7.1 seems very arbitrary and over designed for these experiments. Are these training splits deduplicated from each other?
Idea is pretty interesting and novel and well motivated. The MReQ module is fairly complex, but diagrams and descriptions of the technique are well written. Manuscript is very detailed and training and post-training are well documented. Along with the code release, results look to be easily replicable. 

Authors go through great lengths to train and evaluate everything from scratch and provide extensive results for reconstruction and zero-shot TTS which included both automatic and subjective results. Experiments were run on multiple NAC models, showing that this technique can generalize to pre-existing NAC models.

Benchmark dataset is a welcome addition and the generation and evaluation of the dataset are well documented. Results show RTF improvements, indicating that this technique indeed reduces the bottleneck behind transformer-based TTS.
This paper presents two post-training approaches to resolve the long context length issues for modern transformer-based autoregressive TTS models based on NAC models. MReQ as a framework reduces the framerate by introducing a novel Multi-resolution vector quantization module that allows the user to decompose an existing NAC model into several different code books, each of which operate at a different framerate. This allows them to reduce the bottleneck of the AR component of LLM-based TTS by reducing the frequency of the first component down to 8 Hz. To train this MRVQ module, student-teacher distillation is required. Using this module, this work presents HALL-E, a hierarchical TTS model that generates tokens based on the MReQ tokens. This paper also introduces MinutesSpeech, a benchmark dataset for TTS synthesis that is curated for long-form speech.
* One weakness in the study is that authors do not consider distribution shifts between the different dataset used to train the NAC model and the dataset used for post training. In their experiments, they pretrain their own NAC models (Encodec and Speechtokenizer) on the MinutesSpeech training set and did post training with MReQ with the same training data. However, in practice, researchers will not have access to the datasets used to pretrain these NAC models, so there may be some distribution shift. A similar issue occurs with quantization, where you need some training data to calibrate the model. Additionally, while SpeechTokenizer is meant to tokenize speech, Encodec was originally designed to encode speech along with other kinds of audio. It is unclear whether this capability will be preserved after post-training. Ablation studies do not cover what happens when you try to use this technique with an off-the shelf NAC model like SpeechTokenizer or Encodec off the shelf. 

* The statement on line 211/212 that Training NAC models with the MRVQ module does not seem to be justified or elaborated on. Is this just saying that training this kind of codec model from scratch rather than from a pretrained model is difficult? It seems like from Table 10, that for the MReQ model, starting this kind of training w/o pretraining seems to not affect the WER or RESQ significantly.

* The qualitative results in the ablation study near line 495 are not well explained. The statement that this waveform is unnatural with almost no silence is not clear without a transcription. From a glance, this seems to be a problem with the prosody of the generation, but this is unclear without hearing the samples. Appendix D1 also does not seem to elaborate on this much. I tried to listen to the samples from MinutesSpeech test-90s, and it seems like this is primarily a prosody issue as the Valle model seems to speak at a very consistent rate with much louder inhalation sounds.

* Furthermore, the ablation study for “Can HALL-E handle long audio prompts?” seems trivial, it should be expected that longer voice prompts gives the model more context on the user’s speech. It would be more interesting to see if this effect exists or is less pronounced in the VALLE model.

* There are also some wording errors in the manuscript. For instance, in the implementation details on page 4, lines 199 to 201, the manuscript says that “Figure 2a shows the MRVQ module applied to the Encodec model”, however, Figure 2a actually just shows the normal Encodec model as a teacher model. This should actually be Figure 2b that shows the MRVQ module applied to the Encodec model. Furthermore, Line 201 states that “Figure 2b shows the LRVQ block”, however, it is actually Figure 2c that shows the LRVQ block.  Also, should the statement in line 485/486: “The results indicate that both losses and post-training play important roles in the overall performance.” say that both losses and “pre-training” play important roles, since the second row of Table 10 is about the pre-training of the Encodec model?
Yes
No ethics review needed.
1. Have you tried to train MReQ with a much smaller sampling rate, such as 2 or 4 Hz for the first layer? 
2. What is the accuracy of the sub-encoder E to predict b_{k+1} based on a_{k+1}? Will this accuracy affect the final performance a lot?
3. For the L_{total}， are the weights of the three sub losses equal and all set to 1.0 ? Have you tried other settings?
1. This paper has three contributions 1). A multi-resolution requantization method MReQ is proposed to generate hierarchical codec codes. 2). A hierarchical LLM-base TTS model is proposed to predict MReQ codec codes. 3). A minute-long speech synthesis dataset is introduced.
2. This paper is well written and easy to follow.
To deal with the long-form speech synthesis problem, this paper propose a multi-resolution re-quantization method which hierarchically reorganizes discrete tokens with teacher-student distillations. Based on the hierarchical codec codes, an LLM-based TTS model HALL-E is proposed for speech synthesis. Besides these, a new benchmark dataset MinutesSpeech is introduced for minute-long speech synthesis.

This paper is well written and easy to follow.
1. Some important codec models are not compared with, such as DAC and Vocos.
2. MReQ is only tested on the speech reconstruction, without evaluated with audio dataset.
3. Some latest zero-shot TTS models are not compared with, such as Voicebox, E2 and VALL-E 2.
Yes
No ethics review needed.
Please address the questions raised in the weaknesses section.
The three contributions listed above are valid and well-received. The topic of minute-long speech synthesis is also one that will attract a wide audience.
The paper presents three contributions: (1) Multi-Resolution Requantization (MReQ), a framework that post-trains a conventional RVQ codec into one with a different token rate in each layer; (2) HALL-E, an extension of the VALL-E framework to utilize an MReQ-based codec; and (3) the introduction of MinutesSpeech, a new benchmark for minute-long speech synthesis. Leveraging the MReQ concept, the authors successfully train a codec whose first layer operates at only 8Hz. With this low-token rate modeling, the HALL-E model generates minutes-long speech with significantly better WER and RTF compared to the original VALL-E model.
While the paper is generally well-written, there are several points that seem unclear to me. It is difficult to fully accept their proposal unless these points are clarified. I am open to reconsidering the score later.

- Unreasonable usage of PESQ raises doubts about their experiments.
  - Why and how can PESQ be used to filter out noisy data in the data curation pipeline? PESQ requires reference audio to compute the score, and I cannot understand how it could be applied for data curation purposes.
  - Why is the PESQ score for the ground truth so low in Table 3? The PESQ score for the ground truth should be 4.64.
- Questionable inference and evaluation settings
  - Zero-shot TTS experiment was conducted by “continual” setting or “cross-utterance” setting? From the description, it’s unclear.
  - Assuming that a "continual" setting is used, did the authors exclude the audio prompt portion of the synthesized audio when computing the speaker SIM? This is the approach taken in the VALL-E paper and other related works. The increase in speaker SIM in Table 9 with respect to the prompt length is unusually large, raising concerns that the authors may not have excluded the prompt from the speaker similarity computation.
  - Why is DNSMOS used? DNSMOS is specifically designed and trained to assess the cleanness of speech rather than its naturalness. I strongly suggest replacing it with a naturalness MOS metric, such as UTMOS.
- Other minor request to assess the quality of the model
  - Please present the speaker similarity and MOS scores (UTMOS?) for Table 3 and Table 10.
  - Please provide a breakdown of the AR and NAR model RTFs for Table 7.

Additionally, I found several minor issues in the description. They are not critical, but they need to be revised:
  - Equation 2 is likely incorrect. I believe it should be x_l = x_{l-1} - \tilde{z_l}
  - Similarly, Equation 9 is likely incorrect.
  - In Table 8, the upsampling ratio should be explicitly described instead of simply listing "HALL-E" in the last row.
Yes
No ethics review needed.
--
1. This paper introduces the Multi-Resolution Requantization (MReQ) model, designed to enhance signal processing across multiple granularities.  
2. It presents the development of HALL-E, an autoregressive language model capable of generating extended speech waveforms with high fidelity.  
3. Furthermore, the study proposes a novel benchmark dataset, MinutesSpeech, specifically curated to evaluate the performance of models on long-duration speech sequences.
The paper introduces a novel text-to-speech (TTS) model designed to generate extended speech sequences. To address the inherent limitations of autoregressive language models in handling long outputs, the authors propose the use of *multi-resolution audio codes* (MReQ). Furthermore, the paper presents HALL-E, an autoregressive language model specifically developed to generate these MReQ audio codes efficiently. In addition to the model, the authors construct a new benchmark dataset comprising 40,000 hours of curated speech data. The experimental results demonstrate that the proposed approach can achieve a low frame rate of 8 Hz, highlighting its potential for generating high-quality long-form speech with reduced computational overhead.
1. The concept of time-varying audio codes is not a new development. For reference, please see the work by [Variable-rate Discrete Representation Learning (2021)] [https://arxiv.org/pdf/2103.06089].

2. A notable issue lies in maintaining temporal coherence and acoustic fidelity, as evidenced by the declining performance observed in the SIM score.

3. The model continues to face difficulties in handling spontaneous speech, highlighting an ongoing challenge in this area.
We introduce MReQ and HALL-E, methods that reduce frame rates in TTS models to enable minute-long speech synthesis, and present the MinutesSpeech dataset.
@inproceedings{
anonymous2024halle,
title={{HALL}-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=868masI331},
note={under review}
}
Recently, Text-to-speech (TTS) models based on large language models (LLMs) that translate natural language text into sequences of discrete audio tokens have gained great research attention, with advances in neural audio codec (NAC) models using residual vector quantization (RVQ).
However, long-form speech synthesis remains a significant challenge due to the high frame rate, which increases the length of audio tokens and makes it difficult for autoregressive language models to generate audio tokens for even a minute of speech.
To address this challenge, this paper introduces two novel post-training approaches: 1) Multi-Resolution Requantization (MReQ) and 2) HALL-E.
MReQ is a framework to reduce the frame rate of pre-trained NAC models.
Specifically, it incorporates multi-resolution residual vector quantization (MRVQ) module that hierarchically reorganizes discrete audio tokens through teacher-student distillation.
HALL-E is an LLM-based TTS model designed to predict hierarchical tokens of MReQ.
Specifically, it incorporates the technique of using MRVQ sub-modules and continues training from a pre-trained LLM-based TTS model.
Furthermore, to promote TTS research, we create MinutesSpeech, a new benchmark dataset consisting of 40k hours of filtered speech data for training and evaluating speech synthesis ranging from 3s up to 180s.
In experiments, we demonstrated the effectiveness of our approaches by applying our post-training framework to VALL-E.
We achieved the frame rate down to as low as 8 Hz, enabling the stable minitue-long speech synthesis in a single inference step.
Audio samples, dataset, codes and pre-trained models are available at  https://anonymous.4open.science/w/halle_demo.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Text-to-speech synthesis LLM-based TTS neural audio codec long-form generation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/85f9fb39fd4c117de648cc6a9cb9f3f64c84e1e5.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/5fc5da675426a210c699d4fd553d761197df2875.zip
HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission