Parrot: Seamless Spoken Dialogue Interaction with Double-Channel Large Language Models
Speech Language Models;Generative Spoken Dialogue Language Modeling
Yes
No ethics review needed.
* I would appreciate it if some of the weaknesses related to describing the evaluation study are resolved.

 * "encodes each second of audio into 30-50 discrete tokens from a codebook of size 2048." Why is the token rate variable?
* The paper proposes a simplification of a two-tower approach described by dGSLM, by using a single Transformer predicting two tokens.
This paper introduces Parrot, an audio LLMs models designed for modeling two-channel dialog audio. Parrot is built by fine-tuning an off-the-shelf text LLM on tokenized audio. At first, a single channel audio is used, as given in multiple standard datasets (eg LibriLight). Next, it is fine-tuned on a dialog dataset, Fisher. Here, the model is trained to simultaneously predict two tokens, one for each channel. According to the evaluation, this leads to a more natural dialog flow than in baseline model, dGSLM.
1. In a few places, the paper misrepresents related work. Examples:
  * "the academic community primarily utilizes open-sourced models (Zhang et al., 2023a; Xie & Wu, 2024; Rubenstein et al., 2023; Huang et al., 2024; Wang et al., 2023a; Nachmani et al., 2024; Wang et al., 2023b) following a cascading approach." For instance, [Rubenstein et al., 2023] and [Nachmani et al., 2024] are not cascaded models. They are also not open-sourced.
 * "Much of the prior research has utilized the encoder-decoder architecture to enhance pre-training (Borsos et al., 2023; Lakhotia et al., 2021; Kharitonov et al., 2022; Polyak et al., 2021; Chen et al., 2023; 2022; Hsu et al., 2021; Zeghidour et al., 2022; Defossez et al., 2023; Agostinelli et al., 2023; Ao et al., ´ 2022; Tang et al., 2022; Wu et al., 2023).". ** The first three models are decoder-only LMs, and perhaps many others.

2. The evaluation of the proposed model is limited and is insufficiently described. 
 * S4.5.1 evaluates some pause-prediction accuracy metrics, but the metrics are not really introduced in the text. These metrics are evaluated on some synthetic data --- is there a reason a hold-out subset of Fisher is not used?
 * AudioQA evaluation and comparison to AudioQWEN-2 is only mentioned in a single sentence without any discussion or description. At the same time, it does require some discussion. What are the metrics used? The AudioQA figure indicates some reasonable performance on CoVost2 and FLEURS, I assume better than AudioQWEN-2. However, the training data does not include speech-to-speech translation examples nor contains non-tier1 languages. Should we assume the model somehow picked it up from purely text base model?
* Audio-QWEN2 reports 90+% accuracy on VocalSound. In Figure 8c it is reported to be below 80%. What is the reason for the difference?

3. The paper should at least mention Moshi https://arxiv.org/abs/2410.00037
Yes
No ethics review needed.
As listed in the limitation section
**Summary Of Strengths**:
- Comprehensive Presentation: The paper includes all necessary sections, diagrams, and tables to demonstrate their contribution, the usefulness of the work, while also acknowledging its limitations.
- Research Direction: The application of LLMs for audio token prediction and the double-channel mechanism are interesting and challenging directions in spoken language modeling.
**Paper Summary**:
- The authors present a textless spoken dialogue language model and corresponding training pipelines. The work involves two training stages: during pre-training, the LLM model is used to instruct the prediction of the next audio token, and in SFT, a double-channel mechanism is applied to predict the next token pair. Ablation studies demonstrate that this method yields more natural and fluid dialogue generation compared to baselines.
**Summary Of Weaknesses**:
- Novelty and Clarity: In the contribution summary, the authors list (1) the Parrot model and its innovative pre-training and SFT pipeline, (2) the paradigm of double-channel spoken language modeling, and (3) the evaluations. In my opinion, textless spoken language models are not particularly rare, especially in machine translation tasks (e.g., [Seamless: Multilingual Expressive and Streaming Speech Translation](https://arxiv.org/abs/2312.05187), [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://arxiv.org/abs/2212.08055)). Additionally, the pipeline and the double-channel modeling mechanism appear to be two perspectives on the same concept. It is difficult to consider the potential readiness for future exploration and evaluation as innovative contributions.
- Limitations: Besides the limitation regarding the inability to integrate audio tokens, further analysis of the audio tokenizer should be elaborated on (e.g., how it impacts computational efficiency or downstream inference latency). More case studies should also be included to demonstrate the effectiveness of this work, as A.5 seems unfinished.
Yes
No ethics review needed.
Check weaknesses
1. The authors state that they will open-source their training and inference framework, which would be a valuable contribution to the community, especially since existing end-to-end spoken dialogue models either do not disclose their training methodologies (e.g., Moshi) or lack sufficient documentation (e.g., Mini-OMNI).
2. The authors also evaluate their approach on turn-taking properties, such as recognizing when the user has paused and determining appropriate moments to interrupt the user.
The paper introduces a spoken dialogue model, Parrot, which leverages large-scale single-channel audio data for pre-training and moderate-scale dual-channel dialogue data for supervised fine-tuning. The authors also propose a “next token-pair prediction” approach for spoken dialogue language modeling. The study claims that Parrot facilitates more natural and fluid conversations compared to Dialog GSLM and traditional cascaded spoken dialogue systems.
1. The paper lacks specific details on the cascaded system used for evaluation. DialogGSLM used a relatively weak cascaded system in their work, so it would strengthen this study if the authors evaluated a cascaded system with state-of-the-art ASR, LLM, and TTS models, such as Hugging Face's Speech-to-Speech (https://github.com/huggingface/speech-to-speech), for a fairer comparison. Additionally, latency in cascaded systems can be minimized by parallel threading each module, as demonstrated in the Hugging Face repository.

2. Given the primary objective of this work is to enable engaging and naturally fluid conversations, a human study would be valuable for a thorough evaluation—similar to assessments in text-based dialogue systems. When motivating their approach, the authors point out limitations in cascaded methods; more analysis is needed to clarify if these limitations genuinely impact user experience in human-AI conversations. User study could focus on user satisfaction compared to baseline systems as well as other auxiliary metrics such as naturalness of turn-taking and semantic coherence of response.

3. Although the code is public, it lacks sufficient documentation, and I was unable to run their demo. For example, an inference.py file appears to be missing. 

4. The paper would benefit from further discussion on design choices:

a. I was curious about the choice to use a single codebook. Table 4’s synthesized audio quality results lack clarity on the dataset used. Multiple codebooks often improve results—did the authors experiment with this option?

b. The motivation for using next-token pair prediction, rather than multi-stream prediction as seen in Moshi, is also unclear. While streaming audio generation is effective, the paper doesn’t address the length limits for audio modeling. Given the potential length of audio sequences, does the model implement techniques to reduce sequence length?

c. The authors use only human-human conversation data for supervised fine-tuning. Adding human-AI conversation data, even synthetic, could be beneficial as there are nuance differences in how humans communicate with AI versus other humans.

5. Clarity

a. Section 4.5.1 is difficult to follow. My understanding is that the ground truth is generated using GPT-4. Did the authors verify this ground truth with human judgments? I would recommend the authors to clarify their methodology for generating and validating the ground truth data, and include this information in the paper.

b. Section 4.5.3 presents an interesting observation. Do the authors have any intuition as to why this occurs?

6. In addition to evaluating turn-taking properties, the paper could also benefit from evaluating "speaking-while-listening" capabilities (https://arxiv.org/pdf/2408.02622).

7. Did the authors assess whether the LLM undergoes catastrophic forgetting when fine-tuned on audio data? For example, is Parrot still capable of instruction-following or answering factual questions at a level comparable to LLAMA 3.1?

8. In section A.1, the paper makes claims about prior works that lack experimental or discussion-based support. For instance, the statement that RQ transformers reduce model efficiency is not substantiated. Unsubstantiated claims should be avoided, as they could lead to incorrect community conclusions.
Yes
No ethics review needed.
* L473-476: The discussion on "layer-wise" and "consistent" channel embeddings is unclear. These terms appear only once and lack explanation, leaving their meanings and relevance ambiguous to the reader.
* L1162-1187: The purpose of the GPT score is not clear. It is mentioned in the appendix, but its usage and significance are not explained in the main text, making it difficult to understand its role in the evaluation.
8 483-505: Since AIR-bench questions are in text format, it is unclear how the proposed model, which is audio-based, handles text input. Without further clarification, it is difficult to interpret the evaluation results accurately.
* Section A.1: The essential distinctions between the proposed method and closely related prior work should be concisely summarized in the main related works section. The current related works section lacks sufficient comparison, especially with the most relevant prior methods.
The framework presents a relevant approach to double-channel spoken dialogue, aiming to improve latency by avoiding text generation stages, which could contribute to real-time applications.
The Parrot framework aims to address double-channel spoken dialogue modeling by implementing a pipeline that includes pre-training on single-channel audio and fine-tuning on double-channel audio data. The framework introduces a "next-token-pair prediction" approach within a decoder-only model architecture. However, the proposed solution lacks substantial originality and leaves questionable details for evaluation, which ultimately weakens the paper.
* Minimal Novelty: The framework essentially serves as a decoder-only adaptation of dGSLM combined with a textually pre-trained model (TWIST), offering limited technical innovation. This incremental change does not justify the need for a separate model or paper.

* No Human Evaluation: The absence of human assessment significantly limits the validity of the framework's claims about improving conversational fluidity and natural interaction, which are central to spoken dialogue applications.

* Lack of Established Benchmark Comparisons: Despite the existence of standard benchmarks like ZeroSpeech [1] and StoryCloze [2] for textless spoken language models, the paper does not include comparisons with these datasets. This omission raises concerns about the thoroughness of the experimental validation.

* Poorly Defined Evaluation Methodology: The evaluation details, especially for the reflective pause and interruption response accuracy (Section 4.5.1), are incomplete. Key information, such as the evaluation metric definitions like interaction accuracy, is missing, making it hard to verify the claimed improvements.

* Insufficient Explanation of Key Evaluation Components: See the comments below.

1. Zerospeech 2021 benchmark, https://arxiv.org/abs/2011.11588
2. StoryCloze, https://arxiv.org/abs/2305.13009
Yes
No ethics review needed.
1. In section 4.1, the authors claim to have used 14,000 hours of single-channel for pretraining and 2,200 hours of double-channel for sft. However, in table 1, there are over 70,000 hours of single-channel data and 2,000 hours of double-channel data. At the same time, InstructS2S-200K is a single-channel data, how do you use it in stage 2?

2. In section 4.5.3, the author claims that the embedding of two channels is in different spaces, but in Figure 7, there are two completely different feature spaces, with the right side appearing consistent with the author's statement, so what is the left side of the figure?

3. In Appendix A.6.3, the author lists "prompt for gpt score," but there is no mention in the main text of the need for gpt evaluation.

4. How much data was used to train the speech tokenizer? And what's the output rate of the speech tokenizer?
1. This work explores an interesting topic, namely dual-channel speech input modeling. Previous work has focused on user-assistant turn-based interactions, but in real interactions, immediate processing is required. Therefore, there is a need for a listening channel to continuously process user's speech input.

2. The author made reasonable reviews and citations to related work.
This work presents an approach to spoken dialogue interaction through the development of a large language model (LLM) named Parrot. The authors propose a two-stage training pipeline that leverages single-channel audio data for pre-training and double-channel audio data for supervised fine-tuning (SFT). The key innovation lies in the "next-token-pair prediction" paradigm, which aims to enhance the model's ability to comprehend and generate natural human conversations in real-time.
1. The writing of this paper is very poor, with some tables and figures not being referenced in the main text, and inconsistent statements in the context, making the entire article very difficult to understand. (see Questions for details)

2. The structure design of the model and "next-token-pair prediction" paradigm are not well-motivated, as there is a significant gap between training and inference stages.

    -  The author inputs listening tokens and speaking tokens as pairs into the LLM, which doubles the context length. LSLM [1] has demonstrated the effectiveness of modeling double-channel through embedding fusion and is more efficient in context. Therefore, the author needs to prove the effectiveness of their modeling approach
    -  In the training process, the prediction of the next token depends on the previous predicted tokens. But in the inference process, according to section 3.3, listening tokens are sent to LLM in chunks. At this point, predicting the next speaking tokens no longer depends on the previous predicted speaking tokens. This inference method will disrupt the casual modeling during training.

3. The lack of evaluation details.

      -  In section 4.5.1, LLAMA-Omini and SpeechGPT do not have the ability to interrupt and pause. How does Reflective Pause and Interruption caculated?

      -  In section 4.5.2, all these evaluation metrics cannot reflect the linguistic quality of the model. For an open-ended instruction-following task, there is no evaluation of response quality.

      -  In section 4.6, the evaluation of speech-to-text tasks, such as ASR and ST, is discussed. However, this model is a textless speech-to-speech model and does not have text generation capability, so it is unclear how these tasks are evaluated.

4. Some results are very strange, which it‘s hard for me to believe that it's real.

      -  According to table 1, the model is only trained on English-Only speech data, but in figure 8(c), the model can perform Chinese ASR task on aishell, SER task on MELD, and sound-related task on AIRBench-Sound. This is very strange.
      -  In appendix A.4.2, the larger latency, the worse performance. There is no explanation for this strange results.
We propose a novel spoken dialogue language model that uses an innovative pre-training and SFT pipeline with dual-channel audio data and next-token-pair prediction paradigm, achieving real-time streaming interaction.
@inproceedings{
anonymous2024parrot,
title={Parrot: Seamless Spoken Dialogue Interaction with Double-Channel Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=73EDGbG6mB},
note={under review}
}
Recent advancements in large language models (LLMs) have demonstrated significant potential in enhancing real-time spoken interactions. Presently, open-source methodologies predominantly depend on intermediate generative text-based translations to manage real-time spoken dialogues. However, these techniques often struggle with providing seamless interactions that involve real-time streaming audio inputs. In this research, we unveil an innovative spoken dialogue language model, Parrot, distinguished by its unique pre-training and supervised fine-tuning (SFT) pipeline. This pipeline deviates from conventional methodologies by utilizing both single-channel audio data and double-channel spoken dialogue data to train the textless speech language model. During pre-training, we transmute single-channel audio input into a sequence of discrete tokens, thereby instructing the LLM to identify audio tokens via next-token predictions. In the SFT phase, we pioneer a novel approach to double-channel generative spoken dialogue language modeling with a unique ``next-token-pair prediction" objective, facilitating the LLM's comprehension of natural human conversations. Our inventive pipeline equips the LLM to produce spoken interactions that are more natural and fluid than those generated by previous text-based approaches, as substantiated by thorough evaluations.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Speech Language Models Generative Spoken Dialogue Language Modeling
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/70b01d06841500e0d679045822beb85f50011434.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/e1f415440db2fdae1fe5e5a91c5a38df8677c4f7.zip
Parrot: Seamless Spoken Dialogue Interaction with Double-Channel Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission