T2V2: A Unified Non-Autoregressive Model for Speech Recognition and Synthesis via Multitask Learning
ASR;TTS;Non-Autoregressive;Conformer;Multitask Learning
Yes
No ethics review needed.
* (Lines 47-49): "While multitask learning offers potential for parameter sharing, the performance of individual tasks might suffer due to the inherent complexity of training multiple tasks simultaneously." — The authors should provide references to prior work to support this claim. Additionally, if multitask learning poses such challenges, how were the authors able to handle both text-to-speech and speech-to-text effectively in the proposed unified model?

* Simplifying Figure 1 would make it easier for readers to understand the proposed framework.

* (Line 64): Why was the Conformer architecture with rotary positional embeddings (RoPE) chosen? Could the authors expand on the reasoning for using RoPE? Does RoPE offer benefits for both ASR and TTS tasks?

* (Line 261): It is unclear how the authors optimized the individual losses within their multitask learning framework. Was any weighting applied to the individual losses, and how did each loss contribute to the optimization of each task?

* There is some confusion regarding the dataset subsets used to train the unified model. In lines 311-315, the authors mention a 509-hour subset of LibriHeavy, while in lines 484-485 they refer to training a larger version of the model on a 50K-hour LibriHeavy subset for discrete ASR. What is the relationship between these two training stages? In the second ASR training, is the model initialized from the first training stage, or is it a separately trained ASR model?

* In Table 8, could the authors include baselines for the discrete Zipformer-CTC model for completeness?

* How does the proposed unified model perform on other TTS attributes, such as pitch, volume, and speed?
* This paper explores a challenging problem that enables the unification of ASR and TTS tasks, allowing the framework to benefit from the underlying complexities of each task.
* The claims were tested, demonstrating improvements in TTS performance as intended, while achieving comparable performance for the ASR task.
This paper addresses the challenge of unifying automatic speech recognition (ASR) and text-to-speech synthesis (TTS) using discrete speech tokens derived from self-supervised models. The authors introduce a CTC error correction task to enhance ASR performance by refining raw CTC outputs. Additionally, they propose a monotonic alignment search (MAS) technique with intermediate CTC outputs to achieve TTS alignment without relying on external tools. Experiments were conducted on the LibriHeavy dataset, with both objective and subjective evaluations of the synthesized speech.
It is unclear how much each task contributed to the overall improvement across tasks. Additionally, the performance of the proposed model on different attributes—such as pitch, volume, and speed of synthesized speech—has not been clearly detailed.
Yes
No ethics review needed.
1. What's the rational of doing the token length prediction by feeding text embedding through conformer backbone? This is odd because this is only task/step that conformer has to directly deal with text embedding and there seems to be risk of confusing the model.
2. The length prediction loss is not included in the final training objective in eqs 16, how is the length prediction head trained?
3. When the paper introduces CTC error correction task, it is stated in section 2.4.1 that the purpose is to address the independence assumption in CTC loss. From the mechanism of CTC error correction, it doesn't seem like the training breaks such assumption because the CTC logits was already detached from the computational graph. The author should elaborate more how this method essentially improves the performance. And how is it different from simply masking ground truth text embedding (or randomly augmented text) and do error correction.
* The paper presents a unified framework capable of both ASR and TTS, which may streamline resource use and advance multitask learning for speech models.
* The proposed auxiliary tasks such as CTC error correction and unconditional speech MLM demonstrate a solution for future work to address challenges in multitask learning.
* The paper provides comprehensive experimental results include some ablation studies to justify the design choice and demonstrate T2V2's effectiveness.
The paper introduces a unified non-autoregressive model designed to perform named T2V2 for both of ASR and TTS task. The backbone of T2V2 is Conformer with RoPE which operates on discrete speech tokens. The model achieves the tasks the it's designed for by applying CTC for ASR and conditional masked LM approach for TTS, along with auxiliary tasks like CTC error correction and nconditional
speech masked LM. With extensive experiment and evaluation, the author achieves competitive TTS and ASR performance. The main contributions claimed in the paper include:
* The paper proposes the first unified framework based on discrete speech tokens handling ASR and TTS task.
* The introduction of CTC error correction auxiliary task potentially addresses conditional independence limitation in CTC objective and improves ASR performance.
* The MAS-based alignment solution is innovative and allows for effective TTS alignment without external alignment tools.
* There is lack of discussions about or comparison with other multi-task models equipped with ASR & TTS capabilities such as [1] [2], etc. The discussion of related work isn't sufficient enough.
* The author states that the motivation of adopting non-autoregressive approach is the superior decoding time, however, there is lack of comparison of inference efficiency compared with its autoregressive counterparts.
* Presentation needs to be improved: for example: there is a mixed terminologies of "CTC token" "Text token" "Content token" "Acoustic token" used in Figure 1 and section 2/3, there should be a clear definition of the core terms used in the paper.



[1] Toyin, Hawau Olamide, Hao Li, and Hanan Aldarmaki. "STTATTS: Unified Speech-To-Text And Text-To-Speech Model." arXiv preprint arXiv:2410.18607 (2024).

[2] Yang, Runyan, et al. "PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models." arXiv preprint arXiv:2406.07801 (2024).
Yes
No ethics review needed.
The idea of unifying ASR and TTS in one model is interesting. However, my personal feeling is that the work is still in an early stage and is not ready for publication in ICLR as a top conference. Here are my concerns on the paper.

1. The clarity of the paper needs improvement.
- The paper involved many topics in the field of ASR and TTS such as discrete tokens, vocoder, non-autoregressive generation, etc. I understand it is hard to clarify each topic in detail. I'd suggest the authors to at least detail the used method for the claimed contributions. For example, MAS is claimed to be a major contribution. However, it is not clear how MAS is applied to the framework. How does it compare to the external alignment? In addition, I am wondering how it is compared to CTC alignment, which also provides speech-text alignments for non-autoregressive ASR (R. Fan, W. Chu, P. Chang and J. Xiao, "CASS-NAT: CTC Alignment-Based Single Step Non-Autoregressive Transformer for Speech Recognition," ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 5889-5893, doi: 10.1109/ICASSP39728.2021.9413429.). 
- It is suggested to clarify the loss for each head in Figure 1, especially the conditional TTS and unconditional MLM. Please differentiate the input and output, maybe with different colors?
- The experiments of hyperparameter tuning can be presented in the appendix.

2. Validation and performance
- A very important issue of the paper is the comparison of latency. One of the advantages of using both non-autoregressive generation and discrete token for ASR is the inference speed. The authors have no such comparison. In addition, the paper seems to use iterative generation during inference, the model should be rigorously classified as semi-autoregressive model instead of non-autoregressive model.
- The CTC error correction does not seem to be novel except that it is multi-tasked training with TTS loss. The Lcorr is not in-depth explored in this study. For example, according to the description of the paper, only the substation errors in CTC logits are considered to be corrected, while deletion and insertion errors are considered in literature. (Y. Higuchi, H. Inaguma, S. Watanabe, T. Ogawa and T. Kobayashi, "Improved Mask-CTC for Non-Autoregressive End-to-End ASR," ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 8363-8367, doi: 10.1109/ICASSP39728.2021.9414198.)
- For TTS, the ablation shows that the impacts of MLM and CORR are low. It is interesting to see the performance of training the model with TTS only. This would provide us insights whether the joint ASR and TTS modeling is beneficial.
- For ASR, the performance on test-clean doesn't seem to be competitive, compared to non-discrete ASR (almost twice of the WER). The advantage of discrete ASR is not stated in the paper.
- What is the motivation to use CER for the English ASR evaluation?

Overall, the paper has large room for improvements.
The strengths of this paper are:

1. The first paper to unify ASR and TTS in one model that operates on discrete tokens.
2. Extensive experiments to show the performance of the proposed model.
This paper proposed a T2V2 framework that uses a shared Conformer backbone to jointly train non-autoregressive ASR and TTS tasks with Connectionist Temporal Classification (CTC) and masked language modeling (MLM) loss. Two axillary losses are additionally introduced to improve the model performance, which are CTC error correction to refine raw ASR outputs using contextual information from speech embeddings, and unconditional speech MLM, enabling classifier free guidance. In the proposed framework, Monotonic Alignment Search (MAS) was used with intermediate CTC outputs for TTS alignment without relying on external tools. The experiments show that T2V2 markedly improves synthesized speech robustness achieving state-of-the-art TTS performance and competitive discrete ASR performance.
The weaknesses of the paper are:

1. Need improvement in clarity.
2. Lacking validations on metrics like RTF or latency.
3. The performance does not seem to be competitive.

See below for details.
Yes
No ethics review needed.
1. What is the time increase for 16 iterations compared to 1 iteration in Table 7?
2. What is the performance of the large model for TTS tasks?
3. Table 2 seems to have the best overall performance with iter=1.
1. It is interesting to use the intermediate CTC output to align text and speech via monotonic alignment search without relying on an external aligner.
2. The CTC error correction method is useful and may be applied to other state-of-the-art ASR systems.
3. The paper is well-structured and written.
This paper proposes a unified non-autoregressive model to perform both automatic speech recognition (ASR) and text-to-speech (TTS) synthesis tasks. CTC error correction and classifier free are used to improve ASR and TTS performance respectively. Intermediate CTC outputs are used to align text and speech by Monotonic Alignment Search, without relying on external aligners.
1. Figure 1 is too complicated, with the arrows flowing in interlocking directions, so that readers may feel difficult to follow it.
2. The content tokens employed in the T2V2 model are derived from HuBERT. However, converting these tokens into acoustic tokens and subsequently into speech requires additional components, namely a semantic-to-speech model (specifically SoundStorm) and a codec decoder. In my view, the efficacy of the speech synthesis process is largely dependent upon the performance of SoundStorm.
3. The models being compared in the TTS experiment should have models that only predict content tokens + SoundStorm, which is the most similar setting to T2V2 + SoundStorm.
4. In Tables 1 and 6, when comparing the performance of (with SMLM, without COR) and (without SMLM, without COR), it appears that SMLM harms both TTS and ASR tasks.
5. The change in Corr. thresholds and iterations in Table 7 doesn't seem to make the CER change much.
T2V2 is a unified non-autoregressive model for ASR and TTS using discrete tokens, a shared backbone, and auxiliary tasks like CTC error correction, achieving SOTA TTS and competitive ASR performance.
@inproceedings{
anonymous2024tv,
title={T2V2: A Unified Non-Autoregressive Model for Speech Recognition and Synthesis via Multitask Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=TtKN1TpvUu},
note={under review}
}
We introduce T2V2 (**T**ext to **V**oice and **V**oice to **T**ext), a unified non-autoregressive model capable of performing both automatic speech recognition (ASR) and text-to-speech (TTS) synthesis within the same framework. T2V2 uses a shared Conformer backbone with rotary positional embeddings to efficiently handle these core tasks, with ASR trained using Connectionist Temporal Classification (CTC) loss and TTS using masked language modeling (MLM) loss. The model operates on discrete tokens, where speech tokens are generated by clustering features from a self-supervised learning model. To further enhance performance, we introduce auxiliary tasks: CTC error correction to refine raw ASR outputs using contextual information from speech embeddings, and unconditional speech MLM, enabling classifier free guidance to improve TTS. Our method is self-contained, leveraging intermediate CTC outputs to align text and speech using Monotonic Alignment Search, without relying on external aligners. We perform extensive experimental evaluation to verify the efficacy of the T2V2 framework, achieving state-of-the-art performance on TTS task and competitive performance in discrete ASR.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
ASR TTS Non-Autoregressive Conformer Multitask Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9e168eb3f3e91568f8654eb0580a1993e08ea51e.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/1594853e55a1d68631a42bf22f509e6f22535a55.zip
T2V2: A Unified Non-Autoregressive Model for Speech Recognition and Synthesis via Multitask Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission