JOOCI: A FRAMEWORK FOR LEARNING COMPREHENSIVE SPEECH REPRESENTATIONS
SSL;Speech Representation Learning;Joint Optimization
Thank authors for the response.

Unfortunately, I'm confused by the author response, stating that my summary misunderstood the intention of the paper. "JOOCI is not designed as "speech representation learning for disentangling content information from non-content information," " -- isn't that the motivation stated in the abstract of the paper?
### We sincerely thank the reviewer for their comments and for the opportunity to clarify certain aspects of the JOOCI framework's design and novelty.

* We will clarify the main claim and clearly mention specific tasks instead of the using the SUPERB benchmark term. 

* Regarding the reviewer's remarks on JOOCI's novelty and its alleged similarity to previous work, we would appreciate further clarification. The reviewer has highlighted two specific papers, which both employ a disentanglement module and use supervised data during training in some way. However, JOOCI is fundamentally different: it does NOT include any disentanglement module, nor is it trained with supervised data during pre-training. This distinction contrasts with the reviewer’s assertion about JOOCI’s similarity to the referenced works.

* Additionally, we respectfully note that the reviewer's summary of JOOCI appears to contain a misunderstanding. Specifically, JOOCI is not designed as "speech representation learning for disentangling content information from non-content information," as the summary suggests. In fact, the concept of disentanglement is only mentioned in Section 4.3 of the paper (in the second-to-last section before the conclusion) as part of an ablation study on understanding learned representations. This brief remark reflects an observed behavior in the model’s learned representations rather than a deliberate design goal or imposed structure during pre-training.

* We will add the details of the model as suggested. Lastly, as suggested we will improve the writing of the paper. We have updated the rebuttal, which does address this issue to a major extent. 

We hope this clarification addresses the reviewer's concerns and would be grateful for any additional feedback they might provide on this matter.
Addressing Misunderstandings Regarding JOOCI’s Methodology and Novelty.  JOOCI is not doing any disentanglement.
Yes
No ethics review needed.
See Weaknesses section.
The proposed method is sound. The experimental results on a subset of SUPERB benchmark are strong.
This paper proposes a method for speech representation learning, particularly, for disentangle content information from non-content information. The paper reported strong experimental results on SUPERB benchmark.
- The novelty is limited. The proposed method is very close to a number of existing works, e.g.:
  - Chan et al., Content-Context Factorized Representations for Automated Speech Recognition, InterSpeech 2022.
  - Zhao et al., CCSRD: Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech Translation, EMNLP 2023.
- The main claim is flawed. The paper claims SOTA on SUPERB. However, it only reports experimental results on a subset of the tasks from SUPERB (7 out of 10).
- The writing needs improvements:
  - Importantly, the name "other encoder" is a poor choice, which causes a lot of confusion for reading. Some simple choices such as "non-content encoder" would do a much better job.
   - Secondly, many small claims are questionable throughout the paper. A few examples:
     - Abstract: content and non-content information are orthogonal -- in the words from the paper, “how it is expressed” depends on “what is being said”
     - Sec 2.2: "Since JOOCI uses separate learnable parameters, the losses are summed directly without requiring
additional hyperparameter tuning." -- The previous paragraph said the opposite: " The GRL scale the gradients during backpropagation by a factor of 1/10, preventing interference with the other loss."
  - Lacks details of the model. While references to prior works is great, for completeness of the paper, you should describe the details of you model clearly, so that the readers understand your approach without having to jumping to many other papers.
Yes
No ethics review needed.
- The paper states that WavLM the "previous SOTA method".  By what measure is WavLM SOTA?  On what task(s)?

- I don't follow the sentences "As a result, the model cannot fully leverage all layers ... within a single embedding." nor the following sentence "The strategy of dividing the layers..."  Can you clarify what is meant there?

- The description of the split and append layer is a bit hard to follow.

- In Eq. 1, the index d is never used in the summand.  Also, should "MPL" be "L_MPL"?

- In Eq. 3, what exactly are Student^PN and Teacher^RDINO?

- In Table 1, where are the results for FBANK and other competitor methods obtained from?  Citations should be provided.  I also suggest including MS-HuBERT since JOOCI is based on it, and ideally also data2vec which has good results on many SUPERB tasks (but please let me know if you think these would not be relevant for some reason).

- I don't quite follow the sentence "We augment the data very lightly, so not to interfere with the content encoder a lot and divide its capacity."

- The description of the main results in Section 3.2 seems a bit misleading.  The paper states that the "results clearly indicate that JOOCI outperforms the current state-of-the-art (SOTA) models on the majority of tasks, except few ...".  However, in Table 1 JOOCI appears to outperform other models on exactly half the tasks, and it is never explained in what sense those models are SOTA (though they are clearly commonly used models).

- I do not understand the purpose of the comparison in Table 2, since JOOCI is not an alternative to adapters.  Also, "Houlsby" and "CHAPTER" need to be defined.  

- How does Figure 2 show the effect of data augmentation?  Is there a pair of curves that differs only in the use of data augmentation?

- For Figure 2, more information is needed about the y-axis.  How is CCA similarity defined?  How are the word labels encoded and how many words are there?  There has been prior work using CCA similarity for layer-wise analyses, e.g. Pasad et al., "Comparative layer-wise analysis of self-supervised speech models," ICASSP 2023.  Figure 2 seems similar to some of this prior work, and so it would also be helpful to state how your CCA-based analysis is the same or different, and whether your HuBERT results are similar to Pasad et al.'s.

- The ablation study in Sec. 4.1 is a bit confusing to me.  It claims to separately show the effect of DGRL and data augmentation, but as far as I can tell these two variables are changed simultaneously in the experiments.

- In Table 3, why are the "-" results not included?  If those could be included, they could help to show to what extent JOOCI-C and JOOCI-O specialize for linguistic vs. non-linguistic information.

- In Section 4.2, I have trouble following the first paragraph.  What kind of information is considered "higher-level" in the "other" branch, and what is the "same trend" that is referred to here?

- Section 4.3 claims to "prove that JOOCI is able to disentangle content and other information", but I don't follow how the results show this.  (Also, the word "prove" is too strong here, as in most descriptions of empirical findings.)

- In Table 5, what is the difference between the experiments in the last two lines (both labeled "JOOCI (6-11)"?
+ Addresses an important need to account for both linguistic and non-linguistic content in speech representation learning.

+ Obtains impressive results on several tasks, including speech recognition and speaker identification.
The paper proposes a self-supervised speech representation model that combines two encoders, one intended to encode linguistic content and the other intended for "other" content like speaker and emotion information, trained with different losses.  The idea is that, by training a single encoder with a single loss, previous approaches have trouble encoding these two types of information equally well.  The various elements of the model are largely borrowed from previous work, but combined in a new way.  The model is compared in terms of performance on 8 common tasks (from the SUPERB benchmark) to other commonly used models (HuBERT, WavLM), finding improved performance on 4 of the tasks.  The paper also includes some ablation studies and analyses of several model components.
- Presentation of many details is unclear.  For example, the definition of "content" and "other" is never clearly stated.  Also, the model description is very brief, leaving many details to cited papers or the imagination (for example, is prosody ever/always/sometimes considered "content"?).  Either the writing should be much more precise or the paper should include equations specifying all of the model components.  See some other specific questions below.

- The key claimed contribution is that the model encodes both linguistic and non-linguistic information and that these are disentangled into the two encoders' representations.  However, the results don't quite show this, since the results on tasks are mixed and the analyses don't really demonstrate disentanglement (again see questions below).  Overall, I don't see the community starting to use this model as a replacement for other currently popular models.

- Some of the experiments do not, as far as I can tell, show the claimed findings (see details in "Questions" below).  

- The writing is in general hard to follow at times, in part due to many grammatical errors.
Yes
No ethics review needed.
* Can you offer insights into the relationship between SUPERB downstream task performance and model architecture designs?
* How do your results compare to recent speech representation work that has also been evaluated on SUPERB?
* The research community is highly interested in the topic of speech representation learning.
* The proposed method's evaluation on certain SUPERB tasks yielded better results compared to the cited systems.
* The discussions and comparisons presented are technically sound.
This submission introduces a framework for distinct representation learning of "content" and "other" properties in speech. The authors report improved performance on certain SUPERB tasks compared to other systems. Additionally, the submission includes comparisons with adapters, ablation studies on encoders, data augmentation, and learned representations.
Major issues:

* The model's effectiveness is unconvincing. The baselines cited are outdated and not state-of-the-art, and the model's performances on the semantic tasks are not better.
* The paper's discussion of different model architectures is shallow, limiting its contribution and making it difficult to draw general conclusions. 

Minor:

* Figure 1 could be simplified by removing the hyperparameters.
* The discussion of "Data augmentation" in Line 52 seems out of place, as the initial focus was on model architecture for speech representation learning.
Yes
No ethics review needed.
1) How does the JOCCI compare to the contentvec, SPIN, and Data2vec models?

2) What is the impact of initialization on the model performance e.g. random vs Ms-HuBERT initialization 

3) Can the model be trained with just the GRL loss and without the RDINO teacher?
1) The paper is well-written and addresses an important challenge of disentangling content and other information in the speech representation.

2) The model performs well on the SUPERB benchmark and outperforms HuBERT and wavLM.
The paper proposes to disentangle the content "what is being said" and other “how it is expressed” information present in the speech data. The paper proposes the JOCCI framework, which uses two submodules, focused on maximizing the content and the other information. The content module is trained with a self-supervised objective whereas, the other module is optimized with a teacher-student objective. A regularization loss is added to minimize the information overlap in the two submodules.
1) The baseline comparisons are limited. There have been other attempts to remove other information such as speaker information from the self-supervised representations such as contentvec[1] and SPIN[2]. Even the MS-HuBERT model used for initializing JOCCI is missing from Table 1.

2) JOCCI relies on a pretrained method RDINO for training the other encoder whereas baseline methods such as HuBERT do not. 

[1] “Contentvec: An improved self-supervised speech representation by disentangling speakers

[2] Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering
A framework capable of jointly learning comprehensive speech representations i.e., the content and other information.
@inproceedings{
anonymous2024jooci,
title={{JOOCI}: A {FRAMEWORK} {FOR} {LEARNING} {COMPREHENSIVE} {SPEECH} {REPRESENTATIONS}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=DnfPX10Etk},
note={under review}
}
Information in speech can be divided into two categories: what is being said (content) and how it is expressed (other). Current state-of-the-art (SOTA) techniques model speech at fixed segments, usually 10-25 ms, using a single embedding. Given the orthogonal nature of other and content information, attempting to optimize both within a single embedding results in suboptimal solutions. This approach divides the model's capacity, limiting its ability to build complex hierarchical features effectively. In this work, we present an end-to-end speech representation learning framework designed to jointly optimize the other and \enquote{content} information (JOOCI) in speech. By using separate learnable parameters, JOOCI addresses this optimization challenge by modeling other and content information independently. Our results show that JOOCI consistently outperforms other SOTA models of similar size (100 million parameters) and pre-training data used (960 hours) by a significant margin when evaluated on a range of speech downstream tasks in the SUPERB benchmark. Code and models are available at TBA.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
SSL Speech Representation Learning Joint Optimization
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f9c99e5be347095d17936a18c4c62c6306dfd01c.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
JOOCI: A FRAMEWORK FOR LEARNING COMPREHENSIVE SPEECH REPRESENTATIONS
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission