Enhancing Audio--Language Models through Self--Supervised Post--Training with Text--Audio Pairs
Self-supervised learning;Contrastive learning;Post-training;Audio;Zero-shot evaluation;Audio-retrieval.
Yes
No ethics review needed.
1. In the TerminAL A training stage, the model is fed both single-event and dual-event samples. Do these samples also include overlay cases? For instance, will "Dog" and "Dog while Cat" be presented in the same batch?

2. In Table 2, T-AB appears to improve the 2A and 2C tasks compared to T-B. This suggests that T-A introduces a bias that enhances performance. However, I am unclear on how the T-A training stage leads to this bias. If "Dog" and "Dog while Cat" are presented in the same batch, the samples for "Dog while Cat" will still be penalized under the "Dog" label. Could you provide more insight into how the model effectively retrieves single-class captions when presented with dual-event (overlay) audio samples?

3. Were additional temporal-enhanced captions considered for training, or was the model exclusively trained with prompts like [after, before, while]? For instance, were alternatives such as [then, followed by, and then] also utilized? 

4. The use of prompts when combining two audio events in CompA-CLAP [1] differs from the proposed TerminAL approach. Do you believe this discrepancy will affect the performance results presented in Table 3 and whether the comparison remains fair? For example, it is possible that the keywords [before, after] were not employed in the training of CompA-CLAP.

[1] CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models
The strengths of this paper lie in two main areas:

(1) **Two-Stage Post-Training**: The paper introduces a two-stage post-training method based on an assumption about the learning challenge in CLAP models, which is the difficulty gap related to the number of events versus their temporal order. By addressing these learning challenges progressively through distinct post-training stages, the paper demonstrates performance improvements. Additionally, it contributes by adapting the training objectives of CLAP to TNCE loss terms, though this is primarily referred by prior work in computer vision.

(2) **Zero-Shot Temporal Evaluation**: The introduction of ZSTE establishes a benchmark for audio understanding. The paper provides a range of tasks to thoroughly evaluate CLAP's performance in understanding temporal relations in audio samples.
This paper proposes a post-training method to enhance CLAP's (Contrastive Language-Audio Pretraining) understanding of temporal relations in audio samples. This post-training method consists of two stages: (1) post-training on single and dual audio events, and (2) post-training on varying temporal sequences of the same event combinations. The paper also presents a detailed evaluation on zero-shot temporal evaluation (ZSTE) to demonstrate that CLAP, with the proposed two-stage training, achieves a better understanding of audio semantics in the temporal relations, especially on the understanding of both events in the overlapping cases.
However, the weaknesses of this paper are substantial and can be categorized into three main points:

(1) **Limited Novelty and Generalization**:  The primary contribution lies in the two-stage post-training approach and its experiments; however, the concept of using augmented captions (such as concatenation and overlay) to improve CLAP’s understanding of temporal audio relations has already been explored in prior works [1][2]. Moreover, the paper lacks sufficient evidence and explanation to justify the necessity of a two-stage post-training approach (see point (2) below). Generalization is also a critical concern, as the paper primarily focuses on two-event audio scenarios and restricts prompts to a narrow set of captions—[before, after, while]. Real-world audio scenarios are significantly more complex, involving a wider variety of events and more natural language descriptors. Scaling this method to accommodate more events and captions poses a significant challenge, as the size of the contrastive matrices would become prohibitively large for processing.

(2) **Inadequate Experimental Setup to Demonstrate Effectiveness**:

a. While the proposed post-training method could be applied to existing CLAP models (e.g., LAION-CLAP, Microsoft-CLAP, CompA-CLAP), the authors did not showcase this flexibility. As mentioned in Section 3.4, “the encoders need to be pretrained", it would have been more effective to apply the post-training method to 2-3 different CLAP models and compare their performance with and without the post-training. This approach would demonstrate the post-training method's effectiveness without the need to train the CLAP model from scratch (as it appears the authors did) to save time. It also strengthens the evidence supporting the proposed method.

b. Effectiveness of Two-Stage Training (i.e., TerminAL A and B): Table 2 presents a partial ablation study but omits the results for TerminAL A-only. Including this result would provide a clearer picture of how TerminAL A alone impacts ZSTE performance, offering more comprehensive evidence for the individual effectiveness of each training stage.

c. Overall Performance: According to Table 3, the proposed T-CLAP does not consistently outperform previous models or yield comparable results on non-temporal tasks. CompA-CLAP and T-CLAP each achieve the top-1 performance on approximately half of the tasks. While CompA-CLAP struggles with tasks 2A and 2C, which require predicting both events, T-CLAP sacrifices substantial accuracy on task 1A (dropping from 82% to 75%), which is the most fundamental audio understanding task. These results suggest two conclusions: first, the single-dual contrastive training (T-A) step is crucial, as it drives improvements in tasks 2A and 2C. Second, a trade-off remains between temporal enhancement and core audio classification accuracy, as the proposed method fails to preserve the performance on basic classification task, even falling below ML-ACT, (the baseline before CLAP).

(3) **Poor Presentation**:  This paper lacks several necessary introductions on its regular pages. The paper should emphasize more on explaining the T-A and T-B training paradigms (see further details in the question section) instead of reiterating TNCE details, which are largely sourced from [3]. Additionally, the experimental section is hard to follow due to minimal guidance on the five tasks and their A-B(-C-D) variations. The analysis of these experiments is too brief, making it difficult to validate the effectiveness of the two-stage training method and assess the overall performance. Furthermore, the title of the paper is vague and somewhat misleading; it would benefit from a clearer focus, such as emphasizing **temporal relation enhancement** in **contrastive language-audio pretraining** models.

Above all, I found this paper falls below the standard of ICLR, for its limited novelty and generalizability to enhance CLAP performance; insufficient experimental setup to effectively validate the method’s impact and stability in audio understanding through post-training; and a lack of writing organization that hinders clear communication of the essential training paradigms.

[1] CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models

[2] T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining

[3] Test of Time: Instilling Video-Language Models with a Sense of Time
Yes
No ethics review needed.
Additional Questions:
1. As shown in Figure 1, what is the advantage of 2-stage training vs 1stage training? Doing a comparative study will be useful in understanding which stage is contributing more towards temporal understanding.
2. The performance on standard order understanding benchmarks in audio like COMPA-Order and COMPA-attribute is missing in the main paper.
1. The paper shows a critical gap in current ALMs for lacking temporal reasoning and acting as a "bag of words" while mapping audio and textual information using cosine similarity. To address this, the paper proposes Temin AL, a post-hoc alignment approach focusing on injecting temporal ordering among diverse acoustic events in existing ALMs
2. To evaluate temporal understanding in current ALMs in a zero-shot fashion, the authors also propose a multistage zero-shot temporal evaluation scheme, called ZSTE
This paper introduces an essential gap in contrastively pre-trained Audio-Language Models (ALMs) by showing their failure to capture correct temporal relationships between audio and various acoustic events. To address this, the authors propose TeminAL—a two-step post-training framework for infusing temporal awareness in ALMs. Additionally, the paper introduces ZSTE, a Zero-Shot Temporal Evaluation scheme designed to evaluate temporal understanding in ALMs in a zero-shot fashion.
1. The curriculum learning approach introduced by the authors while showing decent performance in audio-text retrieval tasks, still remains very similar to the CompA's modular contrastive pre-training approach. I will request the authors to address this in the main paper. 
2. The authors have used ESC50 for synthesizing temporal-rich audio segments (e.g., dog after cat, etc.). Did the authors explore more diverse audio sources like AudioSet Strong, which contains time-aligned information and more than 500+ labels? I will request the authors to add a few lines in the paper to explain the rationale behind using ESC50 when compared to other audio data sources.
3. I find important ZS experiments on standard audio-classification tasks like AudioSet, FSD50K, USD8K, TUTUrban, etc. missing in the main paper.
Yes
No ethics review needed.
- In table 1, are the results from different model in comparison trained by the authors? Or are they taking from publicly available models from each work?
- Temporal modeling capability in audio is a fairly novel and under-explored research area. The proposed zero shot evaluation benchmark is reasonable and can be a good contribution to the community.
The authors propose to include both data curation and loss function design in helping the CLAP model to understand better the temporal relation. They also propose a zero shot temporal evaluation framework with a series of tasks in order to benchmark specific temporal modeling capability.
- The use of equations in section 3 to explain the proposed loss function is not easy and straightforward to follow, and they do not connect well with figure 3. Consider adding some of the variables, loss terms from the equation into figure 3 to improve the connection for clearer explanation.
- TeminAL A can benefit from more detailed explanation. It is mentioned that TeminAL A is trained to distinguish between single and multiple sounds,  are they trained with similar contrastive loss? if so, how are the audio and text pairs curated? or are they trained with classification loss separately? It might worth adding a figure to explain the TeminAL A stage training, similar to figure 4 for TeminAL B.
- The more detailed explanation of ZSTE should be included in the main paper, current appendix B.4 contains most of the information for the reader to understand what proposed ZSTE is? Since these are one of the main contributions of this work, it is worth integrate into the main narratives.
- The explanation of the categories A, B, C, D used in Table 2 is buried in the caption of Table 3, these are also important information and might worth moving into the narrative in the sections explaining the evaluation and benchmarks.
- Minor suggestion: consider making the fonts in the figures larger, there are still space in the figure that can be adjusted.
- For section 3.3, in the l_text, x_c and x_a are referred to the raw inputs, if the loss function takes embeddings as inputs, should these variables be z instead of x?
Yes
N/A
No ethics review needed.
See weakness.
1. This paper proposes a novel 2-stage objective TeminAL AB for training audio-language contrastive models on temporal relationships.
2. This paper proposes a benchmark ZSTS to evaluate various audio-language zero-shot understanding tasks.
This paper presents a novel method for training audio-language contrastive models with sensitivity to temporal relations. This paper first constructed augmented data pairs that contain temporal relations, then trained with a novel 2-stage objective TeminAL AB. TeminAL AB first trains the model to distinguish single sounds and multiple sounds, then in the next stage it trains the model to distinguish between specific temporal relationships of the events. The objectives in TeminAL A and B consist of modified infoNCE loss which upweights the training of temporal relations. Last, this paper proposes a benchmark ZSTS to evaluate various audio-language zero-shot understanding tasks.
This paper lacks sufficient justification for its contributions, particularly due to limited comparison with prior works. Here are specific issues with the paper:

1. Comparison with Related Work: To my knowledge, the most relevant prior works are CompA [1] and T-CLAP [2]. The paper briefly introduces CompA in lines 90-104 but does not directly discuss how this work differs from or improves upon CompA.

2. Insufficient Comparative Analysis with CompA: CompA is only included for benchmarking performance. CompA generated and used a larger dataset with up to 110k samples, while this paper does not discuss the impact of dataset size or the types of datasets used in fine-tuning. CompA has its own method of defining tasks, such as using prompts like "before" and "after," while CompA uses "succeeded," "preceded," and "amidst." I am curious if the authors compared the effects of prompt design between these two models. Since CompA has a public benchmark, a fairer comparison would involve evaluating this model on CompA’s benchmark to highlight differences in approach. Perhaps the authors intend to show that their objective is superior to CompA’s, but there is no evidence provided to support this claim.

3. Comparison with T-CLAP: T-CLAP is also highly relevant, as both papers use the ESC-50 dataset to generate data. Although it seems the model and dataset in T-CLAP are not yet public, this work should at least include a discussion of T-CLAP’s approach.

4. Limitations in Contributions: Regarding the three contributions listed in this paper:
The first contribution has already been demonstrated in prior works [1-3].
The second requires more evidence and direct comparison with existing methods.
The third contribution lacks comparison with other benchmarks.

5. Discussion on Objective Functions: Although the authors note in the appendix that training with only the Terminal B objective is ineffective, I believe they should discuss why both Objectives A and B cannot be trained together.

Issues with Paper Presentation:
1. The text in the middle of Figure 3 is too small to read. I suggest placing Figures 3 and 4 together or at least referring to Figure 4 within Figure 3.
2. Line 334 contains a repeated "equation."
3. In line 334, it seems the numbering of the equation is incorrect. Eq 3 does not contain C^{cr} and C^{co}.

[1] Ghosh, S., Seth, A., Kumar, S., Tyagi, U., Evuru, C. K., Ramaneswaran, S., ... & Manocha, D. (2023). Compa: Addressing the gap in compositional reasoning in audio-language models. arXiv preprint arXiv:2310.08753.

[2] Yuan, Y., Chen, Z., Liu, X., Liu, H., Xu, X., Jia, D., ... & Wang, W. (2024). T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining. arXiv preprint arXiv:2404.17806.

[3] Wu, H. H., Nieto, O., Bello, J. P., & Salamon, J. (2023, June). Audio-text models do not yet leverage natural language. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.
The paper introduces TeminAL, a two-stage training method that enhances ALMs with temporal understanding, improving performance by 5.28% on ESC-50. ZSTE, a zero-shot evaluation strategy, shows TeminAL outperforms existing models on downstream tasks.
@inproceedings{
anonymous2024enhancing,
title={Enhancing Audio--Language Models through Self--Supervised Post--Training with Text--Audio Pairs},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=nplYdpc1Pm},
note={under review}
}
Research on multi-modal contrastive learning strategies for audio and text has rapidly gained interest. Contrastively trained Audio-Language Models (ALMs), such as CLAP, which establish a unified representation across audio and language modalities, have enhanced the efficacy in various subsequent tasks by providing good text aligned audio encoders and vice versa. These improvements are evident in areas like zero-shot audio classification and audio retrieval, among others. However, the ability of these models to understand natural language and temporal relations is still a largely unexplored and open field for research. In this paper, we propose to equip the multi-modal ALMs with temporal understanding without loosing their inherent prior capabilities of audio-language tasks with a temporal instillation method $\textbf{TeminAL}$. We implement a two-stage training scheme TeminAL A \& B, where the model first learns to differentiate between multiple sounds in TeminAL A, followed by a phase that instills a sense of time, thereby enhancing its temporal understanding in TeminAL B. This approach results in an average performance gain of $5.28$\% in temporal understanding on the benchmark ESC-50 dataset, while the model remains competitive in zero-shot retrieval and classification tasks on the AudioCap/Clotho datasets. We also note the lack of proper evaluation techniques for contrastive ALMs and propose a strategy for evaluating ALMs in zero-shot settings. The general-purpose Zero-Shot Temporal Evaluation $\textbf{(ZSTE)}$ strategy , is used to evaluate various prior models. ZSTE demonstrates a general strategy to evaluate all ZS contrastive models. The model trained with TeminAL successfully outperforms current models on most downstream tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Self-supervised learning Contrastive learning Post-training Audio Zero-shot evaluation Audio-retrieval.
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d712fcbe933afc260843f3f654437eb62c0f4030.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Enhancing Audio--Language Models through Self--Supervised Post--Training with Text--Audio Pairs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission