ADIFF: Explaining audio difference using natural language
audio processing; audio-language; multimodal learning
Yes
No ethics review needed.
- Its unclear from the paper why  Qwen-Audio Chat is a valid baseline for subjective evaluation in Table 3. The authors mention that its the only ALM that supports two audio inputs, but thats not a reason enough to use this as a baseline for subjective evals if it is not designed to do audio difference explanation task? 

- Insufficient information on human evaluation - its unclear how many evaluations were done here, how many reviewers were involved and how it was ensured that the evaluation is fair. Its unclear what the numbers in Table 3 mean, and hence deriving insights out of them is hard.

- Authors mention "ADIFF sees the largest improvement in granularity metric", but this is not very clear from the table 3 itself, a more detailed table, maybe in appendix that highlights this could be more readable.

- Its unclear why the datasets used for Table 3 different from ACD and CLD datasets?

- Its unclear how the LLM is finetuned. Is a full finetune performed? How about using PEFT  instead and utilizing larger models?
- Introduction of a new task - The paper addresses a previously underexplored area by defining the audio exploration task and the paper motivates this task well.

- Dataset creation and availability -  The authors took great care in creating new datasets ACD and CLD for this task - which is a significant contribution for motivating further research on this task. 

- Novelty in Model Architecture - the authors build upon existing literature to introduce a new cross projection layer that helps compare the two audios properly - a component which is absent in existing literature on audios.

- Comprehensive Evaluation - The authors do both subjective and objective evaluations to establish the models performance. The authors also include an ablation with random weights for Audio encoder - thus establishing a lower bound on the objective evaluations.

- Hallucination detection tools - Authors provide means to detect hallucinations in the models responses using audio event predictions via a HTSAT encoder.
This paper proposes a new task  - Audio Difference Explanation, which involves generating natural language description of differences between two audio recordings. They motivate this new task by pointing its significance in fields like audio forensics, audio quality assessment, and synthetic audio generation. They point out that  the current landscape of Audio-Language models like SALMONN and GAMA are geared towards applications like audio question answering, audio captioning, etc but havent addressed audio difference explanation. 

The authors introduce new datasets Audio Caps Difference (ACD) and Clotho Difference (CLD). These contain pair of audios with text representing difference between the audios generated by an LLM. The text descriptions are at three levels of detail: concise, brief, detailed. These descriptions are human verified and hallucination corrected for the test subset. 

The authors propose a model ADIFF to perform this task. This model encodes the two audios, followed by a projection layer to project these embeddings in LLM space. inserts a separator token in the latent space and concatenates the BPE tokenized representations from a question of the form "Explain the difference between the two audios". They pass this through a cross-projection layer, the output of which serves as an prompt to the LLM backbone (GPT2 in their work), which in turn produces a textual description of the difference between the audios. The authors indicate a 3 stage training of their model - 1. unimodal training of audio encoder, LLM (using pretrained models suffice), 2. audio grounding in language and 3. finetuning all components.

The authors provide ablations on how the cross projection layer is important. As a baseline, authors propose a "naive" baseline - which is similar to existing work in literature. The authors also perform both objective and subjective evaluations of their proposed model, and perform various ablations to understand it better.
- Absence of a concrete baseline - the baseline the authors compare ADIFF against is a clearly inferior version of the same model, and thus guaranteeing that ADIFF would perform better than this baseline. This baseline is more of an ablation of the components of the model.


- A major contribution of this work is "cross projection" layer that distinguishes ADIFF from baseline and existing literature. However there is Insufficient evidence of importance of the cross projection layer due to two factors:

1. The authors motivate that cross projection layer is very important by utilizing the bottom two sections of Table 4 where middle section represents  baseline model with frozen LLM, and lower section indicate baseline model + cross projection layer and sep token. But in the main model ADIFF, they finetune the LLM too, which could also learn what cross-projection learns, hence making having a cross projection layer redundant. If the comparison was between  baseline + LLM finetuned vs baseline + cross projection layer + LLM finetuned, it would indicate more conclusively that cross posting is useful.

2. Authors use GPT2 as the LLM model which ranges from a few hundred million params to 1.5 billion params. Its unclear which LLM was used for the cross projection ablation. Arguably, a larger LLM + finetuning could learn the task of cross projection layer. Its been shown that prompt tuning is very powerful for models like Llama3 8B models, and this ablation might not yield any difference there.


- The paper’s structure and presentation can be improved. There is redundancy and lack of detail in tables and captions —such as the overlapping results in Tables 2 and 4, missing key information about the LLM model used, and insufficient explanations of scores presented in Table 3, which collectively make it challenging to interpret the results accurately.

- The paper inaccurately equates compute across different-sized models in section 5.3 by training them for the same number of epochs, failing to account for the higher per-epoch computational cost of larger models. This undermines the  validity of their scaling comparisons in this section.
Yes
No ethics review needed.
1.Regarding the structure of the sequence sent to the cross-projection layer in Figure 3, what is the length of the audio prefixes (i.e., [prefix 1] and [prefix 2])? From my understanding, HTS-AT provides only one audio embedding for the entire segment of audio samples. Do [prefix 1] and [prefix 2] consist of just a single token representing the latent embedding from HTS-AT?

I suggest including a more detailed explanation of this aspect, rather than solely focusing on the right side of Figure 3.
The paper contains three strengths:

1.  The paper introduces a considerable novelty by defining a new audio task inspired by the development of multimodal language models. The approach to explaining audio differences effectively enhances the LLM's ability to understand audio inputs at both the semantic and acoustic levels. Additionally, the method incorporates primitives of human perception into the dataset creation process, progressively decomposing audio elements from audio events to acoustic scenes, signal-level analysis, emotions, and other properties. This enriches the data quality and practical usability.

2. The evaluation is thorough, primarily focusing on ablation studies. The experimental framework is well-structured, starting with a baseline model and a reference state-of-the-art model (Qwen-Audio Chat), followed by ablation studies that investigate different design choices within the proposed model.

3. The paper includes thoughtful discussions regarding the hallucination tendencies of the LLM when generating annotations. It provides insights into how the authors filter these data using existing methods to enhance dataset quality.

Overall, I find this paper valuable and believe it makes significant contributions to the audio community.
This paper introduces a novel task focused on understanding and explaining the differences between audio recordings through various human perception primitives. It proposes a language model-based (LM-based) approach to generate explanations for the differences between two audio samples, establishing a pipeline that utilizes a large language model and existing audio metadata to create different tiers of audio differences. The experiments evaluate various configurations of the proposed audio difference model, considering factors such as text-encoder size, model components, and multi-stage training paradigms. Additionally, the paper discusses the challenges associated with creating annotations and explores potential real-world applications for this task.
There is one weakness of this paper:

1. The presentation of the experimental section requires further refinement. The experimental tables are somewhat difficult to follow, as the model names are not consistently listed in most of them. Additionally, highlighting or bolding the best values for each metric would enhance clarity and emphasize the differences more effectively.
Yes
No ethics review needed.
I suggest the authors complete the ablation experiments and additional finetuning experiments suggested.
The paper proposes a valid methodology in generating "audio difference" captions through LLMs using human captions; the methodology is verified through experiments.
The authors introduces a new task, to explain differences between audio recordings using natural language. The authors show that current audio language models cannot complete this task, and propose AudioCaps and Clotho "Difference" sets, generated by LLM and verified by human in test set. These sets are trained using ADIFF, the proposed model, to bring performance boost.
1. The three-tier captioning process seems arbitary; no ablation study is conducted for this (i.e. if trained only on tier-3, can the models have good performance on tier-1? If they are purely hierarchical, then training on tier-3 should in theory yield good results on lower tiers.)
2. Both AudioCaps and Clotho's captioning quality is questionable; WavCaps and other larger caption datasets contain more diversity, yet no experiment is conducted on them.
3. The SOTA model, Qwen-Audio, seems to not be finetuned for this task as well. It prompts the question that finetuning a stronger model with out ADIFF may solve the problem entirely just through the proposed dataset.
Yes
No ethics review needed.
- What are the types of differences described in different tiers? Is there a distribution around differences in sounding objects, differences in sounding characteristics within the same category of sound? It would be really helpful to provide these analysis from LLM generated captions.
- For table 3, is the human evaluation also based on the difference between two audios? If so, other than the three datasets utilized (Studio, FSD50K, GTZAN), is there a reason why proposed ACD and CLD are not included in the human evaluation?
- The proposed task in audio difference is novel and interesting. This is a good contribution to the audio reasoning research community. The three-tier approach makes sense and these generated descriptions can be utilized in various research topics.
- The authors provide thorough ablation studies in proposed architectures including architectural choices, language model sizes, and training scenarios, etc.
The authors propose a new derived dataset from AudioCaps and Clotho for describing differences between audios in three different tiers corresponding to the complexities of outputted descriptions. They also propose ADIFF, an architecture improving upon baseline by adding a separator token and cross-projection layer for various captioning metrics.
- The choices of captioning metrics seem insufficient. As also observed by the authors in section 5.1, these metrics can be biased to the words irrelevant to the differences in audio. To complement the metrics, it might worth adding other benchmark such that the differences in sounding objects can be captured from the SED output from each audio, or the characteristics of audio can be derived from other signal processing techniques?
- Showing the complete captioning metrics in all the tables (2, 4, 5, 6) does not seem to help in the narratives, especially not many of them are referred in the text. It might worth consolidate and only show a couple, for example, in the audio captioning research, the community is starting to converge to several more representative metrics that are more relevant to the task 1. Also it might be helpful to bulk the high performant numbers that are also referred to in the text.
- In table 1, is the number of examples from the number of audio in each dataset? It seems that the proposed task requires two audio, therefore the dataset should be consisted of {a_i, a_j, e_i}? If this is the case, how many pairs are included in each of the split? Also, in the original dataset, each audio can include more than 1 caption (except the train split in AudioCaps), how are these captions selected as the seed for LLMs to generate descriptions, these are not clearly explained in the main paper nor in the appendix.

1 https://dcase.community/challenge2024/task-automated-audio-captioning#evaluation
A new task, dataset and model for explaining audio differences using natural language
@inproceedings{
anonymous2024adiff,
title={{ADIFF}: Explaining audio difference using natural language},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=l4fMj4Vnly},
note={under review}
}
Understanding and explaining differences between audio recordings is crucial for fields like audio forensics, quality assessment, and audio generation. This involves identifying and describing audio events, acoustic scenes, signal characteristics, and their emotional impact on listeners. This paper stands out as the first work to comprehensively study the task of explaining audio differences and then propose benchmark, baselines for the task. First, we present two new datasets for audio difference explanation derived from the AudioCaps and Clotho audio captioning datasets. Using Large Language Models (LLMs), we generate three levels of difference explanations: (1) concise descriptions of audio events and objects, (2) brief sentences about audio events, acoustic scenes, and signal properties, and (3) comprehensive explanations that include semantics and listener emotions. For the baseline, we use prefix tuning where audio embeddings from two audio files are used to prompt a frozen language model. Our empirical analysis and ablation studies reveal that the naive baseline struggles to distinguish perceptually similar sounds and generate detailed tier 3 explanations. To address these limitations, we propose ADIFF, which introduces a cross-projection module, position captioning, and a three-step training process to enhance the model’s ability to produce detailed explanations. We evaluate our model using objective metrics and human evaluation and show our model enhancements lead to significant improvements in performance over naive baseline and SoTA Audio-Language Model (ALM) Qwen Audio. Lastly, we conduct multiple ablation studies to study the effects of cross-projection, language model parameters, position captioning, third stage fine-tuning, and present our findings. Our benchmarks, findings, and strong baseline pave the way for nuanced and human-like explanations of audio differences.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
audio processing; audio-language; multimodal learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2e2090bbfb1a11aafa6ca4ed358ac29b2fdbe5c4.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/4a707bb5d6b25b4a7169d0818df8da59c30e9e37.zip
ADIFF: Explaining audio difference using natural language
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission