AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation
Audio-Visual Segmentation;real-time model;efficient architecture
Yes
No ethics review needed.
1. Line 53-76 tries to explain the issues of exiting AVS method. Although it's reasonable that effective multimodal fusion should be critical for AVS, it's not clear why this claim hold: "attention variants generally do not exhibit the same expressive capacity as the default mechanism" (line 75). Further explanation is needed.
2. In Fig. 2, the paper illustrates of attention dissipation. How the audio token is generated? Is this only a one case issue or does it happen all the time in all types of cross attention based AVS? More details are needed to verify the existence of the attention dissipation issue, especially when audio is defined as both key and value in the current cross attention setting. Moreover, where does the cross attention AVS model in equation 1 come from? Citation(s) are needed if it comes from existing literatures, or explanations are needed to explain why QKV is defied in the current way (compared to Ref1, Ref2, Ref3).
3. "Attention maps at early decoder stages tends to capture short-term local correlation features, leading to undesired low utilization of attention" (line 89-91) is used to explain the "narrow attention" issue, which is also not very clear. Why call it "narrow"?
4. In line 123, it's not clear why the existing AVS methods fail to work on the "single frame image + audio" setting.
5. what is the audio backbone in line 184?
6. F_audio=k=v, not clear (line 194). Also, audio feature dimension is D in line 185 and c in line191. It's better to use consistent symbols.
7. The prompt query generator (sec. 4.1) augments the audio feature. How does it bring extra useful information given the augmentation is based on the raw audio feature?
8. The early focus decoder is used to solve the "narrow attention" issue. However, it's not clear how eq. 6 and eq. 7 combined can solve the narrow attention issue.
9. In Table 1, why Ref1 is not compared?
10. In Table 4 and Table 5, it seems the proposed two strategies work better for MS3 dataset, explanations are needed.

Ref1: Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation. CVPR 2024

Ref2: AVSegFormer: Audio-Visual Segmentation with Transformer. AAAI 2024

Ref3: Audio–visual segmentation. ECCV 2022
1. Solving AVS via attention analysis is reasonable given the multimodal nature of AVS.
2. The attention dissipation issue seems interesting given the current QKV setting.
3. The experiments are comprehensive.
This paper finds two primary challenges in existing audio-visual segmentation models, namely attention dissipation caused by anomalous attention weights after Softmax over limited frames, and narrow attention patterns in shallow decoder stages leading to inefficient utilization of attention resources. An AVESFormer is then presented as the first real-time audio-visual efficient segmentation transformer. Particularly, the proposed method leverages a prompt query generator to rectify cross-attention behavior, and an early focus decoder to enhance efficiency. Extensive experiments demonstrate superiority of AVESFormer in mitigating cross-attention issues and achieving a trade-off between model effectiveness and efficiency.
1. The two observations which motivate the proposed method are not clear explained (see details in the Questions section).
2. The setting of the paper for real time AVS should also be verified, especially it should be compared to naive settings, e.g. defining the rest of the frames as an empty token. Most importantly, as temporal information is totally discarded, more analysis is needed on how the proposed method achieve smooth real time AVS.
3. Comparison with other attention based techniques is needed (Ref 1 in the Questions section.)
Yes
No ethics review needed.
See weaknesses
1. The paper identifies critical issues in existing audio-visual segmentation (AVS) models, such as attention dissipation and narrow attention patterns, which are valid challenges in the field.

2. The proposed AVESFormer model aims to address efficiency in real-time AVS tasks, which could be valuable for applications needing low-latency responses.

3. The model demonstrates competitive performance on standard benchmarks with a noted improvement in latency, supporting claims of efficiency gains.
The paper introduces AVESFormer, a transformer model designed for real-time audio-visual segmentation (AVS). It addresses attention dissipation and narrow attention patterns in AVS by implementing a Prompt Query Generator (PQG) and Early Focus (ELF) Decoder. These components aim to improve cross-attention efficiency and reduce computational costs. AVESFormer shows competitive performance on AVSBench with noted speed improvements.
While the issues identified in attention mechanisms are relevant, the approach—specifically the Prompt Query Generator and Early Focus Decoder—largely leverages incremental modifications rather than substantial innovations. Many elements, such as convolutional layers for early-stage feature extraction, are already established techniques in efficient transformers.
Yes
No ethics review needed.
The authors should consider optimizing the audio backbone, Vggish, which currently comprises about 60% of the model parameters, to enable better deployment on mobile devices. Next, the authors are encouraged to explore how incorporating temporal information might influence model performance and inference speed. Additionally, evaluating AVESFormer on a broader range of datasets beyond AVSBench would provide insights into its generalization capabilities. The paper currently lacks a detailed quantitative analysis of attention maps, so including metrics to evaluate the focus and spread of attention could enhance understanding of the model's behavior.
A key innovation is the Prompt Query Generator PQGPQG, which corrects cross-attention behavior and mitigates attention dissipation, enhancing segmentation accuracy and efficiency. Additionally, the Early Focus ELFELF decoder incorporates convolutional operations for local feature extraction, reducing computational demands by replacing attention operations in early transformer stages. The model achieves state-of-the-art performance, surpassing previous models on metrics like the Jaccard index and F-score across datasets such as S4, MS3, and AVSS.
The paper presents AVESFormer, a transformer-based model for real-time audio-visual segmentation AVSAVS. It tackles two main issues in existing AVS models: attention dissipation and limited attention patterns in early decoders. AVESFormer uses a Prompt Query Generator PQGPQG to improve cross-attention and an Early Focus ELFELF decoder that integrates convolutional operations for efficient local feature extraction, reducing computational costs. Experiments show that AVESFormer addresses cross-attention problems, enhances attention utilization, and outperforms previous state-of-the-art models in both performance and speed. Additionally, the paper analyzes attention dissipation and the shortcomings of standard transformer decoders in real-time AVS.
One concern is the significant parameter count of the audio backbone, Vggish, which limits deployment on mobile devices, suggesting future work could optimize this component. Additionally, the model currently ignores temporal information from multiple frames, and incorporating this data could enhance its ability to track moving objects. There is also a need to test the model's generalization to unseen datasets beyond AVSBench to evaluate its robustness in diverse scenarios. While qualitative analyses of attention maps are presented, a quantitative investigation could offer deeper insights into the model's attention mechanisms. Furthermore, comparing AVESFormer with non-transformer models would provide a broader context for its performance. The paper could also benefit from a more detailed discussion on the impacts of model complexity on real-time performance and ethical considerations regarding its application.
Yes
No ethics review needed.
see Weaknesses
- The paper is well-written and easy to follow.

- The authors offer comprehensive ablation studies, which aid in comprehending the various design decisions involved in solving cross-attention issues.

- The proposed method shows a strong performance improvement as well as a reduction in the inference time.

- Sufficient comparison with SOTA methods
The authors proposed AVESFormer, an efficient framework for audio-visual segmentation tasks. By solving two issues of using cross-attention as attention dissipation and narrow attention patterns, they can effectively improve the utilization of cross-attention between audio and visual. AVESFormer improves attention utilization, improves the performance as well as reduces the running time.
- Some typo: Table 3: lartency -> latency

- Regarding the PQG, what is the result if we set the number of queries to the number of objects in the category?

- How much of the flops and memory that AVESFormer save from replacing the early attention with convolution?

- Does the number of stages in the decoder affect the performance?

- Missing references [1], [2], and [3] in Table 1

[1] CVPR2024: Audio-Visual Segmentation via Unlabeled Frame Exploitation 

[2] Referred by multi-modality: A unified temporal transformer for video object segmentation

[3] Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation
We present AVESFormer, the first real-time transformer for audio-visual segmentation, overcoming attention dissipation and inefficient decoding. AVESFormer significantly improves the trade-off between performance and speed.
@inproceedings{
anonymous2024avesformer,
title={{AVESF}ormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=u8SYRtXDsZ},
note={under review}
}
Recently, transformer-based models have achieved remarkable performance on audio-visual segmentation (AVS) tasks.
  However, the cross-attention mechanism exhibits suboptimal behavior and unsatisfactory results in real-time scenarios.
  By analyzing attention maps, we identify two primary challenges in existing AVS models: 1) *attention dissipation,* caused by anomalous attention weights after Softmax over limited frames, and 2) *narrow attention patterns* in shallow decoder stages leading to inefficient utilization of attention resources.
  In this paper, we introduce *AVESFormer*, the first real-time audio-visual efficient segmentation transformer that achieves fast, efficient and light-weight simultaneously.
  Our model leverages an efficient prompt query generator to rectify cross-attention behavior. 
  Moreover, we propose ELF decoder, which enhances efficiency by incorporating convolutional operations tailored for local feature extraction, thus reducing computational overhead.
  Extensive experiments demonstrate that AVESFormer effectively mitigates cross-attention issues, substantially improves attention utilization, and outperforming previous state-of-the-art, achieving a superior trade-off between performance and speed.
  Code can be found in supplementary material.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Audio-Visual Segmentation real-time model efficient architecture
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a20ec002252ca15ccb14b5d9616c74af7d4f64a1.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/7f2dc8e3a1f75a2054539cf8d4808d9918a8f1c2.zip
AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission