EVLM: An Efficient Vision-Language Model for Visual Understanding
multi-modal language models;cross-attention;moe
violation of formatting guideline (margin, length)
Submission Desk Rejected by Program Chairs
an efficient vision-language model for visual understanding.
@misc{
chen2024evlm,
title={{EVLM}: An Efficient Vision-Language Model for Visual Understanding},
author={Kaibing Chen and Dong Shen and Hanwen Zhong and Huasong Zhong and Kui Xia and Di Xu and Wei Yuan and Yifei Hu and Bin Wen and Tianke Zhang and Changyi Liu and Dewen Fan and Huihui Xiao and Jiahong Wu and Fan Yang and Tingting Gao and Di ZHANG},
year={2024},
url={https://openreview.net/forum?id=S7M1iqFLVm}
}
In the field of multi-modal language models, the majority of methods are built on an archi-
tecture similar to LLaVA. These models use a single-layer ViT feature as a visual prompt,
directly feeding it into the language models alongside textual tokens. However, when dealing
with long sequences of visual signals or inputs such as videos, the self-attention mechanism
of language models can lead to significant computational overhead. Additionally, using
single-layer ViT features makes it challenging for large language models to perceive visual
signals fully. This paper proposes an efficient multi-modal language model to minimize
computational costs while enabling the model to perceive visual signals as comprehensively
as possible. Our method primarily includes: (1) employing cross-attention to image-text
interaction similar to Flamingo. (2) utilize hierarchical ViT features. (3) introduce the
Mixture of Experts (MoE) mechanism to enhance model effectiveness. Our model achieves
competitive scores on public multi-modal benchmarks and performs well in tasks such as
image captioning and video captioning.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Kaibing_Chen2 ~Dong_Shen1 ~Hanwen_Zhong2 ~Huasong_Zhong2 ~Kui_Xia1 ~Di_Xu8 ~Wei_Yuan2 ~Yifei_Hu2 ~Bin_Wen3 ~Tianke_Zhang1 ~Changyi_Liu1 ~Dewen_Fan1 ~Huihui_Xiao1 ~Jiahong_Wu2 ~Fan_Yang30 ~Tingting_Gao1 ~Di_ZHANG3
Kaibing Chen Dong Shen Hanwen Zhong Huasong Zhong Kui Xia Di Xu Wei Yuan Yifei Hu Bin Wen Tianke Zhang Changyi Liu Dewen Fan Huihui Xiao Jiahong Wu Fan Yang Tingting Gao Di ZHANG
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
multi-modal language models cross-attention moe
I certify that there is no acknowledgement section in this submission for double blind review.
chen|evlm_an_efficient_visionlanguage_model_for_visual_understanding
/pdf/f08136569bd163123344812329cdba93ffd55a16.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
EVLM: An Efficient Vision-Language Model for Visual Understanding
ICLR 2025 Conference Desk Rejected Submission
ICLR.cc/2025/Conference/Desk_Rejected_Submission