MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Kaining Ying Fanqing Meng Jin Wang Zhiqian Li Han Lin Yue Yang Hao Zhang Wenbo Zhang Yuqi Lin Shuo Liu Jiayi Lei Quanfeng Lu Runjian Chen Peng Xu Renrui Zhang Haozhe Zhang Peng Gao Yali Wang Yu Qiao Ping Luo Kaipeng Zhang Wenqi Shao


[
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI














equal*
corr†




Kaining Yingequal,pjlab
Fanqing Mengequal,sjtu,pjlab
Jin Wangequal,hku
Zhiqian Lipjlab,hku
Han Linsjtu,pjlab
Yue Yangsjtu,pjlab
Hao Zhangpjlab
Wenbo Zhanguoa
Yuqi Linpjlab,zju
Shuo Liupjlab
Jiayi Leipjlab,sjtu
Quanfeng Lupjlab
Runjian Chenpjlab,hku
Peng Xupjlab,hku
Renrui Zhangpjlab
Haozhe Zhangzju
Peng Gaopjlab
Yali Wangcas
Yu Qiaopjlab
Ping Luohku,pjlab
Kaipeng Zhangcorr,pjlab
Wenqi Shaocorr,pjlab






pjlabShanghai Artificial Intelligence Laboratory
hkuThe University of Hong Kong
sjtuShanghai Jiao Tong University
casShenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
zjuZhejiang University
uoaThe University of Adelaide


Wenqi Shaoshaowenqi@pjlab.org.cn
Kaipeng Zhangzhangkaipeng@pjlab.org.cn




Machine Learning, ICML

0.3in
]










 


Large Vision-Language Models (LVLMs) show significant strides in general-purpose multimodal applications such as visual dialogue and embodied navigation. However, existing multimodal evaluation benchmarks cover a limited number of multimodal tasks testing rudimentary capabilities, falling short in tracking LVLM development. In this study, we present MMT-Bench, a comprehensive benchmark designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, reasoning, and planning. MMT-Bench comprises 31,325 meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering 32 core meta-tasks and 162 subtasks in multimodal understanding. Due to its extensive task coverage, MMT-Bench enables the evaluation of LVLMs using a task map, facilitating the discovery of in- and out-of-domain tasks. Evaluation results involving 30 LVLMs such as the proprietary GPT-4V, GeminiProVision, and open-sourced InternVL-Chat, underscore the significant challenges posed by MMT-Bench. We anticipate that MMT-Bench will inspire the community to develop next-generation multimodal foundation models aimed at achieving general-purpose multimodal intelligence.





§ INTRODUCTION



In recent years, Large Vision-Language Models (LVLMs) <cit.> have emerged as powerful tools for advancing artificial intelligence, demonstrating remarkable progress in various domains such as visual dialogue, video analysis and document understanding. Driven by diverse and high-quality instruction fine-tuning data mined from various fields, LVLMs will continue to advance towards multitask AGI <cit.>. As pointed out in Levels of AGI <cit.>, the breadth (generality) of tasks is a fundamental criterion for different levels of AGI. A multitask AGI model can perform a wide range of tasks across different domains with human-like proficiency, which could revolutionize many fields such as personalized education <cit.> and medical diagnosis <cit.>. Therefore, it is crucial to build a comprehensive evaluation benchmark to track multitask AGI development.

However, evaluating LVLMs significantly lags behind their development <cit.>. A line of work attempts to bridge this gap by proposing various multimodal evaluation benchmarks. Examples include LVLM-eHub <cit.>, MMBench <cit.>, MME <cit.>, and SEED-Bench <cit.>, which propose dimensions of multimodal capabilities and corresponding test samples. However, these benchmarks have limited coverage of multimodal tasks while testing rudimentary capabilities like visual recognition and text-scarce OCR. Therefore, they cannot fulfil the requirement of the breadth of tasks <cit.>. Moreover, recent LVLMs continue to excel in these benchmarks. For instance, InternLM-XComposer2 <cit.> achieved 2242.7/2800 and 79.6/100 overall performance on MME and MMBench, respectively. Other works, such as MathVista <cit.> and MMMU <cit.>, focus on discipline knowledge understanding and reasoning but are constrained to visual questions with scientific diagram images, limiting their breadth for benchmarking multitask AGI.

  



To address this challenge, we introduce MMT-Bench, a new benchmark designed to comprehensively assess LVLMs in multimodal multitask understanding. The breadth of MMT-Bench features in three aspects. First, MMT-Bench is meticulously curated and comprises 32K multi-choice visual questions covering 32 core meta-tasks and a total of 162 subtasks (Fig. <ref>), which is 8.1 times larger than MM-Bench <cit.>. 

Second, it encompasses 13 image types such as natural scenes, synthetic images, depth maps, text-rich images, paintings, screenshots, point clouds, medical images, et al. (Fig. <ref>). Such diversity demands the model to be capable enough to interpret various visual inputs.

Third, MMT-Bench spans multimodal scenarios such as vehicle driving, GUI navigation, and embodied AI, testing 14 kinds of multimodal capabilities including visual recognition, localization, reasoning, OCR, counting, 3D perception, temporal understanding, et al. (Fig. <ref>).

We assess 30 publicly available LVLMs under various input modes for best evaluation performance. Our findings highlight the significant challenges posed by MMT-Bench. For instance, GPT-4V only achieves 62.0/100 and 55.6/100 overall scores across all subtasks and subtasks except for visual recognition tasks, respectively, indicating significant room for improvement towards multitask AGI. Thanks to the extensive coverage of multimodal tasks, MMT-Bench enables the evaluation of LVLMs using a task map. This facilitates the discovery of both in- and out-of-domain tasks, providing valuable insights for multimodal commercial applications and ongoing efforts to enhance LVLMs. We summarize the findings as follows:



    
  * The open-source model InternVL-chat has taken a leading position in MMT-Bench, surpassing other closed-source models such as QWen-VL-Plus, GPT-4V, and GeminiProVision.
    
  * The comprehensive error analyses conducted on 162 multimodal tasks reveal that top-performing LVLMs such as InternVL-chat, GPT4V, and GeminiProVision are predominantly prone to perception, reasoning, and knowledge errors. 
    
    
  * The taxonomy analysis shows that current LVLMs perform well in tasks related to visual recognition and description which are in-domain tasks, yet fall short in tasks related to localization and pixel perception which are out-of-domain tasks.
    
  * BLIP2 that does not undergo instruction tuning even outperforms most LVLMs that are tuned by millions of instruction-following data, implying that instruction-tuning with data in some tasks even hurts the generalization on other tasks.
    
  * Certain tasks show improved performance with specific prompting methods, such as multi-image and coordinate-related tasks, as well as those involving visual referring prompts. However, most models do not exhibit improved performance with visual prompting, suggesting potential areas for future enhancement.
    
  * Model performance significantly improves with an
increase in size (7B to 13B) for both LLaVA-v1.5 and LLaVA-
v1.5-Xtuner. Upgrading LLMs, from InternLM to InternLM2,
also enhances the performance of LLaVA.



Overall, the contributions of this work are three-fold. i) We build a new evaluation benchmark called MMT-Bench for multimodal multitask comprehension, allowing us to measure the progress on the path to multitask AGI. ii) We evaluate various publicly available LVLMs on MMT-Bench, revealing that current LVLMs including InternVL-Chat, GPT-4V, and GeminiProVision achieve plain performance in multitask intelligence. iii) We present a taskonomy analysis by evaluating LVLMs on a task map built upon MMT-Bench, facilitating the discovery of both in- and out-of-domain tasks relative to current LVLMs. We anticipate that MMT-Bench will inspire the community to push the boundaries of LVLM research and development, driving us closer to the realization of truly intelligent multimodal systems. The MMT-Bench is open-sourced at <https://github.com/OpenGVLab/MMT-Bench>.








§ RELATED WORK


LVLM. As the Large Language Models (LLMs) continue to garner impressive achievements <cit.>, academic emphasis is increasingly shifting towards the exploration and development of Large Visual Language Models (LVLMs), to bolster the multimodal understanding and generative capabilities of models. Some notable open-source LVLMs, such as mPLUG-Owl2 <cit.>, LLaVA <cit.>, and LLaMA-Adapter <cit.>, have adopted LLMs as their backbone, processing visual features through these LLMs, ultimately achieving an innovative integration of text and visuals. In addition, closed-source models like Gemini <cit.> and GPT-4V <cit.> have demonstrated remarkable results across numerous tasks, making groundbreaking contributions. 
We aim to undertake an in-depth and comprehensive exploration of LVLMs and their capabilities by testing them on massive multimodal tasks.


LVLM Evaluation.
Recently, LVLMs have demonstrated remarkable capabilities to handle many visual-language tasks, which makes previous single-task benchmarks <cit.> insufficient to provide comprehensive evaluations of current LVLMs.
To this end, current LVLM evaluation benchmarks aimed to provide relatively holistic evaluations for the overall reasoning capabilities of LVLMs, such as OwlEval <cit.>, LVLM-eHub <cit.>, SEED-Bench <cit.>, LAMM <cit.>, MM-Vet <cit.> and MMBench <cit.>.
However, these benchmarks only covered a small range of multimodal tasks and vision-language skills, making them not comprehensive enough to asses multitask AGI capabilities.
Besides, recent studies also presented benchmarks of LVLMs which required expert-level domain knowledge, such as Mathvista <cit.> and MMMU <cit.>.
In comparison, our proposed MMT-Bench covers an extensive range of multimodal reasoning capabilities with sufficient test samples from various modalities as shown in Table  <ref>, which requires expert knowledge and deliberate visual
recognition, localization, reasoning, and planning. Our MMT-Bench poses significant challenges for the current state-of-the-art LVLMs.

Multitask Analysis.

Characterizing various tasks and establishing inter-task relationships is an effective means for multitask analysis <cit.>, with wide applications in areas such as meta-learning and transfer learning. A substantial amount of research has been conducted in Taskonomy <cit.>. It utilizes transfer learning to model the structure of the space of visual tasks, thereby harnessing the interconnections among visual tasks to avoid redundancy in learning. Task2Vec <cit.> extracts fisher information as task vectors, which is used in meta-learning. In our paper, thanks to the vast amount of task data collected, we evaluate LVLMs on a task map and conclude challenging tasks for the current LVLMs.





§ MMT-BENCH

In this section, we describe how to build the task hierarchy in Sec. <ref> and the pipeline of data collection in Sec. <ref>.


 §.§ Tasks


Hierarchical Task Structure. We utilize a hierarchical structure to include as more as multimodal tasks to build the MMT-Bench. First, all co-authors come up with meta-tasks for multimodal understanding by brainstorming. We then collect 32 meta-tasks by deduplication and filtering for important tasks as depicted in Fig. <ref>. Second, we decompose each meta-task into several subtasks. The subtask is kept in the MMT-Bench by three criteria. i) Whether the subtask examines the basic multimodal capability. ii) Whether the subtask challenges the current LVLMs. iii) Whether the test sample for the subtask can be publicly accessible. After selection, MMT-Bench comprises 162 sub-tasks, which is 3.8 times larger than TinyLVLM-eHub which previously contained the most tasks <cit.>. The detailed comparison between MMT-Bench and previous benchmarks is provided in Table <ref>. We also present the whole hierarchical structure in Table <ref> of the Appendix.





 §.§ Data Collection


We design an efficient pipeline (see Fig. <ref>) to construct multi-choice visual questions evaluation data for each subtask and the data collection is completed by dozens of co-authors specializing in artificial intelligence.

Datasets Search.
We conduct comprehensive searches for related datasets using various sources such as Google, Paper With Code, Kaggle, and ChatGPT, based on the name of the subtask. After downloading the datasets, we meticulously assess their suitability for evaluating the subtask, ensuring usability and relevance. While most tasks have multiple datasets available, a few may only have one dataset publicly accessible.


Metadata Construction.
We define a uniform format, the metadata, to collate downloaded datasets. It enables the further generation of visual questions and answers. Each sample of metadata consists of images and meta-information. The meta-information (see Fig. <ref>) includes the necessary information to generate questions and answers for the evaluation and also includes manual annotations of required capabilities and the type of visual prompt (i.e., input image). For evaluation efficiency, in each task, we keep the maximum number of samples at 200 by random sampling, and each dataset comprises the same number of samples.


Question and Answer Generation.
For each subtask, we generate multi-choice (maximum eight choices depending on the task) visual questions with choices and answers from their metadata. Specifically, depending on a specific task, we manually design rules or use ChatGPT with well-designed prompts for efficient and high-quality generation. For example, in sketch2image retrieval, we use the corresponding image as a ground-truth answer and generate other choices by randomly sampling other images from metadata. In video captioning, we use ChatGPT to write confused wrong choices.

Dataset Statistics. MMT-Bench comprises 31,325 meticulously curated multi-choice questions with 13 input image types such as natural scenes, synthetic images, text-rich images, medical images, et al. (see Fig. <ref>), covering 32 core meta-tasks and 162 subtasks for multitask multimodal understanding.
Compared to previous LVLMs benchmarks <cit.> addressing limited image types and skills, questions in MMT-Bench span diverse multimodal scenarios such as GUI navigation and document understanding, testing 14 kinds of capabilities including visual recognition, localization, reasoning, OCR, counting, 3D perception, temporal understanding, et al., as shown in Fig. <ref>. These features ensure that MMT-Bench meets the requirement of task breadth for evaluating multitask AGI.








§ EXPERIMENTS

In this section, we conduct a comprehensive evaluation of 30 LVLMs on the MMT-Bench. Sec. <ref> presents the selected LVLMs zoo and the evaluation methods. The quantitative evaluation of each meta-task is provided in Sec. <ref>. We present the analysis of specific tasks with different prompt methods in Sec. <ref>. Furthermore, we give an error analysis of three representative LVLMs in Sec. <ref>.





 §.§ Evaluation Details



Selected LVLMs. 
For completeness, we test 30 representative LVLMs varying in parameters, vision encoders (InternVL <cit.>, EVA-CLIP-ViT <cit.>, CLIP-ViT <cit.>), and LLMs (QWen <cit.>, InternLM <cit.>, LLaMA <cit.>, Vicuna <cit.>, Flan-T5 <cit.>). For details, see Appendix <ref>.






Evaluation Methods. 
In MMT-Bench, samples are in a multi-choice format, e.g., `What is this? Options: (A) Dog (B) Cat'. To extract the choice from LVLMs' responses, we follow OpenCompass' protocol <cit.>: 1) Check if the response includes option letters (A/B); 2) Check for option content (`dog'/`cat'); 3) Use ChatGPT for extraction. If these steps fail, we set the model selection as option letter Z to avoid random assignment <cit.>. 
Accuracy is the primary metric. 





 §.§ Overall Evaluation





This section evaluates LVLMs on MMT-Bench alongside Random Choice and Frequent Choice baselines. We report the overall score for all meta-tasks as well as the best performance on each meta-task in Table <ref>. The detailed results of each sub-task are provided in the Sec. <ref> of the Appendix. Various prompt settings for all tasks are investigated. We summarize the key findings as follows.


i) The Comprehensive Challenge of MMT-Bench: The benchmark poses significant challenges, with even advanced models like InternVL-Chat, GPT-4V and GeminiProVision achieving just 63.4%, 62.0% and 61.6% accuracy, respectively, indicating substantial room for improvement. Notably, removing its strongest area, Visual Recognition (VR), where it scores 84.7%, GeminiProVision's overall performance drops to 55.1%, below satisfactory. The varied task dimensions of the MMT-Bench demand wide-ranging capabilities for optimal performance, emphasizing the benchmark's extensive and rigorous criteria. 

ii) The comparison between Open-source LVLMs and close-source LVLMs. 
The performance of most open-source models lags behind that of closed-source models.
However, leading open-sourced LVLM InternVL-Chat-V1.2-34B have demonstrated remarkable performance, outperforming sophisticated proprietary models such as GPT-4V and GeminiProVision in overall accuracy. 

This achievement suggests that by scaling model size, optimizing training regimes, and leveraging diverse high-quality data, open-sourced LVLMs can rival and even exceed the capabilities of advanced proprietary models. It brings a sense of pride to the open-source community and paves the way for more high-performance yet cost-effective solutions in academia and industry.









iii) The Influence of LLMs and Model Scaling. 
As shown in Table <ref>, model performance significantly improves with an increase in size (7B to 13B) for both llava-v1.5 and llava-v1.5-tuner. Upgrading LLMs, from internlm to internLM2, also enhances the performance of LLaVA, suggesting that larger or improved LLMs boost multi-task performance, with unchanged training data and visual encoders.

iv) Model Performance across Different Meta-Tasks. 
Most LVLMs excel in Visual Recognition (VR) tasks and Visual Captioning (VC), highlighting the ability of LVLMs to recognize `what' an object is and describe the content shown in the image. However, for fine-grained perception tasks (localization, pixel-level perception, etc) or complex reasoning tasks (image evaluation judgment), most LVLMs struggle.

v) BLIP2 impresses in open-source models without instruction-following training, outdoing LLaVA models trained with extensive instruction-following data. Although instruction-tuned models can give responses aligning better with human preference than BLIP2 in open-set QA on some tasks <cit.>, they perform worse than BLIP2 in close-set settings in MMT-Bench. This reflects MMT-Bench's multi-task challenges and hints at using the taxonomy of MMT-Bench to expand the dataset in supervised fine-tuning for future advancement.
















































































































































 §.§ Specific Task and Prompt Methods Analysis


In this section, we evaluate specific tasks using different prompts for LVLMs. 































































Prompting LVLMs with multi-images vs single-image. Here we explore the effects of exploiting multi-image prompts and single-image prompts on the performance of LVLMs.
To this end, we summarized 28 tasks in our MMT-Bench, which usually require multiple images as input, such as image retrieval and video captioning.

For multi-images prompting, we first evaluated LVLMs which are inherently designed to support multiple images as input (dubbed Multi-Images LVLMs), including mPLUG-Owl2, QWen-VL-chat and Gemini-Pro-Vision.
Besides, we also assessed LVLMs which mainly learned on single-image prompts (dubbed Single-Image LVLMs) for more comprehensive comparisons, including BLIP2, SharedCaptioner, ShareGPT4V-7B, Monkey and LLaVA-v1.5-7B.
Following previous studies <cit.>, we input each image individually to Single-Image LVLMs and concatenated all output visual embeddings before feeding into LLMs. 
The designed multi-image prompts for Multi-Images LVLMs and Single-Image LVLMs are summarized in Appendix Sec.<ref>.
As for single-image prompting, we manually combine multiple images into one image and feed it into LVLMs (see examples in Fig. <ref>).

The detailed performance comparisons are presented in Fig. <ref>(a)-(h). We have several observations:
i) Multi-images tasks posed significant challenges to current LVLMs, where the best accuracy achieved by GeminiProVision is only 53.8. 

ii) For Multi-Images LVLMs, providing multiple images as prompts instead of a single image boosted the overall performance on these tasks, demonstrating their capabilities to extract beneficial information from multiple images. 
For instance, for the task of face retrieval (FR), the performance of GeminiProVision increased from 30.5 to 92.5 when providing multiple images as visual prompts.

iii) For Single-Image LVLMs, multi-image prompts also help improve the overall performance of most models, except for Monkey. 
To our surprise, BLIP2 achieved significant performance gain when switching to a multi-image prompt setting, especially on tasks like general action recognition (GAR) and video captioning (VC).
These results highlight the potential of LVLMs to learn more robust unified representations of multiple modalities.


























Most LVLMs Show Poor Generalization in Visual Referring Prompting.
Visual referring prompting is an impressive prompting technique that entails direct image edits like drawing bounding boxes or masks to guide LVLMs to focus on specific regions <cit.>.
We select 14 tasks (see Sec. <ref>) involving visual referring prompting to explore the influence of different prompting methods on the final results. 
We compared three additional settings: using text prompts for bounding boxes in normalized ([0,1]) and pixel ([0, h or w]) formats, and combining visual and text prompts.
As depicted in Fig. <ref>(i), visual prompting (blue curve) significantly lags behind other settings, a disparity mainly attributed to the lack of visual prompting data in most LVLMs during the Supervised Fine-Tuning (SFT) stage. 




 §.§ Error Analysis






To analyze the error distribution of LVLMs on the MMT-Bench, we examined three LVLMs: GPT-4V, GeminiProVision, and InternVL-Chat-V1.2 (InternVL).
Specifically, we randomly selected up to 5 incorrectly answered questions per subtask for each model. Task-specific experts among the co-authors then analyzed these error samples to identify the underlying reasons for the mistakes, yielding the error distribution presented in Fig. <ref>. For definitions and case studies of these six error types, please refer to Sec. <ref> in the appendix.

As shown in Fig <ref>, perception error stands out as the most common type of error across all models, with GPT-4V exhibiting a significantly lower perception error rate (51%) compared to GeminiProVision (76.9%) and InternVL (67.2%), indicating its superior performance in perception tasks. Reasoning error emerges as the second most prevalent error type, with InternVL having the highest reasoning error rate (14.8%), followed by GeminiProVision (10.4%) and GPT-4V (9.94%), highlighting the challenges all models face in complex reasoning tasks.

Additionally, the proportion of lack of knowledge errors is similar across the three models, ranging from 6.99% to 9.0%. It suggests that insufficient knowledge is a common issue. However, GPT-4V has notably higher error rates in lack of capability (19%) and Refusing to Answer (6.11%) compared with the other models, which may be attributed to its more honest approach in acknowledging its limitations and refusing to answer certain questions.

InternVL stands out for its high error rate in failing to follow instructions (6.64%), significantly surpassing GPT-4V (2.99%) and GeminiProVision (1.14%), indicating its struggle in comprehending and executing instructions effectively. On the other hand, annotation error contributes the least to the overall error distribution, implying that the quality of data annotation is high and has a minimal impact on model performance.

To enhance the performance of these large language models, future improvements should focus on addressing the specific error types identified. By targeting perception and reasoning capabilities, tackling the lack of knowledge, and refining the ability to follow instructions, developers can work towards creating more accurate and reliable language models. GPT-4V's honest approach to its limitations also highlights the importance of transparency in AI systems, which can be further explored and incorporated into future model designs.












































§ TASKONOMY ANALYSIS

Thanks to the extensive coverage of tasks in the MMT-Bench, we can evaluate the multimodal performance of LVLMs on a task map. In this way, the roles of different tasks in multimodal capability can be systematically interpreted by analyzing relationships between tasks in the map.






 §.§ Analytical Tools


Task map. 
To investigate the relationships between subtasks, we quantify each subtask as a task vector by following <cit.>. Formally, a task vector is defined by the weight variation between the weight fine-tuned on task data D^t and the initial weight W_0 of a probing model, as given by V^t = min_W ℒ(W|D^t) - W_0



where the subscript t denotes the task and ℒ is the task loss. Three steps are adopted to obtain V^t. First, we use pre-trained QwenVL-Chat as the probing model because QwenVL-Chat achieves good results on most subtasks, which helps acquire promising task vectors. Second, we construct task data D^t by adapting all multi-choice VQA samples into the instruction-following data for each subtask. Third, unlike TaskVec <cit.> that finetunes the whole model, we finetune QwenVL-Chat for 3 epochs using LoRA fine-tuning <cit.> for all 162 subtasks, which reduces the length of task vector from 9.6B to 3.5M and consumes less storage resources.

With task vector, a task map can be constructed as 𝒢={G^st}_s,t=1^T where G^st=1-cos(V^s, V^t) denotes the cosine distance between task s and t and T=162 denoted the total number of subtasks. By definition, we know that 0≤ G^st≤ 2. 




Ranking correlation: Kendall's tau τ.
To quantitatively evaluate LVLMs on a task map, we use the metric of Kendall's tau τ to measure the ranking correlation between performance sequences of LVLMs on different subtasks. 

The intuition is that model A would be superior to model B on task t if model A performs better than model B on task s when task distance G^st is small. 
The Kendall's tau τ is defined as τ^st = 2/M(M-1)∑_1≤ m<n≤ Msign((P^s_m - P^s_n)(P^t_m - P^t_n))





where P^s_m denotes the performance of model m on task s and M is the number of LVLMs. The function sign(·) returns -1 if the argument is negative and 1 otherwise. When τ^st=1, LVLMs have completely consistent performance ranking on task s and t.



 §.§ Findings on Task Map

LVLMs obtain a more consistent performance ranking on tasks closer to each other. 



We assess whether LVLMs achieve consistent performance on two tasks close to each other. To measure this consistency, we employ the Kendall tau metric as introduced in Sec. <ref>. Specifically, we consider all subtask pairs in which two tasks are closer to each other and calculate their average Kendall's tau τ, which can be given by 
    τ_δ = 1/T∑_s=1^T1/|Δ_s|∑_t∈Δ_sτ^st where Δ_s={t:G^st≤δ} 
and δ is a threshold used to control the proximity between two tasks. 

As shown in Table <ref>, as the threshold δ decreases, the task distance becomes smaller, and τ_δ increases. This suggests that LVLMs obtain a more consistent performance ranking on tasks closer to each other. 

Hence, the performance of LVLMs on a new task can be predicted if it is close to one of the MMT-Bench subtasks.












Out-of-Domain (OoD) tasks discovery.

The OoD tasks mean tasks that the current model struggles to handle. Discovering OoD tasks can provide insights for future evaluation efforts and the development of stronger LVLMs. Since model performance on different tasks is related to task distances, we hypothesize that OoD tasks would be grouped in local regions on the task map. Therefore, we conduct hierarchical clustering on the task map to find OoD tasks. Specifically, 162 subtasks are grouped into 12 clusters as shown in Fig. <ref>. We use two criteria to identify clusters containing OoD tasks. First, LVLMs would achieve poor performance on OoD tasks. In this regard, we calculate the average multimodal performance within each task cluster over all LVLM models. Second, the performance of LVLMs on OoD tasks would be inconsistent with the overall multimodal score in Table <ref> because LVLMs with competitive overall scores would even fail to solve OoD tasks. Hence, we calculate the average ranking correlation τ within each cluster. 
We present these statistics in Table <ref> and provide a detailed analysis with the clustering results in Appendix <ref>.
 


We can see that clusters 8, 9, and 11 achieve low multimodal accuracy and ranking correlation τ. In sec <ref>, we find that the model struggles with handling fine-grained visual tasks, such as detection. Through the analysis of these clusters, we similarly find that current multimodal large models cannot perform fine-grained visual cognition and understanding of positional and spatial relationships, such as localization and detection tasks. Moreover, they exhibit poor performance in tasks related to new data structures or types of images, showing a lack of proficiency in handling tasks related to GUI and special data structures like tables.













































In-domain tasks discovery.
In-domain tasks are tasks that most current multimodal large models can handle correctly. Discovering in-domain tasks guides the commercial application of LVLMs in specific scenarios. 
Different from OoD tasks, we identify in-domain tasks by looking for clusters with large ranking correlation τ and high multimodal accuracy.



From Table <ref>, we can see that clusters 2, 3, and 10 achieve relatively high accuracy and large ranking correlation τ. We observe that current multimodal large models possess strong high-level visual comprehension capabilities, enabling them to effectively handle visual recognition tasks, even when dealing with specialized images such as medical images, which is also found in sec <ref>. Moreover, they benefit from the powerful LLMs to accurately describe images.
We provide a detailed analysis along with the clustering results in Appendix <ref>.












§ CONCLUSION AND DISCUSSION

In this work, we introduce MMT-Bench, a comprehensive benchmark designed to evaluate LVLMs in multimodal multitask understanding. The breadth of MMT-Bench is highlighted by its meticulously curated dataset of 31,325 multi-choice questions covering 162 multimodal tasks. Our evaluation reveal significant challenges for current LVLMs posed by our MMT-Bench. We present a taskonomy analysis of LVLMs on the task map, allowing us to predict the performance of a new task. Our goal with MMT-Bench is to measure the progress on the path to multitask AGI. We shall acknowledge that MMT-Bench may not be sufficient as a standard for determining whether multitask AGI has been achieved, as it is impossible to include all multimodal tasks. However, we believe that it should be necessary for a multitask AGI to achieve strong performance on MMT-Bench. We will continue to expand the task set of MMT-Bench. We believe that MMT-Bench will inspire further research and development in LVLMs, bringing us closer to the realization of truly intelligent multimodal systems.

Broader Impact. 
The development and widespread adoption of MMT-Bench as a benchmark for evaluating large vision-language models (LVLMs) have the potential to significantly impact the field of artificial intelligence. While MMT-Bench offers valuable insights and guidance for advancing LVLM research, it is important to consider its broader impact, including ethical considerations and potential societal consequences.

One potential positive impact of MMT-Bench is its role in driving advancements in LVLM technology, leading to improved performance and capabilities in various multimodal tasks. This could benefit numerous applications, such as visual dialogue, video analysis, and document understanding, ultimately enhancing user experiences and productivity. 

However, it is crucial to recognize and address potential negative impacts as well. One of the primary limitations of MMT-Bench is its reliance on curated data, which may inadvertently introduce biases based on the sources and methodologies used for data collection. For example, the performance of each meta-task is obtained by taking the average over all subtasks, which may lead to biased assessment because meta-tasks comprise different numbers of subtasks. Moreover, the selection of tasks and subtasks in MMT-Bench may only partially capture the diversity of real-world scenarios, leading to a limited understanding of LVLMs' capabilities across different domains and populations. Furthermore, the data collection process might disproportionately represent certain demographics or contexts, which can lead to biased evaluations of LVLMs' performance.

The other concern is that the benchmark's emphasis on performance metrics such as overall scores and task-specific accuracies may oversimplify the evaluation process and obscure nuanced differences in LVLMs' performance. This could mask disparities in model performance across demographic groups or domains, contributing to the perpetuation of biases and inequities in AI systems. We are dedicated to collecting as many multimodal tasks as possible into our MMT-Bench for unbiased evaluation.





langley00


icml2024

























In this appendix, we provide further details as follows:


  * Sec. <ref>: Presents hierarchical clustering and more analyses on the task map constructed from our MMT-Bench.

  * Sec. <ref>: Includes
details on sample size, visual input types, and capabilities of LVLMs evaluated for each subtask.

  * Sec. <ref>: Enumerates task abbreviations used throughout the paper.

  * Sec. <ref>: Presents detailed model configurations and experimental details in multi-images and visual prompting.

  * Sec. <ref>: Compares the performance on tasks involving pixel coordinates and normalized coordinates.

  * Sec. <ref>: Compares the performance of LVLMs on different image types and multimodal capabilities.

  * Sec. <ref>: Illustrates error cases of GPT-4V, GeminiProVision, and InternVL-Chat on 32 meta-tasks in MMT-Bench.

  * Sec. <ref>: Gives the comparison of MMT-Bench with Other Benchmarks on OCR-Related Tasks.

  * Sec. <ref>: Presents some Ddtails about the benchmark construction.

  * Sec. <ref>:  Discusses the openCompass protocol used in MMT-Bench and other alternatives.

  * Sec. <ref>:  Gives the computaional resources used in evaluation.




  * Sec. <ref>: Provides the detailed performance of 30 models across all 162 subtasks on MMT-Bench.





















§ TASK MAP



We perform hierarchical clustering on the taskmap, as shown in Fig. <ref>. When selecting the number of clustering clusters as 12, we analyze the clustering results of the task map and the model performance on the corresponding tasks. Here, we list the names of the tasks within each cluster in Table <ref>.



Out-of-Domain (OoD) tasks discovery. We can see that clusters 8, 9, and 11 achieve low multimodal accuracy and ranking correlation τ. From these clusters, we find that current multimodal large models lack the ability to perform fine-grained visual cognition and understanding of positional and spatial relationships, such as localization and detection tasks. Moreover, they exhibit poor performance in tasks related to new data structures or types of images, showing a lack of proficiency in handling tasks related to GUI and special data structures like tables.







  
  
  
  * Cluster 8 mainly involves detection, tracking, and localization tasks, all of which are related to the localization of objects within images. This indicates that current large multimodal models lack fine-grained visual cognition and understanding of positional and spatial relationships.
  
  * Tasks in cluster 9 are centered around GUI navigation, a novel task type requiring strong visual understanding, object localization, and expert knowledge in operating mobile devices <cit.>. This suggests that current large multimodal models need further optimization for GUI-related tasks.
  
  * Apart from detection and localization tasks, cluster 11 also includes tasks involving the recognition of special images or their conversion into structured text. The former requires models to possess spatial cognition and fine-grained visual capabilities, while the latter demands robust OCR abilities and extensive knowledge (such as understanding and outputting the basic structure of code or tables). Our testing LVLMs currently fall short in this aspect.



In-Domain tasks discovery. From Table <ref>, we can see that clusters 2, 3, and 10 achieve relatively high accuracy and large ranking correlation τ. We observe that current multimodal large models possess strong high-level visual comprehension capabilities, enabling them to effectively handle visual recognition tasks, even when dealing with specialized images such as medical images. Moreover, they benefit from the powerful LLMs to accurately describe images.



  
  * Cluster 2 mainly comprises visual recognition tasks, which require the model to possess certain high-level visual capabilities, yet these tasks are relatively simple. Examining Table <ref> and Fig. <ref>, we observe that the model's performance within this cluster is generally good. This validates that the current multimodal large models possess fundamental abilities for visual-semantic understanding, allowing them to fulfil recognition tasks.

  
  * Cluster 3 mainly includes visual recognition tasks as well, yet extends to cover sophisticated visual understanding tasks that require primary specialist knowledge, such as medicine and emotion. Within this cluster, the model demonstrates large τ and high accuracy, suggesting that current multimodal models pay attention to tasks necessitating the infusion of domain-specific knowledge, beyond just natural images. This implies a certain ability to handle problems in specialized fields.

  
  * In Cluster 10, LVLMs achieve good performance on tasks related to the visual description of the image. It indicates that current large multimodal models can describe the image well. It would stem from the fact that these models are typically tuned by massive image-text pairs. 





§ HIERARCHICAL STRUCTURE OF MMT-BENCH


In Table <ref> to Table <ref>, we present all 32 meta-tasks from MMT-Bench, encompassing a total of 162 subtasks. These tables include details on sample size, visual input types, and capabilities of LVLMs evaluated for each subtask.





§ TASK ABBREVIATIONS


Given the extensive number of tasks and models tested within the benchmark, we employ abbreviations to condense the manuscript. The abbreviations used throughout the paper are shown in Table <ref>.




§ MORE EXPERIMENTAL DETAILS



 §.§ LVLMs Model Details


Table <ref> summarizes the LVLMs information used in this paper, including the corresponding parameter sizes, visual encoders, and LLMs. Note that we use follow OpenCompass’ protocol <cit.> to conduct the evaluation process. The inference time varies with different models. For instance, the smaller LLaVA-v1.5-7B <cit.> model takes only 12 minutes to complete the evaluation using 8 GPUs, while the larger InternVL-Chat-V1.2-34B model <cit.> requires 79 minutes and around 80GB of memory. Our open-source codebase supports multi-GPU distributed inference, effectively accelerating the inference process.


 §.§ Multi-Images Prompt Experimental Details


In terms of the 28 tasks requiring multiple images as input, please see Table <ref>-<ref> for the specific task names given task abbreviations.
Besides, we also present the designed prompt examples for Single-Image LVLMs and Multi-Images LVLMs in Table <ref>-<ref> for reference.


 §.§ Visual Referring Prompting Experimental Details


In Section <ref>, we explore the differential efficacy of visual prompting compared to alternative prompting strategies across a spectrum of 14 distinct tasks. These encompass human interaction understanding, social relation recognition, human-object interaction recognition, animal keypoint detection, vehicle keypoint detection, human keypoint detection, clothes keypoint detection, scene text recognition, interactive segmentation, instance captioning, multiple instance captioning, one-shot detection, single object tracking, and counting by visual prompting.



§ PIXEL COORDINATES VS. NORMALIZED COORDINATES
 

In Fig. <ref>, we analyze the performance across 19 detection-related tasks, specifically point tracking, image matting, pixel recognition, polygon localization, pixel localization, depth estimation, MEVIS, remote sensing object detection, rotated object detection, small object detection, camouflage object detection, salient object detection in RGB-D, transparent object detection, face detection, object detection, salient object detection in RGB, referring detection, reason segmentation, and image dense captioning. These tasks span Localization, Pixel-level Perception, and Visual Captioning, comparing outcomes under two different coordinate formats. Notably, GeminiProVision lags behind top open-source LVLMs like BLIP2 and XComposer2, which have been extensively trained with detection data. The preference for normalized coordinates among most models is attributed to their use in the training instruction templates.



§ ANALYSIS ON IMAGES TYPES AND CAPABILITIES

Performance with Different Visual Types.
We compare the performance of 20 LVLMs across 13 types of visual input in Fig. <ref>. 
Most LVLMs struggle with Scientific Diagrams due to task difficulty, as many, including Scientific and "Raven's Progressive Matrices," require complex reasoning, a capability current LVLMs do not possess well.


Performance Across Multimodal Capabilities.
We also compare the performance of 20 LVLMs across 14 types of visual input in Fig. <ref>. As we can see, GeminiProVision once again exhibits strong superiority across most capabilities, especially in retrieval and multi-image analysis (involving the recognition and matching of multiple images), vastly outperforming other open-source LVLMs. This superiority stems from GeminiProVision's support for multi-image mode and its powerful generalization abilities, guiding the future direction of open-source models towards the focus on multi-image and video understanding.

























































    

>=0.4l|>=1.6X|>=0.4X
Details of task clustering on the task map of our MMT-Bench.  

1cMeta-Task     1|c|Subtask     1c# subtasks 
 

3c
  – continued from previous page 

1cMeta-Task     1|c|Subtask     1c# subtasks 
 






3cCluster ID: 1 
 
Visual Prompt Understanding      Visual Prompt Understanding, Som (Set-of-marks) Recognition     2 

Pixel Level Perception     Image Matting      1

 Visual Recognition     Color Recognition, Abstract Visual Recognition     2

 Discipline Knowledge Reasoning     Science, Tech Engineering, Health Medicine, Humanities Social Science, Business, Art Design     6

 3cCluster ID: 2 
 
 Visual Recognition     Waste recognition, Logo and Brand Recognition, Animals Recognition, Weapon Recognition,
Celebrity Recognition, Shape Recognition, Age Gender Race Recognition, Rock Recognition,
Painting Recognition, Gesture Recognition, Vehicle Recognition, Astronomical Recognition,
Fashion Recognition, Musical Instrument Recognition, Disaster Recognition, Sports Recognition,
Building Recognition, Texture Material Recognition, Plant Recognition, Film and Television Recognition,
Animated Character Recognition, Electronic Object Recognition, Scene Recognition,
National Flag Recognition, Profession Recognition, Weather Recognition, Food Recognition     27 

Relation Reasoning     Human Object Interaction Recognition, Human Interaction Understanding     2

Action Recognition     Image-based Action Recognition, Sign Language Recognition, General Action Recognition     4 

Emotion     Scene Emotion Recognition, Artwork Emotion Recognition, Facial Expression Recognition,
Micro Expression Recognition, Body Emotion Recognition     5 

Image Evaluation Judgement     Lvlm Response Judgement     1

Visual Commonsense Reasoning     WhoopsVQA     1

Hallucination     Attribute Hallucination     1

Counting   
Counting by Visual Prompting, Crowd Counting     2 

Medical Understanding   
Other Biological Attributes     1

Autonomous Driving   
Traffic Sign Understanding     1 

OCR   
Font Recognition, Scene Text Recognition     2

Pixel Level Perception   
Pixel Recognition     1

Anomaly Detection   
Face Mask Anomaly Detection     1

Multiple Image Analysis   
Spot the Diff     1

Visual Captioning   
Instance Captioning     1

Doc Understanding   
Clock Reading, Doc VQA     2

Meme Understanding   
Meme Image Understanding     1


 3cCluster ID: 3 
 
 Medical Understanding    Medical Modality Recognition, Lesion Grading, Disease DiagnoseAnatomy Identification    3
 
 Visual Captioning    Multiple Image Captioning, Writing Poetry from Image    2
 
 Emotion    Facial Expression Change Recognition    1
 
 Visual Recognition    Image Season Recognition, Sculpture Recognition, Chemical Apparatus Recognition, Landmark Recognition, Religious Recognition    5
 
 Hallucination    Relation Hallucination    1
 
 Relation reasoning    Social Relation Recognition    1
 
 OCR    Handwritten Text Recognition    1
 
 Temporal Understanding    Temporal Anticipation    1
 
 3cCluster ID: 4 
 
 Intelligence Quotient Test    Ravens Progressive Matrices    1
 
 Temporal Understanding    Temporal Localization    1
 
 Autonomous Driving    Traffic Participants Understanding, Temporal Sequence Understanding, Multiple View Image Understanding    3
 
 Counting    Counting by Category, Counting by Reasoning    2
 
 Hallucination    Order Hallucination    1
 
 Doc Understanding    Visual Document Information Extraction, Chart VQA    2
 
 Action Recognition    Action Quality Assessment,    2
 
 3D    3D Cad Recognition, 3D indoor recognition    2
 
 Anomaly Detection    Industrial Produce Anomaly Detection    1
 
 Image Evaluation Judgement    Image Quality Assessment    1
 
 Low Level Vision    Depth Estimation    1
 


 3cCluster ID: 5 
 
 Multiple Image Analysis    Spot the Similarity    1
 
 Visual Illusion    Color Assimilation, Geometrical Relativity, Color Constancy, Color Contrast, Geometrical Perspective    5
 
 Autonomous Driving    Traffic Light Understanding    1
 
 Visual Recognition    Deepfake Detection    1
 
 Anomaly Detection    Helmet Anomaly Detection    1
 
 
 
  3cCluster ID: 6 
 
 

 
 Image Retrieval    Vehicle Retrieval, Image2image Retrieval, Sketch2image Retrieval, Face Retrieval, Text2image Retrieval, Handwritten Retrieval, Person Reid    7
 
 Image-to-image translation    Image Colorization    1
 
 
 
  3cCluster ID: 7 
 
 

 Visual Code    Eqn2latex,    2
 
 Keypoint Detection    Clothes Keypoint Detection    1
 
 OCR    Handwritten Math Expression recognition    1
 
 Pixel Level Perception    Interactive Segmentation    1
 
 Temporal Understanding    Temporal Ordering    1
 
 Visual Captioning    Image Dense Captioning    1
 
 Action Recognition    Gaze Estimation    1
 

 
 
  3cCluster ID: 8 
 
 

 
 Localization    Salient Object Detection RGB, Camouflage Object Detection, Face Detection, Object Detection, Small Object Detection, Salient Object Detection RGBD, Rotated Object Detection, Remote Sensing Object Detection, Transparent Object Detection    9
 
 Visual Grounding    Referring Detection, Reason Seg    2
 
 Cross Image Matching    Point Tracking, One Shot Detection,    3
 
 Image-to-image Translation    Jigsaw Puzzle Solving    1
 
 Cross Image Catching    Single Object Tracking    1
 
 Pixel Level Perception    Pixel Localization    1
 


  3cCluster ID: 9 
 
 

 GUI Navigation   
Web Shopping, GUI General, Google Apps, GUI Install     4

 
  3cCluster ID: 10 
 
 

 
 Visual Captioning    Multiple Instance Captioning, Image Captioning Paragraph, Image Captioning    3
 
 Anomaly Detection    Traffic Anomaly Detection    1
 
 Doc Understanding    Chart to text    1
 
 Hallucination    Exist Hallucination    1
 
 Relation Reasoning    Scene Graph Recognition    1
 
 Embodied AI    Navigation    1
 
 Anomaly Detection    Behavior Anomaly Detection    1
 
 
  3cCluster ID: 11 
 
 Doc Understanding    Table Structure Recognition, Chart to Table    2
 
 Keypoint Detection    Furniture Keypoint Detection, Vehicle Keypoint Detection, Human Keypoint Detection, Animal Keypoint Detection    4
 
 Pixel Level Perception    Polygon Localization,    2
 
 Temporal Understanding    Next Image Prediction    1
 
 Visual Code    Sketch2code, Screenshot2code    2
 
 
 
  3cCluster ID: 12 
 
 

Meme Understanding    Meme Video Understanding    1
 
 Temporal Understanding    Mevis    1
 
 Visual Captioning    Video Captioning    1
 

 





















































































































































































































































































































































































































































































































































































































 

 










§ CASE STUDY















 
 












In this section, we present a case study analysis of the error types made by GPT-4V, GeminiProVision, and InternVL-Chat on various meta-tasks in MMT-Bench. We classify the errors into the following six categories:

Perception Error: LVLMs fail to recognize, classify or detect the objects or content in images. Most LVLMs are constrained by the representation power of their visual encoders, making this the most common type of error. See examples in Fig. <ref>, Fig. <ref>, etc.

Reasoning Error: LVLMs correctly recognize and perceive the visual content but make errors in reasoning, leading to incorrect answers. See examples in Fig. <ref>, Fig. <ref>, etc.

Lack of Knowledge: LVLMs lack the domain-specific knowledge required to answer specialized questions, such as the location of a landmark (see Fig. <ref>) or the creation date of a particular painting (see Fig. <ref>).

Lack of Capability: LVLMs do not have the capability to solve the corresponding tasks. This error type is particularly evident in GPT-4V, which tends to respond more honestly when it lacks the ability to handle certain tasks. In contrast, other LVLM models are inclined to generate outputs even when the accuracy rate is relatively low. See examples in Fig. <ref>, Fig. <ref>.

Refuse to Answer: LVLMs, such as GPT-4V or Gemini, refuse to answer questions that are anthropocentric or sensitive in nature. See examples in Fig. <ref>, Fig. <ref>.

Fail to Follow Instruct: 
LVLMs fail to correctly understand instructions and provide erroneous answers. For example, LVLMs may not understand the specified conditions in the instruction (see Fig. <ref>) or may ignore the instruction altogether and instead generate a caption for the given image (see Fig. <ref>).











































































§ COMPARISON OF MMT-BENCH WITH OTHER BENCHMARKS ON OCR-RELATED TASKS





To support the claim that previous evaluation benchmarks suffer from text scarcity in OCR tasks, we present a comparative analysis of OCR-related samples from different benchmarks in Table. <ref>. The results demonstrate that datasets like MME and Tiny-LVLM-eHub have relatively short text lengths with limited variations. Furthermore, previous OCR tasks primarily focused on directly outputting text from given scenes or cropped images. In contrast, our proposed MMT-Bench benchmark introduces several new tasks, such as font recognition, handwriting recognition, handwritten formula recognition, and document-based question answering and chart question answering. These additions significantly increase the challenges for evaluating model performance on OCR tasks. Compared to previous benchmarks, MMT-Bench's OCR samples have an average word count and token count more than 5 times that of MME and over 2 times that of MMBench. Additionally, MMT-Bench includes a higher proportion of long text samples with a wider range of text lengths. This demonstrates MMT-Bench's superiority in addressing the text scarcity issue in OCR tasks, providing a more reliable benchmark for comprehensively evaluating the performance of multimodal algorithms on OCR-related tasks.



§ SOME DETAILS ABOUT THE BENCHMARK CONSTRUCTION





 §.§ Metadata







The uniform format of the metadata. As shown in Table. <ref>, the metadata here is a dictionary, and its keys can be divided into two categories. The first category contains the essential keys, including image path, data source, subtask name, meta-task name, specific question template, answer (i.e., ground truth), visual prompt type, and capabilities required to solve the problem. The second category consists of keys that are specific to each sample and may vary accordingly. For animal recognition (example 1), the corresponding category space is retained for generating candidate choices, and referring expressions are preserved in the reasoning segmentation (example 2).

How to get metadata? We obtain the metadata mainly through a two-step process. In the first step, we write a python script for each dataset to directly extract relevant keys from the original dataset, such as image path, data source, and answer. Since samples within the same dataset share some similar characteristics, our co-authors predefined the specific question template, visual prompt, and capabilities for each dataset. The second step involves our co-authors manually performing a sample check (50%, up to 100 samples per dataset) on the samples defined in the first step, primarily focusing on sample-related keys, such as the specific question template and visual prompt.



 §.§ Prepare the Answer and Options




How to ensure the wrong options are difficult distractors? For most tasks, we use prompt engineering with GPT-4V using specific prompts for particular tasks. We employ in-context (2-shot or 3-shot) prompts. We review the options generated for each task and correct some erroneous ones. Interestingly, GPT-4 can generate sufficiently challenging options, which is mainly reflected in two aspects. 
First, the options can be confusing, especially in classification-related tasks (e.g. animal recognition), where the incorrect options tend to generate objects with semantics similar to the ground truth (GT). For example, if the GT is "rat" the generated incorrect options might be "squirrel," "mouse," or "hamster."
Second, the incorrect options can be ambiguous concerning the visual content. For instance, in captioning-related tasks, we use GPT-4V to take the image as input and assist in generating incorrect options, which can effectively alleviate the hallucination problem caused by text-only LLMs. For some problems, such as counting, OCR, and object detection, we can directly add perturbations to the GT using Python scripts to obtain distinguishable options. Throughout this process, we maintain consistency in word length across options, with a standard deviation of 1.1. In summary, we believe that the difficulty level of the options generated by MMT-Bench is reasonable.





The answers in MMT-Bench are visual-grounded. To ensure that the generated ground truth answers are visually grounded, we employ strategies as below.
Firstly, when designing questions for each sample, we carefully craft specific question templates (as mentioned in Sec. <ref>). We strive to avoid including specific information in the templates (for example, instead of asking "Who is the author of the Mona Lisa painting in the image?", we use a more general form such as "Who is the author of the painting in the image?"). This ensures that answering the question requires reliance on the image content.
Secondly, when generating options, we create more complex options to prevent the model from bypassing the visual information and directly providing an answer based on correlations between options or prior knowledge.
Furthermore, Table D demonstrates the visually grounded nature of our MMT-Bench dataset by comparing the performance of text-only LLMs and LVLMs with and without visual information. Without visual input, all models, including state-of-the-art LVLMs, perform similarly to the frequency-based guessing baseline (31.7%) and random guessing (28.5%), indicating the difficulty of answering questions correctly without visual context. For models like LLaVA and QWen that require an image as input, we simulate the absence of visual information by using a pure black image as input. This allows us to evaluate their performance in a text-only setting and compare it to their performance when provided with the actual visual information from our dataset.
However, when provided with visual information, LVLMs show significant improvements, with performance gains ranging from 18.1% to 20.2%. This substantial gap between the text-only and visually-informed settings highlights the strong visual grounding of our dataset, confirming that the questions are designed to require models to rely on visual content and that the ground truth answers are indeed grounded in the visual information.

How to prevent MMT-Bench from becoming overly focused on language understanding? When defining questions and options, the co-authors proficient in each task domain strive to use simple and unambiguous specific question templates. The option generation process follows what is shown above, where we maintain the necessary clarity. To clarify, we report the statistics in Table. <ref>. For some questions, such as PLP (Pixel Level Perception) and Loc (Localization), which are localization problems, their longer question lengths are necessary to introduce the problem definition, answer format ([x, y, x, y]), and detection conditions (detecting corresponding categories, such as the 80 COCO classes).





 §.§ Statistics of Image and Video in MMT-Bench








Table. <ref> presents the number of images and videos in MMT-Bench compared to other multimodal benchmarks. MMT-Bench contains 25,732 single images, 3,800 image pairs (14,800 images), and 1,793 videos (10,572 video frames), demonstrating its comprehensive coverage of various visual data types. Compared to other benchmarks, MMT-Bench has the highest number of single images, more image pairs than MMMU, and more videos and video frames than MVBench and Seed-bench. This extensive coverage and diversity of visual data enable a thorough evaluation of multimodal models across a wide range of tasks and scenarios.




§ OPENCOMPASS' PROTOCOL







How often do steps 1-3 in OpenCompass' protocol fail? We show the result in the following Table G. According to the table, the frequency of failure for steps 1-3 in OpenCompass' protocol varies across different LVLMs, but overall, the extraction of model selections is highly successful. Most models achieve success rates above 87% at Step 1 (checking for option letters), with further improvements in subsequent steps. Models like BLIP-2, InternVL-Chat-V1.2-34B, and LLaVA-1.5-7B demonstrate exceptionally high success rates, consistently providing clear and well-structured responses that include option letters. However, some models, such as GPT-4V, GeminiProVision, Claude-3-Haiku, and QWen-VL-Plus, refuse to answer a small percentage of questions (0.875% to 10.136%), likely due to the sensitive nature of certain questions. In these cases, the model selection is set as option letter Z to differentiate it from a valid answer and to avoid random assignment. Overall, OpenCompass' protocol proves highly effective in extracting model selections from LVLMs' responses, with the multi-step approach ensuring accurate extraction and handling cases where models refuse to answer or provide unclear responses.

Why is OpenCompass’ evaluation protocol chosen over other alternatives? We chose OpenCompass' multi-choice evaluation protocol for MMT-Bench due to its convenience, ease of use, and scalability. As discussed in our response above, OpenCompass' protocol has proven to be highly effective in extracting model selections from LVLMs' responses, with models like BLIP-2, InternVL-Chat-V1.2-34B, and LLaVA-1.5-7B achieving success rates over 99%. The multi-step approach ensures accurate extraction and handles cases where models refuse to answer or provide unclear responses.
While alternative evaluation protocols have their merits, they also have limitations. MME's yes-or-no evaluation simplifies the question-answering process but can lead to biased models and is resource-intensive, resulting in a limited sample size. MMBench's evaluation strategy is resource-intensive and has high testing overhead, making it unsuitable for MMT-Bench's scale. SeedBench calculates the log-likelihood for each candidate option, helping avoid the issue of models not directly answering the options. However, this approach cannot be applied to API-based models.



§ COMPUTAIONAL RESOURCES





We show the resource consumption of some models in Table. <ref>.
The inference times vary among different models. For instance, the smaller LLaVA-v1.5-7B model takes only 12 minutes to complete the evaluation using 8 GPUs, while the larger InternVL-Chat-V1.2-34B model requires 79 minutes and around 80GB of memory. Our open-source codebase supports multi-GPU distributed inference, effectively accelerating the inference process.
Although the prices for API models like Claude-3-Haiku (USD 127) and Qwen-VL-Plus (USD 70.4) are relatively high and their inference times are longer (about 36 hours), the inference costs for smaller open-source models remain manageable. 




§ DETAILED MAIN RESULTS


Table <ref> to <ref> display the performance of 20 models across all 162 subtasks, with accuracy used as the evaluation metric.









































































































