SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
Bohao Li Yuying Ge Yi Chen Yixiao Ge Ruimao Zhang Ying Shan


[

    The Directed Landscape is a Black Noise
    [
    April 29, 2024
=======================================

]

 		
   [1]This technical report is an extension to SEED-Bench-2 <cit.>.
   [2]Correspondence Author.





Comprehending text-rich visual content is paramount for the practical application of Multimodal Large Language Models (MLLMs), since text-rich scenarios are ubiquitous in the real world, which are 
characterized by the presence of extensive texts embedded within images. Recently, the advent of MLLMs with impressive versatility has raised the bar for what we can expect from MLLMs. However, their proficiency in text-rich scenarios has yet to be comprehensively and objectively assessed, since current MLLM benchmarks primarily focus on evaluating general visual comprehension. In this work, we introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating text-rich visual comprehension of MLLMs. Our benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs, each of which covers a wide spectrum of text-rich scenarios in the real world. These categories, due to their inherent complexity and diversity, effectively simulate real-world text-rich environments. We further conduct a thorough evaluation involving 34 prominent MLLMs (including GPT-4V, Gemini-Pro-Vision and Claude-3-Opus) and emphasize the current limitations of MLLMs in text-rich visual comprehension.
We hope that our work can serve as a valuable addition to existing MLLM benchmarks, providing insightful observations and inspiring further research in the area of text-rich visual comprehension with MLLMs. The dataset and evaluation code can be accessed at https://github.com/AILab-CVC/SEED-Benchhttps://github.com/AILab-CVC/SEED-Bench.


Comprehending text-rich visual input is crucial for multimodal large language models (MLLMs), as it enables them to effectively synthesize and interpret intricate information from various modalities, such as text, images, and diagrams. This capability is indispensable for real-world applications, where visual data is often presented in diverse and interconnected formats.
Despite recent progress in MLLMs, the majority of existing benchmarks primarily focus on object and scene information in images, overlooking the importance of text-rich scenarios. To bridge this gap, we introduce SEED-Bench-2-Plus, a comprehensive benchmark designed to assess MLLMs' performance across a wide range of text-rich visual data.
SEED-Bench-2-Plus consists of 2K multiple-choice questions with precise human annotations, spanning three dimensions (maps, charts, and webs) that can be further categorized into 64 finer types. We conduct a thorough evaluation involving 29 prominent MLLMs and summarize key findings, emphasizing the current limitations of MLLMs in text-rich visual comprehension.
We hope that our work serves as a valuable addition to existing MLLM benchmarks, providing insightful observations and encouraging further research in the area of text-rich visual comprehension with MLLMs. The dataset and evaluation code can be accessed at https://github.com/AILab-CVC/SEED-Benchhttps://github.com/AILab-CVC/SEED-Bench.





    


§ INTRODUCTION






In recent years, through leveraging the strong generality of Large Language Models (LLMs) <cit.>, Multimodal Large Language Models (MLLMs) <cit.> have showcased remarkable capabilities in comprehending multimodal data, which aim to mimic human-like understanding through multimodal perception and reasoning. To realize the practical applications of MLLMs in the real world, a significant challenge lies in comprehending text-rich visual data, which is pervasive in various contexts with images intertwined with texts. An MLLM capable of comprehensively comprehending text-rich scenarios should be able to interpret texts, understand visual content, and discern the interactions between textual and visual contexts.



With the emergence of increasingly powerful and versatile MLLMs, such as GPT-4V <cit.>, Gemini-Pro-Vision <cit.>, and Claude-3-Opus <cit.>, it naturally raises the question: How do these models perform in text-rich scenarios? Although recent benchmarks <cit.> are specifically designed to evaluate MLLMs, their primary focus is on general visual comprehension (e.g., images in different domains), leaving a significant gap in a comprehensive and objective evaluation of MLLM in text-rich contexts. 

In this work, we introduce SEED-Bench-2-Plus, a comprehensive benchmark designed specifically to assess MLLMs' performance in comprehending text-rich visual data, which covers a wide spectrum range of text-rich scenarios in the real world. Specially, we meticulously craft 2.3K multiple-choice questions spanning three broad categories including Charts, Maps and Charts, as shown in Figure <ref>. The broad categories are further divided into 63 specific types as shown in Figure <ref>, to capture a more granular view of the challenges presented by text-rich visual comprehension. Each question in our benchmark is answered by human annotators, ensuring the accuracy and reliability of the ground-truth answer.

We further conduct extensive evaluation, encompassing 34 prominent MLLMs, including GPT-4V, Gemini-Pro-Vision, and Claude-3-Opus, which reveals critical insights into the current limitations and strengths of these models in comprehending text-rich information. Our findings underline several key observations: the inherent complexity of text-rich visual data, the varying difficulty levels across different data types, and the performance disparities among leading MLLMs. 

Through this evaluation, we aim not only to benchmark current MLLM performance but also to catalyze further research in enhancing MLLMs' proficiency of multimodal comprehension in text-rich scenarios. Serving as a valuable supplement to Seed-Bench-2 <cit.>, both the dataset and the evaluation code of Seed-Bench-2-Plus are made publicly available. We also consistently maintain a leaderboard to facilitate ongoing contributions to this area.



As we advance towards Artificial General Intelligence (AGI), the significance of Multimodal Large Language Models (MLLMs) <cit.> has become increasingly apparent. These models, which integrate powerful Large Language Models (LLMs) <cit.> with multimodal comprehension capabilities, aim to mimic human-like understanding through multimodal cognition. A notable challenge in this area is the interpretation of complex, text-rich visual data, such as maps, charts, and web pages. This type of data, embodying intricate information across multiple modalities, mirrors the complexity of real-world scenarios where visual and textual elements are deeply intertwined.




Despite progress, there remains a notable gap in MLLMs' ability to effectively understand text-rich visual data. Moreover, the visual part of most existing benchmarks for MLLMs <cit.> primarily focuses on image-based or video-based data <cit.> , leaving the critical area of text-rich visual comprehension underexplored, as illustrated in Table <ref>. 
To address this need, we introduce SEED-Bench-2-Plus, a comprehensive benchmark designed to rigorously assess MLLMs' performance in comprehending text-rich visual data. 
SEED-Bench-2-Plus is meticulously crafted, featuring over 2.3K multiple-choice questions across three distinct dimensions—maps, charts, and webs—further divided into 63 specific types, as shown in Figure <ref> and Figure <ref>. This benchmark is underpinned by precise human annotations, ensuring a nuanced evaluation of MLLM capabilities.


Our extensive evaluation, encompassing 29 prominent MLLMs, reveals critical insights into the current limitations and strengths of these models in processing text-rich information. Our findings underline several key challenges: the inherent complexity of text-rich visual data, the varying difficulty levels across different data types, and the performance disparities among leading MLLMs. These insights underscore the necessity for continued innovation and development in MLLM technologies, guiding us closer to the realization of more intelligent and capable multimodal systems.



Through this evaluation, we aim not only to benchmark current MLLM performance bu also to catalyze further research in enhancing MLLMs' proficiency in complex multimodal comprehension. Serving as a valuable supplement to Seed-Bench-2 <cit.>, both the dataset and the evaluation code of Seed-Bench-2-Plus are made publicly available. We also consistently maintain an objective leaderboard to facilitate ongoing contributions to this area.









§ RELATED WORK

Multimodal Large Language Models.
Following the remarkable success of Large Language Models (LLM) <cit.>, recent studies work on generative Multimodal Large Language Models (MLLMs) <cit.> aim to enhance multimodal comprehension by aligning the visual features of pre-trained image encoders with LLMs on image-text datasets. Some research <cit.> has further incorporated video inputs, leveraging the vast capabilities of LLMs for video understanding tasks. Recent work <cit.> take significant strides in equipping MLLMs with the capacity for generating images beyond texts. However, their capability in text-rich scenarios has yet to be comprehensively and objectively assessed, which is the primary motivation of SEED-Bench-2-Plus.

Benchmarks for Multimodal Large Language Models.

In tandem with the rapid development of Multimodal Large Language Models (MLLMs), several concurrent studies <cit.> have proposed various benchmarks for evaluating MLLMs. 
For instance, GVT <cit.> constructs a benchmark by aggregating two semantic-level understanding tasks (VQA and Image Captioning) and two fine-grained tasks (Object Counting and Multi-class Identification). However, its evaluation is limited to specific aspects of visual understanding. LVLM-eHub <cit.> combines multiple existing computer vision benchmarks and develops an online platform where two models are prompted to answer a question related to an image, and human annotators are employed to compare the models' predictions. The involvement of human annotators during evaluation not only introduces bias but also incurs significant costs. LLaVA-Bench <cit.>, LAMM <cit.>, and Touchstone <cit.> utilize GPT to evaluate the relevance and accuracy of answers in relation to the ground truth. The reliance on entity extraction and GPT metric can affect the accuracy and reliability of the evaluation. MME <cit.> and MMBench <cit.> aim to enhance the objective evaluation of MLLMs by constructing 2194 True/False Questions and 2974 Multiple Choice Questions across various ability dimensions, respectively. MMMU <cit.> generates numerous college-level multi-discipline QAs to evaluate MLLMs' knowledgeability.
In SEED-Bench-2-Plus, we focus on evaluating MLLMs’ performance in comprehending text-rich visual
data, which covers a wide spectrum range of text-rich scenarios in the real world.



§ SEED-BENCH-2-PLUS















Our SEED-Bench-2-Plus benchmark incorporates 2K multiple-choice questions, all of which are accompanied by accurate human annotations and span three broad categories including Chats, Maps and Webs. In this section, we first the broad categories of SEED-Bench-2-Plus in Sec. <ref>. We then introduce the data source in Sec. <ref>, and finally, we describe the evaluation strategy for MLLMs to answer multiple-choice questions in Sec. <ref>.



 §.§ Broad Categories

To thoroughly evaluate the capabilities of MLLMs in comprehending text-rich data, SEED-Bench-2-Plus encompasses 3 broad categories (see Figure. <ref>), which can be further divided into 63 data types (see Figure. <ref>).

Charts. This category pertains to the information contained within the chart image. In this task, the model is required to understand the specific semantics of each chart type, extract relevant information, and answer questions based on the context and spatial relationships. 

Maps. This category corresponds to the information present in the map image. The model is expected to identify symbols, text, and spatial relationships, and use this information to answer questions that often require geographical or domain-specific knowledge.

Webs. In this category, the model needs to understand the layout and design of different websites, extract relevant information from various elements, and answer questions that may relate to the website's content, functionality, or design based on the given website screenshot.


Text-Rich Data. To assess the capabilities of MLLMs in managing text-rich data, we take into account four dimensions that cover different types of text-rich data, namely equations, charts, maps, and webs. A detailed breakdown of the data composition for charts, maps, and webs is depicted in Figure <ref>.

 
    
  * Charts: This dimension pertains to the information contained within the chart image. In this task, the model is required to understand the specific semantics of each chart type, extract relevant information, and answer questions based on the context and spatial relationships. 
    
  * Maps: This dimension corresponds to the information present in the map image. The model is expected to identify symbols, text, and spatial relationships, and use this information to answer questions that often require geographical or domain-specific knowledge.
    
  * Webs: In this dimension, the model needs to understand the layout and design of different websites, extract relevant information from various elements, and answer questions that may relate to the website's content, functionality, or design based on the given website screenshot.





 §.§ Data Source

To construct a comprehensive benchmark that encapsulates a variety of evaluation scenarios, it is imperative to amass an extensive collection of data. This data should predominantly include images that are rich in textual information, thereby creating a text-rich dataset.

Charts. To obtain chart data with rich text information, we manually gather 20 types of charts from the Internet. These types encompass Fishbone Diagram, Flow Chart, Foodchain, Gantt Chart, House Design Diagram, Kanban Board, Knowledge Graph, Matrix Diagram, Mockups, Organizational Chart, Pert Chart, Sankey Diagram, Spider Diagram, Storyboard, Swot Analysis, Timeline, Tree Diagram, Venn Diagram, Wireframes, and Word Cloud. Specifically, we employ GPT-4V <cit.> to generate corresponding questions and use human annotators to enhance the quality of the questions and corresponding options.

Maps. As for map data rich in text information, we manually collect 25 types of maps from the Internet. These types include Astronomical Map, Concept Map, Crime Map, Demographic Map, Environmental Map, Flow Map, Geologic Map, Heat Map, Historical Map, Infographics, Land Use Map, Migration Map, Mindmap, Nautical Map, Political Map, Resource Map, Road Map, Route Map, Site Map, Thematic Maps, Topographic Map, Transit Map, User Journey Map, Water Cycle, and Weather Map. Specifically, we utilize GPT-4V <cit.> to generate corresponding questions and employ human annotators to enhance the quality of the questions and corresponding options.

Webs. For web data rich in text information, we manually gather 18 types of screenshots from various websites on the Internet. These types include Airbnb, Amazon, Ebay, Github, Google Scholar Page, Homepage, Indeed, Linkedin, Papers, Poster, Project Page, Stackoverflow, Steam, Steam Game Setting, Tripadvisor, Twitter, Wikipedia, and Yelp. Specifically, we utilize GPT-4V <cit.> to generate corresponding questions and employ human annotators to enhance the quality of the questions and corresponding options.




 §.§ Evaluation Strategy


Different from MMBench <cit.>, which employs ChatGPT to match a model's prediction to one of the choices in a multiple-choice question (achieving only an 87.0% alignment rate), we adopt the answer ranking strategy <cit.> to evaluate existing MLLMs with multiple-choice questions. Specifically, for each choice of a question, we compute the likelihood that an MLLM generates the content of this choice given the question. We select the choice with the highest likelihood as the model's prediction. Our evaluation strategy does not depend on the instruction-following capabilities of models to output “A”, “B”, “C” or “D”. Also, this evaluation strategy eliminates the impact of the order of multiple-choice options on the model's performance.














§ EVALUATION RESULTS






 §.§ Models

We evaluate a total of 31 open-source MLLMs including BLIP-2 <cit.>, InstructBLIP <cit.>, InstructBLIP Vicuna <cit.>, LLaVA <cit.>, MiniGPT-4 <cit.>, VPGTrans <cit.>,  MultiModal-GPT <cit.>, Otter <cit.>, OpenFlamingo <cit.>, LLaMA-Adapter V2 <cit.>, GVT <cit.>, mPLUG-Owl <cit.>, Qwen-VL <cit.>, Qwen-VL-Chat <cit.>, LLaVA1.5 <cit.>, IDEFICS-9B-Instruct <cit.>, InternLM-Xcomposer-VL <cit.>, VideoChat <cit.>, Video-ChatGPT <cit.>, Valley <cit.>, Emu <cit.>, NExt-GPT <cit.>, SEED-LLaMA <cit.>, CogVLM <cit.>, InternLM-Xcomposer-VL-2 <cit.>, InternLM-Xcomposer-VL-2-4bit <cit.>, LLaVA-Next <cit.>, Yi-VL <cit.>, SPHINX-v2-1k <cit.>, mPLUG-Owl2 <cit.>, SEED-X <cit.> (We evaluate the general instruction-tuned model SEED-X-I), and 3 closed-source MLLMs including GPT-4V <cit.>, Gemini-Pro-Vision <cit.>, and Claude-3-Opus <cit.>, based on their official implementations. 
It is important to note that for Gemini-Pro-Vision, we only report task performance when the model responds to over half of the valid data in the task.




 §.§ Main Results
 
The evaluation results for various MLLMs across different categories of SEED-Bench-2-Plus are presented in Table <ref>. The detailed results of 63 types are presented in Table <ref>, Table <ref>, Table <ref>, Table <ref>, Table <ref>, and Table <ref>, respectively. Notably, GPT-4V surpasses a significant number of MLLMs, demonstrating superior performance across most evaluation types.

To provide a more comprehensive overview of model capabilities across different categories, we have visualized the ranking of each model within each category in Figure <ref>. Here, darker hues represent higher ranks. The top-performing MLLM, GPT-4V, exhibits competitive results across different categories.




 §.§ Observations

Through a comprehensive and objective evaluation of various MLLMs across different text-rich scenarios in SEED-Bench-2-Plus, we have gleaned valuable insights that can guide future research efforts.

Comprehending text-rich data proves to be more complex. The majority of MLLMs achieve lower results on text-rich data, with the average accuracy rate being less than 40%. Considering the potential of MLLMs as multimodal agents, particularly website agents, a crucial capability is to analyze a variety of websites and generate accurate responses. The unsatisfactory results indicate that significant advancements are required before MLLMs can be effectively utilized as multimodal agents. This underlines the complexity of comprehending text-rich data and highlights the need for further research and development in this area to enhance the proficiency of MLLMs in text-rich scenarios.

Maps are more difficult for text-rich comprehension. Maps are inherently complex and multidimensional, frequently featuring multiple overlapping layers of information. They not only display geographical details but also contain various symbols, colors, and texts to represent different types of information, such as topographical features, political boundaries, and points of interest. This complexity can pose challenges for models to accurately interpret and understand the full context of maps, compared with charts and webs.


The performance of MLLMs varies significantly across different data types. Different data types, such as knowledge graph and matrix diagram, which both fall under the category of charts, differ considerably in their complexity and structure. Some data types may require more sophisticated understanding and processing capabilities than others due to their inherent characteristics. As such, it is reasonable to observe a performance discrepancy of models across different data types. This finding underscores the need for MLLMs to be robust and adaptable, capable of handling a diverse range of data types in text-rich scenarios.




§ CONCLUSION

In this study, we present SEED-Bench-2-Plus, an extensive benchmark specifically designed for evaluating Multimodal Large Language Models (MLLMs) in text-rich scenarios. SEED-Bench-2-Plus comprises 2K multiple-choice questions, each annotated by humans, and span across 3 brod categories across 63 data types. We carry out a comprehensive evaluation of 31 noteworthy open-source MLLMs and 3 closed-source MLLMs, examining and comparing their performances to derive valuable insights that can guide future research. Serving as a valuable supplement to SEED-Bench-2 <cit.>, both the dataset and the evaluation code of SEED-Bench-2-Plus are made publicly available. Additionally, we consistently maintain a leaderboard to facilitate the advancements in the field of text-rich visual comprehension with MLLMs.



    ieeenat_fullname
    





