EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark
Benchmark;Evaluation;large multimodal model
For using smaller margin than the official template.
Submission Desk Rejected by Program Chairs
We propose a comprehensive electrical and electronics engineering benchmark to evaluate LMMs.
@misc{
li2024eeebench,
title={{EEE}-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark},
author={Ming Li and Jike Zhong and Tianle Chen and Konstantinos Psounis},
year={2024},
url={https://openreview.net/forum?id=pjfvmZWFfX}
}
Recent studies on large language models (LLMs) and large multimodal models (LMMs) have demonstrated promising skills in various domains including science and mathematics. However, their capability in more challenging and real-world related scenarios like engineering has not been systematically studied. To bridge this gap, we propose EEE-Bench, a multimodal benchmark aimed at assessing LMMs' capabilities in solving practical engineering tasks, using electrical and electronics engineering (EEE) as the testbed. Our benchmark consists of 2860 hand-picked and carefully curated multiple-choice and free-form problems spanning 10 essential subdomains such as analog circuits, control systems, etc. Compared to benchmarks in other domains, engineering problems are intrinsically 1) more visually complex and versatile and 2) less deterministic in solutions. Successful solutions to these problems often demand more-than-usual rigorous integration of visual and textual information as models need to understand intricate images like abstract circuits and system diagrams while taking professional instructions, making them excellent candidates for LMM evaluations. Alongside EEE-Bench, we provide extensive quantitative evaluations and fine-grained analysis of 17 widely-used open and closed-sourced LLMs and LMMs. Our results demonstrate notable deficiencies of current foundation models in EEE, with an average performance ranging from 19.48% to 46.78%. Finally, we reveal and explore a critical shortcoming in LMMs which we term ``laziness": the tendency to take shortcuts by relying on the text while overlooking the visual context. In summary, we believe EEE-Bench not only reveals some noteworthy limitations of LMMs but also provides a valuable resource for advancing research on their application in practical engineering tasks, driving future improvements in their capability to handle complex, real-world scenarios.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Ming_Li28 ~Jike_Zhong1 ~Tianle_Chen2 ~Konstantinos_Psounis1
Ming Li Jike Zhong Tianle Chen Konstantinos Psounis
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Benchmark Evaluation large multimodal model
I certify that there is no acknowledgement section in this submission for double blind review.
li|eeebench_a_comprehensive_multimodal_electrical_and_electronics_engineering_benchmark
/pdf/5786a296f4e670aaf1d2282f06028507bb4f9f7f.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark
ICLR 2025 Conference Desk Rejected Submission
ICLR.cc/2025/Conference/Desk_Rejected_Submission