Evaluating the image recognition capabilities of GPT-4V and Gemini Pro in the Japanese national dental examination
Large Language Models (LLMs) are anticipated to complement traditional learning methods such as textbooks, lectures, and practical exercises through AI-driven learning experiences, thereby supporting students’ learning processes. , , ChatGPT-4V developed by OpenAI (September 2023 model) and Gemini Pro by Google (December 13, 2023 model) are classified as multimodal AIs, equipped with image recognition capabilities as LLMs. , OpenAI has explicitly stated that the current image recognition features are not suited for medical imaging. However, LLMs with image recognition capabilities not only serve as learning support tools but also have the potential of future applications in patient diagnosis and treatment within the medical field. , Therefore, evaluating the current scope of image recognition abilities of GPT-4V and Gemini Pro is beneficial. The Japanese National Dental Examination (JNDE) includes a variety of images such as intraoral and extraoral photographs, CT and MRI scans, ultrasound, panoramic X-ray images, dental X-ray images, dental technical work, charts, illustrations, tables, and orthodontic polygonal charts. This study was conducted to reveal the accuracy rates of GPT-4V and Gemini Pro when using image recognition capabilities to answer questions with various types of image information in the JNDE conducted in January 2023. We aimed to investigate the potentials of GPT-4V and Gemini Pro as educational support tools. 

Obtaining and processing data from the Japanese national dental examination We collected questions and their corresponding correct answers from the 116th Japanese National Dental Examination, administered in March 2023, from the Ministry of Health, Labour and Welfare (MHLW) website of Japan. Specifically, we selected 160 questions from the JNDE, including intraoral and extraoral photographs, CT, and MRI scans, echo, panoramic X-ray images, dental X-ray images, dental laboratory work, diagrams, and charts like orthodontic polygon tables, illustrations, and tables. To input questions with images, charts, and tables, we used ChatGPT-4V and Gemini Pro, both equipped with image recognition functions. We then analyzed the data, including the percentage of correct answers and other relevant information. The JNDE question criteria indicate that the JNDE will have three areas: compulsory, general, and clinical practical. Of the 160 questions used in this study, seven fell under the compulsory area, 55 under the general area, and 98 under clinical practice. Large language models The LLMs used were ChatGPT-4V from OpenAI (model: September 2023) and Gemini Pro from Google (model: December 13, 2023). These LLMs are classified as multimodal AI, capable of analyzing and outputting data based on two or more types of data, such as text, audio, images, videos, and numbers. Standardized prompt input and question text, along with charts and images, were individually copied and pasted into the corresponding web interface of each LLM, and all responses were recorded. The inputs were submitted on December 20, 2023. A prompt is “an instruction given to an LLM to enforce a specific rule, automate a process, or guarantee a certain quality and quantity of the output produced.” The format of the prompts for entering question text and images was standardized as follows: “You are a student taking the National Dental Examination. Please first provide the correct answers based on the text of the questions. Next, please explain your choice and why the other choices are inappropriate." Data and statistical analysis For data analysis, we used Qlik Sense® Enterprise August 2022 Patch 2 (Qlik Technologies Inc., King of Prussia, PA, USA). Statistical analysis was conducted using GraphPad Prism 9.5.1 (GraphPad Software, Boston, MA, USA) employing Fisher's exact test and Mann–Whitney U test. 

We collected questions and their corresponding correct answers from the 116th Japanese National Dental Examination, administered in March 2023, from the Ministry of Health, Labour and Welfare (MHLW) website of Japan. Specifically, we selected 160 questions from the JNDE, including intraoral and extraoral photographs, CT, and MRI scans, echo, panoramic X-ray images, dental X-ray images, dental laboratory work, diagrams, and charts like orthodontic polygon tables, illustrations, and tables. To input questions with images, charts, and tables, we used ChatGPT-4V and Gemini Pro, both equipped with image recognition functions. We then analyzed the data, including the percentage of correct answers and other relevant information. The JNDE question criteria indicate that the JNDE will have three areas: compulsory, general, and clinical practical. Of the 160 questions used in this study, seven fell under the compulsory area, 55 under the general area, and 98 under clinical practice. 

The LLMs used were ChatGPT-4V from OpenAI (model: September 2023) and Gemini Pro from Google (model: December 13, 2023). These LLMs are classified as multimodal AI, capable of analyzing and outputting data based on two or more types of data, such as text, audio, images, videos, and numbers. Standardized prompt input and question text, along with charts and images, were individually copied and pasted into the corresponding web interface of each LLM, and all responses were recorded. The inputs were submitted on December 20, 2023. A prompt is “an instruction given to an LLM to enforce a specific rule, automate a process, or guarantee a certain quality and quantity of the output produced.” The format of the prompts for entering question text and images was standardized as follows: “You are a student taking the National Dental Examination. Please first provide the correct answers based on the text of the questions. Next, please explain your choice and why the other choices are inappropriate." 

For data analysis, we used Qlik Sense® Enterprise August 2022 Patch 2 (Qlik Technologies Inc., King of Prussia, PA, USA). Statistical analysis was conducted using GraphPad Prism 9.5.1 (GraphPad Software, Boston, MA, USA) employing Fisher's exact test and Mann–Whitney U test. 

shows the interface of GPT-4V and Gemini Pro as it processes inputs from JNDE and illustrates a pivotal moment in the present study. The inputs include both textual questions and photographs. Upon submission of these inputs, the system generated corresponding responses. shows the number of correct answers, incorrect answers, and percentage of correct answers obtained for each of the 160 questions using GPT-4V and Gemini Pro. The correct response rates for GPT-4V and Gemini Pro were 35.0% and 28.1%, respectively, indicating that the correct response rate for GPT-4V was higher but not statistically significant. shows the percentage of correct answers for GPT-4V and Gemini Pro across the compulsory, general, and clinical practical areas of the JNDE. The correct response rates for GPT-4V and Gemini Pro were 57.1% and 28.6% in compulsory area, 43.6% and, 29.1% in general area, and 28.6% and 27.6% in clinical practice, respectively. Fisher's exact test was performed on the number of correct and incorrect answers from GPT-4V and Gemini Pro for the compulsory, general, and clinical practice questions; however, no statistically significant differences were found. GPT-4V had a higher percentage of correct answers in all three areas than Gemini Pro. presents the percentage of positive matches between GPT-4V and Gemini Pro in the three areas. Compliance rates were 57.1%, 41.8 %, and 28.6% for compulsory, general, and clinical practice, respectively. The overall compliance rate for the 160 questions was 34.4%. outlines the number of questions by dental subject and the percentage of correct answers for GPT-4V and Gemini Pro, ordered by the percentage of correct answers for GPT-4V. Dental anesthesiology and endodontics had more than 70% correct responses on GPT-4V, whereas Gemini Pro had fewer than 50% correct responses. The correct response rates were generally higher for GPT-4V, with some subjects having identical correct response rates. Only for Partial Dentures did, Gemini Pro exhibit a higher percentage of correct answers than GPT-4V. Overall, GPT-4V showed more correct answers than Gemini Pro. demonstrates the differences in the number of images inputted into GPT-4V and Gemini Pro for correct and incorrect responses. GPT-4V showed that, statistically, fewer images were inputted for correct responses. Gemini Pro tended to answer correctly, using fewer images. However, it showed more images than GPT-4V, and the number of inputted images for correct and incorrect responses was not statistically significant. Lastly, shows the number of images inputted into GPT-4V and Gemini Pro for illustrations, highlighting the significant differences between the images and charts in this study. Both models showed significantly more correct responses when fewer images were inputted. 

In the Japanese national dental examination, the accuracy rates of GPT-4V and Gemini Pro in response to questions were higher for GPT-4V across the mandatory, general, and clinical practice areas, although the difference was not statistically significant. This suggests that while LLMs have evolved to understand and respond to complex question formats, their level of understanding in specialized fields, such as dentistry, remains a challenge. , , According to OpenAI 4, the interpretation of medical images by GPT-4V is inconsistent, and the model sometimes gives accurate and incorrect answers to the same questions. The lack of statistically significant differences between LLMs across different dental areas in the JNDE highlights the need for further enhancement of LLM features, particularly to improve our understanding of domain-specific knowledge. This study demonstrated the importance of image recognition capabilities in answering JNDE questions. The introduction of LLMs with image recognition features represents a significant advancement in LLM technology, enabling the analysis of both textual and visual information. , , However, our data indicate that the current image recognition capabilities of LLMs are insufficient for interpreting images in specialized medical fields. Interestingly, our research data showed that the number of images included in a question affects the accuracy rate of the LLM, with a tendency toward higher accuracy with fewer images. This suggests that the quantity and complexity of the visual information affect the LLM's image recognition capabilities, thereby clearly identifying potential improvements for future model updates. One limitation of our study is that although we assessed the accuracy rate of the LLMs' responses, we should have evaluated the quality of the explanations provided by the LLMs. In future research, it would be beneficial to include an evaluation of these models' explanatory capabilities to provide a more comprehensive assessment of their performance. Moreover, we acknowledge the limitation that only one question was given to some dental subjects, and only one assessment of correct responses was made. However, the results of this study offer valuable insights into the image recognition deficits of the two LLM models. Future studies should consider assessing the correct response rate multiple times on the same dataset to enhance our understanding. This approach could lead to different results, offer a more comprehensive understanding of model performance, and help identify consistent patterns and areas for improvement in image recognition performance. Therefore, it is essential to interpret the results cautiously, as some subjects were given only one question, and the correct response rate results were assessed only once. The overall superior performance of GPT-4V compared to Gemini Pro in our study may be attributed to continuous updates in OpenAI's model. Furthermore, the fact that GPT-4V showed superior accuracy rates in specific areas, such as dental anesthesia and endodontics, suggesting that there are certain fields where the current LLM's image recognition capabilities are more suitable. In conclusion, this study demonstrates the potential of LLMs as educational support tools in dentistry, while also highlighting areas that require further technological development. In addition to the dental medical questions used in this study, it is necessary to evaluate the capabilities of LLMs for questions in other specialized fields. Advancements in LLMs, such as improved image recognition capabilities and the acquisition of specialized knowledge, can significantly transform dental education by complementing textbooks, lectures, and practical exercises with AI-driven learning experiences, thereby supporting students in their learning process. 

The authors have no conflicts of interest relevant to this article. 

