Expert Usability Evaluation of a Mobile Application for Systematic Caries Management in Children and Adolescents: A Cross-sectional Study
ntroduction Recently, the field of mobile healthcare (mHealth), which combines digital technology and healthcare, has attracted attention , and the quality of mHealth applications is also being addressed. In addition, studies are being conducted to verify the effectiveness of mHealth applications. The reason for conducting systematic reviews and evaluations of mHealth applications is that the rapid increase in the development of mHealth applications has raised an important issue about reliability, information, and functionality. It has also been noted that consumers have limited reference indicators for selecting high-quality applications. Developers need to develop useful, accurate, and adaptable apps. Usability evaluation should be conducted to ensure that users can effectively and subjectively perform specific tasks in any environment. The Mobile App Rating Scale (MARS) developed by Stoyanov et al. is a validated tool designed to assess the quality of healthcare-related applications. This scale has the advantage of being scored from an expert perspective, which compensates for the reliability issue of subjective user ratings. The MARS tool has been actively used to evaluate many types of healthcare applications, , including self-management related to the healthcare system. , Heuristic evaluation is a method of identifying usability issues based on usability principles by an expert. , Heuristic evaluation is known to be able to identify many important usability issues with a relatively small number of experts. CAMBRA-kids for caries management in infants and young children, has also been subjected to expert heuristic evaluation and evaluated for effectiveness. As of 2018, there are approximately 1,075 mobile applications focused on oral care. The majority of oral health-related mobile application research has focused on adolescents. Mobile applications are mainly used for oral health promotion and behavior management, with limited use for diagnosis and remote consultation. Smartphone use is a part of adolescents’ daily lives and has a high value to them. Given the accessibility and convenience of smartphone use, mobile applications could effectively manage dental caries in adolescents. Dental caries affects approximately 60–90% of adolescents worldwide. The high frequency of snacking and preference for cariogenic foods during this period can negatively affect dental health, making caries management critical. Mobile oral health applications accessible to adolescents should not only provide simple information but also be evidence-based and facilitate systematic caries management in collaboration with patient-centered clinicians. Therefore, in this study, an expert evaluation was conducted to verify the systematic aspects and usability of the “CAMBRA-students” application for caries risk assessment and systematic caries management in children and adolescents, to ensure the validity of the application, and ultimately to activate the caries management program for children and adolescents using “CAMBRA-students.” 

aterials and M ethods Participants The research participants were computer scientists and clinicians with sufficient knowledge and experience with the CAMBRA program. They were selected by purposive sampling. They had no limitations in installing and using mobile applications. The minimum number of experts for the evaluation was set at three. A total of five participants conducted the evaluation, including two IT specialists, two clinical dental hygienists with a master's degree and >3 years of clinical experience who had completed CAMBRA-related courses, and a professor of dental hygiene. “CAMBRA-students” Mobile Application Evaluated for Usability The mobile application “CAMBRA-students” was developed as an Android, iOS, and web-based application with the aim of identifying caries risk in children and adolescents, preventing dental caries, and ultimately promoting oral health. The implemented application was subjected to a usability evaluation method called “Cognitive Walkthrough” based on work scenarios to assess cognitive ease and potential for misuse, followed by modifications. After initial adjustments were made, the application was selected for evaluation. The “CAMBRA-students” application consisted of the following sections in . Research Tool MARS Evaluation of the “CAMBRA-students” Application The usability evaluation of the “CAMBRA-students” application was conducted using the MARS adapted by Byun. MARS was designed to identify and correct problems during the development process. In addition, it was also used to evaluate the usability of applications. It consisted of a total of 23 items in five subcategories: engagement (five items), functionality (four items), esthetics (three items), information (seven items), and subjective quality (four items), all rated on a five-point Likert scale. The usability of the application was evaluated by calculating the average score for each of the five subcategories with a score of three or more indicating fair to good. , Heuristics Evaluation of the “CAMBRA-students” Application Nielsen's heuristic evaluation is the most widely recognized usability evaluation tool in the field of user interfaces to date, consisting of 10 principles for evaluating technical errors or problems in an application. , For this study, we used a modified tool of Nielsen's heuristic evaluation with eight principles : visibility of the application, reflection of the real world, consistency and connectivity, ergonomic design, readability of the screen and ease of input, usability and efficiency of user settings, emotional aspects, and error management. The tool scored each heuristic on a 5-point scale (0: no usability problem at all; 1: cosmetic problem only, does not need to be fixed unless extra time is available on the project; 2: minor usability problem, fixing it should be given a low priority, 3: major usability problem, important to fix, should be given a high priority; and 4: usability catastrophe, imperative to fix). The experts freely described each heuristic principle. Any problem identified by more than two experts, or any problem rated with a severity level of four was considered for modification in the application. , , In addition, modifications were made based on the researchers’ judgment to improve usability by addressing problems that could improve the user experience. Research Procedure This study was approved by the Institutional Review Board of NAMSEOUL University (NAMSEOUL-202212-001) and conducted from June to July 2023. This is a two-part article due to the nature of a single IRB-reviewed study. In this study, five experts were selected, informed consent was obtained, and a survey was conducted to identify systematic errors or graphical problems in the “CAMBRA-students” application developed to manage dental caries in children and adolescents. Subjects were given instructions on how to download and use the application (Android, IOS) along with the questionnaire. Each of the five experts downloaded and used the application for 1 week before evaluating it. When the experts evaluated the app, they logged in directly from the user's perspective, entered risk and protective factors in the caries risk assessment section, and then used the preprovided administrator account to enter dental records and disease indicators arbitrarily from the expert's perspective. In addition, the application was revisited from the user's perspective to explore “assessment outcome”, “management”, and “oral health care information”. Any usability issues identified during this process were noted in the questionnaire and described freely. Based on the collected questionnaires, the researchers analyzed and discussed the findings according to heuristic standards and finally made modifications to the application according to the identified issues. Statistical Analysis The collected data were analyzed using SPSS version 23.0 (IBM Co., Armonk, New York, United States). General characteristics and heuristic scores of the study subjects were subjected to frequency analysis. MARS was analyzed by calculating the mean and standard deviation for each domain using descriptive statistics. 

The research participants were computer scientists and clinicians with sufficient knowledge and experience with the CAMBRA program. They were selected by purposive sampling. They had no limitations in installing and using mobile applications. The minimum number of experts for the evaluation was set at three. A total of five participants conducted the evaluation, including two IT specialists, two clinical dental hygienists with a master's degree and >3 years of clinical experience who had completed CAMBRA-related courses, and a professor of dental hygiene. 

The mobile application “CAMBRA-students” was developed as an Android, iOS, and web-based application with the aim of identifying caries risk in children and adolescents, preventing dental caries, and ultimately promoting oral health. The implemented application was subjected to a usability evaluation method called “Cognitive Walkthrough” based on work scenarios to assess cognitive ease and potential for misuse, followed by modifications. After initial adjustments were made, the application was selected for evaluation. The “CAMBRA-students” application consisted of the following sections in . 

MARS Evaluation of the “CAMBRA-students” Application The usability evaluation of the “CAMBRA-students” application was conducted using the MARS adapted by Byun. MARS was designed to identify and correct problems during the development process. In addition, it was also used to evaluate the usability of applications. It consisted of a total of 23 items in five subcategories: engagement (five items), functionality (four items), esthetics (three items), information (seven items), and subjective quality (four items), all rated on a five-point Likert scale. The usability of the application was evaluated by calculating the average score for each of the five subcategories with a score of three or more indicating fair to good. , Heuristics Evaluation of the “CAMBRA-students” Application Nielsen's heuristic evaluation is the most widely recognized usability evaluation tool in the field of user interfaces to date, consisting of 10 principles for evaluating technical errors or problems in an application. , For this study, we used a modified tool of Nielsen's heuristic evaluation with eight principles : visibility of the application, reflection of the real world, consistency and connectivity, ergonomic design, readability of the screen and ease of input, usability and efficiency of user settings, emotional aspects, and error management. The tool scored each heuristic on a 5-point scale (0: no usability problem at all; 1: cosmetic problem only, does not need to be fixed unless extra time is available on the project; 2: minor usability problem, fixing it should be given a low priority, 3: major usability problem, important to fix, should be given a high priority; and 4: usability catastrophe, imperative to fix). The experts freely described each heuristic principle. Any problem identified by more than two experts, or any problem rated with a severity level of four was considered for modification in the application. , , In addition, modifications were made based on the researchers’ judgment to improve usability by addressing problems that could improve the user experience. 

The usability evaluation of the “CAMBRA-students” application was conducted using the MARS adapted by Byun. MARS was designed to identify and correct problems during the development process. In addition, it was also used to evaluate the usability of applications. It consisted of a total of 23 items in five subcategories: engagement (five items), functionality (four items), esthetics (three items), information (seven items), and subjective quality (four items), all rated on a five-point Likert scale. The usability of the application was evaluated by calculating the average score for each of the five subcategories with a score of three or more indicating fair to good. , 

Nielsen's heuristic evaluation is the most widely recognized usability evaluation tool in the field of user interfaces to date, consisting of 10 principles for evaluating technical errors or problems in an application. , For this study, we used a modified tool of Nielsen's heuristic evaluation with eight principles : visibility of the application, reflection of the real world, consistency and connectivity, ergonomic design, readability of the screen and ease of input, usability and efficiency of user settings, emotional aspects, and error management. The tool scored each heuristic on a 5-point scale (0: no usability problem at all; 1: cosmetic problem only, does not need to be fixed unless extra time is available on the project; 2: minor usability problem, fixing it should be given a low priority, 3: major usability problem, important to fix, should be given a high priority; and 4: usability catastrophe, imperative to fix). The experts freely described each heuristic principle. Any problem identified by more than two experts, or any problem rated with a severity level of four was considered for modification in the application. , , In addition, modifications were made based on the researchers’ judgment to improve usability by addressing problems that could improve the user experience. 

This study was approved by the Institutional Review Board of NAMSEOUL University (NAMSEOUL-202212-001) and conducted from June to July 2023. This is a two-part article due to the nature of a single IRB-reviewed study. In this study, five experts were selected, informed consent was obtained, and a survey was conducted to identify systematic errors or graphical problems in the “CAMBRA-students” application developed to manage dental caries in children and adolescents. Subjects were given instructions on how to download and use the application (Android, IOS) along with the questionnaire. Each of the five experts downloaded and used the application for 1 week before evaluating it. When the experts evaluated the app, they logged in directly from the user's perspective, entered risk and protective factors in the caries risk assessment section, and then used the preprovided administrator account to enter dental records and disease indicators arbitrarily from the expert's perspective. In addition, the application was revisited from the user's perspective to explore “assessment outcome”, “management”, and “oral health care information”. Any usability issues identified during this process were noted in the questionnaire and described freely. Based on the collected questionnaires, the researchers analyzed and discussed the findings according to heuristic standards and finally made modifications to the application according to the identified issues. 

The collected data were analyzed using SPSS version 23.0 (IBM Co., Armonk, New York, United States). General characteristics and heuristic scores of the study subjects were subjected to frequency analysis. MARS was analyzed by calculating the mean and standard deviation for each domain using descriptive statistics. 

esults General Characteristics of Study Subjects Study participants included two IT specialists, two clinical dental hygienists, and one professor of dental hygiene. Of these, four (80%) had a master's degree or higher. All of them had >10 years of experience in their respective fields ( ). MARS Evaluation of the “CAMBRA-students” Application Mobile App Rating Scale had an average objective score of 3.76 and the average score including subjective application quality was 3.64. The highest-scoring question was “credibility” in the “information” category with a score of 4.80. The lowest scoring question was “customization” in the “engagement” category with a score of 2.20. Other scores ranged from 2.00 to 2.90 for the “interactivity” question in the “engagement” category and the “would you pay for this application?” question in the “application subjective quality” category, with 20 questions scoring above the average of 3.00. When considering the average score for each of the subscales of the tool, the subscale for commitment was the lowest, with an average score of 3.00. The subscale with the highest score was the “information” subscale with a mean score of 4.18, or 83.53% out of a maximum score of 5 ( ). Heuristic Evaluation of the “CAMBRA-students” Application A total of 45 issues were raised in the heuristic evaluation, with heuristic #5 receiving the most feedback from the experts. The highest severity level, four points, was given to the heuristic #3. Among the identified heuristic issues, those raised by two or more evaluators or those with a severity score of four or higher were considered for application modification. This criterion resulted in prioritized modifications for nine issues: including those related to heuristics #1 and #2, which received a severity score of three, and issues under heuristic #3, which received the highest severity score of four. In addition to these nine issues, 14 usability comments were also discussed by the researchers and included in the application modifications. As a result, a total of 23 issues were identified and addressed for revision ( ). Expert Comments Based on the Eight Usability Principles of Heuristic Evaluation The expert evaluation of the “CAMBRA-students” application resulted in a total of 23 problems ( ). The IT specialist described 16 issues. The clinical dental hygienist described six issues and the professor of dental hygiene described one issue. A total of three issues were identified related to heuristic #1, the IT specialist raised the lack of an automatic logout function related to user security and the mislabeling of data related to visibility. The clinical dental hygienist suggested color-coding caries risk to increase visibility. A total of six issues were raised for heuristic #2. The main issues raised by the experts were the lack of instructions for use and the inappropriate wording and location of the notification button. Dental hygienists suggested the same issue and recommended that it be revised to use words that are easily understood by elementary school students. The issues related to heuristic #3 were assigned a severity level of four. An inconsistent wording of notifications was reported, noting that the names of the “accept” and “decline” buttons were different in different notification windows, which could be confusing to users. The issues that fell under heuristic #4 were that the lack of a logout button on the main screen and the lack of an icon to navigate to the home screen hindered user usability. There was a total of six issues that fell under heuristic #5; the IT specialist recommended moving the search bar to the top because it was located at the bottom of the screen and was not easy to use, while the clinical dental hygienist suggested improving the readability by correcting typos and line breaks in the oral health management. The professor of dental hygiene also recommended unifying the English and Korean languages as it could be confusing to users. The IT specialist recommended several changes related to heuristic #6 and heuristic #8. The expert recommended fixing the usability issues related to user settings and changes to the caries risk assessment that could hinder usability. 

Study participants included two IT specialists, two clinical dental hygienists, and one professor of dental hygiene. Of these, four (80%) had a master's degree or higher. All of them had >10 years of experience in their respective fields ( ). 

Mobile App Rating Scale had an average objective score of 3.76 and the average score including subjective application quality was 3.64. The highest-scoring question was “credibility” in the “information” category with a score of 4.80. The lowest scoring question was “customization” in the “engagement” category with a score of 2.20. Other scores ranged from 2.00 to 2.90 for the “interactivity” question in the “engagement” category and the “would you pay for this application?” question in the “application subjective quality” category, with 20 questions scoring above the average of 3.00. When considering the average score for each of the subscales of the tool, the subscale for commitment was the lowest, with an average score of 3.00. The subscale with the highest score was the “information” subscale with a mean score of 4.18, or 83.53% out of a maximum score of 5 ( ). 

A total of 45 issues were raised in the heuristic evaluation, with heuristic #5 receiving the most feedback from the experts. The highest severity level, four points, was given to the heuristic #3. Among the identified heuristic issues, those raised by two or more evaluators or those with a severity score of four or higher were considered for application modification. This criterion resulted in prioritized modifications for nine issues: including those related to heuristics #1 and #2, which received a severity score of three, and issues under heuristic #3, which received the highest severity score of four. In addition to these nine issues, 14 usability comments were also discussed by the researchers and included in the application modifications. As a result, a total of 23 issues were identified and addressed for revision ( ). 

The expert evaluation of the “CAMBRA-students” application resulted in a total of 23 problems ( ). The IT specialist described 16 issues. The clinical dental hygienist described six issues and the professor of dental hygiene described one issue. A total of three issues were identified related to heuristic #1, the IT specialist raised the lack of an automatic logout function related to user security and the mislabeling of data related to visibility. The clinical dental hygienist suggested color-coding caries risk to increase visibility. A total of six issues were raised for heuristic #2. The main issues raised by the experts were the lack of instructions for use and the inappropriate wording and location of the notification button. Dental hygienists suggested the same issue and recommended that it be revised to use words that are easily understood by elementary school students. The issues related to heuristic #3 were assigned a severity level of four. An inconsistent wording of notifications was reported, noting that the names of the “accept” and “decline” buttons were different in different notification windows, which could be confusing to users. The issues that fell under heuristic #4 were that the lack of a logout button on the main screen and the lack of an icon to navigate to the home screen hindered user usability. There was a total of six issues that fell under heuristic #5; the IT specialist recommended moving the search bar to the top because it was located at the bottom of the screen and was not easy to use, while the clinical dental hygienist suggested improving the readability by correcting typos and line breaks in the oral health management. The professor of dental hygiene also recommended unifying the English and Korean languages as it could be confusing to users. The IT specialist recommended several changes related to heuristic #6 and heuristic #8. The expert recommended fixing the usability issues related to user settings and changes to the caries risk assessment that could hinder usability. 

iscussion This study was to evaluate technical errors or issues encountered during the use of the mobile application “CAMBRA-students,” developed for the systematic management of dental caries in children and adolescents. The usability evaluation was performed using the heuristic evaluation , and the MARS , on five experts. The heuristic evaluation followed eight principles and identified a total of 45 issues, with 23 significant issues emerging based on the study's evaluation criteria. Of these, IT specialists identified 16 issues, clinical dental hygienists identified six issues, and a professor of dental hygiene identified one issue. Based on numerous precedents, , , this study ranked the issues raised by experts according to their severity and incorporated them into application modifications. Content suggested by IT specialists and clinical dental hygienists under heuristic #1 included automatic logout upon program closure and corrections to date display errors and improved risk level visibility. The most critically received heuristic was #3, which was related to #2. These related issues were addressed by clarifying messaging and improving screen layout and button placement. Changes also included clarifying the postquestionnaire process and improving error management for a more intuitive user experience. The overall usability evaluation of the “CAMBRA-students” application was conducted using the MARS tool, which could identify problems encountered during the process of using the application for ease of learning and users’ subjective views. The overall average MARS score of this study was 3.64, which was higher than the average MARS score of 3.40 of 37 applications in a previous study that systematically evaluated oral health care applications. In addition, Zhang et al. have stated that the usability of an application is satisfactory when it scores 3.0 (moderate) or higher. Overall, the usability of the application in this study was rated as satisfactory. Among the MARS items, the “customization” item in the “engagement” domain scored the lowest with a score of 2.20. Previous studies , have also shown that “engagement” received low scores overall and is an important area for overall improvement. Siddiqui et al. have suggested that “engagement” is a measure of how engaged users are with the application and that it is important to focus on the application design, user interest, and targeting the right audience and improving interactive features to enable interaction with clinicians. Patients should be provided with oral health education videos and other information that encourages user engagement. Applications should be updated regularly to provide the latest technology and information, rather than being neglected, which is a problem with all applications. The expert evaluation of the “CAMBRA-students” application in this study indicates that the application's systematic quality and usability of the application are generally satisfactory, making it suitable for its intended users. The use of the “CAMBRA-students” application in school-based oral health management programs for children and adolescents is expected to facilitate individualized oral care and ultimately contribute to the improvement of oral health among these groups. This study highlights the potential of using the “CAMBRA-students” application for pediatric dental caries management while acknowledging the limitations of expert-only evaluations. Future research should include user-based usability evaluations and qualitative expert ratings for more in-depth analysis. It is anticipated that the application, continuously updated with evidence-based information, will become a widely used tool for pediatric oral health management. 

onclusion In this study, the expert evaluation of the “CAMBRA-students” application showed that the overall systematic quality and usability of the application were suitable for application users. In the future, when the “CAMBRA-students” application is applied to school-based oral health care for systematic caries management in children and adolescents, it is expected to provide personalized oral health care for each student in the long term, ultimately contributing to improving the oral health of children and adolescents. Clinical Significance The “CAMBRA-students” application can be effectively used in public oral health services to manage caries risk in adolescents. Expert evaluation improved the application's usability and systematic quality. This enables personalized oral health management for adolescents. The application has the potential to enhance adolescent oral health and can be integrated into school-based programs. 

The “CAMBRA-students” application can be effectively used in public oral health services to manage caries risk in adolescents. Expert evaluation improved the application's usability and systematic quality. This enables personalized oral health management for adolescents. The application has the potential to enhance adolescent oral health and can be integrated into school-based programs. 

rcid Yu-Min Kang https://orcid.org/0000-0001-8020-4326 An-Na Yeo https://orcid.org/0000-0003-2500-1408 Su-Young Lee https://orcid.org/0000-0002-5502-3037 

