## Protocol Section
### Identification Module
**NCT ID:** NCT06273579

**Brief Title:** Efficiency of Verbal Intelligent Tutor Instruction in Neurosurgical Simulation

**Official Title:** Efficiency of Verbal Intelligent Tutor Instruction in Neurosurgical Simulation: A Randomized Controlled Trial

#### Organization Study ID Info
**ID:** 2010-270, NEU-09-042-Trial 5
#### Organization
**Class:** OTHER
**Full Name:** McGill University
### Status Module
#### Completion Date
**Date:** 2024-12

**Type:** ESTIMATED

#### Expanded Access Info
#### Last Update Post Date
**Date:** 2024-03-05

**Type:** ACTUAL

**Last Update Submit Date:** 2024-03-03

**Overall Status:** NOT_YET_RECRUITING

#### Primary Completion Date
**Date:** 2024-10

**Type:** ESTIMATED

#### Start Date
**Date:** 2024-03

**Type:** ESTIMATED

**Status Verified Date:** 2024-03

#### Study First Post Date
**Date:** 2024-02-22
**Type:** ACTUAL
**Study First Submit Date:** 2024-02-07
**Study First Submit QC Date:** 2024-02-16
### Sponsor Collaborators Module

#### Lead Sponsor

**Class:** OTHER

**Name:** McGill University

#### Responsible Party

**Investigator Affiliation:** McGill University

**Investigator Full Name:** Rolando Del Maestro

**Investigator Title:** Director of the Neurosurgical Simulation and Artificial Intelligence Learning Centre

**Type:** PRINCIPAL_INVESTIGATOR

### Oversight Module

**Is FDA Regulated Device:** False

**Is FDA Regulated Drug:** False

### Description Module

**Brief Summary:** At the Neurosurgical Simulation and Artificial Intelligence Learning Centre, we seek to provide surgical trainees with innovative technologies that allow them to improve their surgical technical skills in risk-free environments, potentially improving patient operative outcomes. The Intelligent Continuous Expertise Monitoring System (ICEMS), a deep learning application that assesses and trains neurosurgical technical skill and provides continuous intraoperative feedback, is one such technology that may improve surgical education.

In this randomized controlled trial, medical students from four Quebec universities will be blinded and randomized to one of three groups (one control and two experimental). Group 1 (control) will be provided with verbal AI tutor feedback based on the ICEMS error detection. Group 2 will be tutored by a human instructor who will receive ICEMS error data and deliver verbal instruction identical to that which the AI tutor delivers. Group 3 will be tutored by a human instructor who will be provided with ICEMS data but may deliver feedback as they feel is appropriate to correct the error.

The aim of this study is to determine how the method of delivery of verbal surgical error instruction influences trainee response to instruction and overall surgical performance. Evaluating trainee responses to AI instructor verbal feedback as compared to feedback from human instructors will allow for further development, testing, and optimization of the ICEMS and other AI tutoring systems.

**Detailed Description:** Background: Expert surgical technical skill is linked with improved patient outcomes; however, training novices to master these skills remains challenging. The Intelligent Continuous Expertise Monitoring System (ICEMS) is a deep learning application that was developed at the Neurosurgical Simulation and Artificial Intelligence Learning Centre to improve neurosurgical education. The ICEMS assesses and trains bimanual surgical performance by providing continuous feedback via verbal instructions in order to improve trainee performance and mitigate errors.

Rationale: A previous randomized controlled trial (RCT) performed at our centre demonstrated that intelligent tutoring is more effective than expert tutoring in a simulated neurosurgical procedure (NCT05168150). Another RCT revealed that medical students' performance in response to ICEMS instruction to decrease bipolar force application was variable (NCT04700384). An agglomerative clustering algorithm classified these variable student responses into 3 groups: 53% successfully obeyed the instruction to correct the error, 36% did not obey the instruction, and 11% over-responded to the instruction. This response variability could significantly limit the utility of the ICEMS and may be attributed to different learning styles, stress levels, or misinterpretation of AI instruction. During this study, expert trainers were not provided with ICEMS error data. Conducting a new RCT in which expert trainers are provided with ICEMS error data will clarify the reason many trainees did not respond to the AI instruction.

This report follows the Consolidated Standards of Reporting Trials-Artificial Intelligence (CONSORT-AI) as well as the Machine Learning to Assess Surgical Expertise (MLASE) checklist.

Hypotheses:

1. Verbal AI feedback will yield significantly lower success response rates among trainees than identical error feedback provided by human instructors.
2. Trainee performance assessment scores will be significantly higher in the two different human instruction groups assessed.
3. Instruction delivered by the AI tutor will result in increased stress levels and cognitive load as compared to verbal error feedback delivered by human instructors.

Primary Objectives: To determine how the method of delivery of surgical error instruction influences:

1. Trainee response to instruction, i.e., whether they corrected, did not correct, or over-corrected the error (data collected by the ICEMS).
2. Trainee overall surgical performance (average expertise score on practice scenarios calculated by the ICEMS, Objective Structured Assessment of Technical Skills (OSATS) score on realistic scenario determined by two blinded expert raters).

Secondary Objective: To determine how the method of delivery of surgical error instruction influences trainee affective cognitive responses (self-reported via questionnaires on 5-point Likert scales).

Setting: McGill University's Neurosurgical Simulation and Artificial Intelligence Learning Centre.

Participants: Students enrolled in their preparatory, first, or second year at one of four Quebec medical schools.

Design: A three-arm randomized controlled trial.

Intervention: Participants will undergo a training session of approximately 90 minutes on the NeuroVR (CAE Healthcare), a virtual reality (VR) surgical simulator that simulates a subpial brain tumor resection. The NeuroVR has two possible scenarios: a simple practice scenario and a complex realistic scenario. Participants will perform six repetitions of the practice scenario (5 minutes each) followed by the realistic scenario (13 minutes). The ICEMS will continuously assess performance throughout the trial. All participants will receive verbal feedback when the ICEMS detects an error in their performance; however, the method of delivery of this verbal feedback will differ between groups.

* Group 1 (control) will receive verbal feedback directly from the ICEMS when an error is detected.
* Group 2 (experimental) will receive verbal feedback from an expert instructor delivered in the same words as the ICEMS.
* Group 3 (experimental) will receive verbal feedback from an expert instructor delivered in their own words.

Verbal feedback will be based on the following six metrics:

1. Tissue injury risk: When a trainee receives feedback on this metric, the healthy brain tissue has been damaged.
2. Bleeding risk: When a trainee receives feedback on this metric, there is bleeding that must be cauterized.
3. Instrument tip separation distance: Refers to the distance between the tip of the ultrasonic aspirator and the tips of the bipolar forceps. When a trainee receives feedback on this metric, their instruments are too far apart.
4. High bipolar force: Refers to the amount of force applied to the tissue by the bipolar forceps. When a trainee receives feedback on this metric, they are applying too much force with the bipolar.
5. Low bipolar force: Refers to the amount of force applied to the tissue by the bipolar forceps. When a trainee receives feedback on this metric, they are not applying enough force with the bipolar.
6. High aspirator force: Refers to the amount of force applied to the tissue by the ultrasonic aspirator. When a trainee receives feedback on this metric, they are applying too much force with the aspirator.

These metrics will continuously be evaluated by the ICEMS. The ICEMS will only detect an error on one metric at a time according to a predetermined hierarchy (in the order listed above). For example, if a trainee makes an error on both bleeding risk (2) and high aspirator force (6) at the same time, the ICEMS will only detect an error for bleeding risk since this metric is above high aspirator force in the hierarchy.

The first practice scenario will serve as a baseline; thus, no feedback will be given. In the second, third, fourth, and fifth repetitions, feedback will be given according to ICEMS error detection. In the sixth repetition as well as the realistic scenario, no feedback will be provided.

Significance: With surgical education approaches beginning to shift towards competency-based frameworks, the implementation of effective AI educational feedback into surgical training becomes crucial for optimizing surgical learning. The results of this RCT will allow for the evaluation and reengineering of the ICEMS and other AI tutoring systems, which may advance the development of not only standardized competency-based surgical education training curricula, but any AI tutor technology dependent on verbal instruction.

### Conditions Module

**Conditions:**

- Surgical Education

**Keywords:**

- Virtual Reality

- Artificial Intelligence

- Surgical Training

- Surgical Education

- Surgical Simulation

- Neurosurgery

### Design Module

#### Design Info

**Allocation:** RANDOMIZED

**Intervention Model:** PARALLEL

**Intervention Model Description:** Randomized Controlled Trial

##### Masking Info

**Masking:** DOUBLE

**Masking Description:** Double (Participant, Expert Instructor)

Both study participants and expert instructors are unaware of the intervention (varying the method of delivery of verbal error instruction). Moreover, when rating performance in the realistic scenario using the OSATS, experts are unaware of which group any given trainee is part of.

**Who Masked:**

- PARTICIPANT

- OUTCOMES_ASSESSOR

**Primary Purpose:** HEALTH_SERVICES_RESEARCH

#### Enrollment Info

**Count:** 78

**Type:** ESTIMATED

**Phases:**

- NA

**Study Type:** INTERVENTIONAL

### Arms Interventions Module

#### Arm Group 1

**Description:** 26 participants allocated. During their second, third, fourth, and fifth repetition of the practice subpial brain tumor resection scenario, participants will receive verbal ICEMS feedback when the system detects an error on their performance.

**Label:** Tutored by AI

**Type:** NO_INTERVENTION

#### Arm Group 2

**Description:** 26 participants allocated. During their second, third, fourth, and fifth repetition of the practice subpial brain tumor resection scenario, participants will receive verbal feedback from an expert instructor. The expert instructor will deliver this feedback using the same words as the ICEMS.

**Intervention Names:**

- Behavioral: Experimental Group - Verbal expert instructor feedback in AI's words

**Label:** Tutored by human instructor using AI's words

**Type:** EXPERIMENTAL

#### Arm Group 3

**Description:** 26 participants allocated. During their second, third, fourth, and fifth repetition of the practice subpial brain tumor resection scenario, participants will receive verbal feedback from an expert instructor. The expert instructor will deliver this feedback using any wording they feel is appropriate to correct the error.

**Intervention Names:**

- Behavioral: Experimental Group - Verbal expert instructor feedback in expert's own words

**Label:** Tutored by human instructor using wording of choice

**Type:** EXPERIMENTAL

### Interventions

#### Intervention 1

**Arm Group Labels:**

- Tutored by human instructor using AI's words

**Description:** Expert instructor assigned to tutor this group will receive error detection data from the ICEMS. They will also be provided with a list of commands that the ICEMS uses. When the system detects an error in a student's performance for a given metric, the expert instructor must deliver this command in the same words as the ICEMS.

**Name:** Experimental Group - Verbal expert instructor feedback in AI's words

**Type:** BEHAVIORAL

#### Intervention 2

**Arm Group Labels:**

- Tutored by human instructor using wording of choice

**Description:** Expert instructor assigned to tutor this group will receive error detection data from the ICEMS. When the system detects an error in a student's performance for a given metric, the expert will deliver feedback in their own words.

**Name:** Experimental Group - Verbal expert instructor feedback in expert's own words

**Type:** BEHAVIORAL

### Outcomes Module

#### Primary Outcomes

**Description:** After a trainee receives verbal feedback on a specific metric, the ICEMS will record their response to this instruction, i.e., whether they corrected, did not correct, or over-corrected the error.

**Measure:** Response to instruction

**Time Frame:** 1 day of study

**Description:** The ICEMS will continuously assess the trainee's performance and calculate an average expertise score between -1.00 (novice) and 1.00 (expert).

**Measure:** Average Intelligent Continuous Expertise Monitoring System (ICEMS) expertise score

**Time Frame:** 1 day of study

**Description:** While performing the complex realistic scenario, participants will be video recorded. Two blinded expert raters will evaluate these videos using the OSATS global rating scale between 1 (novice) and 7 (expert).

**Measure:** Objective Structured Assessment of Technical Skills (OSATS) global rating

**Time Frame:** Approximately 5 months after start of study

#### Secondary Outcomes

**Description:** Measured using Duffy's Medical Emotions Scale (MES) before, during, and after the intervention (self-reported via questionnaires on 5-point Likert scales).

**Measure:** Difference in the strength of emotions elicited

**Time Frame:** 1 day of study

**Description:** Measured using Leppink's Cognitive Load Index (CLI) after the intervention (self-reported via questionnaire on 5-point Likert scales).

**Measure:** Difference in cognitive load

**Time Frame:** 1 day of study

### Eligibility Module

**Eligibility Criteria:** Inclusion Criteria:

* Medical students who are actively enrolled in medical school at any Quebec institution who do not fit the exclusion criteria.
* Premedical students who are actively enrolled in medical school at any Quebec institution who do not fit the exclusion criteria.

Exclusion Criteria:

* Prior use of the NeuroVR (CAE Healthcare) simulator.

**Healthy Volunteers:** True

**Minimum Age:** 18 Years

**Sex:** ALL

**Standard Ages:**

- ADULT

- OLDER_ADULT

### Contacts Locations Module

#### Central Contacts

**Contact 1:**

**Email:** rolando.del_maestro@mcgill.ca

**Name:** Rolando F Del Maestro, MD, PhD

**Phone:** (519) 708-0346

**Role:** CONTACT

**Contact 2:**

**Email:** bianca.giglio@mail.mcgill.ca

**Name:** Bianca Giglio, BSc

**Phone:** (514) 802-1608

**Role:** CONTACT

#### Locations

**Location 1:**

**City:** Montréal

**Contacts:**

***Contact 1:***

- **Email:** rolando.del_maestro@mcgill.ca

- **Name:** Rolando F Del Maestro, MD, PhD

- **Phone:** (519) 708-0346

- **Role:** CONTACT

***Contact 2:***

- **Email:** bianca.giglio@mail.mcgill.ca

- **Name:** Bianca Giglio, BSc

- **Phone:** (514) 802-1608

- **Role:** CONTACT

**Country:** Canada

**Facility:** Neurosurgical Simulation and Artificial Intelligence Learning Centre

**State:** Quebec

**Zip:** H2X 4B3

### IPD Sharing Statement Module

**Access Criteria:** Researchers who wish to access the data must contact the principal investigator of the trial, Dr. Rolando F. Del Maestro.

**Description:** Data obtained from primary and secondary outcomes may be shared if other researchers have an interest in this data.

**Info Types:**

- STUDY_PROTOCOL

- SAP

- ICF

- CSR

- ANALYTIC_CODE

**IPD Sharing:** YES

**Time Frame:** Data will be available for 5 years following the completion of the trial.

### References Module

#### References

**Citation:** Fazlollahi AM, Bakhaidar M, Alsayegh A, Yilmaz R, Winkler-Schwartz A, Mirchi N, Langleben I, Ledwos N, Sabbagh AJ, Bajunaid K, Harley JM, Del Maestro RF. Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial. JAMA Netw Open. 2022 Feb 1;5(2):e2149008. doi: 10.1001/jamanetworkopen.2021.49008.

**PMID:** 35191972

**Citation:** Yilmaz R, Fazlollahi AM, Winkler-Schwartz A, Wang A, Makhani HH, Alsayegh A, Bakhaidar M, Tran DH, Santaguida C, Del Maestro RF. Effect of Feedback Modality on Simulated Surgical Skills Learning Using Automated Educational Systems- A Four-Arm Randomized Control Trial. J Surg Educ. 2024 Feb;81(2):275-287. doi: 10.1016/j.jsurg.2023.11.001. Epub 2023 Dec 29.

**PMID:** 38160107

**Citation:** Fazlollahi AM, Yilmaz R, Winkler-Schwartz A, Mirchi N, Ledwos N, Bakhaidar M, Alsayegh A, Del Maestro RF. AI in Surgical Curriculum Design and Unintended Outcomes for Technical Competencies in Simulation Training. JAMA Netw Open. 2023 Sep 5;6(9):e2334658. doi: 10.1001/jamanetworkopen.2023.34658.

**PMID:** 37725373

**Citation:** Mirchi N, Bissonnette V, Yilmaz R, Ledwos N, Winkler-Schwartz A, Del Maestro RF. The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine. PLoS One. 2020 Feb 27;15(2):e0229596. doi: 10.1371/journal.pone.0229596. eCollection 2020.

**PMID:** 32106247

**Citation:** Winkler-Schwartz A, Yilmaz R, Mirchi N, Bissonnette V, Ledwos N, Siyar S, Azarnoush H, Karlik B, Del Maestro R. Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation. JAMA Netw Open. 2019 Aug 2;2(8):e198363. doi: 10.1001/jamanetworkopen.2019.8363.

**PMID:** 31373651

**Citation:** Sioufi J, Hall B, Antel R, Moussa S, Subasri M, Fakih M, Islam N, Hamdy RC, Chopra S, Harley JM, Keuhl A, Bassilious E, Sherbino J, Bilgic E, Bondok MS, Bondok M, Martel L, Law C, Posel N, Fleiszer D, Daud A, Hauer T, Carr-Pries N, Hali K, Wolfstadt J, Ferguson P, Ghasroddashti A, Sorefan-Mangou F, Del Fernandes R, Williams E, Choi K, Zevin B, Patterson ED, Kirupaharan S, Mann S, Winthrop A, Zevin B, Bondok M, Ghanmi N, Etherington C, Saddiki Y, Lefebvre I, Berthelot P, Dion PM, Raymond B, Seguin J, Sekhavati P, Islam S, Boet S, Tee T, Pachchigar P, Tarabay B, Yilmaz R, Hamdan NA, Agu C, Almansouri A, Harley J, Del Maestro R, Bondok M, Bondok MS, Nguyen AX, Law C, Nathoo N, Bakshi N, Ahuja N, Damji KF, Grewal K, Azher S, Moreno M, Pekrun R, Wiseman J, Fried GM, Lajoie S, Brydges R, Hadwin A, Sun NZ, Khalil E, Harley JM, Nguyen EL, Patel P, Muaddi H, Rukavina N, Bucur R, Shwaartz C, Islam N, Moussa S, Subasri M, Fakih M, Hamdy RC, Wong E, Tewari A, Brydges R, Louridas M, Balaji S, Patel P, Muaddi H, Gaebe K, Luzzi C, Kay A, Rukavina N, Selzner M, Reichman T, Shwaartz C, Balaji S, Muaddi H, Shahabinezhad A, Patel P, Rukavina N, Reichman T, Jayaraman S, Shwaartz C, Nashed J, Ramelli L, Kolasky O, Dickenson T, Dullege M, Kang A, Winthrop A, Mann S, Lau D, Henkelman E, Jacob J, Watson I, Haji F, McEwen CC, Jaffer I, Sibbald M, Blouin V, Benard F, Pelletier F, Abdo S, Meloche-Dumas L, Kapralos B, Dubrowski A, Patocskai E, Pachchigar P, Agu C, Yilmaz R, Tee T, Maestro RD, Adedipe I, Stephens C, Ghebretatios M, Laplante S, Patel P, Balaji S, Muaddi H, Rukavina N, Shwaartz C, Brodovsky M, Lai C, Behzadi A, Blair G, Almansouri A, Hamdan NA, Yilmaz R, Tee T, Pachchigar P, Eskandari M, Agu C, Giglio B, Balasubramaniam N, Bierbrier J, Collins DL, Gueziri HE, Del Maestro RF, Koonar E, Ramazani F, Hart R, Henley J, Roberts S, Chandarana S, Matthews W, Schrag C, Matthews J, Mackenzie D, Cutting C, Lui J, Delisle E, Cordoba T, Cordoba C, Giglio B, Lacroix A, Cairns J, Alsayegh A, Alhantoobi M, Balasubramaniam N, Safih W, Hamel M, Del Maestro R, Francis G, Moise A, Omar Y, Hathi K, Mavedatnia D, Grose E, Philips T, Schneider C, Corbin D, Lesage F, Pellerin M, Ben-Ali W, Tamani Z, Joly-Chevrier M, Benard F, Meloche-Dumas L, Laflamme L, Boulva K, Younan R, Dubrowski A, Patocskai E, Sticca G, Petruccelli J, Dorion D, Osman Y, Benard F, Habti M, Meloche-Dumas L, Duranleau X, Boulva K, Kaviani A, Younan R, Dubrowski A, Vessella K, Patocskai E, Valji R, Turner S, Lam T, Mobilio MH, Hirsh J, Lising D, Cil T, Marcon E, Moulton CA, D'Souza A, Milazzo T, Datta S, Valiquette C, Avery E, Voineskos S, Musgrave M, Wanzel K, Schneidman J, Armstrong N, Gerardis G, Silver J, Azzam MA, Fisher R, Banks I, Young M, Nguyen LH, Skakum M, Hancock BJ, Min SL, Youssef F, Keijzer R, Morris M, Shawyer A, Retrosi G. C-CASE 2023: Promoting Excellence in Surgical Education: Canadian Conference for the Advancement of Surgical Education, Oct. 12-13, 2023, Montreal, Quebec. Can J Surg. 2023 Dec 8;66(6 Suppl 2):S137-S150. doi: 10.1503/cjs.014523. Print 2023 Nov-Dec. No abstract available.

**PMID:** 38065582

## Derived Section

### Intervention Browse Module - Browse Branches

- Abbrev: Infe

- Name: Anti-Infective Agents

- Abbrev: All

- Name: All Drugs and Chemicals

### Intervention Browse Module - Browse Leaves

- ID: M11767

- Name: Metronidazole

- Relevance: LOW

- As Found: Unknown

### Misc Info Module

- Version Holder: 2024-05-24
