Ying Song,Junjie Hu,Qiang Wang,Chengrong Yu,Jiachong Su,Lin Chen,Xiaorui Jiang,Bo Chen,Lei Zhang,Qian Yu,Ping Li,Feng Wang,Sen Bai,Yong Luo,Zhang Yi
Young oncologists benefit more than experts from deep learning-based organs-at-risk contouring modeling in nasopharyngeal carcinoma radiotherapy: A multi-institution clinical study exploring working experience and institute group style factor
Deep learning,Contouring,Organs-at-risk,Nasopharyngeal carcinoma radiotherapy,Working experience
Highlights • The contouring performance and consistency were improved with various development. • The deep learning-based contouring system was effective for various institutes. • Beginners have better acceptance and more contouring improvement than experts.
Young oncologists benefit more than experts from deep learning-based organs-at-risk contouring modeling in nasopharyngeal carcinoma radiotherapy: A multi-institution clinical study exploring working experience and institute group style factor

• The contouring performance and consistency were improved with various development. • The deep learning-based contouring system was effective for various institutes. • Beginners have better acceptance and more contouring improvement than experts.

Nasopharyngeal carcinoma (NPC) is a typical tumor in the head and neck [1], which is widely observed in southeast Asia and northern Africa, especially in southern China [2]. The high incidence and mortality worldwide of NPC demand enhancing control and prevention [3]. As the primary treatment modality for NPC, radiotherapy plays an important role [4], [5], [6]. NPC radiotherapy requires accurate contouring of organs at risk (OAR) for dose sparing. However, manual OAR contouring for NPC is a time-consuming and tedious procedure that heavily depends on personal experience. Significant observer variance and low group consistency have been reported [7], [8]. Automatic contouring is a promising solution for improving clinical contouring efficiency and interobserver consistency [7], [8]. There were traditional solutions based on image deformation and registration [9], [10] with some unsolved problems, namely, no golden rules for atlas establishment [11], the heavy requirement for computational resources, and long computing time [12]. Deep learning methods provide solutions to automatic radiotherapy contouring from a new point of view. For NPC OAR contouring, there were algorithm design studies for deep learning-based contouring system (DLCS) performance improvement [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25] and clinical studies for DLCS performance evaluation [26], [27]. The DLCS performance ranged from 0.45 to 0.94 for Dice with various models [28]. However, the evaluation based on numerical geometric metrics does not fully reveal the clinical acceptance of the contouring target for different oncologists, and the clinical evaluation research was limited. Deep learning methods and registration algorithm contouring performance were evaluated for swallowing-related organs [27] and 11 organs of the head and neck site [26] by quantitative parameters, drawing the main conclusion that DLCS provided equivalent or better segmentation accuracy and higher efficiency. Besides contouring time reduction and segmentation accuracy, complicated clinical issues remained in DLCS application, e.g., inter- and intra-observer consistency variation, group consistency variation, oncologist performance improvement, and their acceptability, especially for oncologists with diverse experience and institute group contouring styles. These problems pose challenges to the DLCS’s clinical effectiveness. Some research investigated this topic providing proof of lower inter-observer variability and improved consistency for the larynx contouring course [31] and atlas-based NPC OAR edition [29]. Some research purely focuses on manual contouring analysis of the prostate site [30] and multiple lesions and organs [31] with no intervening factors. For the aspect of different expertise, no statistical difference was found for a single institute [31]. However, manual contouring is a task with high variability both within and across institutions requiring end-to-end tests [32]. In clinical practice, DLCS is usually established by the dataset from one relatively large institute restricted by the data quality and patient resources but is used by various authorized oncologists from other institutes. In such circumstances, the DLCS-based contouring performance of oncologists with varied working experiences from multi-institutes is unknown. Therefore, it is critical to evaluate the oncologist’s acceptability towards DLCS, their performance during the contouring edition, and their improvement in clinical practice. In this study, we focus on oncologist behaviors from three institutions marked by A, B, and C, with diverse working experience and institute group contouring styles. We constructed a DLCS for NPC OAR contouring based on CT from one institute. We investigated oncologists’ consistency, performance improvement, and acceptability based on two factors: working experiences and institute group styles.

The study was approved by the ethics committee as a retrospective clinical trial with patient informed consent exempt. All the patients with the pathology diagnosis of nasopharyngeal carcinoma treated by bilateral radiotherapy from September 2018 to December 2019 were collected in the institute of group A, and 188 patients were finally included. The flow diagram of this study is shown in Fig. 1. All the patients were immobilized in the supine position by specific head-and-neck thermoplastic masks with planning CT scanning and reconstruction done by 3 mm thickness and 0.94 × 0.94 cm2 in-plane resolution on a spiral CT (SOMATOM Definition AS+, Siemens). MRI simulation was also done for all the patients at the same immobilization position with the masks (Discovery 750 W, GE).

In our clinical routine, manual contouring was done on planning CT combined with the registered MRI images by a qualified oncologist with 28 OARs. The contoured datasets are then viewed, adjusted, and approved by the corresponding responsible advanced oncologist for all patients. To avoid the oncologist personality influence, all contouring results were adjusted and approved by consensus of two advanced experts according to the anatomical boundaries in registered CT and MRI images by the international guideline [33]. The approved contours were considered as the ground truth and abbreviated as OARGT, later used for network training and the golden reference for result evaluation. The patients’ characteristics are shown in Table 1.

We construct the DLCS based on an encoder-and-decoder U-Net architecture [34], [35], [36], DeepLabv3+ [37], for NPC contouring. The architecture of DeepLabv3+ is illustrated in supplemental file Fig. 1. In application, the ResNet50 network serves as the backbone for the encoder component, extracting the abstract features of multiple layers by downsampling. The decoder component mainly consists of four atrous spatial pyramid pooling modules with different rates, combined with global average pooling and other up-sampling operations. Moreover, low-level features from the encoder are re-fed to the decoder for aggregating more abstract features. For the network training, the cross-entropy criterion was employed as the loss function, and the maximum number of the epoch was set to a constant of 100. Stochastic gradient descent was used as an optimizer, and its parameters were selected according to suggested values, with initial learning rate, momentum, and weight decay values of 0.01, 0.9, and 5*10−4, respectively. The DLCS was set on a server equipped with a Linux operation system, RAM hardware of 64 G, CPU Intel Xeon CPU E5-2620 v3 @ 2.40 GHz with 24 kernels, and NVIDIA TITAN RTX GPUs with 24G memory. 188 patients were randomly assigned to training and validation set (150 cases), and test set (38 cases). The model code can be accessed at https://github.com/hujunjiescu/DeepRT.

Six oncologists majoring in NPC from three institutions were recruited for the study. There were three experts (over 10 years of experience labeled by OA10, OB10, and OC10, respectively) and three beginners (1 years of experience marked by OA1, OB1, and OC1). Two trials, independent contouring labeled as OARmanual, and post-DLCS edition labeled as OARp-DLCS, of 28 OARs were performed for ten randomly-selected test cases on the Pinnacle3 treatment planning system (Philips Healthcare, Hamburg, Germany, version 9.1). Six oncologists were trained according to the international guideline [33], and the contouring consensus were built in advance. All the evaluations were done randomly and double-blindly, and two trials were carried out over two sessions by two months apart. Contouring performance was evaluated by two quantitative metrics, including volumetric Dice [30], [38], [39] for volume overlapping evaluation and surface Dice [30], [39], [40] for border overlapping evaluation with a distance tolerance of 1 mm. To evaluate the subjective acceptability of OARDLCS, an objective volume-based and a surface-based oncologist satisfaction rate (VOSR and SOSR) were defined by the OARDLCS and the corresponding OARp-DLCS to objectively measure the volume and surface deviation of DLCS contouring to oncologists’ desired targets. The definition details are shown in Fig. 2. When no modification was needed for one oncologist, the corresponding satisfaction rate reached the maximum of 1. This way, subjective acceptability was evaluated based on the objective metric derived from the whole edition process. Based on OARGT as a reference, volumetric and surface Dice were calculated to evaluate oncologist performance and group consistency. Based on OARp-DLCS as a reference, VOSR and SOSR were calculated. To explore the oncologist acceptability impact on their post-DLCS edition performance, the correlation relationship between the satisfaction rate and post-DLCS edition performance was investigated. The Pearson linear regression method was adopted for the Dice metric, and the Spearman rank method was employed for ranked group data. Analysis of variance (ANOVA) was used for pair-wise and inter-group comparison (IBM SPSS, version 25.0, New York, NY, USA, P < 0.05). All P-values were from two-sided tests.

The performance of each oncologist and experience-based intra-institute consistency is illustrated in Fig. 3. Post-DLCS edition improved intra-institute consistency significantly for most OARs except the left optical nerve, the pituitary gland, temporal lobes, and the left mandible. The intra-institute inconsistency was eliminated for group C but still existed for groups A and B. To objectively investigate the DLCS-based improvement of individual contouring performance, the post-DLCS edition improvement, defined as the post-DLCS edition metric minus the metric of manual contouring, was listed in Table 2. For those OARs with significant performance differences between manual contouring and post-DLCS edition, all OARs obtained performance improvement without any metric deterioration for all oncologists. The DLCS-based edition performance for eyes, spinal cord, constrictor naris, and larynx was improved for all oncologists. However, no significant improvement was found for the left lens, optical nerves, and the left cochlea. For the left OARs, the performance improvement varied for different institute groups. There were more numbers of OARs with improved performance for beginners compared with experts, 13 v.s. 8 (volumetric Dice metric) and 10 v.s. 5 (surface Dice metric) for group A, and 21 v.s. 14 (volumetric Dice metric) and 15 v.s. 14 (surface Dice metric) for group B. However, this trend reversed for group C with more numbers of OARs improved for experts. Thus, we explored the two-way post-DLCS edition performance improvement analysis illustrated in Fig. 4.

For those OARs with performance improvement significance in Fig. 4, all OARs gained performance improvement for all institute and experienced groups. However, among OARs with experience factor significance or interaction impact, beginners obtained more performance improvement with 7 OARs in volumetric Dice and 5 OARs for surface Dice than experts with only 4 OARs for both volumetric Dice and surface Dice), indicating that beginners benefit more from the post-DLCS edition. Among OARs with institute factor significance, group C benefits most in post-DLCS performance improvement with the highest number of improved OARs (8 for the volumetric Dice and 10 for the surface Dice), followed by group B (4 for both volumetric Dice and surface Dice). Typical two-dimensional slices and three-dimensional visualization for contouring and edition are shown in Fig. 5, demonstrated on MIM Maestro (Version 7.1, MIM Software Inc., Cleveland, OH). For ultimately the same contours, MIM only showed one color to represent the contour.

Two-way variance analysis of performance metrics for working experience and institute contouring styles is shown in Fig. 6. For manual contouring, there were fourteen OARs suffering from inter-institute inconsistency, seven OARs for experience inconsistency, and ten OARs influenced by their interaction effect, qualified by both volumetric Dice and surface Dice. But the number was reduced to one (the right temporal lobe) for the institute contouring style factor and zeros for the working experience factor by post-DLCS edition, indicating that inter-institute and inter-experience consistency could be improved by DLCS. The consistency improvement also can be seen in Fig. 5 (f) and (l). The three-dimensional visualization showed clear consistency improvement with less rendered colors. More analysis details can be found in the supplemental file table 1–28. For all those OARs except larynx with experience factor significance, the expert manual contouring and post-DLCS edition performed better than beginners both by volumetric Dice and surface Dice, indicating that working experience had a significant effect on clinical performance. For 4 OARs with institute contouring style significance in the post-DLCS edition, oncologists from group C performed slightly better than other institute groups.

Volume-based and surface-based oncologist satisfaction rates for different groups were listed in Table 29 in the supplemental file. Most OARs got a satisfaction rate above 0.9, and some OARs got a maximum satisfaction rate of 1 with a deviation of 0, indicating that there was no manual edition for the OARs. The acceptability varied among institute groups. Pituitary gained the lowest VOSR, 0.82 from oncologists of institute A but got 0.97 and 0.96 from oncologists of institutes B and C, respectively. The same was found for optical nerves, with the lowest VOSR from institute B but 0.9 above the satisfaction rate from other oncologist groups. For experienced groups, the beginners had better acceptance of DLCS with a significantly higher satisfaction rate than the experts. A similar situation happened for SOSR. The correlation analysis between oncologist satisfaction rate and oncologist performance is listed in Table 3. All the prominence detection was under 0.05. There was a remarkable positive linear relationship between VOSR and post-DLCS edition volumetric Dice with coefficient of 0.78, and between VOSR and volumetric Dice improvement with coefficient of 0.46. This proves that oncologists’ higher DLCS acceptability produced better contouring performance in clinical practice. Supplemental file Fig. 2 shows the scatter plot of the data for each group. The linear fitting line gradients of the beginners were higher than that of experts except for VOSR versus volumetric Dice improvement for group B. This also indicated that the beginners are more affected by DLCS than the experts. This was a consistent founding derived from Table 29 in the supplemental file.

While the post-DLCS edition helps shorten contouring time, we focused institute contouring style and working experience impact difference. Thus, a two-way variance analysis for saving time between institute groups and working experienced groups was observed, and the result is listed in Table 4. More details of actual manual contouring time and post-DLCS edition time were listed in Table 30 and Table 31 in the supplemental file. Institute differences and the interaction impacted the saving time. However, working experience had no significant impact. It revealed that the DLCS contributes to time reduction with no difference in working experience. However, time reduction varied among institute groups, showing that institute contouring styles greatly influenced clinical applications and affected the edition process.

In this study, we comprehensively explored the clinical impact of a deep-learning contouring system established in one institute on oncologists from three institutions with various working experience and contouring styles in nasopharyngeal carcinoma organs at risk contouring practice. Individual performance, group performance, experience-based intra-group consistency, inter-institute, and inter-experience consistency were improved, but the development varied for different institute groups. The DLCS established in one institute was also effective for oncologists from other institutes, with slightly better post-DLCS performance and greater improvement. All the analyses supported that beginners had better acceptance of DLCS and gained more performance improvement than experts. We gave solid proof of DLCS impact on oncologists with different experiences and contouring styles from various institutes. However, the oncologist number was limited due to the intensive manual labor for the experiments, which might cause bias due to the oncologist’s personal working style. In the future, more oncologists may be invited to enhance the experiment reliability. Interestingly the DLCS failed to develop the consistency and edition performance for all OARs. Experience-based intra-group consistency improvement failed for the left optical nerve, the pituitary gland, temporal lobes, and the left mandible. The individual performance improvement failed for the left lens, optical nerves, and the left cochlea, revealing that DLCS may be not practical for every oncologist’s contouring target. The possible cause may be that the DLCS was trained by CT slices, and OARDLCS was tested on CT slices. However, soft tissues, like the OARs mentioned above, had no clear boundaries on CT slices, which may cause low prediction accuracy and then lead to considerable modifications and low observer consistency. MRI can provide clear boundaries for soft tissues to solve the problem, and we are working on contouring prediction systems based on registered CT and MRI. We believe it will be a better automatic contouring framework.

Individual performance, group performance, experience-based intra-group consistency, inter-institute and inter-experience consistency were improved by deep-learning organs at risk contouring system for nasopharyngeal carcinoma radiotherapy. Still, the improvement varied for different institute group. The DLCS established in one institute was also effective for oncologists from other institutes with slightly better post-DLCS performance and higher improvement. The beginners had better acceptance of DLCS and gained more performance improvement than experts, which could positively impact toxicities avoidance and complication reductions in clinical practice.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.