Chanyoung Ko,Jae-Hyun Lim,JaeSeong Hong,Soon-Beom Hong,Yu Rang Park
Development and Validation of a Joint Attention–Based Deep Learning System for Detection and Symptom Severity Assessment of Autism Spectrum Disorder Deep Learning for Detection and Symptom Severity Assessment of ASD
Key Points Question Can joint attention be quantified for detecting autism spectrum disorder (ASD) and assessing ASD symptom severities? Findings In this diagnostic study of 45 children with ASD and 50 with typical development, a deep learning system trained on videos acquired using a joint attention–eliciting protocol for classifying ASD vs typical development and predicting ASD symptom severity showed high predictive performance. This new artificial intelligence–assisted approach based predictions on participants’ behavioral responses triggered by social cues. Meaning These findings suggest that this method may allow scalable, digitalized measurement of joint attention, enabling deep learning–based analysis and modeling to facilitate development of automated detection and symptom severity assessment tools for individuals with social deficiencies.
Development and Validation of a Joint Attention–Based Deep Learning System for Detection and Symptom Severity Assessment of Autism Spectrum Disorder Deep Learning for Detection and Symptom Severity Assessment of ASD

Can joint attention be quantified for detecting autism spectrum disorder (ASD) and assessing ASD symptom severities?

In this diagnostic study of 45 children with ASD and 50 with typical development, a deep learning system trained on videos acquired using a joint attention–eliciting protocol for classifying ASD vs typical development and predicting ASD symptom severity showed high predictive performance. This new artificial intelligence–assisted approach based predictions on participants’ behavioral responses triggered by social cues.

These findings suggest that this method may allow scalable, digitalized measurement of joint attention, enabling deep learning–based analysis and modeling to facilitate development of automated detection and symptom severity assessment tools for individuals with social deficiencies.

Attending to other people and sharing an attentional focus on objects or events with other individuals facilitates learning to socialize. This is termed joint attention, and despite interindividual differences in joint attention, this ability is observed as early as 6 months in a typically developing infant. Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by abnormal patterns of social interaction and communication. Infants with this condition, but not those with typical development (TD), appear to lack joint attention. Thus, this difference has interested researchers as a means of diagnosis, a prognostic indicator, and a potential intervention target for individuals with ASD. Although validated manuals for observing joint attention exist, for example, the Early Social Communication Scales (ESCS), they are labor-intensive and difficult to implement without trained clinicians and the proper experimental setup. Recently, machine learning and deep learning (DL) artificial intelligence (AI) models using simple behavioral video data to detect ASD have gained momentum, not only to characterize children with autism objectively but also to develop scalable screening or assistive diagnostic tools for ASD. Previous efforts to build ASD-detection AI models showed promising results. However, due to a lack of automation, the requirements for specialized and expensive equipment, calibration, trained personnel, dependence on human rating of autistic behaviors, and low precision and recall, they are unsuitable for the development of ASD detection tools. The low precision and recall scores of these models may be due to a lack of targeted behavioral biomarkers that can readily discriminate ASD from TD. Another valid cause may be the absence of methods for quantitatively measuring the complex hallmark behaviors of ASD. Thus, there is a need for objective measurement of clinically validated autism-related behaviors, such as joint attention, which may be implemented in a screening tool as well as in an objective diagnostic tool for clinicians who have had to rely on subjective assessment scores in diagnosing ASD to date. We developed a digitalized method for joint attention assessment that required a new protocol for specific task administration guidelines to elicit 3 types of joint attention mentioned in the ESCS for video recording of task-related behaviors. The collected video data were then used as input for training a DL model to identify ASD and assess ASD symptom severity. Preliminary results have been obtained previously with this approach in a small sample. In the present study, we assessed whether the joint attention–based DL model could distinguish children with ASD from those with TD and differentiate ASD symptom severity levels based on joint attention behaviors ascertained from input video data using explainable AI techniques.

This prospective diagnostic study involved children aged 24 to 72 months from multiple sites in South Korea. Children with ASD were recruited from a single institution—Seoul National University Hospital (SNUH) Child Psychiatry Outpatient Clinic—where individuals at high risk of ASD were referred from throughout the country, while individuals with TD and no history of developmental delay or psychiatric condition per parent report were recruited from various day care centers across South Korea. Participants were from Seoul (25%), metropolitan cities (25%), or self-governing provinces (50%). All caregiver-reported ethnicity, in accordance with the National Institutes of Health categories, was Korean. Caregivers provided written informed consent. The study was approved by Yonsei University Health System Institutional Review Board and followed the Standards for Reporting of Diagnostic Accuracy Studies (STARD) reporting guideline.

Conservative inclusion criteria for ASD (clinical diagnosis by a child psychiatrist and scores above the cutoff on the criterion standard diagnostic tool) and TD (scores below the cutoff on a screening tool) were used to select individuals belonging to clinically distinct groups to ensure development of an accurate and precise ASD vs TD classification model. Detailed enrollment process is described in eFigure 1 in Supplement 1, and sociodemographic and clinical measures are presented in the Table.

All caregivers completed the Korean Childhood Autism Rating Scale II (K-CARS-2) as part of screening. The K-CARS-2 consists of 15 questions about the presence or absence of autism symptoms, with a total score of 15 to 60. The K-CARS-2 has been also used to assess ASD symptom severity, where scores less than 30 represent non-ASD; 30 to 36, mild or moderate ASD; and 37 to 60, severe ASD.

Children whose K-CARS-2 scores revealed an ASD risk or whose caregiver expressed concern were referred to a child psychiatrist for a diagnostic evaluation at SNUH. The Korean Autism Diagnostic Observation Schedule II (K-ADOS-2) was administered to children who received a clinical diagnosis of ASD as part of routine evaluation.

Cognitive functioning was assessed using various measurements, depending on the child’s age and ability to attend to demanding cognitive tasks. We defined the best estimate IQ using the Korean Bayley Scales of Infant and Toddler Development, Second Edition, and Korean Wechsler Preschool and Primary Scale of Intelligence, Fourth Edition.

We designed a protocol for measuring and video recording 3 types of joint attention, adopting methods from the ESCS manual of Mundy et al and validated behavior extraction techniques. Children were individually tested in a quiet room. Each child was seated on a height-adjustable chair in front of a table. There are 2 types of joint attention: initiation of joint attention (IJA) associated with the child’s motivation for social interaction, and response to joint attention (RJA), associated with the child’s responsiveness to a social cue. Response to joint attention may be classified as low and high level, referring to the child’s ability to maintain attention on objects pointed near them (low-level RJA) and far from them (high-level RJA). Video data were collected from 3 joint attention tasks designed to elicit IJA, low-level RJA, and high-level RJA behaviors. The procedures and video acquisition setup are described in eFigures 2 and 3 in Supplement 1. Video data were acquired in a single 10-minute session per participant. Tasks were filmed from a front-facing viewpoint using a digital camera (DSC-RX100 IV; Sony) with resolution of 1920 × 1080 and 30 frames/second. As a way of monitoring how engaged a participant was for each repeated trial of a given task, we devised a compliance score metric to compute compliance scores that could indirectly show the participants’ task performance and engagement level (eTable 1 in Supplement 1).

We customized a DL classification system consisting of neural network architectures, that is, a convolutional neural network (CNN), long short-term memory, and attention mechanism as illustrated in eFigure 4 in Supplement 1. Size of the input data for the IJA-based DL system is 224 × 224 × 300 (30 frames/s × 10 seconds) and 224 × 224 × 150 (30 frames/s × 5 seconds) for the RJA-based DL system. We used 10-fold group-wise (by individual) cross-validation for development of ASD detection and ASD symptom severity assessment systems. Performances during training and validation are presented in eTable 2 in Supplement 1.

The gradient-weighted class activation mapping (Grad-CAM) technique was used to produce visual explanations of how the system makes its prediction by superimposing a visualization layer at the end of the CNN model. This method uses the gradients of any target concept, which are accumulated in the last CNN layer, to generate a localization heatmap highlighting key areas in the image for predicting the concept. Redder areas suggest more significant features for model prediction. Long short-term memory was developed to better process sequential data; hence, it was incorporated into our DL classification system to account for the time-dependent nature of video data. The addition of the attention mechanism enabled us to access and visualize the attention weights across each video sequence according to which frames or video time points contributed most to the model’s decision-making by plotting the attention weights across each video sequence. To visualize how the DL system distinguishes ASD from TD or distinguishes the different severities of ASD on a data set level and thus verify the consistency of our DL system’s decision-making, we drew cluster maps on the testing data set for each joint attention task. The technique used was agglomerative hierarchical clustering, which is characterized by clustering through iteration, where similar clusters merge with other clusters until k clusters are formed. This can be visualized via a dendrogram.

We used means (SDs) and medians (ranges) to express continuous variables. The χ2 test was used to compare categorical variables. A 2-way mixed analysis of variance was used to explore 2-way interactions between the ASD vs TD effect and the number of repeated trials on the compliance score. The area under the receiver operating characteristic (AUROC), accuracy, recall, and precision were computed to evaluate the performance of the classification models. Statistical analyses and calculations of the validation measures were performed using Python, version 3.6.8, with SciPy, version 1.4.1, and Statsmodels 0.11.1 (Python Software). Cluster maps were drawn using Scikit-learn, version 0.23.2, Seaborn, version 0.11.0, and Matplotlib, version 3.3.1 (Python Software). Deep learning provided a classification score ranging from 0 to 1, and the lowest predicted probability value of the DL model’s output is greater than 0.5 for classifying ASD vs TD and the different ASD symptom severities. The threshold for statistical significance was set at 2-sided P < .05. We estimated CIs with the Hanley and McNeil method at 95% level.

Of the 110 children with screening data, a total of 95 (86.4%) were included for joint attention–based AI model training, 45 (47.4%) with ASD (mean [SD] age, 48.0 [13.4] months; 24 boys [53.3%] and 21 girls [46.7%]) and 50 (52.6%) with TD (mean [SD] age, 47.9 [12.5] months; 27 boys [54.0%] and 23 girls [46.0%]). Detailed descriptions of the 2 groups are shown in the Table.

Using our compliance scoring metric, we discovered that while the overall quality of joint attention behaviors differed between TD and ASD groups independently of trial or task type, each group showed similar within-group compliance (attentiveness) patterns for each joint attention task. While most children with ASD showed an incremental reduction in attentiveness after showing initial interest in the social cue as much as the children with TD for the IJA task, children with ASD showed a contrasting uninterest in the social cue presented at the beginning of RJA tasks. Detailed results of 2-way mixed analysis of variance and a visual representation of the task compliance results are shown in eFigure 5 in Supplement 1.

While all DL models trained on any joint attention task showed promising classification performance in identifying ASD, the IJA-based ASD symptom severity prediction DL model showed superior performance compared with that of the models based on RJA tasks across all validation measures. The validation measures of the DL models are presented in Figure 1.

The IJA-based model showed an AUROC of 99.6% (95% CI, 99.4%-99.7%), accuracy of 97.6% (95% CI, 97.1%-98.1%), precision of 95.5% (95% CI, 94.4%-96.5%), and recall of 99.2% (95% CI, 98.7%-99.6%). The low-level RJA–based model showed an AUROC of 99.8% (95% CI, 99.6%-99.9%), accuracy of 98.8% (95% CI, 98.4%-99.2%), precision of 98.9% (95% CI, 98.3%-99.4%), and recall of 99.1% (95% CI, 98.6%-99.5%). The high-level RJA–based model showed an AUROC of 99.5% (95% CI, 99.2%-99.8%), accuracy of 98.4% (95% CI, 97.9%-98.9%), precision of 98.8% (95% CI, 98.2%-99.4%), and recall of 98.6% (95% CI, 97.9%-99.2%).

The IJA-based model showed an AUROC of 90.3% (95% CI, 88.8%-91.8%), accuracy of 84.8% (95% CI, 82.3%-87.2%), precision of 76.2% (95% CI, 72.9%-79.6%), and recall of 84.8% (95% CI, 82.3%-87.2%). The low-level RJA–based model showed an AUROC of 84.4% (95% CI, 82.0%-86.7%), accuracy of 78.4% (95% CI, 75.0%-81.7%), precision of 74.7% (95% CI, 70.4%-78.8%), and recall of 78.4% (95% CI, 75.0%-81.7%). The high-level RJA–based model showed an AUROC of 84.2% (95% CI, 81.8%-86.6%), accuracy of 81.0% (95% CI, 77.3%-84.4%), precision of 68.6% (95% CI, 63.8%-73.6%), and recall of 81.0% (95% CI, 77.3%-84.4%). To explore whether age affects model performance, we performed additional analysis by age (<48 vs ≥48 months). Even after controlling for the effect of age, the IJA-based model performance was superior to that of the other task-based models. There was an improvement in high-level RJA–based model performance when trained on data sets of older children. The results of model performance by age group are presented in eFigure 6 in Supplement 1.

The Grad-CAM results for TD and ASD are shown in eFigure 7 in Supplement 1. Attention plots are shown in eFigure 8 in Supplement 1. Peaks at certain time steps, representing an increase in attention weights, as shown in the attention plots (y-axis attention weights vs x-axis time steps), were presumed to be video frames capturing important features for decision-making by the DL model. For the time steps at which attention weights peaked, we visualized the gradient weights using Grad-CAM, which revealed differing patterns of motion and behavior between the TD and ASD groups. In the IJA task, which was designed to trigger the participant to initiate social interaction, individuals with TD showed a triadic gaze pattern—gaze shifting from the toy object to the examiner, then back to the toy object—while the individual with ASD failed to gaze on either the toy object or the examiner (eFigure 7A in Supplement 1). In the low-level RJA task, a heatmap around the face and eyes showed that the gaze of the individual with TD on the presented toy object was maintained for long durations, while the gaze of the individual with ASD remained on the toy object only briefly, and then wandered off elsewhere (eFigure 7B in Supplement 1). In the high-level RJA task, individuals with TD immediately turned around to view the poster and then turned back to face the examiner as if seeking approval, while those with ASD showed delayed or no response (eFigure 7C in Supplement 1).

A cluster map hierarchically clusters to order data—in this case, attention weights through the video sequence—by similarity, thereby reorganizing the data for the rows and columns and displaying similar attention weight rise and fall patterns next to one another. Different task types showed different patterns of attention weight peaks between different classes at the data set level. Individuals of the same cluster showed similar peak rise and fall patterns across the time steps, as shown in Figure 2 and Figure 3 and eFigures 9 and 10 in Supplement 1. While all 3 joint attention–based cluster maps sorted ASD vs TD with nearly equal effectiveness (Figure 2 and eFigure 9 in Supplement 1), the IJA-based cluster map (Figure 3) sorted non-ASD vs mild to moderate ASD vs severe ASD more effectively than the RJA-based cluster maps (eFigure 10 in Supplement 1).

To the best of our knowledge, no previous study has demonstrated that DL models for the detection and symptom severity assessment of ASD could be developed using a complex behavioral biomarker such as joint attention. Previous attempts to distinguish ASD from TD using simple gestures or eye-gaze patterns were limited by high false-positive rates due to the lack of targeted ASD-related behavioral biomarkers with high discriminatory power or the lack of a replicable method for objectively measuring target behaviors. In this diagnostic study, our methodical acquisition of input video data for the DL system yielded predictions with high accuracy and precision. By adopting operational definitions of joint attention from well-validated methods, we designed joint attention tasks for easy administration and replicability, eliciting distinctive patterns of social interaction that differed according to diagnosis status and ASD symptom severity. In addition, when acquiring joint attention videos, we monitored the task compliance of every participant at each repeated measure of joint attention. Children with ASD showed qualitatively different behaviors in response to socially salient information. Both ASD and TD showed habituation after 10 repetitions of the IJA task and after 5 repetitions of the RJA tasks. Based on these results, we assume that a joint attention–driven DL model should be trained on videos of the first 5 to 10 repetitions of the same joint attention behavior. By implementing explainable AI tools, we were able to show that our DL systems made predictions based on salient behavioral differences, such as turning one’s head or shifting one’s eye gaze to view a shared interest better, in the same way trained specialists diagnose ASD, and for assessing the different symptom severities of ASD. Children with TD showed triadic gaze patterns during the IJA task, maintained gaze on a visual stimulus during the low-level RJA task, and immediately turned their heads to view a visual stimulus located far away during the high-level RJA task. Children with ASD seldom or never showed triadic gaze patterns during the IJA task, were not able to maintain gaze on visual stimuli during the low-level RJA task, and showed delayed or no response to the examiner’s directing attention to visual stimuli across the room during the high-level RJA task. Although DL systems trained on any given joint attention task showed high and comparable detection performances in distinguishing ASD vs TD, the IJA-based DL model performed significantly better than the RJA-based models in ASD symptom severity assessment. The RJA tasks, while effective for checking the likelihood of having ASD, may not be sensitive enough to detect subtle motivational nuances pertinent for differentiating one type of ASD from another. Performance of RJA tasks may be associated with development, as model training using the data set of older children showed improved performance. Based on our compliance subanalysis results, it is also plausible that due to early loss of interest during RJA data collection, the DL models could not extract enough feature patterns across individuals with different levels of ASD.

This study has some limitations, including the small sample size and a statistically significant difference in IQ between children with ASD and TD. IQ affects one’s ability to perform behavioral tasks; however, joint attention is a strong trait differentiating ASD from TD even after controlling for IQ. External validation using a larger sample of IQ-matched cohorts with ASD and TD is still warranted.

In this diagnostic study, we developed DL models for identifying ASD and differentiating levels of ASD symptom severity. We believe our research opens possibilities for gathering large data sets on behavioral biomarkers through standardized video data acquisition setup amenable to computer vision and DL and applicable to a wide range of neuropsychiatric conditions. Moreover, our findings suggest that our method may be a good alternative for assisting clinicians in making well-informed referrals to a child development specialist or in making ASD diagnosis.