--- author: - - - title: 'Machine Learning Workflow to Explain Black-box Models for Early Alzheimer's Disease Classification Evaluated for Multiple Datasets' --- Introduction {#Sec:Introduction} ============ Alzheimer's Disease (AD) is a neurodegenerative disease @10.1002/alz.12068 and the most frequent cause of dementia. As the number of dementia patients increases continuously, AD is a globally growing health problem @WAR2018. Currently, there is no causal therapy to cure AD @10.1002/alz.12068. To recruit and monitor subjects for therapy studies, it is important to identify patients at risk to develop AD early and to develop preclinical markers. Subjects with cognitive impairments that do not interfere with everyday activities are considered as having Mild Cognitive Impairment (MCI) due to AD @10.1016/j.jalz.2011.03.003. The risk to develop AD is increased for subjects with MCI in comparison to cognitively normal controls (CN). However, not all subjects with MCI prospectively convert to AD. One possibility for early AD detection is to find patterns distinguishing between progressive MCI subjects (pMCI) who will develop AD and subjects with stable MCI (sMCI). Multiple Machine Learning (ML) workflows were implemented for this differentiation. Some used models like Decision Trees (DTs) or Logistic Regression (LR), which were interpretable by design. However, black-box models like eXtreme Gradient Boosting (XGBoost) @10.1145/2939672.2939785, Random Forests (RFs) @10.1023/A:1010933404324, or Convolutional Neural Networks (CNNs) @10.1038/nature14539 often outperform those models. Black-box models are designed to identify highly complex associations and are challenging to interpret. Thus, the risk of learning spurious decision functions caused by patterns occurring in the training dataset is increased for black-box models @10.1038/s41467-019-08987-4. This research is an extended version of earlier work @Bloch2021 and thus expands the previously developed ML workflow. The previously developed workflow enabled the interpretation of black-box models based on model-agnostic Shapley values. Shapley values give individual explanations for the prediction of each subject and visualize complex relationships between features and model predictions. In this research, the previous experiments are expanded by using three AD datasets and three adjusted feature sets. In addition to the previously trained tree-based models, Support Vector Machines (SVMs) @SVMs and LR models were implemented and explained. In this work, Shapley-based explanations were compared to classical feature importance methods, absolute log odd's ratios, and permutation importance. In comparison to previous work @Bloch2021, an improvement of the classification results for ADNI and AIBL was achieved for the differentiation between Cognitive Normal (CN) controls and MCI subjects as well as for MCI vs. AD classification and models trained without cognitive test scores and validated for AIBL. Additionally, the ADNI and AIBL results achieved for sMCI vs. pMCI classification, trained with cognitive test scores, outperformed previous work. This article is structured as follows: In Section \[Sec:RelatedWork\] related work is described. Section \[Sec:MaterialsAndMethods\] introduces the datasets and methods used to implement the ML workflow and the details of the experiments. Section \[Sec:Results\] elaborates on the experimental results. Those results are discussed including the mentioning of limitations in Section \[Sec:Discussion\]. Finally, Section \[Sec:Conclusion\] concludes the overall work. Related Work {#Sec:RelatedWork} ============ Interpretable ML was developed to explain black-box models @Molnar2019. As the heterogeneous etiology of AD is not completely understood yet, interpretability is important and enables the validation of the biological plausibility of ML models. Recently, some studies have used interpretable ML in AD detection. For example, Long Short-Term Memory- (LSTM-) @Hochreiter1997 based Recurrent Neural Networks (RNN) @10.1038/323533a0 were trained to classify CN vs. MCI subjects in @10.1371/journal.pone.0236868. The experiments included multiple techniques to fuse sociodemographic and genetic data with Magnetic Resonance Imaging (MRI) scans. The resulting models were evaluated for two AD datasets – the AD subset @10.1159/000320988 of the Heinz Nixdorf Risk Factors Evaluation of Coronary Calcification and Lifestyle (RECALL) (HNR) @10.1067/mhj.2002.123579 (61 MCI and 59 CN) and 624 subjects (397 MCI, 227 CN) of the Alzheimer's Disease Neuroimaging Initiative (ADNI) @10.1212/WNL.0b013e3181cb3e25 study phase 1. To visually explain individual model decisions, Gradient-weighted Class Activation Mapping (Grad-CAM) @10.1109/ICCV.2017.74 was used. A focus on biologically plausible regions was observed. Four heatmap visualization methods – sensitivity analysis @Simonyan2014, guided backpropagation @Springenberg2014, occlusion @10.1007/978-3-319-10590-1_53, and brain area occlusion inspired by @Yang2018 – were compared for 3D-CNNs in @10.1007/978-3-030-02628-8_3. The CNN models were trained using 969 MRI scans of 344 ADNI subjects (151 CN, 193 AD). However, it was unclear whether the described workflow ensured independent training and test sets by using multiple scans per subject @10.1016/j.media.2020.101694. Thus, the Cross-Validation (CV) accuracy of $77~\%\pm6~\%$ might be affected by data leakage. All heatmaps focused on AD-related anatomical brain areas. An interpretable deep learning model, consisting of a Generative Adversarial Network @Goodfellow2014 to extend the training dataset, a regression network to generate feature vectors from adjacent visits, and a classification model was introduced in @10.1101/230.733-60-2639784. Firstly, the regression model iteratively estimated the feature vector at the following visit. The resulting feature vector was used as input for the classification model, which predicted the final diagnosis. To classify 101 pMCI vs. 115 sMCI ADNI subjects, longitudinal volumetric MRI features were used. The model outperformed SVMs and artificial neural networks. A new interpretable model, based on distinct weighted rules was introduced in @10.7717/peerj.6543 and evaluated for 151 subjects (97 AD and 54 CN) of the ADNI cohort. The framework is called Sparse High-order Interaction Model with Rejection option (SHIMR) and consists of two hierarchical stages. In the first stage, the interpretable model was trained using plasma features. The data of subjects with an unclear prediction in this stage were propagated to the second stage. In this stage, an SVM @SVMs was trained using invasive Cerebrospinal Fluid (CSF) markers. The evaluation included both, CV and an independent test set. The described model reached an Area Under the Receiver Operating characteristics Curve (AUROC) of 0.81 for the test set. SHapley Additive exPlanations (SHAP) @Lundberg2017 were used in @10.1186/s13195-021-00879-4 to explain differences in models trained using coreset selection methods. The idea was, to determine coresets of subjects with the most informative data. RF and XGBoost models were trained on these coresets to avoid overfitting and improve ML models. The results of Data Shapley @Ghorbani2019data coreset selection were compared to Leave-One-Out @10.2307/1268249 selection and random exclusion. All models were trained and validated for the ADNI dataset (400 sMCI, 319 pMCI) and externally validated for a subset of the AIBL dataset (16 sMCI, 12 pMCI). SHAP summary plots showed that models trained for both the entire training set and the coreset learned biologically plausible associations. To examine the predictive influence of -amyloid plaques, tau tangles, and neurodegeneration during the disease progression, RF feature importance was used in @10.1038/s42003-020-1079-x. The experimental data included 405 ADNI subjects (148 CN, 147 MCI, 110 AD). -amyloid Positron Emission Tomography (PET) detected -amyloid plaques, invasive CSF features surrogated tau tangles, and MRI and Fluorodeoxyglucose (FDG) PET scans were used to determine neurodegeneration. The experimental results showed that models trained to classify the early AD stages preferred features representing tau tangles and -amyloid plaques. Models trained to predict later stages favored surrogates for neurodegeneration. SHAP @Lundberg2017 and Gradient Tree Boosting (GTB) @10.1214/aos/1013203451 reproduced those observations. The RF and the entire feature set reached accuracies of 73.17 % (CN vs. MCI), 71.01 % (MCI vs. AD), and 90.34 % (CN vs. AD). SHAP values were also used in @10.3389/fdata.2021.613047 to explain population-based and individual predictions of XGBoost models and RFs. Models were trained using sociodemographic and lifestyle factors to predict the patient's risk to develop AD based on medical history. Transfer learning applied information extracted from the Survey of Health, Ageing, and Retirement in Europe (SHARE) @10.1093/ije/dyt088 (80,699 CN, 4,157 AD) to the PREVENT cohort @10.1136/bmjopen-8702028617 (109 subjects with high risk to develop AD, 364 subjects with low risk). The PREVENT cohort was younger than the SHARE cohort. The models support the hypothesis that age is the most important risk factor in AD detection. Consistent with previous research @10.1016/s0140-6736(20)30367-6, amongst other factors, less education, physical inactivity, diabetes, and infrequent social contact were identified as potential risk factors. A two stage-based classification workflow that used SHAP values to interpret RFs was developed in @10.1038/s41598-021-82098-3. In the first stage, CN vs. MCI vs. AD classification was performed. The second stage implemented the differentiation of sMCI and pMCI subjects. The models were based on multiple modalities including MRI, PET, CSF biomarkers, cognitive tests, medical history, genetics, and many more. The RFs were trained and tested using 1,048 subjects (294 CN, 254 sMCI, 232 pMCI, and 268 AD) of the ADNI dataset. For CN vs. MCI vs. AD classification, the model almost exclusively selected cognitive test scores as the most important features. The model learned bad cognitive test results increased the risk of AD and MCI. The most important features for sMCI vs. pMCI classification also were cognitive test scores followed by PET and MRI features. Bad cognitive test scores, small MRI volumes, and small PET uptakes were associated with disease progression. <span>p<span>0.5cm</span>p<span>2.5cm</span>p<span>4cm</span>p<span>4cm</span>p<span>3cm</span>p<span>3cm</span></span> Ref.&Task&Subjects&Modality&ML method&Explanability method\ @10.1371/journal.pone.0236868&CN vs. MCI&HNR: 61 MCI, 59 CN; ADNI-1: 397 MCI, 227 CN&MRI, socio-demography, ApoE&LSTM based RNN&GradCAM\ @10.1007/978-3-030-02628-8_3&CN vs. AD&ADNI: 151 CN, 193 AD&MRI&CNN&sensitivity analysis, guided backpropagation, occlusion, brain area occlusion\ @10.1101/230.733-60-2639784&sMCI vs. pMCI&ADNI: 101 pMCI, 115 sMCI&MRI volumes&Neural network&intrinsic\ @10.7717/peerj.6543&CN vs. AD&ADNI: 54 CN, 97 AD&CSF, Plasma&SHIMR& intrinsic\ @10.1186/s13195-021-00879-4&sMCI vs. pMCI&ADNI: 400 sMCI, 319 pMCI; AIBL: 16 sMCI, 12 pMCI&MRI volumes, demography, ApoE&RF, XGBoost&SHAP\ @10.1038/s42003-020-1079-x&CN vs. MCI, CN vs. AD, MCI vs. AD&ADNI: 148 CN, 147 MCI, 110 AD&Amyloid-PET, MRI, FDG-PET, CSF&RF, GTB&RF-Feature importance, SHAP\ @10.3389/fdata.2021.613047&high vs. low risk&SHARE: 80,699 CN, 4,157 AD; PREVENT: 364 low risk, 109 high risk&sociodemography, lifestyle&RF, XGBoost&SHAP\ @10.1038/s41598-021-82098-3&CN vs. MCI vs. AD, sMCI vs. pMCI&ADNI: 294 CN, 254 sMCI, 232 pMCI, 268 AD&MRI, CSF, PET, cognitive tests, medical history, genetics&RF&ensemble of surrogat models, SHAP\ Materials and Methods {#Sec:MaterialsAndMethods} ===================== The ML workflow, implemented using the programming language Python v3.6.9 @Python3 is shown in Figure \[fig:MachineLearningWorkflow\]. It enables the interpretation of black-box models trained to detect early AD. In the following, the workflow and the methods used for implementation are elucidated. ![Implemented ML workflow. Volumetric features were extracted for one baseline (BL) MRI scan per subject. The ADNI dataset was randomly split into an 80 % training and 20 % test set. The most important MRI features were selected using forward feature selection, those were concatenated with sociodemographic features, number of ApolipoproteinE$\epsilon$4 (ApoE$\epsilon$4) alleles, and cognitive test scores. Bayesian optimization implemented hyperparameter tuning. Black-box RFs, XGBoost models, LR models, as well as polynomial and radial SVMs, were trained and validated. Shapley values were calculated for black-box model interpretation. An evaluation was performed for the independent ADNI test set and for the external AIBL and OASIS datasets.<span data-label="fig:MachineLearningWorkflow"></span>](Fig1.pdf){width="100.00000%"} Datasets -------- Data used in the preparation of this article were obtained from the ADNI @10.1212/WNL.0b013e3181cb3e25, the AIBL @10.1017/S1041610209009405, and the OASIS @10.1101/230.733-60-26394902 cohorts. ADNI[^1] was launched in 2003 as a public-private partnership. The primary goal of ADNI is to test whether a combination of biomarkers can measure the progression of MCI and AD. Those biomarkers include serial MRI, PET, biological markers, as well as clinical and neuropsychological assessments. The ongoing ADNI cohort recruited subjects from more than 60 sites in the United States and Canada and consists of four phases (ADNI-1, ADNI-2, ADNIGO, and ADNI-3). The subjects were assigned to three diagnostic groups. CNs have no problems with memory loss. Subjects with AD meet the criteria for probable AD defined by the National Institute of Neurological and Communicative Disorders and Stroke–Alzheimer's Disease and Related Disorders Association (NINCDS-ADRDA) @10.1212/WNL.34.7.939. The diagnostic criteria of ADNI were explained in @10.1212/WNL.0b013e3181cb3e25. The dataset was downloaded on 27 Jul 2020 and initially included 2,250 subjects. <span>p<span>1.65cm</span>|p<span>0.4cm</span>p<span>0.9cm</span>p<span>0.7cm</span>p<span>1cm</span>p<span>1.3cm</span>p<span>0.9cm</span>p<span>2cm</span></span> &n&Age&Gender&Education&MMSCORE&CDR&ApoE$\epsilon$4\ &&years&f in %&years&$\bar{x}\pm\sigma$&$\bar{x}\pm\sigma$&0/1/2 in %\ CN&512&74.2$\pm$5.8&51.8&16.3$\pm$2.7&29.1$\pm$1.1&0.0$\pm$0.0&71.3/26.2/ 2.3\ MCI&853&73.1$\pm$7.6&40.8&15.9$\pm$2.9&27.6$\pm$1.8&0.5$\pm$0.0&49.4/39.5/10.8\ sMCI&400&73.2$\pm$7.5&40.2&15.8$\pm$3.0&27.8$\pm$1.8&0.5$\pm$0.0&56.8/34.0/ 9.2\ pMCI&319&74.0$\pm$7.1&40.1&15.9$\pm$2.8&27.0$\pm$1.7&0.5$\pm$0.0&34.2/49.5/16.3\ AD&335&75.0$\pm$7.8&44.8&15.2$\pm$3.0&23.2$\pm$2.1&0.8$\pm$0.3&33.1/47.2/19.1\ $\Sigma_{CN, MCI,AD}$&1,700&73.8$\pm$7.2&44.9&15.9$\pm$2.9&27.2$\pm$2.7&0.4$\pm$0.3&52.8/37.0/ 9.9\ AIBL[^2] is the largest AD study in Australia and was launched in 2006. AIBL aims to discover biomarkers, cognitive test results, and lifestyle factors associated with AD. As AIBL focuses on early AD stages, most of the subjects are CN. The MCI subjects of AIBL met the criteria described in @10.1111/j.1365-2796.2004.01380.x, AD diagnoses following the NINCDS-ADRDA criteria @10.1212/WNL.34.7.939 for probable AD. The diagnostic criteria of AIBL were described in @10.1017/S1041610209009405. Approximately half of the CN subjects recruited in AIBL show memory complaints @10.1017/S1041610209009405. AIBL data version 3.3.0 was downloaded on 19 Sep 2019 and originally included 858 subjects. <span>l|rrrrrr</span> &n&Age&Gender&MMSCORE&CDR&ApoE$\epsilon$4\ &&years&f in %&$\bar{x}\pm\sigma$&$\bar{x}\pm\sigma$&0/1/2 in %\ CN&446&72.5$\pm$6.1&57.0&28.7$\pm$1.2&0.0$\pm$0.1&69.3/26.5/ 2.7\ MCI&95&75.4$\pm$7.0&47.4&27.1$\pm$2.2&0.5$\pm$0.1&47.4/36.8/12.6\ sMCI&16&77.8$\pm$6.9&37.5&28.0$\pm$1.7&0.4$\pm$0.2&56.2/37.5/ 6.2\ pMCI&12&75.3$\pm$5.8&33.3&26.2$\pm$1.6&0.5$\pm$0.0&16.7/50.0/33.3\ AD&71&73.1$\pm$6.6&59.2&20.5$\pm$5.7&0.9$\pm$0.6&29.6/49.3/18.3\ $\Sigma_{CN, MCI,AD}$&612&73.0$\pm$6.6&55.7&27.5$\pm$3.5&0.2$\pm$0.4&61.3/30.7/ 6.0\ The aim of the Open Access Series of Imaging Studies (OASIS) 3[^3] @10.1101/230.733-60-26394902 dataset is, to investigate the effects of healthy ageing and AD. The subjects of OASIS-3 were recruited from several ongoing studies in the Washington University Knight Alzheimer Disease Research Center [^4]. The longitudinal dataset included MRI scans, fMRI scans, Amyloid- and FDG-PET scans, neuropsychological test results, and clinical data for 1,098 subjects. OASIS focuses on the preclinical stage of AD. All OASIS subjects had a Clinical Dementia Rating (CDR) less than or equal to 1. The OASIS dataset provides multiple target values. In this research, CN subjects had normal cognition and absence of MCI or AD diagnosis, MCI subjects had amnestic MCI with memory impairment and AD diagnosis follows the NINCDS-ADRDA criteria @10.1212/WNL.34.7.939 for probable AD. <span>l|rrrrrr</span> &n&Age&Gender&MMSCORE&CDR&ApoE$\epsilon$4\ &&years&f in %&$\bar{x}\pm\sigma$&$\bar{x}\pm\sigma$&0/1/2 %\ CN&704&68.3$\pm$9.3&58.7&29.1$\pm$1.2&0.0$\pm$0.1&65.8/29.7/4.1\ MCI&19&76.7$\pm$7.0&36.8&28.1$\pm$1.4&0.3$\pm$0.2&57.9/42.1/0.0\ AD&198&75.6$\pm$7.9&48.5&24.8$\pm$4.0&0.7$\pm$0.3&38.9/51.5/9.1\ $\Sigma_{CN, MCI,AD}$&921&70.1$\pm$9.5&56.0&28.1$\pm$2.8&0.2$\pm$0.3&59.8/34.6/5.1\ ### Subject Selection {#SEC:SubjectSelection} For the ADNI dataset, all subjects with an MRI scan at the baseline visit were included. 521 subjects who have no MRI scan at the baseline visit were excluded, 29 subjects failed the MRI feature extraction described in Section \[SEC:FeatureExtraction\]. The demographics of the resulting 1,700 subjects are summarized in Table \[Table:ADNIBaseline\]. The 853 MCI subjects were divided into two groups. The sMCI subjects had a stable MCI diagnosis at all follow-up visits and the pMCI subjects converted to a stable AD diagnosis at any visit. 38 subjects with no follow-up visits and 96 subjects who reverted to CN or MCI were excluded from this separation, resulting in 400 sMCI and 319 pMCI subjects. For AIBL, the same exclusion criteria were applied. Therefore, 170 subjects had no MRI scan at the baseline visit, the baseline MRI scans of 76 subjects failed for the MRI feature extraction pipeline described in Section \[SEC:FeatureExtraction\]. The demographics of the resulting 612 subjects are summarized in Table \[Table:AIBLBaseline\]. Similar to the ADNI dataset, the 95 MCI subjects were divided into two groups. In this step, 60 subjects with no follow-up visits and 7 subjects who reverted to CN or MCI were excluded from this separation, resulting in 16 sMCI and 12 pMCI subjects. The exclusion criteria were similarly applied for the OASIS-3 dataset, which originally included 1,098 subjects. For 983 subjects, a diagnosis of CN, MCI, or AD was assigned for at least one visit. The MRI feature extraction pipeline failed for all MRI scans of five subjects, no MRI scan was successfully matched to a diagnosis with a tolerance of 365 days for 57 subjects. In contrast to the ADNI and AIBL datasets, which exclusively included baseline visits, the first visit with an MRI scan and a diagnosis was used for OASIS. The demographics of the remaining 921 subjects are summarized in Table \[Table:OASISBaseline\]. The number of subjects with MCI as baseline diagnosis is 19. This number was decreased if subjects without follow-up diagnoses were excluded. Thus, no experiments were executed to separate sMCI and pMCI subjects in OASIS-3. For reproducible research, the supplementary material contains lists with the subject and MRI IDs and the diagnoses for all datasets. ### MRI Scan Selection From the ADNI dataset, T1-weighted MRI scans recorded at the baseline visit were included. The acquisition parameters differ between scanners. During the ADNI-1 study phase, scans were recorded using a field strength of 1.5 T. In the remaining study phases, MRI scans with a field strength of 3.0 T were recorded. From the AIBL dataset, T1-weighted MRI scans following the protocol of the ADNI 3D T1-weighted sequences were included. All AIBL scans had a resolution of $1\times1\times1.2$ mm. For the OASIS-3 dataset, T1-weighted MRI scans, recorded on three scanners, were included. The field strengths of those scanners are 1.5 T and 3.0 T @10.1101/230.733-60-26394902. MRI Feature Extraction {#SEC:FeatureExtraction} ---------------------- Using FreeSurfer v6.0 @10.1016/j.neuroimage.2012.01.021, volumetric features were extracted for each MRI scan. These include the volumes of 34 cortical areas per hemisphere of the Desikan-Killiany atlas @10.1016/j.neuroimage.2006.01.021, 34 subcortical areas @10.1016/S0896-6273(02)00569-X, and the estimated Total Intracranial Volume (eTIV). As recommended for volumes in @10.1007/s10548-012-0246-x, the volumetric features were normalized by eTIV. This results in 103 MRI volumes, which were split into 49 features of the left hemisphere, 49 features of the right hemisphere, and five additional unpaired segmentations (3rd ventricle, 4th ventricle, brain stem, CSF, eTIV). After the normalization, for paired volumes, the sum (described in Equation \[EQN:SUM\]), the difference (described in Equation \[EQN:DIFF\]), and the ratio (described in Equation \[EQN:RATIO\]) of both hemispheres are calculated to investigate symmetry and to decrease feature interactions. This results in 152 MRI features (49 sums, 49 differences, 49 ratios, and five unpaired features). Brain asymmetry was previously associated with AD @10.3390/technologies5020016 [@10.1038/s41467-021-21057-y; @10.3389/fnins.2020.00602; @10.1016/j.dadm.2019.08.001]. Equation \[EQN:DIFF\] shows that differences were calculated by subtracting the right from the left volume similar to @10.1038/s41467-021-21057-y, where the cortical thickness was used instead of volumetric features. $$\label{EQN:SUM} sum_{ROI}=\frac{vol_{ROI}^{left}}{eTIV}+\frac{vol_{ROI}^{right}}{eTIV}$$ $$\label{EQN:DIFF} diff_{ROI}=\frac{vol_{ROI}^{left}}{eTIV}-\frac{vol_{ROI}^{right}}{eTIV}$$ $$\label{EQN:RATIO} ratio_{ROI}=\frac{vol_{ROI}^{left}}{vol_{ROI}^{right}}$$ Manual Feature Preselection {#SEC:ManualFeaturePreselection} --------------------------- Three feature sets were investigated in the experiments. The manual feature selection aims to choose less invasive, accessible examination techniques which were able to detect early signs of AD. Feature set 1 (FS-1) includes all MRI features, and sociodemographic features including age, gender, and years of education. However, the years of education are only available for the ADNI dataset. Feature set 2 (FS-2) expands FS-1 by the number of ApoE$\epsilon$4 alleles, a genetic risk factor associated with AD, which can be obtained from blood samples or via less invasive swab tests from the inside surface of the cheek. Feature set 3 (FS-3) extended FS-2 by three cognitive tests including the score of the Mini-Mental State Examination (MMSCORE) and two logical tests to evaluate the long-term (Logical memory, delayed – LDELTOTAL) and the short-term memory (Logical memory, immediate – LIMMTOTAL). The CDR was strongly associated with AD diagnosis and was not included in the experiments. Dataset Splitting ----------------- The ADNI dataset was split on the subject level into two distinct subsets. The training set included 80 % of the data and the test set consisted of the remaining 20 %. The splitting was executed within each diagnostic group to ensure similar distributions. The AIBL and OASIS datasets were used as external test sets. None of the AIBL and OASIS subjects was used in the training or model selection process. Feature Selection ----------------- Initially, 152 MRI features were extracted from the MRI scans. Those features are reduced to focus the ML models on the most important features. For this reason, feature forward selection was implemented. In comparison to feature selection methods like RF feature importance, this method avoids correlated features in the dataset @10.1186/1471-2105-8-25. Forward selection is a greedy procedure that iteratively identifies the best new feature until no improvement was reached. The training dataset was split into an 80 % training set and 20 % validation dataset. The training dataset was used to train the ML model used for classification with default hyperparameters on the feature set, and the validation dataset was used to calculate the validation accuracy for this feature set. The selected MRI features were expanded using the features described in Section \[SEC:ManualFeaturePreselection\]. Hyperparameter Tuning --------------------- To tune the hyperparameters of the ML models, Bayesian optimization @10.1007/3-540-07165-2_55 was implemented using the Python package scikit-optimize v0.8.1 @10.5281/zenodo.4014775. Bayesian optimization maps the dependency of the hyperparameters and the model performance using a Gaussian Process. Initially, ten nearly random hyperparameter combinations were selected by a Latin Hypercube Design (LHD) @10.2307/1268522. Bayesian optimization with LHD initialization was successfully used in previous research @10.1007/978-3-030-70569-5_18 to optimize the parameters for early AD detection. Each parameter was split into ten equidistant intervals and one sample was randomly chosen per interval. This results in ten samples per parameter, which were randomly matched. A stratified $10\times10$-fold CV @10.1007/978-0-387-39940-9_565 was applied to the training dataset to estimate the model accuracy for an independent test set. Stratified $10\times10$-fold CV was implemented by splitting each diagnostic group of the training dataset into ten distinct folds using the Python package scikit-learn v0.23.2 @scikit-learn. Ten iterations were performed, each with a different fold used as a validation dataset (10 %). The training dataset included the remaining nine folds (90 %). With shuffled data in each run, this procedure was repeated ten times. The ML model was initially evaluated for ten LHD combinations. To predict the average CV accuracy for the initial parameter combinations, the Gaussian Process was fitted. Afterward, an optimization selected the next promising parameter combination. As an acquisition function, the Lower Confidence Bounds (equation \[Eq:LCB\]) was used. In this equation, $\hat{\mu}_\Theta$ is the Gaussian Process estimation of the CV accuracy and $\hat{\Sigma}_\Theta$ is the covariance at parameter combination $\Theta$. $$\label{Eq:LCB} LCB(\Theta)=\hat{\mu}_\Theta-\hat{\Sigma}_\Theta$$ The hyperparameter combination selected in the previous step was again evaluated using CV. Afterward, to refine the Gaussian Process and to determine the following combination, the respective tuple of hyperparameter and mean CV accuracy was added to the Gaussian Process. The procedure was repeated 25 times. The best hyperparameter combination was chosen to train the final model. Model Training -------------- During hyperparameter training and final model generation, XGBoost models, RFs, radial SVMs, polynomial SVMs, DTs, and LR models were trained. The preprocessing pipeline included centering, scaling, and median imputation. The entire preprocessing pipeline was implemented within the CV to avoid over-optimistic performance estimations @10.1109/MCI.2018.2866730. The parameters were calculated for the CV training set and reused for the test and external datasets. The preprocessing was implemented using the Python package scikit-learn v0.23.2 @scikit-learn. Ensemble-based black-box XGBoost @10.1145/2939672.2939785 models, follow the idea of gradient boosting models @10.1214/aos/1013203451. It means the combination of multiple weak classifiers results in a strong, joint classifier. By learning the gradients of the previous classifier, Gradient boosting fulfills this assumption. The final prediction consisted of the sum of weak classifier predictions. XGBoost is distributed as an open-source software library and the main advantages are scalability, parallelization, and distributed execution. The hyperparameters and intervals used during Bayesian optimization are summarized in Table \[Table:Parameter\]. The hyperparameter `n_estimators` sets the number of boosting iterations, `learning_rate` was the learning rate that preferences weak classifiers at early iterations, the minimum loss reduction required to split a node is defined by `gamma`, the hyperparameter `max_depth` sets the maximum depth of an individual tree, and the minimum number of observations in a child node was denoted as `min_child_weight`, `subsample` and *colsample\_bytree* set the proportion of randomly subsampled training instances and features per iteration. The Python package xgboost v1.2.0 @Chen2019 implemented the XGBoost algorithm. RF @10.1023/A:1010933404324 training was implemented using the Python package scikit-learn v0.23.2 @scikit-learn. The RF algorithm is based on multiple DTs. Each DT was trained using randomly chosen features and subjects. Those subjects were selected using bootstrap sampling @10.1214/ss/1177013815 on the training dataset. RF inference was computed by summarizing the individual DTs using a majority voting. The RF hyperparameters are summarized in Table \[Table:Parameter\]. `n_estimators` sets the number of DTs, each split used a random subset of `max_features` features, the hyperparameter `min_samples_leaf` describes the minimum number of samples in a leaf node. Support Vector Machines (SVMs) @SVMs were implemented using the Python package scikit-learn v0.23.2 @scikit-learn. SVMs separate two classes using a decision boundary which was referred to as an $n$-dimensional hyperplane. Here, $n$ is the number of features. To increase the robustness of the hyperplane for unknown observations, SVMs select the hyperplane with the largest distance from the observations. For this reason, the distance between the hyperplane and the observations was maximized using the hinge loss function @10.1162/089976604773135104. The support vectors describe the observations closest to the hyperplane. Removing support vectors from the dataset directly influences the hyperplane. The cost parameter `C` enables SVMs to avoid overfitting, the higher `C`, the less complex an SVM is. Kernel functions help to model complex interactions. In this research, a polynomial and a radial kernel were implemented. The `degree` hyperparameter of the polynomial kernel controls the degree of the kernel, high values lead to more complex hyperplanes. The `gamma` hyperparameter constraints the influence, a single observation has on the hyperplane. If `gamma = scale`, $\frac{1}{\# features\cdot \sigma}$ was used as a value of gamma, if `gamma = auto`, a value of $\frac{1}{\# features}$ was used. The SVM hyperparameters and their ranges are summarized in Table \[Table:Parameter\]. <span>llrr</span> Model&Hyperparameter&Minimum&Maximum\ LR&`C`&$10^{-4}$&$10^2$\ &`penalty`&`l2`&`none`\ DT&`criterion`&`gini`&`entropy`\ &`splitter`&`best`&`random`\ &`max_depth`&1&100\ &`min_samples_split`&0&1\ RF&`n_estimators`&250&1,250\ &`max_features`&2&\# features\ &`min_samples_leaf`&1&20\ XGBoost&`n_estimator`&1&500\ &`max_depth`&1&20\ &`learning_rate`&$10^{-10}$&1\ &`gamma`&0&20\ &`min_child_weight`&1&30\ &`subsample`&0&1\ &`colsample_bytree`&0&1\ polynomial SVM&`C`&$10^{-4}$&$10^2$\ &`degree`&1&10\ &`gamma`&`scale`&`auto`\ radial SVM&`C`&$10^{-4}$&$10^2$\ &`gamma`&`scale`&`auto`\ In contrast to the black-box models, DTs @10.1201/9781315139470 and LR models were selected as simple and interpretable comparison models. DTs were implemented using the Python package scikit-learn v0.23.2 @scikit-learn. A DT consists of successively learned decision rules of the form $x \leq t$ for numerical or $x\in t$ for categorical features $t$ is a threshold or a subset of values. The next decision rule was selected by the `splitter` which ranked all possible rules using a `criterion`. Decision rules were iteratively expanded until a maximum depth of `max_depth` was met or a percentage `min_samples_split` of samples were in a split. LR @cox1958regression is a Generalized Linear Model (GLM) with a logistic link function. This link function allows the processing of binomial output variables. The logistic model function is given in equation \[Eq:logReg\]. The model predicts the probability $P(Y=1\vert X=x,\Theta)$ of observation $x$ with given parameters $\Theta$ being in the positive class $Y=1$. The LR algorithm was implemented using the Python package scikit-learn v0.23.2 @scikit-learn. $$\label{Eq:logReg} P(Y=1\vert X=x,\Theta)=\frac{1}{1+\exp(x\cdot\Theta)}$$ Model Interpretation with Shapley Values ---------------------------------------- There are multiple methods to interpret ML models. An overview can be found in @Molnar2019. For example, DTs and LR models are interpretable by design. However, black-box models often outperform those interpretable models but the interpretation of black-box models is complicated. In this research, model-agnostic Shapley values were used. Shapley values are local models, which explain the predictions of individual observations and thus enable high clinical benefit and high adaption to the black-box model. Shapley values @Shapley1953 are affiliated with coalition game theory and aim to decompose the prediction of a subject into the contributions of each feature. For this aim, Shapley values are based on the additive linear explanation model shown in equation \[Eq:AdditiveLinearModel\]. For a subject $x$, the model prediction $f(x)$ is decomposed into the feature contributions $\Phi_j$, a simplified representation of the feature values $x'$, and the average model prediction $\Phi_0$. A binned binary feature representation was used for tabular data, with $N$ being the number of simplified features. $$\label{Eq:AdditiveLinearModel} f(x)=\Phi_0+\sum_{j=1}^{N}\Phi_j x'_j$$ The idea of using Shapley values to explain black-box ML models is, to fairly decompose the contribution of each feature for the subject's prediction. Due to this fairness, the sum of all Shapley values is equal to the difference between the average model prediction and the probability prediction of a subject. Equation \[Eq:ShapleyValueOrig\] shows Shapley values are defined as the average, weighted contribution, a simplified feature has in all subsets. For the exact calculation of a Shapley value $\Phi_i$ for a given subject and feature $i$, it is required to determine the contribution of this feature for all subsets $S$ of the entire feature set $F$. The investigation of each subset $S$ requires the retraining and evaluation of the black-box model $f_{S}(S)$. With the help of the model performances trained with ($f_{S\cup i}(S\cup i)$) and without ($f_S(S)$) the feature at interest $i$, their differences were calculated. The weighted average difference across subsets builds the Shapley value. The weighting depends on the relative number of features $\vert S\vert$ in subset $S$. High weights were assigned to subsets with few and many features. In this way, the estimation of the main individual effects and the total effects are supported. $$\label{Eq:ShapleyValueOrig} \Phi_i=\sum_{S\subseteq F\setminus\{i\}} \frac{\vert S\vert! (\vert F\vert-\vert S\vert-1)!}{\vert F\vert!}\bigl(f_{S\cup\{i\}}(S\cup i)-f_S(S)\bigr)$$ However, the number of subsets increases exponentially with the number of input features, leading to high computational effort for the exact calculation of Shapley values. Kernel SHapley Additive exPlanations (SHAP) @10.1007/s10115-013-0679-x avoid time-consuming repeated training and evaluation by estimating Shapley values. This algorithm is based on Local Interpretable Model-agnostic Explanations (LIME) @10.18653/v1/n16-3020 and was implemented using the Python package shap v0.38.1 @Lundberg2017. A new dataset containing variants of the observation at interest is created by permuting selected features. An additive linear model (Eqn. \[Eq:AdditiveLinearModel\_SHAP\]) with $x'$ is a simplified representation of the black-box input features and $g(x')$ is the explanation model was fitted to the generated dataset. $$\label{Eq:AdditiveLinearModel_SHAP} g(x')=\Phi_0+\sum_{i=1}^{M}\Phi_i\cdot x'_i$$ The weights $\Phi_i$ of the explanation model estimates the SHAP values for each subject and each feature. For tabular data, the simplified features are binned binary feature representations that represent if the original feature value or a permutation was used. SHAP force plots @Lundberg2018 explain the model prediction of individual subjects using Shapley values. Features with positive Shapley values show strong positive effects on the prediction and small negative Shapley values represent small negative effects. SHAP force plots can be found in Figure \[fig:SHAP\_force\_plots\]. SHAP summary plots @Lundberg2018 summarize the explanations for the entire training dataset. Each point visualizes the feature value of a subject and the associated Shapley value. The color of a point depends on the subject's feature value. On the vertical axis, the features are ordered by the mean absolute Shapley values. The plots were limited to the top ten features. SHAP summary plots can be found in Figure \[fig:FeatureSet\_comp\_CNAD\_svmPoly\_rfMean\_SHAP\], Figure \[fig:Reproducibility\_Comparison\_RF\_CNAD\_SHAP\], Figure \[fig:Reproducibility\_Comparison\_RF\_CNMCI\_SHAP\], and Figure \[fig:Comparison\_ClassificationModels\_sMCI\_FS-3\_RFMean\_SHAP\]. There are some reasons, including out-of distribution sampling during Shapley value approximation and not taking into account feature correlation, why Shapley values should be used with caution for black-box model interpretability @kumar2020problems. Therefore, it is important to compare Shapley value results with other ML explanation methods, or to reduce or consolidate correlated features @Pekala2021. In this work, forward selection was implemented to reduce the number of correlated features in the dataset, Shapley values were compared to classical feature importance measurements (Section \[SEC:ClassificationModel\]), and correlated features are consolidated to aspects. Evaluation ---------- The models were evaluated for the ADNI test set and the external AIBL and OASIS datasets. The performance was measured using accuracy (ACC) (equation \[EQN:ACCURACY\]), balanced accuracy (BACC) (equation \[EQN:BALANCEDACCURACY\]), F1-Score (F1) (equation \[EQN:F1\]), and Matthews correlation coefficient (MCC) (equation \[EQN:MCC\]). Table \[TAB:CONTINGENCY\_TABLE\] visualizes the contingency table used for the calculation of those scores. Providing multiple scores for evaluation increased the comparability to other research. In comparison to accuracy, which focuses on correctly classified cases, the F1-Score focuses on incorrectly classified cases. The macro averaging F1-Score was calculated to address both, the diseased and the healthy subject classification. Balanced accuracy is based on both, sensitivity and specificity and thus is suitable to evaluate imbalanced class problems. The MCC returns a value between 0 and 1 and is also suitable to handle imbalanced datasets. <span>c|cc</span> &patient&control\ patient&True positive (TP)& False positive (FP)\ control&False negative (FN)&True negative (TN)\ $$\label{EQN:ACCURACY} ACC=\frac{TP+TN}{TP+TN+FP+FN}$$ $$\label{EQN:BALANCEDACCURACY} BACC=\frac{\frac{TP}{TP+FN}+\frac{TN}{TN+FP}}{2}$$ $$\label{EQN:F1} F1=\frac{TP}{TP+\frac{1}{2}(FP+FN)}$$ $$\label{EQN:MCC} MCC=\frac{TP\cdot TN -FP \cdot FN}{\sqrt{(TP+FP)\cdot(TP+FN)\cdot (TN+FP)\cdot (TN+FN)}}$$ Additionally, the Area under the Receiver Operating Curve (AUROC), which models the relationship between the True Positive Rate (TPR – equation \[EQN:TPR\]) and the False Positive Rate (FPR – equation \[EQN:FPR\]) for different classification thresholds was computed for all models. AUROC is suitable to investigate classification tasks with imbalanced datasets. $$\label{EQN:TPR} TPR=\frac{TP}{TP+FN}$$ $$\label{EQN:FPR} FPR=\frac{FP}{TN+FP}$$ Results {#Sec:Results} ======= In the following, the experimental results are presented. The MRI features selected using forward selection and the performances achieved for CN vs. AD, CN vs. MCI, MCI vs. AD, and sMCI vs. pMCI classification were given. SHAP summary plots compared the models trained using different feature sets, validation datasets, and classification models. The results of SHAP summary plots are compared to natural RF- and XGBoost-based feature importance scores and permutation importance scores. The influence of feature interactions for Shapley values is investigated and SHAP force plots explain individual model predictions of interesting subjects. Feature Selection ----------------- The MRI features selected during forward selection for CN vs. AD classification and different ML methods used as base classifiers are summarized in Table \[Table:ForwardSelectionCNAD\]. In this research, feature forward selection was used to reduce the number of MRI features and the influence of correlated features. <span>ccc</span> LR&DT&RF\ sum\_entorhinal&sum\_Amygdala&sum\_Amygdala\ sum\_Amygdala&sum\_entorhinal&diff\_parstriangularis\ ratio\_lingual&sum\_Hippocampus&diff\_superiorparietal\ sum\_middletemporal&ratio\_supramarginal&sum\_lateralorbitofrontal\ diff\_Lateral.Ventricle&&sum\_medialorbitofrontal\ ratio\_entorhinal&&\ XGBoost&SVM poly&SVM radial\ sum\_Amygdala&sum\_entorhinal&sum\_Amygdala\ sum\_middletemporal&sum\_inferiorparietal&sum\_entorhinal\ sum\_entorhinal&diff\_Cortex&diff\_Cortex\ diff\_lateralorbitofrontal&sum\_Amygdala&sum\_VentralDC\ &ratio\_paracentral&\ For the CN vs. AD detection task, the RF, and the polynomial SVM chose five features, the XGBoost, the DT, and the radial SVM chose four features and the LR chose six features. Overall, the six methods chose 16 different features. The most important feature for the RF, the XGBoost, the DT and the radial SVM was the sum of the left and right amygdalae. For the polynomial SVM and the LR, the most important feature was the sum of the entorhinal cortices. Both features were previously associated with AD detection @10.1038/nrneurol.2009.215 [@10.1002/hbm.20934; @10.1016/j.neurobiolaging.2003.12.007; @10.1016/j.pscychresns.2011.06.014]. Previous research also shows, that most of the selected features are associated with atrophy patterns in AD @10.1136/gpsych-8702028617. All methods also selected at least one difference or ratio of the left and right cortical or subcortical areas. Those features describe the asymmetry of both hemispheres. Brain asymmetry measurements were associated with AD @10.3390/technologies5020016 [@10.1038/s41467-021-21057-y; @10.3389/fnins.2020.00602; @10.1016/j.dadm.2019.08.001] and were also successfully applied for ML models in this field @10.3390/s21030778. The rankings of the forward selection for CN vs. MCI detection and different base classifiers are given in Table \[Table:ForwardSelectionCNMCI\]. For CN vs. MCI detection, the RF, XGBoost, and LR base classifiers chose four features, the DT chose three features, the polynomial SVM chose eight features, and the radial SVM chose six features. Overall, the six ML methods chose 25 different features. Thus, in comparison to the CN vs. AD classification, the ML models show less agreement about the selected features. Consequently, the feature which was selected first in the forward selection process differed in five out of six methods. For the RF and the DT, the sum of the insular cortices was selected, the XGBoost classifier chose the ratio of the inferior parietal lobule, the polynomial SVM selected the sum of the lingual gyri, the SVM with the radial kernel chose the sum of the temporal pole volumes and the LR selected the sum of the left and right middle temporal gyri. Those features were previously associated with AD progression @10.1136/gpsych-8702028617 [@Foundas1997-ik; @10.1016/j.neurobiolaging.2010.04.026; @10.1371/journal.pcbi.1001006; @10.1038/nrneurol.2009.215; @10.1002/hbm.20934; @10.1016/j.neurobiolaging.2003.12.007; @10.1016/j.pscychresns.2011.06.014]. Similar to the CN vs. AD classification, all models selected at least one feature describing the asymmetry of the cortical and subcortical brain regions. The forward feature selection results of the six ML models for MCI vs. AD classification are summarized in Table \[Table:ForwardSelectionMCIAD\]. Four of the six models, namely RF, XGBoost, SVM poly, and LR selected five features. The DT chose six different features and the radial SVM selected two MRI features. Overall, the six methods selected 22 unique features. The most important features were the sum of the left and right hippocampi for the RF and DT model, the difference of the lateral ventricles for the XGBoost model, the sum of the inferior temporal gyri for both SVMs, and the sum of the entorhinal cortex volumes for the LR. Those features were previously associated with AD detection @10.1038/nrneurol.2009.215 [@10.1002/hbm.20934; @10.1016/j.neurobiolaging.2003.12.007; @10.1016/j.pscychresns.2011.06.014; @10.3233/jad-2011-101782; @10.1136/jnnp.72.4.491]. <span>ccc</span> LR&DT&RF\ sum\_entorhinal&sum\_Hippocampus&sum\_Hippocampus\ sum\_precuneus&sum\_cuneus&sum\_Amygdala\ sum\_VentralDC&sum\_posteriorcingulate&diff\_entorhinal\ diff\_frontalpole&ratio\_Putamen&sum\_isthmuscingulate\ diff\_rostralanteriorcingulate&sum\_Cortex&ratio\_lateralorbitofrontal\ &ratio\_parstriangularis&\ XGBoost&SVM poly & SVM radial\ diff\_Lateral.Ventricle&sum\_inferiortemporal&sum\_inferiortemporal\ diff\_Cortex&Brain.Stem&ratio\_frontalpole\ sum\_Cortex&sum\_entorhinal&\ sum\_pericalcarine&sum\_precuneus&\ sum\_precentral&ratio\_precuneus&\ The results of the forward selection for the sMCI vs. pMCI classification task are summarized in Table \[Table:ForwardSelectionsMCIpMCI\]. Five features were selected by the RF model, the XGBoost model chose six features, the DT selected only one feature, both SVMs chose four features, and the LR selected three features. Overall, the six methods picked 19 unique features. Three methods, namely the RF, the DT, and the SVM with the radial kernel selected the sum of the left and right amygdalae as the most important feature. The forward selection with the XGBoost base model first picked the sum of the hippocampi. The polynomial SVM selected the sum of the left and right precuneus and the LR chose the sum of the inferior temporal gyri. Those features were previously associated with AD detection @10.1038/nrneurol.2009.215 [@10.1002/hbm.20934; @10.1016/j.neurobiolaging.2003.12.007; @10.1016/j.pscychresns.2011.06.014; @10.1136/jnnp.72.4.491; @10.3389/fnagi.2018.00304]. <span>ccc</span> LR&DT&RF\ sum\_inferiortemporal&sum\_Amygdala&sum\_Amygdala\ diff\_middletemporal&&sum\_inferiorparietal\ sum\_precentral&&sum\_entorhinal\ &&sum\_lateraloccipital\ &&diff\_superiorparietal\ XGBoost&SVM poly& SVM radial\ sum\_Hippocampus&sum\_precuneus&sum\_Amygdala\ diff\_lateralorbitofrontal&sum\_inferiortemporal&diff\_Inf.Lat.Vent\ Brain.Stem&sum\_rostralanteriorcingulate&sum\_precuneus\ sum\_caudalmiddlefrontal&ratio\_rostralanteriorcingulate&diff\_middletemporal\ diff\_precentral&&\ sum\_postcentral&&\ Classification Tasks -------------------- In the following the classification performances achieved for the four classification tasks are elaborated. ### CN vs. AD The results achieved for CN vs. AD classification are summarized in Table \[Table:ResultsCNAD\_FS1\]. The no information rates were 60.36 % for the independent ADNI test set, 86.27 % for AIBL, and 78.05 % for OASIS. CN was the most frequent class in all datasets. The best accuracy during CV of 99.68 % $\pm$ 0.74 was achieved for the DT trained with feature selection and FS-3. This model also reached a perfect classification for the ADNI test set. All models trained for the CN vs. AD task reached accuracies higher than the no information rate for the ADNI dataset. The best AIBL accuracy of 95.94 % was achieved for the XGBoost model trained for FS-3 and with feature selection. This model also reached the best F1-Score (91.48 %) and the best MCC (0.830) for the AIBL dataset. The best balanced accuracies of 95.45 % for the AIBL dataset were reached for both SVMs trained with feature selection and FS-3. The LR model trained with feature selection for FS-3 reached the best AIBL AUROC of 99.55 %. Overall, two models achieved AIBL accuracies smaller than the no information rate of 86.27 %. Those models were the DTs trained with feature selection for FS-1 and FS-2. The best OASIS accuracy of 90.58 % was achieved for the polynomial SVM which was trained with feature selection and FS-3. For OASIS, four models achieved accuracies worse than the no information rate of 78.05 %. Three of those models were trained on FS-1 and with feature selection, namely, the RF, the DT, and the SVM. The last model reaching an accuracy worse than the no information rate was the DT trained with FS-2 and feature selection. ### CN vs. MCI The results achieved for CN vs. MCI classification are summarized in Table \[Table:Results\_CNMCI\_FS-1\]. The no information rate for this task was 62.64 % for the ADNI test set, 82.44 % for AIBL, and 97.39 % for OASIS. MCI was the most frequent class in the ADNI dataset, whereas, for AIBL and OASIS, CN was. The results achieved for CN vs. MCI classification were worse than those for the CN vs. AD task. The best accuracy during CV of $90.21~\%\pm2.72$ was achieved for the XGBoost model trained for FS-3 and without feature selection. The best accuracy for the ADNI test set was 91.58 % reached for two models. Both models, the radial SVM and the XGBoost model were trained for FS-3 and with feature selection. The latter model also reached the best ADNI balanced accuracy and ADNI F1-Score. Overall, none of the models reached an ADNI accuracy worse than the no information rate of 62.51 %. The results achieved for AIBL and OASIS were worse than the ADNI results. The best AIBL accuracy was 68.95 % achieved for two DTs trained with forward feature selection for FS-1 and FS-2. These models also reached the best AIBL balanced accuracies, AIBL F1-Scores, and AIBL MCCs. The best OASIS accuracy of 55.05 % was reached for the DT trained without feature selection for FS-3. For the CN vs. MCI classification, all models achieved accuracies worse than the no information rates for OASIS and AIBL. ### MCI vs. AD The MCI vs. AD classification results are summarized in Table \[Table:Results\_MCIAD\_FS1\]. The no information rate was 71.85 % for the ADNI test set, 57.23 % for AIBL, and 91.24 % for OASIS. The most frequent class was MCI for ADNI and AIBL as well as AD for OASIS. The best CV accuracy of 89.39 % $\pm$ 2.99 was achieved for the RF trained without feature selection and FS-3. For the independent ADNI test set, the best accuracy was 88.66 %, reached by the RF and LR models trained with feature selection and FS-3. The first of those models also reached the best ADNI AUROC of 95.50 %, whereas the second model achieved the best ADNI F1-Score (85.50 %), and ADNI MCC (0.712). None of the models reached an ADNI accuracy worse than the no information rate. However, the DT trained without feature selection for FS-1 as well as the XGBoost and the DT both trained for FS-2 and with feature selection exactly achieved the no information rate of 71.85 % for the independent ADNI test set. The two DT models mentioned predicted the MCI class for all subjects and thus represented random classifiers. The best AIBL accuracy was 84.94 % reached by the RF model trained without feature selection and FS-3. This model also reached the best AIBL balanced accuracy (83.64 %), AIBL F1-Score (84.24 %), and AIBL MCC (0.693). Except for the previously mentioned random classifiers, all models outperformed the no information rate for the AIBL dataset which was 57.23 %. The performances for OASIS were worse than those achieved for ADNI and AIBL. The best OASIS accuracy of 57.14 % was achieved for the radial SVM trained without feature selection and FS-3. The random classifiers achieved a worse accuracy of 8.76 % for OASIS. These accuracies correspond to the ratio of MCI subjects in the dataset. MCI was the most frequent class for the ADNI dataset and the rarest class for OASIS. All models achieved OASIS accuracies worse than the no information rate of 91.24 %. ### sMCI vs. pMCI The results reached for sMCI vs. pMCI classification with no information rates of 55.56 % for the ADNI test set and 57.14 % for AIBL are summarized in Table \[Table:ResultssMCIpMCI\_FS1\]. As previously mentioned, due to the lack of available data, OASIS was not used for this comparison. The best CV accuracy of $70.75~\%~\pm~5.94$ was achieved for the RF trained with feature selection and FS-3. For the independent ADNI test set, the radial SVM which was trained with forward feature selection for FS-3 reached the best accuracy (75.00 %), balanced accuracy (74.38 %), F1-Score (74.51 %), and MCC (0.491). The best AUROC of 80.14 % was reached for the XGBoost model trained with forward feature selection for FS-3. None of the models achieved worse results than the ADNI no information rate of 55.56 %. The best AIBL accuracy was 82.14 % reached for the radial SVM which was trained for FS-3, and without feature selection. Five models achieved AIBL accuracies worse than the no information rate. Those models were all DTs trained with FS-1 and FS-3, and the LR trained with feature selection and FS-3. Feature Sets ------------ As can be seen in Table \[Table:ResultsCNAD\_FS1\], for CN vs. AD classification, all models achieved the best scores using FS-3. Thus, adding cognitive test results to the dataset improved the overall classification results. The SHAP summary plots for the polynomial SVMs trained with feature selection for all three feature sets are shown in Figure \[fig:FeatureSet\_comp\_CNAD\_svmPoly\_rfMean\_SHAP\]. SHAP summary plots @Lundberg2018 explain the predictions for the subjects of the entire ADNI, AIBL, and OASIS datasets. Each point plots a Shapley value for a subject and a feature and is colored depending on the feature value. The vertical axis represented the features, ordered by the mean absolute Shapley values and their distribution. The higher a Shapley value is, the more the feature expression increases the probability the model classifies the subject as an AD subject. All SHAP summary plots were limited to the top ten features. Following the biological processes of AD, small brain volumes @10.1016/j.dadm.2015.11.002 [@10.1371/journal.pone.0066367; @10.1038/s41598-018-29295-9; @10.1038/nrneurol.2009.215], large ventricular volumes @10.1016/j.neuroimage.2004.03.040 [@10.1212/01.WNL.0000110315.26026.EF], the presence of ApoE$\epsilon$4 alleles @10.1126/science.8346443 [@10.1073/pnas.90.5.1977; @10.1136/jnnp.2010.231555], and bad performances in cognitive test scores were expected to have a pathogenic effect on the disease progression. The left hemisphere was expected to be more affected by atrophy than the right @10.3390/s21030778 one. However, some investigations for MCI subjects also showed the right hippocampus was more affected @10.1002/hbm.23772. This asymmetry occurs primarily in the hippocampi and amygdalae @10.1093/brain/aww243 [@10.1016/j.neuroimage.2015.01.032]. For FS-1 and FS-2, the most important feature was the sum of the left and right amygdalae. Consistently with the previously mentioned atrophy patterns @10.1016/j.dadm.2015.11.002 [@10.1371/journal.pone.0066367; @10.1038/s41598-018-29295-9; @10.1038/nrneurol.2009.215], small volumes of the amygdalae (colored in blue) increased the probability the model classifies a subject as an AD subject. Large amygdala volumes (colored in red) were associated with CN subjects. The second most important feature for both models was the sum of the left and right entorhinal cortex. The model, trained using FS-1, learned that large volumes (colored in red) of the amygdalae, the entorhinal cortices, and the inferior parietal lobules had protective effects on the disease progression (negative Shapley values). Those assocications correspond to previous research @10.1016/j.dadm.2015.11.002 [@10.1371/journal.pone.0066367; @10.1038/s41598-018-29295-9; @10.1038/nrneurol.2009.215]. The model additionally learned that a large difference between the left and right cortex volume (colored in red) was associated with CN. Large differences were reached if the volume of the left hemisphere was larger than the right one. The same observation applies to the ratio of the left and right paracentral lobules. Considering the sociodemographic features show, that the model learned, young age (colored in blue) was associated with disease progression. However, the summary of the ADNI dataset in Table \[Table:ADNIBaseline\] shows the mean age of the CN group was younger than the AD group. Additionally, the model learned that females (colored in blue) and subjects with low education more likely develop AD. FS-2 added the number of ApoE$\epsilon$4 alleles to FS-1. The additional feature was the third most important feature of this model. The model learned that a large number of ApoE$\epsilon$4 alleles (no ApoE$\epsilon$4 alleles are colored in blue, one ApoE$\epsilon$4 allele is colored in purple, two ApoE$\epsilon$4 alleles are colored in red) led to an increased risk. Previous research also identified ApoE$\epsilon$4 as an AD risk factor @10.1126/science.8346443 [@10.1073/pnas.90.5.1977; @10.1136/jnnp.2010.231555]. Biologically plausible associations @10.1016/j.dadm.2015.11.002 [@10.1371/journal.pone.0066367; @10.1038/s41598-018-29295-9; @10.1038/nrneurol.2009.215] were noted for the summed volumes of the left and right amygdalae, the entorhinal cortices, and the inferior parietal lobules. The ratio of the left and right paracentral lobule volumes showed an increased risk of AD if the left hemisphere was smaller than the right one. The same applies to the differences in cortex volumes. FS-3 added the results of three cognitive tests to FS-2. The LDELTOTAL cognitive test score achieved the best feature importance for this model, followed by the MMSCORE. The LIMMTOTAL score achieved the sixth-best feature importance. For all cognitive test scores, the model associated good scores (colored in red) with healthy subjects. The third most important feature was the summed volume of the entorhinal cortices. Consistently with AD atrophy, small volumes (colored in blue) were associated with AD progression. The same applied to the sum of the inferior parietal lobules and the amygdalae. Similar to the FS-1 and FS-2 models, the FS-3 model also learned young age (colored in blue) was associated with AD, although, the mean age of the ADNI-CN group was younger than the mean age of the ADNI-AD group. \[b\]<span>0.49</span> ![SHAP summary plots of the polynomial SVM trained with feature selection for CN vs. AD. The plots visualize the Shapley values of *n=2,266* subjects from the ADNI, AIBL, and OASIS datasets and the ten most important model features. Each subject is represented by a dot. The colors decode the subject's feature expression. High feature expressions are colored in red and small expressions in blue. The model learned that features with high Shapley values increased the patient's AD risk. Each plot shows a model trained on a different feature set<span data-label="fig:FeatureSet_comp_CNAD_svmPoly_rfMean_SHAP"></span>](Fig2a.pdf "fig:"){width="\textwidth"} \[b\]<span>0.49</span> ![SHAP summary plots of the polynomial SVM trained with feature selection for CN vs. AD. The plots visualize the Shapley values of *n=2,266* subjects from the ADNI, AIBL, and OASIS datasets and the ten most important model features. Each subject is represented by a dot. The colors decode the subject's feature expression. High feature expressions are colored in red and small expressions in blue. The model learned that features with high Shapley values increased the patient's AD risk. Each plot shows a model trained on a different feature set<span data-label="fig:FeatureSet_comp_CNAD_svmPoly_rfMean_SHAP"></span>](Fig2b.pdf "fig:"){width="\textwidth"} \ \[b\]<span>0.49</span> ![SHAP summary plots of the polynomial SVM trained with feature selection for CN vs. AD. The plots visualize the Shapley values of *n=2,266* subjects from the ADNI, AIBL, and OASIS datasets and the ten most important model features. Each subject is represented by a dot. The colors decode the subject's feature expression. High feature expressions are colored in red and small expressions in blue. The model learned that features with high Shapley values increased the patient's AD risk. Each plot shows a model trained on a different feature set<span data-label="fig:FeatureSet_comp_CNAD_svmPoly_rfMean_SHAP"></span>](Fig2c.pdf "fig:"){width="\textwidth"} As can be seen in Table \[Table:Results\_CNMCI\_FS-1\], for the CN vs. MCI classification, FS-3 outperformed FS-1 and FS-2 for the ADNI and AIBL performance scores. The best accuracies for OASIS were reached for FS-3, whereas FS-2 models outperform those models for F1-Score, balanced accuracy, AUROC, and MCC. For the MCI vs. AD task, which is summarized in Table \[Table:Results\_MCIAD\_FS1\], the same applied to all ADNI and AIBL scores. For the OASIS dataset, the best accuracy and F1-Score were reached by FS-1 and
{'source': 'arXiv_out/2205'}