In recent years, many papers have reported state-of-the-art performance on Alzheimer's Disease classification with MRI scans from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset using convolutional neural networks. However, we discover that when we split that data into training and testing sets at the subject level, we are not able to obtain similar performance, bringing the validity of many of the previous studies into question. Furthermore, we point out that previous works use different subsets of the ADNI data, making comparison across similar works tricky. In this study, we present the results of three splitting methods, discuss the motivations behind their validity, and report our results using all of the available subjects.
@cite_5: Automatic extraction of features from MRI brain scans and diagnosis of Alzheimer's Disease (AD) remain a challenging task. In this paper, we propose an efficient and simple three-dimensional convolutional network (3D ConvNet) architecture that is able to achieve high performance for detection of AD on a relatively large dataset. The proposed 3D ConvNet consists of five convolutional layers for feature extraction, followed by three fully-connected layers for AD NC classification. The main contributions of the paper include: (a) propose a novel and effective 3D ConvNet architecture; (b) study the impact of hyper-parameter selection on the performance of AD classification; (c) study the impact of pre-processing; (d) study the impact of data partitioning; (e) study the impact of dataset size. Experiments conducted on an ADNI dataset containing 340 subjects and 1198 MRI brain scans have resulted good performance (with the test accuracy of 98.74 , 100 AD detection rate and 2,4 false alarm). Comparisons with 7 existing state-of-the-art methods have provided strong support to the robustness of the proposed method.
Unlike most of the aforementioned papers that report high performance but do not explicitly mention their training and testing data split methodology, @cite_5 pointed out the problem of potentially giving away information from the training set into the testing set when splitting randomly by MRI scans, and showed that splitting MRI data by patient led to worse model performance. However, they only report two-class classification (Normal vs. Alzheimer's) in their results. They also only used a subset that is less than half the size of the MRIs available in ADNI, even though they reported empirical studies showing that .