A Deep Unsupervised Learning Approach Toward
MTBI Identification Using Diffusion MRI

arXiv:1802.02925v1 [cs.CV] 8 Feb 2018

Shervin Minaee1 , Yao Wang1 , Anna Choromanska1 , Sohae Chung2 , Xiuyuan Wang2 , Els Fieremans2 ,
Steven Flanagan3 , Joseph Rath3 , Yvonne W. Lui2 ,
1 Electrical and Computer Engineering Department, New York University
2 Department of Radiology, New York University
3 Department of Rehabilitation Medicine, New York University

Abstract—Mild traumatic brain injury (mTBI) is a growing
public health problem with an estimated incidence of one million
people annually in US. Neurocognitive tests have been used to
both assess the patient condition and to monitor the patient
progress. This work aims to directly use diffusion MR images
taken shortly after injury to detect whether a patient suffers from
mTBI, by incorporating deep learning techniques. To overcome
the challenge due to limited training data, we describe each brain
region using the bag of word representation, which specifies
the distribution of representative patch patterns. We apply a
convolutional auto-encoder to learn the patch-level features, from
overlapping image patches extracted from the MR images. to
learn features from diffusion MR images of brain using an
unsupervised approach. Our experimental results show that the
bag of word representation using patch level features learnt by
the auto encoder provides similar performance as that using the
raw patch patterns, both significantly outperform earlier work
relying on the mean values of MR metrics in selected brain
regions.

the raw MRI volume data as the input. We have proposed a
new approach for feature extraction from MR images, where
we first learn the feature representation of patches using a
deep unsupervised learning approach [8], and then aggregate
the features from different patches through a bag of word
representation, and use them along with demographic and
neuro-cognitive test features as the overall feature vector.
We then use feature selection followed by a classification
algorithm to identify mTBI patients. The block diagram of
the overall algorithm is shown in Fig. 1. Through experimental
study, we show that by learning patch level deep features and
aggregating them through a bag of word representation for
each brain region, we get much higher accuracy compared to
using mean values of the various MR metrics in each region.

I. I NTRODUCTION
Mild traumatic brain injury (mTBI) is a growing public health problem, which can cause loss of consciousness,
headache, difficulty thinking, memory problems, attention
deficits, mood swings and frustration. Recent work using MRI
revealed that there are areas of subtle brain injury after mTBI;
however, no single imaging metric has been shown to be
sufficient as an independent biomarker [1]-[3].
While diffusion MRI has been extremely promising in the
study of mTBI, identifying patients with recent mTBI remains
a challenge. The gray matter such as the thalamus and white
matter including the corpus callosum and frontal deep white
matter have been repeatedly implicated as areas at high risk for
injury in the literature. There have been some previous studies
using machine learning techniques for mTBI identification
from MR images [4]-[7], however the features used in those
works are mainly hand-crafted features and may not be the
most discriminative feature for this task.
In this work, we develop a machine learning framework to
classify mTBI patients and controls using features extracted
from diffusion MRI in the thalamus and corpus callosum.
The main challenge is that we only have limited samples
(only 114), and each sample has a very high dimensional
raw representation (multiple 3D MR volumes). Therefore, it
is not possible to directly train a classification network using

Fig. 1. The block-diagram of the proposed mTBI identification algorithm

The rest of the paper is organized as follows. Section II
provides the description of the proposed framework. Section
III provides the experimental studies and comparison to other
works. And finally the paper is concluded in Section IV.
II. T HE P ROPOSED F RAMEWORK
There have been some previous studies on mTBI classification using different sets of features, from demographic (such
as age and gender) and neurocognitive to hand-crafted features
from medical images. In this work we propose a framework
for mTBI identification based on imaging, demographic, and
neuro-cognitive features. We derive the imaging features from
the specialized MR imaging protocols and related image
metrics that are developed and shown to be promising for

distinguishing mTBI patients from controls [1]-[3]. These
metrics are summarized in Table I.
TABLE I
MRI METRICS DESCRIPTION
MRI Metric
AWF
DA
De-par
De-perp
FA
MD
AK, MK, RK

Metric Description
Axonal Water Fraction
Diffusivity within Axons
Diffusion parallel to the axonal tracts in the
extra-axonal
Diffusion perpendicular to the axonal tracts in
the extra-axonal
Fractional Anisotropy
Mean Diffusion
Axial Kurtosis, Mean Kurtosis, Radial Kurtosis

Fig. 2. The block-diagram of the proposed convolutional auto-encoder

convolutional auto-encoder. We use the same model for all
regions (Thalamus and CC).
B. Bag of Visual Words

Because of the limitation of the number of samples, it is
not possible to train a deep convolutional network to directly
classify the brain images. To tackle this problem, and also
based on the intuition that mTBI would only impact some
local regions in the brain, we propose to represent each brain
region by a bag of words (BoW) representation, which is
the histogram of different clustered patch patterns among all
patches in the region.
A main challenge in using such a representation is how to
describe each local patch. Because mTBI does not necessarily
impact all patches, we cannot infer the patch-level labels from
subject-level labels. Therefore we cannot learn patch-level features through supervised learning. To overcome this problem,
we apply an unsupervised learning approach at the patch level,
and train an auto-encoder to learn features that can be used
to reconstruct patches. In the following, we first explain how
do we learn patch level features through training an autoencoder, and then describe how do we aggregate the patchlevel features using a bag of words representation for each
region, and finally how to combine the region representations
and other information for patient-level classification.

Once the features are extracted from each patch, we use
the bag of words (BoW) representation [9] to describe each
brain region, which calculates the histogram of representative
patterns (or visual words) over all patches in this region. To
find the visual words, we apply the K-means clustering algorithm to the patch features obtained for all training patches.
Given the MR images of a subject, we extract overlapping
patches from each of two brain regions (Thalamas and CC)
Each patch is quantized to its closest visual words, and each
metric in each region is described by a histogram of different
visual words among all patches in this region. In the case
where we trained a single auto-encoder to generate a single
feature representation for all metrics together, each region is
represented by a single histogram. The block diagram of the
BoW approach is shown in Figure 3.

A. Patch Feature Learning Using an Auto-Encoder
In order to learn patch-level features without having labels,
we use a convolutional auto-encoder [10]-[11], which is an
unsupervised feature learning approach. Convolutional autoencoder gets an image patch as the input and performs multiple
convolution plus downsampling layers to encode the image
into some latent representation, and then uses these features
to reconstruct the original patch through deconvolution. By
doing so, the network is forced to learn some representative
information that is sufficient to recover the image. The overall
architecture of a convolutional auto-encoder is shown in Figure
2. Here, each layer of the network performs three operations:
convolution, nonlinear transformation, and pooling (downsampling). After training this model, the latent representation in
the mid-layer is used as patch feature representation.
In our study we consider two scenarios when training the
convolutional auto-encoder. In one scenario, we train one
model for each metric (such as FA, MK, RK, etc.). In the
other scenario, we concatenate all metrics together (treat them
as different channels) to form a 3D patch and train one

Fig. 3. The block-diagram of the proposed BoW approach

C. Feature Selection and Classification
After deriving deep-bag-of-words features from diffusion
MR images, we will get two feature vectors, one for Thalamus
region and the other for Corpus Callosum. We concatenate
the features from both regions, with demographic and neurocognitive test features, to form the final feature vector. We
then perform feature selection [12]-[13] to minimize the
risk of over-fitting before classification. We tried multiple
feature selection approaches such as max-relevance and minredundancy (MRMR) [14], maximum correlation, greedy forward selection, and it turns out that the greedy forward
feature selection works best for our problem. This approach

selects the best features one at a time with a given classifier,
through a cross-validation approach. For classification, we
tried different classifiers (such as SVM, neural network, and
random forest) and SVM was chosen because it generally gave
better performance.
III. E XPERIMENTAL R ESULTS
We evaluate the performance of the proposed approach on
our dataset of 114 subjects. This dataset contains 69 mTBI
subjects between 18 and 64 years old, within 1 month of mTBI
as defined by the American College of Rehabilitation Medicine
(ACRM) criteria for head injury and 40 healthy age and sexmatched controls.
To evaluate the model performance, we use a cross validation approach, where each time we randomly take 20% of the
samples for validation, and the rest for training. We repeat this
procedure 50 times (to decrease sampling bias), and report the
average validation error as the model performance.
For the convolutional auto-encoder, we learn features on
patches of 16x16 pixels in each image slice. The encoder and
decoder each have 4 layers, and the kernel size is always set
to (3,3). The latent feature dimension is 32 for the networks
which are trained on each metric, and 64 for the network which
is trained on the stack of all metrics. To train the model, the
batch size is set to 500, and the model is trained for 10 epochs.
The learning rate for the stochastic gradient descent is set
to 0.0003. The learnt auto-encoder is used to generate latent
features on each overlapping patch in the training images. The
resulting features are further clustered to N words using Kmeans clustering. N was varied among 20, 30, and 40. Each
MR metric in each region is represented by a histogram of
N dimension For SVM, we use radial basis function (RBF)
kernel. The hyper-parameters of SVM model (kernel width
gamma, and the mis-classification penalty weight, C) are tuned
based on the validation set.
In the first experiment, we compare the proposed approach
with some previous works. For the proposed approach, the
initial image feature representation is 260 dimensional, with
20 words for each of 8 MR metrics (AWF, DA, De par, FA,
MD, AK, MK, RK) in Corpus Callosum, and 5 MR metrics in
Thalamus (FA, MD, AK, MK, RK). Together with additional 2
demographic and clinical features, the total feature dimension
is 266. We compare our approach with previous works that
only used mean values of each MR metric in each region
[6], and also the work in [4], as well as BoW approach on
raw patches [6] (that is, we find the visual words by applying
the K-means clustering algorithm on the raw image patches
directly). It is worth to mention that the work in [4] was using
a different dataset, and set of metrics.
With forward feature selection using the SVM classifier, the
optimal feature subset contains 10 features FA in Thal, DA in
CC, AWF in CC, AWF in CC, AWF in CC, FA in CC, RK
in Thal, MK in Thal, RK in CC, FA in Thal. The comparison
with previous works is provided in Table II. As we can see,
Deep-BoW approach achieves reasonable improvement over
previously used features, except for BoW on raw patches

where it performs slightly worse than using the raw pixel
values. We believe the performance of Deep-BoW could be
further improved by better tuning the deep network hyperparameters.
TABLE II
P ERFORMANCE COMPARISON OF DIFFERENT APPROACHES
The Algorithm
Single best feature [6]
The selected subset with 8 features [6]
The algorithm in [4] using selected features
with MRMR, and Baysian Network
BoW on raw patches and 10 metrics [7]
The algorithm in [4] using selected features
with MRMR, and Neural Network
BoW on raw patches with 20D histograms [6]
The proposed Deep-BoW with 20D histograms
The proposed Deep-BoW with 30D histograms

Classification Accuracy
on Validation Set
72%
80%
82%
85.5%
86%
92%
87.8%
90.1%

We have also evaluated the classification performance for
feature subset of different size. Besides classification accuracy, we also report the sensitivity and specificity, which are
important in the study of medical data analysis. The sensitivity
and specificity are defined as in Eq. (1), where TP, FP, TN, and
FN denote true positive, false positive, true negative, and false
negative respectively. In our evaluation, we treat the mTBI
subjects as positive.
TP
TN
Sensitivity =
,
Specificity =
(1)
TP+FN
TN+FP
Figure 4 denotes the classification accuracies, sensitivities
and specificities achieved by optimum subset of feature of
dimension 1 to 10.

Fig. 4. The model performance for feature set of different size

In Table III, we provide the comparison between the two
scenarios, where in one of them one network is trained per
metrics, and in the other one a single network is trained over
the stack of multiple metrics (for different number of words).
To have a better generalization accuracy analysis, we also
provide the comparison in terms of the accuracy on a heldout
set. In each run, we randomly pick 20 samples as the heldout
set. We then run cross validations 50 times within the remaining data, to generate 50 models, and use the ensemble of 50

TABLE III
P ERFORMANCE COMPARISON FOR DIFFERENT APPROACHES
Auto-Encoder
Scenario

Bow-Hist
Dimension

Image Feature
Dimension

Multiple Network
Multiple Network
Single Network
Single Network

20
30
130
190

260
390
260
380

CrossValidation
Accuracy
87.8%
90.1%
89.4%
90%

models to make prediction on the held-out set and calculate
the classification accuracy. We repeat this 6 times, each time
with a different set of 20 heldout samples chosen randomly
and report the average accuracy. For this evaluation, we only
compare two BoW based approaches, one derived by applying
K-means on the raw patch images, and another by applying Kmeans on the patch level deep features. The average accuracy
for the heldout sets for these two approaches are given in Table
III. It is interesting that, although BoW using deep patch level
features had lower cross validation accuracy in Table I, it had
slightly higher heldout set accuracy. For BoW on raw patches,
the selected subset of features includes: Age, MK in Thal, RK
in Thal, Depar in CC, AWF in CC, Depar in CC, Depar in
CC, RK in Thal, MK in Thal, FA in CC. For BoW using the
deep features, the selected features include: FA in Thal, DA
in CC, AWF in CC, AWF in CC, AWF in CC, FA in CC, RK
in Thal, MK in Thal, RK in CC, FA in Thal. This comparison
is provided on Table IV.
TABLE IV
P ERFORMANCE COMPARISON OF DIFFERENT APPROACHES
The Algorithm
BoW on raw patches [6] using 20
words for each metric and region
The proposed Deep-BoW

Heldout Set Accuracy
86.2%
86.4%

Finally, we present the the average histogram representation of patients, and compare it with the average histogram
representation of control subjects. These histograms and their
difference are shown in Fig. 5. As we can see mTBI and
control subjects have clear differences in some part of these
representations.

Fig. 5. Deep-BoW histograms of patients and controls

IV. C ONCLUSION
In this work, we propose a machine learning framework
for mTBI identification from diffusion MR images. First,
deep unsupervised learning approach is used to learn feature
representation for image patches, and then patch level features
are aggregated in the form of the bag of word representation to form the overall image feature. These features are
used along with demographic and neuro-cognitive features for
patient identification. Then greedy forward feature selection
and support vector machine are used to perform classification.
Through experimental studies, we show that by learning deep
visual features, we obtain significant gain over using mean
values of various MR metrics in each brain region.
ACKNOWLEDGMENT
Research reported in this publication was supported by
National Institute of Neurological Disorders and Stroke of National Institutes of Health under award number R21NS090349.
The content is solely the responsibility of the authors and does
not necessarily represent the official views of the National
Institutes of Health.
R EFERENCES
[1] E Raz, JH Jensen, Y Ge, et al. “Brain iron quantification in mild traumatic
brain injury: a magnetic field correlation study”, American journal of
neuroradiology, pp.1851-1856, 2011.
[2] L Tang, Y Ge, DK Sodickson, L Miles, et al. “Thalamic resting-state
functional networks: disruption in patients with mild traumatic brain
injury”, Radiology, 260(3), pp.831-840, 2011.
[3] Zhou Y, Kierans A, Kenul D, Ge Y, Rath J, Reaume J, Grossman RI, Lui
YW, “Longitudinal Regional Brain Volume Changes in Mild Traumatic
Brain Injury Patients”, Radiology, in press 2013.
[4] YW Lui, Y Xue, D Kenul, Y Ge, RI Grossman, Y Wang, “Classification
algorithms using multiple MRI features in mild traumatic brain injury”,
Neurology 83.14: 1235-1240, 2014.
[5] VM Vergara, AR Mayer, E Damaraju, KA Kiehl, V Calhoun, “Detection
of mild traumatic brain injury by machine learning classification using
resting state functional network connectivity and fractional anisotropy”,
Journal of neurotrauma, 2017.
[6] S Minaee, S Wang, Y Wang, et al. “Identifying Mild Traumatic Brain
Injury Patients From MR Images Using Bag of Visual Words”, Signal
Processing in Medicine and Biology Symposium, IEEE, 2017.
[7] S Minaee, Y Wang, S Chung, et al. “A Machine Learning Approach For
Identifying Patients with Mild Traumatic Brain Injury Using Diffusion
MRI Modeling”, ASFNR 12th Annual Meeting, 2017.
[8] I Goodfellow, Y Bengio, A Courville, “Deep learning”, Cambridge: MIT
press, 2016.
[9] J Yang, YG Jiang, AG Hauptmann, CW Ngo, “Evaluating bag-ofvisual-words representations in scene classification”, Proceedings of the
international workshop on multimedia information retrieval, ACM, 2007.
[10] J Masci, U Meier, D Ciresan, J Schmidhuber, “Stacked convolutional
auto-encoders for hierarchical feature extraction”, International Conference on Artificial Neural Networks, 2011.
[11] B Leng, S Guo, X Zhang, Z Xiong, “3D object retrieval with stacked
local convolutional auto-encoder”, Signal Processing 112, 2015.
[12] I Guyon, A Elisseeff, “An introduction to variable and feature selection”,
Journal of machine learning research, 3(Mar), 1157-1182, 2003.
[13] G Chandrashekar, F Sahin, “A survey on feature selection methods”,
Computers and Electrical Engineering, Elsevier, 2014.
[14] H Peng, F Long, C Ding, “Feature selection based on mutual information
criteria of max-dependency, max-relevance, and min-redundancy”, IEEE
Transactions on pattern analysis and machine intelligence, 2005.

