Multi-modal analysis of genetically-related
subjects using SIFT descriptors in brain MRI
Kuldeep Kumar1,2 , Laurent Chauvin1 , Mathew Toews1 ,
Olivier Colliot2,3 and Christian Desrosiers1

arXiv:1709.06151v1 [cs.CV] 18 Sep 2017

1

LIVIA, École de technologie supérieure (ÉTS), Montreal, Canada
Aramis project-team, Inria Paris, Sorbonne Universités, UPMC Univ Paris 06,
Inserm, CNRS, Institut du cerveau et la moelle (ICM) - Hôpital Pitié-Salpêtrière,
Boulevard de l0 hôpital, F-75013, Paris, France
3
AP-HP, Departments of Neurology and Neuroradiology, Hôpital Pitié-Salpêtrière,
75013, Paris, France
2

Abstract. So far, fingerprinting studies have focused on identifying features from single-modality MRI data, which capture individual characteristics in terms of brain structure, function, or white matter microstructure. However, due to the lack of a framework for comparing across multiple modalities, studies based on multi-modal data remain elusive. This
paper presents a multi-modal analysis of genetically-related subjects to
compare and contrast the information provided by various MRI modalities. The proposed framework represents MRI scans as bags of SIFT
features, and uses these features in a nearest-neighbor graph to measure subject similarity. Experiments using the T1/T2-weighted MRI and
diffusion MRI data of 861 Human Connectome Project subjects demonstrate strong links between the proposed similarity measure and genetic
proximity.

1

Introduction

The human brain shows a significant amount of variability in terms of structure
[7], function [9], and white matter architecture [10]. Recently, various studies
have focused on characterizing this variability using brain fingerprints, for instance, based on shape [15], functional connectivity [1], white matter fiber geometry [6] or voxel-wise diffusion density [16]. The importance of these fingerprint
studies is motivated by the fact that brain characteristics are largely determined
by genetic factors [11], and that various neurological disorders like Parkinson [2]
and autism [4] have been linked to brain abnormalities.
Most fingerprint studies use magnetic resonance imaging (MRI) data, which
provides a non-invasive way to probe the structure, function and white matter
connectivity of the brain. Studies focusing on structural MRI modalities like T1or T2-weighted images derive their fingerprints mainly based on morphometry. In
[15], Wachinger et al. quantify the shape of cortical and sub-cortical structures
via the spectrum of the Laplace-Beltrami operator. The resulting representation, called Brainprint, is used for subject identification and analyzing potential

genetic influences on brain morphology. In [13], Toews et al. represent images
as a collection of localized image descriptors, and apply scale-space theory to
analyze their distribution at the characteristic scale of underlying anatomical
structures. This morphometry-based representation is employed to identify distinctive anatomical patterns of genetically-related individuals or subjects with
a known brain disease.
Various methods have also been proposed to characterize brain properties derived from diffusion MRI (dMRI) or functional MRI (fMRI). In [5], Kochunov et
al. analyze heritability of fractional anisotropy (FA) in white matter and compare
findings across subjects from different datasets. Likewise, Yeh et al. [16] build
a local connectome fingerprint using dMRI data, based on the diffusion density
at each voxel, and use their proposed fingerprint to analyze genetically-related
subjects. In [1], Finn et al. consider the correlation between time courses of atlasdefined nodes to generate a functional connectivity profile, and use this profile
to identify individuals across scan sessions, both for task and rest conditions.
Moreover, Miranda et al. [8] propose a linear model to describe the activity of
brain regions in resting-state fMRI as a weighted sum of its functional neighboring regions. Their functional fingerprint, derived from the model’s coefficients,
has the ability to predict individuals using a limited number of non-sequential
frames.
So far, brain fingerprinting studies have used single-modality imaging data
to characterize the structural, functional or white-matter connectivity profiles of
individuals. Combining multi-modal data in a single fingerprint would provide
a more discriminative characterization of individuals and help understanding
the unique contribution of each modality in this characterization. However, such
multi-modal fingerprinting analysis has been elusive, due to the lack of a common framework for comparing subjects across modalities. Motivated by this, we
propose a multi-modal analysis approach that combines information from structural MRI (T1 and T2) and diffusion MRI into a single framework. The idea is
to represent each image using local features and approximate image manifold by
computing pairwise subject similarity in an efficient way. Building from the work
presented in [12], this framework represents images as a bags of features (BoF),
where features are defined based on scale-invariant feature transform (SIFT) descriptors. This technique has the advantage of being alignment independent and,
thus, facilitates comparisons across different subjects. To find similar subjects,
we first build a nearest-neighbor (NN) graph by matching SIFT descriptors, and
then compute the pairwise geodesic distance between BoFs using the Jaccard
distance metric. In this work, we extend the framework described in [12] to include T1-weighted (without skull), T2-weighted, and diffusion-weighted MRI,
and present a comprehensive quantitative and qualitative analysis using 861
subjects from Human Connectome Project. To our knowledge, this study is the
first step towards using multi-modal information for subject fingerprinting and
analyzing genetically-related subjects.

2

Materials and Methods

We start by describing the data and preprocessing steps of our study. In Section
2.2, we then present the proposed approach for comparing subjects using multimodal data.
2.1

Data and preprocessing

We have used the pre-processed structure and diffusion MRI of 861 subjects (482
females, 378 male and 1 unknown, age 22-35) from the Q3 release of the Human
Connectome Project [14]. These images were acquired on a Siemens Skyra 3T
scanner and pre-processed as described in [3]. Further details can be obtained
from HCP Q3 data release manual1 .
For structural MRI we considered high-resolution T1-weighted (0.7 mm),
high-resolution T2-weighted (0.7 mm), and Freesurfer processed T1-weighted
(skull stripped, 1 mm). In the case of dMRI data, signal reconstruction was
performed with the freely available DSI Studio toolbox [17]. All subjects were
reconstructed in MNI space (at 1 mm resolution) using the Q-Space diffeomorphic reconstruction (QSDR) option in DSI Studio. Three diffusivity measures
were extracted to characterize white matter micro-structure: Generalized Fractional Anisotropy (GFA) and Normalized Quantitative Anisotropy (NQA0 and
NQA1). GFA extends the standard FA measure to orientation distribution functions (ODF) based on spherical harmonics. NQA is a scaled version of quantitative anisotropy, which is calculated from the peak orientations on a spin distribution function (SDF). More information about these measures can be found
in [17].
2.2

Multi-modal subject similarity

Computing similarities between pairs of subjects, based on their multi-modal
data, involves multiple steps. In the first step, 3D keypoints are located in the
scans of each subject by finding the local extrema (i.e., maxima or minima) of the
difference of Gaussians (DoG) occurring at multiple scales. Keypoints with low
contrast or corresponding to edge response are discarded, and remaining ones
encoded into a feature vector (i.e, the descriptor) using the histogram of oriented
gradients (HOG) within a small neighborhood. Note that these descriptors are
robust to changes in illumination, scale and rotation, and are thus efficient for
comparing images acquired using different scanners or imaging parameters. Each
subject is then represented as an orderless bag of features (BoF), containing
all the descriptors found in this subject’s scans. This representation provides a
simple and extensible way of incorporating data from multiple modalities.
Because the BoFs of two subjects may contain different numbers of descriptors, they cannot be directly compared. To circumvent this problem, we construct
1

http://www.humanconnectome.org/documentation/Q3/

an intrinsic manifold of subject appearance using a feature-to-feature nearestneighbor (NN) graph. In this graph, each descriptor is represented by a node
and is connected to its K most similar descriptors based on Euclidean distance.
Let A and B be the BoF of two subjects, the similarity between these subjects
can then be evaluated directly via the Jaccard measure:
J(A, B) =

|A ∩ B|
,
|A| + |B| − |A ∩ B|

(1)

where |A ∩ B| is the number of edges between nodes in A and those in B.

3

Experiments and results

We use the MotherID, Age, TwinStat, and Zygosity fields of the HCP twin
data to obtain 84 mono-zygotic (MZ) twin pairs, 84 di-zygotic (DZ) twin pairs,
and 217 non-twin sibling subjects (NT). The proposed method is applied on
each type of structure or diffusion image (i.e., T1, T2, GFA, NQA0, NQA1)
to identify pairs of genetically-related subjects. For simplification purposes, we
refer to those image types as modalities. These modalities are compared using
the recall@k measure, which is the proportion of genetically-related subjects in
the set of k most similar subjects, as defined in Eq. (1). For a given sibling type,
recall@k values are computed for all subjects having at least one sibling of this
type, and averaged across these subjects. We then asses the complementarity
of different modalities, by using them together for twin/sibling identification. A
qualitative analysis on a single non-twin sibling pair is also performed to visualize
the location and scale of features matches.
Parameter K determines the number of connections in the nearest-neighbor
graph, which can impact the proposed similarity measure. In our experiments,
we tested K ∈ {10, 20, . . . , 50} and found nearly the same results for all tested
values. In what follows, we report results obtained with K = 20.
3.1

Identification of genetically-related subjects

Figure 1 shows the mean recall@k, k = 1, . . . , 50, obtained for identifying MZ, DZ
and NT siblings. Note that we perform the task of sibling identification over all
861 subjects. The top row gives results obtained using single-modality data. T1FS represents lower-resolution T1 images pre-processed with FreeSurfer. These
images were included to evaluate the impact of scan resolution and for combining
structural and diffusion data. The bottom row gives results obtained using a
combination of two different modalities, except for T1+dMRI which corresponds
to T1-FS combined with all diffusion measures (i.e., GFA, NQA0, NQA1). To
consider the chance factor, we also provide the recall@k resulting from a random
subject similarity measure (denoted as rnd in the bottom row plots).
For the identification of MZ twins, we observe a mean recall near 100% for all
modalities, illustrating the high impact of genetic similarity on both structural
and diffusion geometry in the brain. Comparing MZ, DZ and NT siblings, we

MZ

DZ

NT

Fig. 1: Mean recall@k for MZ, DZ and NT sibling identification using singlemodality (top row) multi-modality data (bottom row).

see higher recall values for MZ twins compared to DZ twins or NT siblings,
supporting the fact that MZ twins share more genetic information. We also note
higher recall values for DZ twins compared to NT siblings, although both sibling
types share the same genetic proximity. These differences are significant with
p < 0.01 in a Wilcoxon signed-rank test. To rule out the impact of age difference,
we further divided NT sibling pairs into two groups, based on the median age
difference of 3 years, and repeated the same experiment with T1 images. Figure 2
show no statistical difference between these two NT sub-groups. Similarly, there
is no difference between MZ and DZ twins divided based on median age of 29.

MZ

DZ

NT

Fig. 2: Age impact on Recall@k for MZ, DZ and NT sibling identification using
T1 data. MZ and DZ twins are divided into two groups based on median age of
29. NT sibling pairs are divided based on median age difference of 3 years.

Comparing individual modalities, we note that T1 leads to the highest recall
values for identifying all sibling types. While T1 and T2 are highly correlated

modalities, more feature matches were obtained for T1, suggesting this structural
modality to be more relevant for the task of twin/sibling identification (see Figure
3). Moreover, the benefit of using a higher resolution is illustrated by the better
performance of T1 compared to T1-FS, although these differences could also be
explained by the lack of skull in T1-FS images.
In general, diffusion measures (GFA, NQA0, NQA1) lead to smaller recall
values than T1 or T2. This could be due to the fact that, for such images, the
features are mainly located inside or near to white matter, as opposed to the
whole brain for T1/T2 (see Figure 3). Comparing diffusion measures, GFA and
NQA0 perform nearly the same, GFA having slightly higher recall values for
MZ and NT sibling identification. This is possibly due to the fact that GFA
contains information from the full diffusion profile, while NQA0 is based on the
most prominent diffusion direction. NQA1, which contains information along the
second most prominent diffusion direction, performs poorly compared to other
diffusion measures.
Considering results obtained with multiple modalities, we note that the combination of T1 and T2 gives a higher recall than using these modalities individually. For instance, T1+T2 gives a recall@50 value of 0.706 for NT identification,
compared to 0.564 and 0.549 for T1 and T2, respectively. Similarly, combining
T1 and all diffusion measures (i.e., T1+dMRI) improves the mean recall values
significantly, for the identification of all sibling types (e.g., a recall@50 value of
0.722 for NT identification). These results validate our hypothesis that different
modalities provide complementary information.
3.2

Scale-space visualization of features matches

Figure 3 provides a scale-space visualization of feature matches for a single pair of
MZ twins and NT siblings, where scale information is represented using the circle
radius. Note that circles represent the intersection of 3D spheres with the visible
slice and, thus, non-intersecting features are hidden in this 2D visualization.
It can be seen that different image modalities generally result in distinct,
complementary feature correspondences throughout the brain, allowing a rich
characterization of both anatomical and connectivity structure. In T1 and T2
images, features are mainly located in the frontal lobe, corpus callosum and brain
stem. Smaller-scale features are also visible along various cortical regions, as well
as in sub-cortical structures near the basal ganglia. Although highly correlated,
T1 images show significantly more feature matches than T2 images. Moreover,
images based on diffusion measures have less matches than in structural modalities. These matches are located mostly inside or near to white matter: larger-scale
features in the corpus-callosum, and smaller-scale ones in the brain stem and
along white matter bundles. While not shown in the figure, the set of matches
found by combining two modalities (e.g., T1 + T2) generally corresponds to the
union of those obtained with these individual modalities.
Comparing different sibling types, we observe a greater number of matches
between MZ twins than NT siblings. This observation, which is easier to visualize in T2 and GFA images, is consistent with other analyses on twin datasets.

T1

T2

GFA

Fig. 3: Example of feature matches for a single pair of MZ twins (first and second
row) and non-twin siblings (third and fourth row). Scale space is represented
using circle radius (for the visible slice).

In terms of feature location and scale, no obvious pattern can be seen when
comparing these two sibling types. However, a more detailed analysis would be
required to validate this assertion.

4

Conclusion

We presented a framework based on SIFT descriptors for the multi-modal analysis of genetically-related subjects. This framework represents each subject as
an order-less sets (i.e., bag) of local invariant features, which can be extracted
from different modalities. An efficient strategy, computing Jaccard similarity on
a feature-to-feature nearest-neighbor graph, was proposed to evaluate the simi-

larity between subjects having a different number of features. In a quantitative
analysis, mean recall@k was used to measure the relation between the proposed
subject similarity and genetic proximity, and the contribution/complementarity
of information from different MRI modalities. In agreement with other twin
data analysis in the literature, results show mono-zygotic twins to be more similar than di-zygotic twins and non-twin siblings. Likewise, it was found that
di-zygotic twins are more similar than non-twin siblings, and that this similarity
is not entirely explained by age difference. Our analysis also showed structural
modalities, in particular T1, to be more useful than modalities measuring diffusion. However, combining both structural and diffusion data leads to a better
sibling identification.
This work could be extended by further investigating the differences, in terms
of feature location and similarity, between di-zygotic twins and non-twin siblings.
A deeper analysis of aging effects could also be performed, for instance, using
longitudinal data. Such analysis would help understand the effect of neuroplasticity on individual brain characteristics.
Acknowledgements: Data were provided in part by the Human Connectome
Project, WU-Minn Consortium (Principal Investigators: David Van Essen and
Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers
that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University.

References
1. Finn, E.S., Shen, X., Scheinost, D., Rosenberg, M.D., Huang, J., Chun, M.M., Papademetris, X., Constable, R.T.: Functional connectome fingerprinting: identifying
individuals using patterns of brain connectivity. Nature neuroscience (2015)
2. Geevarghese, R., Lumsden, D.E., Hulse, N., Samuel, M., Ashkan, K.: Subcortical structure volumes and correlation to clinical variables in parkinson’s disease.
Journal of Neuroimaging 25(2), 275–280 (2015)
3. Glasser, M.F., Sotiropoulos, S.N., Wilson, J.A., Coalson, T.S., Fischl, B., Andersson, J.L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J.R., et al.: The minimal
preprocessing pipelines for the human connectome project. Neuroimage 80, 105–
124 (2013)
4. Goldman, S., O’Brien, L.M., Filipek, P.A., Rapin, I., Herbert, M.R.: Motor stereotypies and volumetric brain alterations in children with autistic disorder. Research
in autism spectrum disorders 7(1), 82–92 (2013)
5. Kochunov, P., Jahanshad, N., Marcus, D., Winkler, A., Sprooten, E., Nichols, T.E.,
Wright, S.N., Hong, L.E., Patel, B., Behrens, T., et al.: Heritability of fractional
anisotropy in human white matter: a comparison of human connectome project
and enigma-dti data. Neuroimage 111, 300–311 (2015)
6. Kumar, K., Desrosiers, C., Siddiqi, K., Colliot, O., Toews, M.: Fiberprint: A subject
fingerprint based on sparse code pooling for white matter fiber analysis. NeuroImage 158, 242 – 259 (2017), http://www.sciencedirect.com/science/article/
pii/S1053811917305669

7. Mangin, J.F., Riviere, D., Cachia, A., Duchesnay, E., Cointepas, Y., PapadopoulosOrfanos, D., Scifo, P., Ochiai, T., Brunelle, F., Regis, J.: A framework to study
the cortical folding patterns. Neuroimage 23, S129–S138 (2004)
8. Miranda-Dominguez, O., Mills, B.D., Carpenter, S.D., Grant, K.A., Kroenke, C.D.,
Nigg, J.T., Fair, D.A.: Connectotyping: model based fingerprinting of the functional
connectome. PloS one 9(11), e111048 (2014)
9. Mueller, S., Wang, D., Fox, M.D., Yeo, B.T., Sepulcre, J., Sabuncu, M.R., Shafee,
R., Lu, J., Liu, H.: Individual variability in functional connectivity architecture of
the human brain. Neuron 77(3), 586–595 (2013)
10. de Schotten, M.T., Bizzi, A., Dell’Acqua, F., Allin, M., Walshe, M., Murray, R.,
Williams, S.C., Murphy, D.G., Catani, M., et al.: Atlasing location, asymmetry
and inter-subject variability of white matter tracts in the human brain with MR
diffusion tractography. Neuroimage 54(1), 49–59 (2011)
11. Thompson, P.M., Ge, T., Glahn, D.C., Jahanshad, N., Nichols, T.E.: Genetics of
the connectome. Neuroimage 80, 475–488 (2013)
12. Toews, M., Wells, W.M.: How are siblings similar? how similar are siblings? largescale imaging genetics using local image features. In: 2016 IEEE 13th International
Symposium on Biomedical Imaging (ISBI). pp. 847–850 (April 2016)
13. Toews, M., Wells, W., Collins, D.L., Arbel, T.: Feature-based morphometry: Discovering group-related anatomical patterns. NeuroImage 49(3), 2318–2327 (2010)
14. Van Essen, D.C., Smith, S.M., Barch, D.M., Behrens, T.E., Yacoub, E., Ugurbil,
K., Consortium, W.M.H., et al.: The wu-minn human connectome project: an
overview. Neuroimage 80, 62–79 (2013)
15. Wachinger, C., Golland, P., Kremen, W., Fischl, B., Reuter, M., Initiative, A.D.N.,
et al.: Brainprint: a discriminative characterization of brain morphology. NeuroImage 109, 232–248 (2015)
16. Yeh, F.C., Badre, D., Verstynen, T.: Connectometry: A statistical approach harnessing the analytical potential of the local connectome. NeuroImage 125, 162–171
(2016)
17. Yeh, F.C., Wedeen, V.J., Tseng, W.Y.I.: Generalized q-sampling imaging. IEEE
transactions on medical imaging 29(9), 1626–1635 (2010)

