JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

1

Unsupervised End-to-end Learning
for Deformable Medical Image Registration

arXiv:1711.08608v2 [cs.CV] 20 Jan 2018

Siyuan Shan, Wen Yan, Xiaoqing Guo, Eric I-Chao Chang, Yubo Fan and Yan Xu*

Abstract—We propose a registration algorithm for 2D CT/MRI
medical images with a new unsupervised end-to-end strategy
using convolutional neural networks. The contributions of our
algorithm are threefold: (1) We transplant traditional image
registration algorithms to an end-to-end convolutional neural
network framework, while maintaining the unsupervised nature
of image registration problems. The image-to-image integrated
framework can simultaneously learn both image features and
transformation matrix for registration. (2) Training with additional data without any label can further improve the registration
performance by approximately 10%. (3) The registration speed
is 100x faster than traditional methods. The proposed network
is easy to implement and can be trained efficiently. Experiments
demonstrate that our system achieves state-of-the-art results on
2D brain registration and achieves comparable results on 2D liver
registration. It can be extended to register other organs beyond
liver and brain such as kidney, lung, and heart.
Index Terms—Image registration, Unsupervised, Convolutional
networks, End-to-end, Image-to-image

I. I NTRODUCTION

M

EDICAL image registration plays an important role
in medical image processing and analysis. As far as
brain registration is concerned, accurate alignment of the
brain boundary and corresponding structures inside the brain
such as hippocampus is crucial for monitoring brain cancer
development. As illustrated in Figure 1, image registration
refers to the process of revealing the spatial correspondence
between two images. Several image registration toolkits such
as ITK [1], ANTs [2] and Elastix [3] have been developed to
facilitate research reproduction.
A wide variety of medical registration algorithms have been
developed in the past [4], [5], [6], [3], [7], [8], focusing
primarily on unsupervised methods. These algorithms select
a transformation model, define a metric that measures the
This work is supported by Microsoft Research under the eHealth program,
the National Natural Science Foundation in China under Grant 81771910, the
National Science and Technology Major Project of the Ministry of Science
and Technology in China under Grant 2017YFC0110903, the Beijing Natural
Science Foundation in China under Grant 4152033, the Technology and
Innovation Commission of Shenzhen in China under Grant shenfagai2016627, Beijing Young Talent Project in China, the Fundamental Research Funds
for the Central Universities of China under Grant SKLSDE-2017ZX-08 from
the State Key Laboratory of Software Development Environment in Beihang
University in China, the 111 Project in China under Grant B13003. Asterisk
indicates corresponding author.
Siyuan Shan, Wen Yan, Xiaoqing Guo, Yubo Fan and Yan Xu are with
State Key Laboratory of Software Development Environment and Key Laboratory of Biomechanics and Mechanobiology of Ministry of Education and
Research Institute of Beihang University in Shenzhen, Beihang University,
Beijing 100191, China (email: shansiliu@outlook.com; yanwen@buaa.edu.cn;
guoxiaoqing@buaa.edu.cn; yubofan@buaa.edu.cn; xuyan04@gmail.com).
Eric I-Chao Chang, and Yan Xu are with Microsoft Research, Beijing
100080, China (email:echang@microsoft.com; xuyan04@gmail.com).

(a)
Fixed

(b)
Moving

(c)
Deformation

(d)
Warped

Fig. 1: Illustration of image registration. Given a fixed image (a) and a moving image
(b), a deformation field (c) is predicted to warp the moving image so that (d) and (a)
are spatially aligned. Note that the deformation field is color coded according to .

similarity of two images to be registered, and iteratively update
the transformation parameters or deformation field to optimize
the defined metric. A fraction of registration algorithms is
learning-based [9]. For learning-based approaches: (1) informative feature representations are difficult to obtain directly
from learning and optimizing morphing or similarity function;
(2) unlike image classification and segmentation, registration
labels are difficult to collect. These two reasons limit the
development of learning-based registration algorithms.
Recently, the field of computer vision has witnessed a
tremendous advancement triggered by deep learning technologies like convolutional neural networks (CNNs). CNNs have
proven their mettle in handling image classification [10], [11],
object detection [12] as well as pixel-wise prediction tasks like
semantic segmentation [13] and edge detection [14]. Apart
from these tasks where only a single image is processed,
CNNs also have the capacity to tackle image matching and
registration problems. For example, Zbontar and LeCun [15]
train a CNN to predict the similarity of two image patches for
subsequent stereo matching. Wei et al. [16] utilize CNNs as
a feature extraction tower and compute dense human body
correspondence according to the feature vectors extracted.
These works qualify CNNs as a potential tool for medical
image registration.
There have been few works to use CNNs for medical
image registration [17]. Yang et al. [18] design a deep
encoder-decoder network to initialize the momentum of the
large deformation diffeomorphic metric mapping (LDDMM)
registration model. However, their method is a patch-based
algorithm and thus requires postprocessing that cannot be
handled inside CNNs. Wu et al. [19] adopt unsupervised deep
learning to obtain features for image registration. Though
good performance is achieved, their method is also a patchbased learning system and relies on other feature-based registration methods to perform image registration. Miao et al.
[20] adopt CNN regressors to directly predict transformation

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

2

Fig. 2: Illustration of the unsupervised training strategy of our fully convolutional image-to-image registration network. The registration network takes two images and outputs a
deformation field, which is used to produce the sampling grid. The moving image is then warped by the sampling grid via bilinear interpolation. The loss function is defined as the
photometric difference between the warped image and the fixed image. The registration error can be efficiently back propagated to update the learnable parameters of the registration
network for end-to-end training.

parameters for 2D/3D images and achieve higher registration
success rates than traditional methods. But their model is not
trained end-to-end and cannot perform deformable registration.
Compared with patch-based training systems, image-to-image
prediction can be performed by fully convolutional neural
networks (FCNs) [13] where pixel-wise features are predicted.
Therefore, a CNN model that can perform image-to-image
deformable registration through end-to-end FCNs is desired.
FlowNet [21] is an appropriate CNN that can directly
predict optical flow from two input images using end-toend fully convolutional networks (FCNs) with competitive
accuracy at frame rates of 5 to 10 fps. FlowNet is trained on a
synthetic dataset in a supervised manner, where natural image
pairs with ground-truth registration parameters are generated
via computer graphic techniques. However, unlike natural
images, realistic medical images are difficult to generate. Consequently, the learning-based methods have not been widely
used to solve medical image registration problems [17]. To
this point, it is highly desired to develop an unsupervised
learning framework with end-to-end CNNs for medical image
registration, which implicitly learns to predict registration
parameters or deformation without ground-truth supervision.
The spatial transformer network (STN) proposed by Jaderberg et al. [22] enables neural networks to spatially transform
feature maps. The process of STN is as follows: STN first
generates a sampling grid according to the transformation
parameters produced by neural networks. An input image can
be spatially warped by the sampling grid. The warping process
is implemented by bilinear interpolation, which makes STN
fully-differentiable. Several STN-based approaches have been
proposed to address similar problems in natural scenes, such as
optical flow estimation [23], [24] depth estimation [25], [26]
and single-view reconstruction [27]. Inspired by the recent
success of STN [22], we develop an unsupervised learning
framework by combining the spatial transformer with fully
convolutional neural networks for 2D medical image registration. The integrated framework can simultaneously learn both
image features and transformation matrix for registration. We
define the pixel-wise difference between the warped moving
image and the fixed image as the loss function, the registration

error can be effectively backpropagating to CNNs for learning the optimal transformation parameters that minimize the
registration error. As shown in Figure 2, this training strategy
is very similar to the mechanism of traditional registration
algorithms where no ground-truth deformation is required.
In this paper, we build an end-to-end unsupervised learning
system with fully convolutional neural networks in which
image-to-image medical image registration is performed as
illustrated in Figure 2. Compared with FlowNet, the algorithm
does not require a synthetic dataset for supervised learning.
Compared with STN, our method can perform image registration in a deformation field form while STN can only perform
classification; our method is for template alignment while STN
is for class alignment. Besides, as an unsupervised learning
model, its registration performance can be easily improved by
introducing additional training data without any label.
To summarize, in this paper we develop unsupervised
convolutional neural networks for 2D tissue registration via
direct deformation field prediction. The contributions of our
algorithm are threefold: (1) Our algorithm is an end-to-end
CNN-based learning system under an unsupervised learning
setting that performs image-to-image registration. (2) Training
with additional data without any label can further improve
the registration performance. (3) We achieve a 100x speed-up
compared to traditional image registration methods.
II. R ELATED W ORK
Here we first describe research directly related. Then, the
key components of the traditional algorithms are summarized
and several works that tackle image registration problems with
CNN are outlined.
A. Directly Related Works
Three existing approaches that are closely related to our
work are discussed below.
Dosovitskiy et al. [28] propose an end-to-end fully convolutional neural net FlowNet for optical flow estimation in real
time. FlowNet has an encoder-decoder architecture with skip
connections. It predicts optical flow at multiple scales and each
scale is predicted based on the previous scale. Compared with

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

the nature of supervised learning of Flownet, an unsupervised
architecture is utilized in this work to predict deformation field
that aligns two images.
Jaderberg et al. [22] propose the spatial transformer networks (STN) which focuses on class alignment. It shows that
spatial transformation parameters (e.g. affine transformation
parameters, B-Spline transformation parameters, deformation
field, etc) can be implicitly learned without ground-truth
supervision by optimizing a specific loss function [22]. STN is
a fully differentiable module that can be inserted into existing
CNNs, which makes it possible to cast the image registration
task as an image reconstruction problem. Few papers focus on
the registration task using STN. In this paper, we use the STN
to make registration alignment in the medical image field.
B. Traditional Medical Image Registration Algorithms
A variety of traditional medical image registration algorithms have been proposed over the past few decades [4], [5],
[6], [3], [7]. A successful image registration application requires several components that are correctly combined, namely
the definition of the cost function, the multiresolution strategy,
and the coordinate transformation model.
Cost functions, also called similarity metrics, measure how
well two images are matched after transformation. The cost
function is one of the most crucial parts of a registration
algorithm. It is selected with regards to the types of objects to
be registered. Commonly adopted cost functions are the mean
squared difference [29], mutual information [30], normalized
mutual information [31] and normalized correlation [32]. A
regularization term is often required to penalize undesired
deformations [33].
The multiresolution strategy [34] is a widely adopted technique to increase registration speed and improve the stability of
the optimization. A sequence of reduced resolution versions of
input images is created, which forms a pyramid representation.
Then registration is performed at each level of the pyramid
from coarse to fine resolution consecutively, with the initial
transformation of the next level being the resulting transformation of the previous level.
Coordinate transformation models are determined according
to the complexity of deformations that need to be recovered.
Though in some cases parametric transformation models (such
as rigid, affine and B-Splines transformation) are enough to
recover the underlying deformations [30], [35], a more flexible
non-parametric transformation model allowing for arbitrary
local deformations is usually needed [4], [5]. Non-parametric
registration aims to find a dense deformation field where each
pixel is individually displaced to get a reasonable alignment
of the images. In this work, we only consider non-parametric
transformation models.
To date, traditional registration algorithms have achieved
satisfactory performance on various datasets. However, they
have a non-negligible drawback. For each pair of unseen
images to be registered, traditional registration methods iteratively optimize the cost function from scratch, which seriously
limits the registration speed and totally neglects the inherent
registration patterns shared across images from the same

3

dataset. In this work, we propose a fully convolutional and
image-to-image registration framework to overcome the above
mentioned drawbacks while maintaining competitive registration performance. It is also shown that the three components of
classical methods can be easily transplanted to existing CNN
frameworks.
C. Supervised Learning Methods
There have been few works to use supervised CNNs in a
patch-based manner for medical image registration. Yang et
al. [18] design a deep encoder-decoder network to initialize
the momentum of the large deformation diffeomorphic metric
mapping registration model. Sokooti et al. [36] train a 3D
CNN to register chest CT data using artificially generated
displacement vector field. Their method is also patch-based.
Compared with patch-based training systems, image-toimage prediction can be performed by fully convolutional
neural networks (FCNs) [13] where pixel-wise features are
predicted. Fischer et al. [28] propose a novel CNN model for
optical flow prediction. This model is trained end-to-end on a
synthetic dataset and can perform image-to-image optical flow
prediction.
Though all of these works achieve competitive performance,
they are trained on synthetic datasets [36], [28] or datasets
using the results of classical methods as ground truth [18].
D. Unsupervised Learning Methods
To obviate the need to collect real data with abundant
and reliable ground-truth annotations, unsupervised learning
methods become prevalent. Wu et al. [19] adopt unsupervised
deep learning to obtain features for image registration. Though
good performance is achieved, their method is a patch-based
learning system and relies on other feature-based registration
methods to perform image registration. Ren et al. [23] and
Yu et al. [24] use the spatial transformer networks (STN)
[22] and optical flow produced by a CNN to warp one frame
to match its previous frame. The difference between two
frames after warping is used as the loss function to optimize
the parameters of CNN. Their unsupervised methods do not
require any ground-truth optical flow. Similarly, Garg et al.
[25] use an image reconstruction loss to train a network for
monocular depth estimation. This work is further ameliorated
by incorporating a fully differentiable training loss and leftright consistency check [26]. We follow the idea of these
works to train a model for image-to-image registration in
an unsupervised manner. An auxiliary loss function is also
introduced to regularize the deformation field as will be
discussed in the next section.
III. M ETHODS
This section introduces the problem of image registration
and describes our image registration network. We introduce
a novel training loss for the problem that does not require
supervision in the form of ground truth deformation. Figure 3
illustrates our neural networks for medical image registration.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

4

Fig. 3: Illustration of the detailed structures of our image registration network. The
size and number of channels of each feature map are shown. The network takes two
concatenated images to be registered as input and predicts the deformation field at 7
different scales during training. During testing, only the deformation field at the largest
scale is used. The skip architecture of FlowNet is not shown for simplicity.

A. Problem Statement
In image registration, one image IM called the moving
image is deformed to match another image IF called the fixed
image according to a two-dimensional dense deformation field
u. The deformed image I˜ is expressed as
I˜ (x) = IM (x + u(x)) ,

(1)

where x denotes a two-dimensional location. In this work,
we attempt to estimate the optimal deformation field u(x) for
accurate image registration.
B. Unsupervised Image Registration Network
A fully convolutional network is adapted to model the
complex non-linear transformation from two input images to a
deformation field that aligns the input images. The deformation
field prediction module is inspired by FlowNet [28]. FlowNet
is a fully convolutional neural net originally proposed to solve
the optical flow estimation problem. It takes two input images
and outputs a dense optical flow/deformation field that aligns
the two input images. FlowNet consists of a contracting part to
capture context and an expanding part of intelligent flow field
refinement. Skip connections are also included to combine
high-level and low-level features. FlowNet predicts optical
flow at multiple scales and each scale is predicted based on
the previous scale. This design resembles the multiresolution
strategy [34] adopted by traditional registration algorithms and
improves the robustness of deformation field prediction. We
adopt FlowNetSimple architecture [28] with one modification.
The output optical flow field of FlowNet is 4 times smaller
than the input. To obtain a dense deformation field that has the
same resolution as the input image, we repeat the upsampling
block of FlowNet twice. The detailed structure of the proposed
image registration network is shown in Figure 3.
FlowNet [28] is originally trained in a supervised manner
by minimizing the endpoint error (EPE) between the predicted
flow vector and the ground truth flow averaged over all
pixels. For image registration problems, however, ground truth
deformation is difficult to collect. Though we can alternatively regard the result of traditional registration algorithms
as ground truth to train the network, they are not universal
solutions considering the variability of medical images and
the inaccuracy of traditional algorithms.
The problem mentioned above is mainly caused by the
inherent incongruity between the unsupervised nature of the
image registration problem and the supervised training strategy

of CNN. Concretely, traditional registration algorithms are not
learning-based and thus unsupervised. A similarity metric that
forces two images to appear similar to each other is optimized
and there are no learnable parameters in traditional algorithms. The philosophy of CNN, however, is utterly different.
CNN is a high-capacity learning model containing millions
of learnable parameters. It is usually trained in a supervised
manner where ground-truth class labels or segmentation masks
are provided. Therefore, a modification of the classical CNN
architecture is required to transplant traditional registration
algorithms to the deep learning framework.
In this work, the spatial transformer network (STN) [22] is
inserted to our image registration network for unsupervised
learning. STN is selected for two reasons. First, as it can
spatially warp feature maps or images inside neural networks,
the warped moving image can be successfully produced to
constitute the photometric loss function. Second, its fullydifferentiable property makes it possible to train the registration network end-to-end. STN contains a regular spatial
grid generator and a sampler. The deformation field predicted
by our image registration network is used to transform the
regular spatial grid into a sampling grid. Then, the sampler
uses the sampling grid to warp the input image. Bilinear interpolation is adopted during the sampling, which makes STN
fully differentiable for backpropagation. By defining the pixelwise differences (photometric differences) between the warped
moving image and the fixed image as the loss function, the
image registration problem becomes an image reconstruction
problem. The loss functions of our image registration network
are defined as follows.
Photometric Difference Loss We define a photometric loss
Lsphotometric at each output scale s of FlowNet and let us (x)
denote the predicted deformation field at output scale s.
Lsphotometric is L1 photometric image reconstruction error
defined as
X
Lsphotometric =
||I˜s (x) − IFs (x)||,
(2)
x∈Ω
s
where I˜s (x) = IM
(x + us (x)) is the moving image resized
to scale s and warped by STN according to deformation field
us (x), IFs (x) is the fixed image resized to scale s and Ω is the
two-dimensional image plane. This loss function encourages
the warped image to appear similar to the fixed image.
Deformation Field Smoothness Loss A regularization term
Lsmooth is generally needed to encourage the estimated deformation field to be locally smooth. In this work two types of
regularization terms LsmoothN and LsmoothE are compared.
LsmoothN is a normal L1 penalty on the deformation field
gradient ∂us (x),

LssmoothN =

X

|∂x us (x)| + |∂y us (x)|,

(3)

x∈Ω

where ∂x and ∂y respectively denote partial derivatives along
horizontal and vertical directions.
LsmoothE is the L1 penalty weighted by an edge-aware term
[37] as deformation field discontinuities often occur at image

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

5

gradients,

C. Parameters of Traditional Registration Algorithms

LssmoothE =

X

|∂x us (x)|e

s
−||∂x IF
(x)||

s
−||∂y IF
(x)||

+|∂y us (x)|e

x∈Ω

(4)
Total Loss The total loss L is the weighted sum of the above
defined losses,
L=

7
X

αs Lsphotometric + βs Lssmooth ,

(5)

s=1

where there are seven output scales of the proposed deformation field prediction module.
Our framework is an end-to-end learning system and allows
for fast and accurate deformation field prediction. In our
experiment, we show that the proposed registration network
performs well compared to traditional methods both in terms
of accuracy and speed.

.

The registration parameters used for the experimentation can
be found at the source codes of ITK [41]. ‘DeformableRegistration x.cxx’ in this website corresponds to ‘itk x’ in Table II
and IV. For Elastix, BSpline is selected as the transformation
model. The source code of Elastix used for registering MRI
brain dataset can be found at [42] and for registering liver
CT dataset can be found at [43]. For ANTs, the following
parameter settings are adopted to register MRI brain dataset:
antsRegistration -d 2 -m CC[${fixed},${moving},1,4]
-t SyN[0.5] -c [600x600x50x10x0,0,5]
-s 5x4x3x2x1vox -f 5x4x3x2x1 -u 1 -o ${prefix}

we adopt the following parameter settings to register liver CT
dataset:
antsRegistration -d 2 -m CC[${fixed},${moving},1,4]
-t SyN[0.5] -c [600x600x50x10x0,0,5]
-s 8x6x4x2x1vox -f 8x6x4x2x1 -u 1 -o ${prefix}

D. Evaluation
IV. E XPERIMENT
We evaluate the proposed algorithm with extensive experiments on two datasets with the ground truth of the corresponding landmarks and segmentation boundaries. These two
datasets respectively contain MRI brain images and CT liver
images. The proposed unsupervised registration algorithm is
compared to traditional registration algorithms provided by
off-the-shelf toolboxes like Advanced Normalization Tools
(ANTs) [38], Elastix [3] and Insight Segmentation and Registration Toolkit (ITK) [1]. We also report the results of our
baseline methods (supervised) where the outputs of the above
mentioned traditional algorithms are regarded as ground truth
to train the registration network.

We use the same set of evaluation metrics for all datasets.
1) Jaccard Coefficient: The first metric is Jaccard coefficient that measures the overlap of ground truth segmentation
masks. It is defined as |A ∩ B|/|A| ∪ B| where A is the
segmentation mask of the fixed image and B is the deformed
segmentation mask of the moving image.
2) Distance Between Corresponding Landmarks: The second metric is introduced to measure the capacity of algorithms
to register fine-grained structures. The registration error on
a pair of images is quantified as the average 2D Euclidean
distance between a landmark in the warped image and its
corresponding landmark in the fixed image.
E. Experiments on MRI Brain Registration

A. Our baseline (Supervised Method)
The registration network is also trained in a supervised
manner as done in FlowNet [28] to verify the efficacy of the
unsupervised method. More specifically, the deformation field
produced by traditional registration algorithms is used as the
ground truth to train the registration network. We regard the
network trained by this supervised strategy as the baseline of
our work.
B. Implementation
Our model is trained using Caffe [39]. K40 GPU and CUDA
7.0 are used for training acceleration. We choose Adam [40]
as the optimization method with β1 = 0.9 and β2 = 0.999.
The weight decay is 0.0005 and the batch size is 32. FlowNet
is finetuned from the FlowNetSimple model pretrained on the
Flying Chair dataset [28].
For the unsupervised method, the learning rate is 10−5 when
the training begins. We halve the learning rate after 10 epochs
and keep training for another 7 epochs. The parameters α
and β introduced in Section III-B that balance different loss
functions are set to 1 and 0.05 respectively for all datasets.
For the supervised method, the learning rate is 10−4 when
the training begins. We halve the learning rate after 10 epochs
and keep training for another 7 epochs.

1) Dataset: The T1-weighted MRI brain data with groundtruth segmentation are selected from the LONI Probabilistic
Brain Atlas (LPBA40) [44], which consists of images from 40
subjects. We discard twenty images with tilted head positions
and select the remaining twenty subjects. This dataset provides ground-truth segmentation masks. Eighteen well-defined
anatomic landmarks (see Figure 4) that are distributed mainly
in the lateral ventricle and the median sagittal plane [45]
are manually annotated by three doctors, and the average
coordinates from three doctors are considered as the groundtruth positions of the landmarks. The original size of the
3D brain MRI volume is 256×124×256 voxels, which are
zero-padded to 256×128×256 and resized to 256×256×256
voxels. Affine transformation (implemented by ANTs with
mutual information as the metric) is applied to each 3D brain
image before we slice 3D volume into 2D images. During the
training phase, a pair of slicing planes A and B at the same
position of the MRI volumes are interchangeably treated as a
pair of fixed and moving images. This procedure produces
a total of 291,840 (20×19×256×3, 20×19 subject pairs,
256 positions and 3 directions) 2D images for training and
evaluation.
Additional MRI brain data is needed to improve the performance of our unsupervised methods. We randomly select 35
normal patients provided by ADNI [46] (Alzheimer’s Disease

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

6

TABLE I: Performance of various methods with brain data. In our method, <
P, E, N > denote the inclusion of photometric loss Lphotometric , edge-aware
smothness loss LsmoothE and normal smothness loss LsmoothN , respectively. Then
the size of training data is enlarged to improve the performance of unsupervised method.
No registration means that no deformation occurs. Variations of various unsupervised
methods proposed in the paper are usually different in the training but mostly share the
same architecture in testing; their run time speeds are therefore approximately the same.

Fig. 4: This sketch illustrates the 18 landmarks selected in the brain dataset. L1: right
lateral ventricle posterior, L2: left lateral ventricle posterior, L3: anterior commissure
corresponds to the midpoint of decussation of the anterior commissure on the coronal
AC plane, L4: right lateral ventricle superior, L5: right lateral ventricle inferior, L6: left
lateral ventricle superior, L7: left lateral ventricle inferior, L8: middle of lateral ventricle,
L9: posterior commissure corresponds to the midpoint of decussation, L10: right lateral
ventricle superior, L11: left lateral ventricle superior, L12: middle of lateral ventricle,
L13: corpus callosum inferior, L14: corpus callosum superior, L15: corpus callosum
anterior, L16: corpus callosum posterior tip of genu corresponds to the location of the
most posterior point of corpus callosum posterior tip of genu on the midsagittal planes,
L17: corpus callosum fornix junction, L18: pineal body.

Neuroimaging Initiative) to enlarge our training set. The
original size of additional MRI brain data is 196×256×256
voxels, which is rotated into 256×196×256 and resized into
256×256×256 voxels. Then, the additional MRI brain data
is preprocessed in the same way as the data of the LONI
Probabilistic Brain Atlas (LPBA40) [44].
2) Experiment Results: Fourfold cross validation is adopted
in the experiment and the results are reported on the 20,520
(20×19×18×3) 2D slice pairs containing the same corresponding landmarks.
Table I quantitatively shows the performance of our unsupervised methods, our best baseline (supervised methods)
and the best traditional registration algorithms. Jaccard Coefficient (Jacc) and Distance Between Corresponding Landmarks
(Dist) are used as evaluation metrics. The running time (Rt)
for each algorithm to register a pair of images is reported.
The unsupervised methods PN and PE are 100x faster than
traditional methods while achieving superior registration performance (Dist and Jacc). Besides, the unsupervised methods
are superior to supervised methods. Figure 5 illustrates the
registration results of different methods.
Table II compares our supervised baseline methods to traditional registration algorithms, namely Insight Segmentation
and Registration Toolkit (ITK) [47], Elastix [3] and Advanced
Normalization Tools (ANTs) [38]. Note that our baseline
models are trained in a supervised manner by regarding the
results of the traditional algorithms in the same row of Table
IV as the ground truth.
F. Training with additional unlabeled data
One advantage of unsupervised learning algorithms is that
they do not require labeled training data. Therefore, the
training set can be easily enlarged to further improve the

Method
no registration
traditional method :
traditional best (Dist) - itk16
traditional best (Jacc) - ants
supervised (our baseline) :
our best baseline (Dist) - itk16
our best baseline (Jacc) - itk16
unsupervised (our method):
PN
PE
more unlabeled data (our method):
PN
PE

Dist
4.04

Jacc
0.908

Rt (s)
/

3.21
3.33

0.948
0.955

7.396
14.221

3.14
3.14

0.956
0.956

0.053
0.053

3.06
3.05

0.953
0.951

0.053
0.053

3.02
3.01

0.956
0.958

0.053
0.053

TABLE II: Performance of various traditional methods and our supervised baseline
methods with brain data.

itk1 (FEM)
itk2 (Demons)
itk3 (Demons)
itk5 (Demons)
itk13 (BSplines)
itk16 (Demons)
itk17 (Demons)
elastix
ants

Traditional method
Rt
Dist
Jacc
(s)
3.40
0.948 25.860
3.33
0.934 5.426
3.22
0.936 10.428
4.27
0.910 6.812
3.22
0.938 50.205
3.21
0.948 7.396
3.93
0.941 13.541
3.45
0.951 25.840
3.33
0.955 14.221

Our baseline
Dist

Jacc

4.54
3.34
3.14
3.59
3.26
3.14
3.62
3.43
3.16

0.897
0.934
0.940
0.920
0.936
0.956
0.940
0.950
0.954

Rt
(s)
0.053
0.053
0.053
0.053
0.053
0.053
0.053
0.053
0.053

registration performance without human labeling effort. As
mentioned in IV-E1, Additional MRI brain data of 35 patients
provided by ADNI [46] (Alzheimer’s Disease Neuroimaging
Initiative) are selected to enlarge our training set. In the
experiment, we retrain the unsupervised PN and PE model
with this enlarged dataset and obtain an improvement in the
registration performance. Notably, PE model achieves the best
performance with Dist decreasing from 3.05 to 3.01 and Jacc
increasing from 0.951 to 0.958, as is shown in Table I.
G. Experiments on CT Liver Registration
1) Dataset: The 3D liver CT dataset is provided by the
MICCAI 2007 Grand Challenge [48], which consists of images
from 20 subjects. We discard two anomalous subjects and
select the remaining 18 subjects. This dataset only provides
ground-truth segmentation masks, and the coordinates of the
landmarks are manually annotated by three doctors. Four
landmarks (L1, L2, L3, L4) on the liver portal vein are
selected (see Figure 6). Each doctor labels the coordinates
of the landmarks separately in 3D volumes via the ITKSNAP tool. The average coordinates from the three doctors are
considered the ground-truth positions of the landmarks. The
regions of interest (128×128×128 voxels) containing the liver
are extracted for further processing. As our network currently

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

7

Fig. 5: Illustration of the brain registration performance of the proposed unsupervised methods, our best supervised baseline methods and the best traditional registration algorithms
(itk16) with the best Dist: (a) Moving image, (b) Ground truth segmentation mask of moving image, (c) Fixed image, (d) Ground truth segmentation mask of fixed image. (e), (g),
(i) respectively denote the moving images warped by the best traditional registration algorithm (itk16), our best supervised baseline method (itk16), our best unsupervised method
(PE). The translucent red masks in (f), (h) and (j) respectively correspond to (e), (g), (i) and denote the warped ground truth segmentation mask of the moving images. The white
masks in (f), (h), (j) are the ground-truth segmentation mask of the fixed image. The red and yellow crosses denote landmarks of moving image and fixed image, respectively. Dist
in (c) and Jacc in (d) denote no registration.

only supports 2D input images, we slice 3D CT volume along
three orthogonal axes. Affine transformation (implemented by
ANTs with mutual information as the metric) is applied to
each 3D CT image before we slice 3D volume into 2D images.
During the training phase, a pair of slicing planes A and B
at the same position of the CT volumes are interchangeably
treated as a pair of fixed and moving images. This procedure
produces a total of 117,504 (18×17×3×128, 18×17 subject
pairs, 3 directions and 128 positions) 2D images for training
and evaluation.
Additional CT liver data provided by LiTS [49] (Liver
Tumor Segmentation Challenge) from Training Batch 1 (it
contains 3D CT volumes from 28 patients) are selected to
enlarge our training set. The regions of interest ( resized to
128×128×128 voxels) containing the liver are extracted for
further processing. These data are preprocessed in the same
way as the data of MICCAI 2007 Grand Challenge [48].
2) Experiment Results: Trifold cross validation is adopted
in the experiment and the results are reported only on the
3,672 (18×17×4×3×1) 2D slice pairs containing the same
corresponding landmarks. Table III quantitatively shows the
performance of our unsupervised methods, our best baseline
(supervised methods) and the best traditional registration algorithms. We retrain the unsupervised PN and PE model with this
enlarged dataset and obtain an improvement on the registration
performance. Notably, Dist of PE model decreases from 13.79
to 13.54 and Jacc increases from 0.837 to 0.845, as is shown in
Table III. Figure 7 illustrates the registration results of different
methods.
Table IV compares our baselines with several traditional registration algorithms. Our baselines run faster than traditional

TABLE III: Performance of various methods with liver data. In our method, <
P, E, N > denote the inclusion of photometric loss Lphotometric , edge-aware
smothness loss LsmoothE and normal smothness loss LsmoothN , respectively.Then
the size of training data is enlarged to improve the performance of unsupervised method.
No registration means that no deformation occurs. Variations of various unsupervised
methods proposed in the paper are usually different in the training but mostly share the
same architecture in testing; their run time speeds are therefore approximately the same.

Method
no registration
traditional method :
traditional best (Dist) - elastix
traditional best (Jacc) - itk17
supervised (our baseline) :
our best baseline (Dist) - itk16
our best baseline (Jacc) - ants
unsupervised (our method):
PN
PE
more unlabeled data (our method):
PN
PE

Dist
13.46

Jacc
0.665

Rt (s)
/

11.77
12.68

0.817
0.836

29.935
6.886

11.78
12.09

0.780
0.791

0.032
0.032

12.51
13.79

0.822
0.837

0.032
0.032

12.35
13.54

0.831
0.845

0.032
0.032

TABLE IV: Performance of various traditional methods and our supervised baseline
methods with liver data.

itk1 (FEM)
itk2 (Demons)
itk3 (Demons)
itk5 (Demons)
itk13 (BSplines)
itk16 (Demons)
itk17 (Demons)
elastix
ants

Traditional method
Rt
Dist
Jacc
(s)
12.84 0.729 11.014
13.39 0.704 2.954
13.28 0.772 5.103
13.90 0.785 4.312
12.95 0.725 15.540
11.94 0.807 8.809
12.68 0.836 6.886
11.77 0.817 30.602
12.51 0.822 5.131

Our baseline
Dist

Jacc

13.13
13.39
12.82
13.35
13.24
11.78
12.79
12.73
12.09

0.677
0.688
0.753
0.705
0.705
0.780
0.781
0.790
0.791

Rt
(s)
0.032
0.032
0.032
0.032
0.032
0.032
0.032
0.032
0.032

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

Fig. 6: This sketch illustrates the landmarks selected in the CT liver dataset. P1: hepatic
portal, P2: superior branch of left lobe, P3: inferior branch of left lobe, P4: medial branch
of left lobe, P5 and P8: anterior branch of right, P6: inferior branch of right lobe, P7:
superior branch of right lobe. L1, L2, L3, L4 are the selected landmarks, where L1 is
the intersection point of P6 and P7, L2 is the intersection point of P5 and P8, L3 is the
intersection point of P6 and P7 and L4 is the first bifurcation point of left portal.

registration algorithms and achieve superior performance both
in terms of Dist and Jacc.
Although our methods outperform traditional methods, the
performance of liver registration is still far from clinical
requirements. In order to enhance the accuracy of registration,
we introduce ROI segmentation mask into our unsupervised
pipeline as an extra component. The details of ROI segmentation mask are illustrated in supplement material.
V. C ONCLUSION
In this paper, we have developed an end-to-end framework
using unsupervised fully convolutional neural networks to
perform medical image registration. The proposed network
is trained in an unsupervised manner without any groundtruth deformation. Experiments demonstrate that our methods
achieve the state-of-the-art results on the MRI brain dataset
in both accuracy and registration speed, and achieve the
comparable results on the CT liver dataset. We achieve a 100x
speed-up compared to traditional image registration methods.
The scope of our proposed methods is quite broad and can
be widely applied to various medical image registration and
computer vision applications.
S UPPLEMENT M ATERIALS
Compared with brain images, the background of liver images is more complicated as there are various anatomical structures in the abdomen. The background noise, such as other
visceral organs or vessel, has an extremely negative impact
on liver registration. For such tasks, ROI segmentation is a
prerequisite for successful image registration as shown in [48],
[50], [51], as it can focus the registration process on specific
regions of interest (ROI) and avoid the undesired alignment of
artifacts. We also introduce ROI segmentation mask module,
which reduces background noise and interference. That is, we
first segment tissues to be registrated. After that, we registrate
the tissues based on the segmentation masks. Our system
pipeline with ROI segmentation mask module is shown in
Figure 8.
Our motivation to introduce ROI segmentation mask is twofold. First, our unsupervised flow networks under an imageto-image paradigm allow the ROI mask to be conveniently

8

implemented through back-propagation. Second, having ROI
mask theoretically and mathematically greatly enhances registration performance [48]; this is evident in our experiments
where a significant performance boost is observed using the
module. Our ROI segmentation mask module is performed
under a Holistically-nested nets (HNN) [14], [52] paradigm,
as it can perform multi-scale and multi-level learning under
deep supervision within FCNs. HNN is finetuned from the
pretrained 5-stage VGG [11]. The learning rate of HNN is
10−4 when the training begins for quick mask initialization.
After 4 epochs, the learning rate of the HNN part decreases
to 10−7 .
We find that using photometric difference loss alone results
in poor quality deformation field in some extreme cases where
there are serious illumination changes. To overcome these
problems, the ROI segmentation mask produced by HNN is
leveraged to guide the deformation field learning process. The
predicted segmentation mask is found to be both accurate and
robust to illumination changes, which motivates us to add a
ROI boundary overlapping loss Lsoverlap to our system,
X
Lsoverlap =
||D̃s (x) − DFs (x)||,
(6)
x∈Ω

(x + us (x)) denotes the ROI segmentawhere D̃ (x) =
tion mask of the moving image resized to scale s and warped
by STN according to deformation field us (x), DFs (x) is the
ROI segmentation mask of the fixed image resized to scale s.
s

s
DM

A. Experiments on CT Liver Registration
1) Experiments Result: As is illustrated in Table III, the
performance of liver registration is beyond satisfying, which
is far from clinical requirements. Thus, we add another
extra experiment by introducing ROI segmentation module
into the image-to-image deformable regsitration module. To
demonstrate the contribution of the ROI segmentation module,
we compare the performance of different models with the
ROI segmentation module, denoted by w/ mask. Table V
quantitatively shows the performance of our unsupervised
methods with mask, our best baseline (supervised methods)
and the best traditional registration algorithms with mask.
The unsupervised methods PMN w/ mask and PME w/
mask exhibit capability to align object boundaries by achieving
high Jaccard Coefficient 0.903 and 0.905 respectively. And
it’s evidenced that ROI segmentation mask has significantly
increased the performance of liver registration both in Dist and
Jacc. Notably, methods based on convolution neural networks
achieve 100x speedup compared to traditional methods. Figure
7 illustrates the registration results of different methods on
liver CT data.
Table VI shows the performance of various traditional
methods and our supervised baseline methods with mask.
2) Training with additional unlabeled data: Given that ROI
segmentation model is able to boost the performance of liver
registration, in the experiment we retrain the PMN w/ mask
model and PME w/ mask model with this enlarged dataset
and observe an improvement on the registration performance,
with Dist decreasing from 11.74 to 11.21 and Jacc increasing

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

9

Fig. 7: Illustration of the liver registration performance of the proposed unsupervised methods, our best supervised baseline (w/ mask) methods and one the best traditional registration
algorithms (Elastix) with the best Dist: (a) Moving image, (b) Ground truth segmentation mask of moving image, (c) Fixed image, (d) Ground truth segmentation mask of fixed
image. (e), (g), (i) and (k) respectively denote the moving images warped by the best traditional registration algorithm (elastix), our best supervised baseline method (itk16), our
best unsupervised method PN. The translucent red masks in (f), (h), (j), (l) respectively correspond to (e), (g), (i), (k) and denote the warped ground truth segmentation mask of the
moving images. The white masks in (f), (h), (j), (l) are the ground-truth segmentation mask of the fixed image. The red and yellow crosses denote landmarks of moving image and
fixed image, respectively. Dist in (c) and Jacc in (d) denote no registration.

TABLE VI: Performance of various traditional methods and our supervised and unsupervised methods with mask using liver data.

Fig. 8: This illustrates a brief structure of our fully convolutional image-to-image
registration network. The ROI segmentation network predicts the ROI segmentation
masks of the input fixed and moving images to reduce background noises and reserve
the ROI. The registration network takes two images and outputs a deformation field.

TABLE V: Performance of various methods with mask using liver data. In our method,
< P, M, E, N > denote the inclusion of photometric loss Lphotometric , ROI
boundary overlapping loss Loverlap , edge-aware smothness loss LsmoothE and
normal smothness loss LsmoothN , respectively. Then the size of training data is
enlarged to improve the performance of unsupervised method with mask. Variations of
various unsupervised methods proposed in the paper are usually different in the training
but mostly share the same architecture in testing; their run time speeds are therefore
approximately the same.

Method
trditional method: w/mask :
traditional best (Dist) - itk16
traditional best (Jacc) - itk17
supervised w/mask (our baseline) :
our best baseline (Dist) - itk16
our best baseline (Jacc) - itk16
unsupervised w/ mask (our method):
PN
PE
PMN
PME
more unlabeled data w/ mask (our
method):
PMN
PME

Dist

Jacc

Rt (s)

11.00
11.25

0.934
0.981

3.274
8.690

11.49
11.49

0.868
0.868

0.057
0.057

12.14
13.63
11.74
12.87

0.872
0.872
0.903
0.905

0.057
0.057
0.057
0.057

11.21
12.54

0.917
0.918

0.057
0.057

itk1 (FEM)
itk2 (Demons)
itk3 (Demons)
itk5 (Demons)
itk13 (BSplines)
itk16 (Demons)
itk17 (Demons)
elastix
ants

Traditional method /w
Rt
Dist
Jacc
(s)
12.82 0.734 11.748
13.35 0.709 2.793
13.21 0.865 4.985
13.49 0.948 4.555
12.56 0.783 8.390
11.00 0.934 3.274
11.25 0.981 8.690
11.26 0.967 30.543
11.68 0.967 3.710

Our baseline /w
Rt
Dist
Jacc
(s)
12.92 0.682 0.057
13.38 0.687 0.057
13.26 0.762 0.057
12.82 0.803 0.057
12.04 0.755 0.057
11.49 0.868 0.057
11.87 0.845 0.057
11.53 0.847 0.057
11.52 0.846 0.057

from 0.903 to 0.917, as shown in Table V. Noted that we
do not use any label during this processing, neither the
liver segmentation mask provided by LiTS nor optical flow
label generated by traditional methods. The ROI segmentation
modules are trained only by original data of MICCAI 2007
Grand Challenge [48], and the registration modules are trained
in a completely unsupervised manner.
Figure 9 illustrates the registration results of unsupervised
methods w/ mask (in Table V) and without mask (in Table
III).
B. Experiments on MRI brain Registration
We also validate the effect of ROI segmentation module on
brain MRI dataset, as is shown in Table VII and Table VIII.
We retrain the unsupervised PN w/ mask model and PE w/
mask model with additional dataset provided by ADNI [46]
(Alzheimer’s Disease Neuroimaging Initiative, and observe an
improvement in the registration performance. Note that the
ROI segmentation modules are trained only by original data
provided the LONI Probabilistic Brain Atlas (LPBA40) [44].

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

10

Fig. 9: Illustration of the liver registration performance of the proposed unsupervised methods with and without mask: (a), (c), (e), (g), (i) and (k) respectively denote the moving
images warped by PN, PE, PE with more unlabeled data, PMN w/ mask and PME w/ mask, PME w/ mask with more unlabeled data. The translucent red masks in (b), (d), (f), (h),
(j), (l) respectively correspond to (a), (c), (e), (g), (i), (k) and denote the warped ground truth segmentation mask of the moving images. The white masks in (b), (d), (f), (h), (j),
(l) are the ground-truth segmentation mask of the fixed image. The red and yellow crosses denote landmarks of moving image and fixed image, respectively.

The result demonstrates that the ROI segmentation module
can improve the performance of brain registration limitedly
compared to liver. It is due to that the background noise of
brain images is much lower than liver images.
Figure 10 illustrates the registration results of unsupervised
methods w/ mask (in Table VII) and without mask (in Table
I).
TABLE VII: Performance of various methods with mask using brain data. In our method,
< P, E, N > denote the inclusion of photometric loss Lphotometric , edge-aware
smothness loss LsmoothE and normal smothness loss LsmoothN , respectively. Then
the size of training data is enlarged to improve the performance of unsupervised method.
Variations of various unsupervised methods proposed in the paper are usually different
in the training but mostly share the same architecture in testing; their run time speeds
are therefore approximately the same.

Method
traditional method w/ mask:
traditional best (Dist) - itk3
traditional best (Jacc) - itk17
supervised (our baseline) :
our best baseline (Dist) - itk3
our best baseline (Jacc) - itk16
unsupervised w/ mask (our method):
PN
PE
more unlabeled data w/ mask (our
method):
PN
PE

Dist

Jacc

Rt (s)

3.14
3.54

0.998
1.000

10.123
12.450

3.04
3.18

0.964
0.967

0.094
0.094

3.37
3.35

0.971
0.951

0.094
0.094

3.32
3.31

0.974
0.955

0.094
0.094

ACKNOWLEDGMENT
The authors would like to thank all the dataset providers for
making their databases publicly available.
R EFERENCES
[1] L. Ibanez, W. Schroeder, L. Ng, and J. Cates, “The itk software guide,”
2005.

TABLE VIII: Performance of various traditional methods and our supervised and
unsupervised methods with mask using brain data.

itk1 (FEM)
itk2 (Demons)
itk3 (Demons)
itk5 (Demons)
itk13 (BSplines)
itk16 (Demons)
itk17 (Demons)
elastix
ants

Traditional method /w
Rt
Dist
Jacc
(s)
3.44
0.973 26.740
3.26
0.958 5.412
3.14
0.998 10.123
4.31
1.000 6.654
3.14
0.968 51.739
3.18
0.998 5.717
3.53
1.000 12.450
3.38
0.988 25.981
3.57
0.994 7.767

Our baseline /w
Rt
Dist
Jacc
(s)
3.36
0.954 0.094
3.30
0.943 0.094
3.04
0.964 0.094
3.45
0.939 0.094
3.15
0.960 0.094
3.18
0.967 0.094
3.43
0.951 0.094
3.23
0.955 0.094
3.24
0.966 0.094

[2] B. B. Avants, N. J. Tustison, G. Song, and J. C. Gee, “Ants, advanced
normalization tools,” http://www.picsl.upenn.edu/ANTS/, 2010.
[3] S. Klein, M. Staring, K. Murphy, M. A. Viergever, and J. P. Pluim,
“Elastix: a toolbox for intensity-based medical image registration,” TMI,
vol. 29, no. 1, pp. 196–205, 2010.
[4] J. Ashburner, “A fast diffeomorphic image registration algorithm,”
Neuroimage, vol. 38, no. 1, pp. 95–113, 2007.
[5] T. Vercauteren, X. Pennec, A. Perchant, and N. Ayache, “Diffeomorphic demons: Efficient non-parametric image registration,” NeuroImage,
vol. 45, no. 1, pp. S61–S72, 2009.
[6] G. Song, N. Tustison, B. Avants, and J. C. Gee, “Lung ct image
registration using diffeomorphic transformation models,” Medical image
analysis for the clinic: a grand challenge, pp. 23–32, 2010.
[7] J.-P. Thirion, “Image matching as a diffusion process: an analogy with
maxwell’s demons,” Medical image analysis, vol. 2, no. 3, pp. 243–260,
1998.
[8] Y. Xu, C. Xu, X. Kuang, H. Wang, E. I. Chang, W. Huang, Y. Fan
et al., “3d-sift-flow for atlas-based ct liver image segmentation,” Medical
physics, vol. 43, no. 5, pp. 2229–2241, 2016.
[9] C. Guetter, C. Xu, F. Sauer, and J. Hornegger, “Learning based nonrigid multi-modal image registration using kullback-leibler divergence,”
MICCAI, pp. 255–262, 2005.
[10] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in NIPS, 2012, pp. 1097–
1105.
[11] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

11

Fig. 10: Illustration of the brain registration performance of the proposed unsupervised methods with and without mask: (a), (c), (e) , (g), (i) and (k) respectively denote the moving
images warped by PN, PE, PE with more unlabeled data, PN w/ mask and PE w/ mask, PE w/ mask with more unlabeled data. The translucent red masks in (b), (d), (f), (h), (j),
(l) respectively correspond to (a), (c), (e), (g), (i) and (k) denote the warped ground truth segmentation mask of the moving images. The white masks in (b), (d), (f), (h), (j), (l) are
the ground-truth segmentation mask of the fixed image. The red and yellow crosses denote landmarks of moving image and fixed image, respectively.

[12] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards realtime object detection with region proposal networks,” in NIPS, 2015,
pp. 91–99.
[13] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks
for semantic segmentation,” in CVPR, 2015, pp. 3431–3440.
[14] S. Xie and Z. Tu, “Holistically-nested edge detection,” in ICCV, 2015,
pp. 1395–1403.
[15] J. Zbontar and Y. LeCun, “Stereo matching by training a convolutional
neural network to compare image patches,” JMLR, vol. 17, no. 1-32,
p. 2, 2016.
[16] L. Wei, Q. Huang, D. Ceylan, E. Vouga, and H. Li, “Dense human
body correspondences using convolutional networks,” in CVPR, 2016,
pp. 1544–1553.
[17] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi,
M. Ghafoorian, J. A. van der Laak, B. van Ginneken, and C. I. Sánchez,
“A survey on deep learning in medical image analysis,” arXiv preprint
arXiv:1702.05747, 2017.
[18] X. Yang, R. Kwitt, and M. Niethammer, “Fast predictive image registration,” in LABELS, 2016, pp. 48–57.
[19] G. Wu, M. Kim, Q. Wang, Y. Gao, S. Liao, and D. Shen, “Unsupervised
deep feature learning for deformable registration of mr brain images,”
in MICCAI, 2013, pp. 649–656.
[20] S. Miao, Z. J. Wang, and R. Liao, “A cnn regression approach for realtime 2d/3d registration,” TMI, vol. 35, no. 5, pp. 1352–1363, 2016.
[21] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov,
P. van der Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical
flow with convolutional networks,” in ICCV, 2015, pp. 2758–2766.
[22] M. Jaderberg, K. Simonyan, A. Zisserman et al., “Spatial transformer
networks,” in NIPS, 2015, pp. 2017–2025.
[23] Z. Ren, J. Yan, B. Ni, B. Liu, X. Yang, and H. Zha, “Unsupervised deep
learning for optical flow estimation,” in AAAI, 2017.
[24] J. Y. Jason, A. W. Harley, and K. G. Derpanis, “Back to basics:
Unsupervised learning of optical flow via brightness constancy and
motion smoothness,” in ECCV, 2016, pp. 3–10.
[25] R. Garg, G. Carneiro, and I. Reid, “Unsupervised cnn for single view
depth estimation: Geometry to the rescue,” in ECCV, 2016, pp. 740–756.
[26] C. Godard, O. Mac Aodha, and G. J. Brostow, “Unsupervised monocular
depth estimation with left-right consistency,” in CVPR, 2017.
[27] A. Kanazawa, D. W. Jacobs, and M. Chandraker, “Warpnet: Weakly
supervised matching for single-view reconstruction,” in CVPR, 2016,
pp. 3253–3261.
[28] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov,
P. van der Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical
flow with convolutional networks,” in ICCV, 2015, pp. 2758–2766.
[29] J. Kybic and M. Unser, “Fast parametric elastic image registration,”
IEEE transactions on image processing, vol. 12, no. 11, pp. 1427–1442,
2003.

[30] P. Viola and W. M. Wells III, “Alignment by maximization of mutual
information,” International journal of computer vision, vol. 24, no. 2,
pp. 137–154, 1997.
[31] T. Rohlfing, C. R. Maurer, D. A. Bluemke, and M. A. Jacobs, “Volumepreserving nonrigid registration of mr breast images using free-form
deformation with an incompressibility constraint,” TMI, vol. 22, no. 6,
pp. 730–741, 2003.
[32] G. P. Penney, J. Weese, J. A. Little, P. Desmedt, D. L. Hill et al., “A
comparison of similarity measures for use in 2-d-3-d medical image
registration,” TMI, vol. 17, no. 4, pp. 586–595, 1998.
[33] M. Staring, S. Klein, and J. P. Pluim, “A rigidity penalty term for
nonrigid registration,” Medical physics, vol. 34, no. 11, pp. 4098–4108,
2007.
[34] H. Lester and S. R. Arridge, “A survey of hierarchical non-linear medical
image registration,” Pattern recognition, vol. 32, no. 1, pp. 129–149,
1999.
[35] D. Rueckert, L. I. Sonoda, C. Hayes, D. L. Hill, M. O. Leach, and
D. J. Hawkes, “Nonrigid registration using free-form deformations:
application to breast mr images,” TMI, vol. 18, no. 8, pp. 712–721,
1999.
[36] Hessam Sokooti and Bob de Vos and Floris Berendsen and Boudewijn
P.F. Lelieveldt and Ivana Igum and Marius Staring, “Nonrigid Image
Registration Using Multi-Scale 3D Convolutional Neural Networks,” in
MICCAI, Quebec,Canada, 2017.
[37] P. Heise, S. Klose, B. Jensen, and A. Knoll, “Pm-huber: Patchmatch with
huber regularization for stereo matching,” in ICCV, 2013, pp. 2360–
2367.
[38] B. B. Avants, N. Tustison, and G. Song, “Advanced normalization tools
(ants),” Insight j, vol. 2, pp. 1–35, 2009.
[39] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for
fast feature embedding,” in Proceedings of the 22nd ACM international
conference on Multimedia, 2014, pp. 675–678.
[40] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014.
[41] “Source codes of itk,” https://github.com/InsightSoftwareConsortium/
ITK/tree/master/Examples/RegistrationITKv4.
[42] “Source codes of elastix,” http://elastix.bigr.nl/wiki/images/d/d5/
Par0035.Hammers.MI.bs.1.ASGDPrime.txt.
[43] “Source codes of elastix,” http://elastix.bigr.nl/wiki/images/c/c9/
Parameters BSpline.txt.
[44] D. W. Shattuck, M. Mirza, V. Adisetiyo, C. Hojatkashani, G. Salamon,
K. L. Narr, R. A. Poldrack, R. M. Bilder, and A. W. Toga, “Construction
of a 3d probabilistic atlas of human cortical structures,” Neuroimage,
vol. 39, no. 3, pp. 1064–1080, 2008.
[45] I. D. Grachev, D. Berdichevsky, S. L. Rauch, S. Heckers, D. N.
Kennedy, V. S. Caviness, and N. M. Alpert, “A method for assessing the

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

[46]
[47]
[48]

[49]
[50]

[51]
[52]

accuracy of intersubject registration of the human brain using anatomic
landmarks,” Neuroimage, vol. 9, no. 2, pp. 250–268, 1999.
“The alzheimer’s disease neuroimaging initiative (adni),” http://adni.loni.
usc.edu/methods/mri-analysis/adni-standardized-data.
“Insight segmentation and registration toolkit (itk),” https://itk.org/.
K. Murphy, B. Van Ginneken, J. M. Reinhardt, S. Kabus, K. Ding,
X. Deng, K. Cao, K. Du, G. E. Christensen, V. Garcia et al., “Evaluation
of registration methods on thoracic ct: the empire10 challenge,” TMI,
vol. 30, no. 11, pp. 1901–1920, 2011.
“Lits (liver tumor segmentation challenge),” https://competitions.
codalab.org/competitions/17094#learn the details-overview.
W. H. Greene, S. Chelikani, K. Purushothaman, J. Knisely, Z. Chen,
X. Papademetris, L. H. Staib, and J. S. Duncan, “Constrained non-rigid
registration for use in image-guided adaptive radiotherapy,” Medical
image analysis, vol. 13, no. 5, pp. 809–817, 2009.
E. Ferrante and N. Paragios, “Slice-to-volume medical image registration: A survey,” Medical Image Analysis, vol. 39, pp. 101–123, 2017.
C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu, “Deeply-supervised
nets,” in Artificial Intelligence and Statistics, 2015, pp. 562–570.

12

