Retinal Vasculature Segmentation Using Local
Saliency Maps and Generative Adversarial
Networks For Image Super Resolution

arXiv:1710.04783v2 [cs.CV] 16 Oct 2017

Dwarikanath Mahapatra, Behzad Bozorgtabar, Sajini Hewavitharanage, Rahil
Garnavi
IBM Research Australia
[dwarim,sydb,sajinihe,rahilgar]@au1.ibm.com.

Abstract. We propose an image super resolution(ISR) method using
generative adversarial networks (GANs) that takes a low resolution input
fundus image and generates a high resolution super resolved (SR) image
upto scaling factor of 16. This facilitates more accurate automated image
analysis, especially for small or blurred landmarks and pathologies. Local
saliency maps, which define each pixel’s importance, are used to define
a novel saliency loss in the GAN cost function. Experimental results
show the resulting SR images have perceptual quality very close to the
original images and perform better than competing methods that do
not weigh pixels according to their importance. When used for retinal
vasculature segmentation, our SR images result in accuracy levels close
to those obtained when using the original images.

1

Introduction

Normal retinal fundus images have high resolution to detect and segment prominent landmarks and pathologies, but not sufficient for small and indistinct pathologies (microaneurysms, haemorrhages) and smaller vessel branches. Image super
resolution (ISR) produces highly accurate super resolved (SR) images from single
field of view(FOV) images that are comparable with the original HR images. This
is particularly relevant for tele-ophthalmology requiring transmission of acquired
LR images. SR images improves detection of neovascularization, segmentation
of small vessel branches and small pathologies not visible in the original LR images. Ophthalmologist may also use it to closely analyze suspicious regions with
minute abnormalities.
Medical ISR methods using examples [110] and self similarity [101] were reliant on external data which put them at a disadvantage. Subsequently, parametric generative models learned the mapping between the original and LR version
to upscale MR brain [113] and cardiac [9] images. These approaches are computationally demanding as candidate patches are searched in the training dataset
to find the most suitable HR sample. Other methods using random forest regressors [124] convolutional neural networks (CNNs) [17] produce high quality

2

images. Recent work using Fourier burst accumulation [34], generative adversarial networks (GANs) [40] and CNNs [104] also highlight the importance of ISR
for medical image analysis problems.
GANs [40] are state-of-the-art for ISR primarily due to the ResNet based generator architecture, but are less effective with retinal images for scaling factors
greater than 4 due to cost functions that do not explicitly include local structure information. To overcome this limitation we propose a image SR method
based on GANs that has the following novelties in its cost function: 1) using
local saliency maps computed from curvature maps (that highlight local structures) and 2) entropy filtering (to highlight compact regions). Together they
outperform [40] for 4 − 16x magnification and the resulting SR images preserve
information content and perceptual information of the LR image. Our method’s
effectiveness is demonstrated in segmenting the retinal vasculature of SR images.

2

Saliency Map Calculation

Existing saliency methods highlight a globally salient region while ‘local’ saliency
maps are essential to compute individual pixel importance values. Inspired by
Perazzi et al [106] we combining abstraction, element distribution and uniqueness
to generate a local saliency map for retinal images.
Abstraction: Using superpixels for abstraction (as in [106]) provides high
level global information. Instead curvature maps (Icurv ) capture local structural
information based on vessel curvature and other small elements.
ICurv =

fxx fy2 + fyy fx2 − 2fxy fx fy
,
3/2
fx2 + fy2

(1)

where fx , fy are image gradients and fxx , fyy indicate second derivatives.
Element Distribution: Element distribution captures compactness
P and
continuity of retinal structures. Pixel (s) entropy is given by IEnt (s) = − i∈Ns pi log pi .
IEnt is the entropy image, Ns = 7 × 7 is s’s neighborhood and determines compactness, pi is the probability of intensity i in Ns calculated using a 8 bin histogram (experimentally determined). IEnt assigns low values for regions with
compact objects. Hence its values are normalized to [0, 1] and transformed as
1 − IEnt to highlight compact regions. IEnt is smoothed using a Gaussian low
pass filter of size 3 and standard deviation 0.5 to remove any isolated noisy
regions or pixel clusters.
Uniqueness: The sum of weighted difference of pixel feature maps is,
X
DF (s) =
exp (− ks − si k) |F (s) − F (si )|,
(2)
i

where DF indicates the difference or uniqueness map for feature map F (ICurv
or 1 − IEnt ); ks − si k is the Euclidean distance between s and its ith neighbor
si . Squared difference of feature values in [106] introduces blur which is undesirable for image super resolution. Instead we use the absolute difference and also

3

(a)

(b)

(c)

(d)

(e)

Fig. 1. (a) Original fundus image (b) grayscale image; (c) 1 − IEnt ; (d) ICurv ; (e) local
saliency map ISal at scale 7 × 7. Warmer colours indicate higher values

normalize the difference map to [0, 1]. The final saliency map is,
ISal = w1 × DCurv + (1 − w1 ) × D1−Ent .

(3)

where w1 balances the relative contribution of each feature. w1 = 0.4 was experimentally set by varying it between [0, 1] in steps of 0.01 and comparing
the quality of the resulting SR images on a subset of 50 images. The resulting
saliency map in Figure 1 (e) clearly highlights the local retinal structures and
thus justifies its use in the GAN cost function.

3

Generative Adversarial Networks

ISR estimates a high-resolution, super resolved image I SR from a low-resolution
input image I LR . For training, I LR is the low-resolution version of the high resolution counterpart I HR , obtained by applying a Gaussian filter to IHR followed
by downsampling with factor r. The generator network is a feed-forward CNN
(GθG ) whose parameters θG = W1:L ; b1:L are obtained by,
N

1 X SR
θb = arg min
l
GθG (InLR ), InHR ,
θG N
n=1

(4)

where lSR is the loss function and InHR , InLR are HR and LR images. The adversarial min-max problem is defined by,
min max
θG

E

[log DθD (I HR )] +

θD I HR ptrain (I HR )

E

I LR pG (I LR )

[log(1 − DθD (GθG (I LR ))]

(5)
This trains a generative model G with the goal of fooling a differentiable discriminator D that is trained to distinguish SR images from real images. G creates
solutions that are very similar to real images and thus difficult to classify by D.
This encourages perceptually superior solutions than obtained by minimizing
pixel-wise mean square error (MSE). G employs residual blocks (Figure 2 (a)).
Each block has two convolutional layers with 3 × 3 filters and 64 feature maps,
followed by batch normalization and ReLU activation.
D solves the maximization problem in Eqn. 5. It has eight convolutional
layers with the kernels increasing by a factor of 2 from 64 to 512 (Figure 2 (b)).

4

(a)

(b)

Fig. 2. (a) Generator Network; (b) Discriminator network. n64s1 denotes 64 feature
maps (n) and stride (s) 1 for each convolutional layer.

Leaky ReLU is used and strided convolutions reduce the image dimension when
the number of features is doubled. The resulting 512 feature maps are followed
by two dense layers and a final sigmoid activation to obtain a probability map.
3.1

Loss Function

SR
) and adversarial or generative loss
lSR is a combination of content loss (lCont
SR
), balanced by a factor α = 0.01, and is given by :
(lGen
SR
SR
lSR = lCont
+ αlGen

(6)

Content Loss: MSE results in smooth SR images that are perceptually
unsatisfying and lack high frequency content. Perceptually important details in
SR images is preserved by our saliency weighted MSE loss (lw−M SE ),
lw−M SE =

W H
1 XX
HR
×(wIHR Ix,y
− wISR GθG (I LR )x,y )2 ,
W H x=1 y=1

(7)

where wIHR , wISR are saliency values of HR (I HR ) and SR (GθG (I LR )) images.
A CNN loss [40] is calculated as the L2 distance between SR image and groundtruth HR image using all 512 feature maps of Relu 4 − 1 layer of a pre-trained
V GG − 16 [120].
SR
lCN
N =

Wi,j Hi,j
XX
1
(φi,j (I HR )x,y − φi,j (GθG (I LR ))x,y )2
Wi,j Hi,j x=1 y=1

(8)

φi,j the feature map obtained by the j−th convolution (after activation) before
the i−th max pooling layer and Wi,j and Hi,j are the dimensions of φ.
Local Saliency Loss: Our novel saliency loss measures the difference in
saliency maps of GθG (I LR )) and I HR by comparing their local landmarks. This
enables inclusion of important structural information in the cost function.
SR
lSal
=

Wi,j Hi,j
XX
1
((I HR )x,y − (GθG (I LR )Sal )x,y )2
Wi,j Hi,j x=1 y=1 Sal

(9)

5
Scaling factor(r) = 4
Scaling factor (r) = 8
r=16
SSIM RMSE PSNR S3
p
SSIM RMSE PSNR S3
p
SSIM
(10−6 ) dB
(10−6 dB
SRGANSal 0.89 6.2
44.3 0.83
0.84 7.5
39 0.74
0.80
SRGANLedig 0.78 8.1
36.4 0.65 < 0.001 0.73 9.3
31 0.60 < 0.001 0.69
SRCN N
0.75 9.1
34.3 0.61 < 0.009 0.67 10.9
28 0.57 < 0.001 0.64
SR-RF
0.71 10.3 30.2 0.57 < 0.009 0.62 12.3
25 0.55 < 0.001 0.59
SSR
0.67 11.2 27.1 0.54 < 0.001 0.60 13.7
22 0.21 < 0.001 0.56
Table 1. Comparative results of different methods for image super resolution.

HR
ISal
and GθG (I LR )Sal denote the saliency maps of I HR and GθG (I LR ).
SR
Adversarial Loss: The generative loss lGen
[40] over all training samples is

SR
lGen
=

N
X

− log DθD (GθG (I LR ))

(10)

n=1

DθD (GθG (I LR )) is probability that GθG (I LR ) is a natural HR image. This network favours solutions in the manifold of retinal images. Convergence is facilitated by minimizing − log DθD (GθG (I LR )) instead of − log[1 − DθD (GθG (I LR ))].
Training: The model in Fig. 2 (a) generates images upscaled by 2×. For
higher scale factors we feed the output of this network to another identical
network to get 4× images. Consecutive networks can be combined to get 8×,
16×, etc image resolution. In each subsequent upsampling step the corresponding
networks are trained on images of size 4×, 8×, etc.

4

Experiments And Results

Dataset: We apply our algorithm on 5000 retinal fundus images from multiple
sources with different image dimensions [4], and augmented 100 times by rotation and translation. The dark borders were removed and the images resized to
1024 × 1024 pixels. Our method was implemented with Python and TensorFlow
(for GANs). For GAN optimization we use Adam with β1 = 0.93 and batch
normalization. The ResNet was trained with a learning rate of 0.001 and 105
update iterations. MSE based ResNet was used to initialize G. The final GAN
was trained with 105 update iterations at learning rate 10−3 . The average training time using the augmented version from 4000 images was 14 hrs for scaling
factor(r) 2, 26 hours for r = 4, and 40 hours for r = 8. Time to generate a super
resolved image is 1 ms for r = 2, 1.4 ms for r = 4, and 1.9 ms for r = 8. Training
and test was performed on a NVIDIA Tesla K40 GPU with 12 GB RAM.
4.1

Image Super Resolution Results

The following ISR methods are used for comparison: 1) SRGANLedig : - the
baseline GAN using MSE and CNN loss [40]; 2) SRGANSal : - our proposed

6

(a)

(b)

(c)

(d)

Fig. 3. Super resolution results for r = 4. (a) original HR; SR images from: (b)
SRGANSal ; (c) SRGANLedig ; and (d) SRCN N .

method; 3) SRCN N :- CNN based method of [17]; 4) SR − RF :- random forest
based method of [114]; 5) SSR :- self super resolution method of [34]. Resized
1024 × 1024 images are ground truth HR images, I HR , which are downsampled
by different r to obtain I LR from which I SR are generated. Y −channel images
of I HR and I SR are used to compute: 1) peak signal to noise ratio (PSNR); 2)
structural similarity (SSIM) [129]; 3) S3 - the sharpness metric of [128]; and 4)
root mean square error (RMSE). Higher values of 1, 2, 3 and lower values of 4
indicate better performance.
Results of 5−fold cross validation for r = 4, 8 are presented in Table 1. Due to
space constraints only SSIM values are shown for r = 16. For r = 2, performance
difference of all methods is small but becomes more pronounced for higher r.
SRGANSal gives the best results for all r, and the improvement over competing
methods is significant as is evident from the p−values of Wilcoxon signed-rank
tests. Figure 3 shows results of the top 3 methods (due to space constraints) for
r = 4. SRGANSal shows the best performance as is evident from the SR image
in Fig. 3 (b) where one of the minor retinal branches (indicated by yellow arrow)
is clearly visible. On the other hand the SR image by SRGANLedig (Fig. 3 (c))
is blurry and does not clearly show this retinal branch. Other methods perform
much worse, with significant blur visible for the main branches as well. Clearly,
SRGANSal gives the closest reconstruction to the HR image of Fig. 3 (a).
SR
SR
Importance of Saliency Maps: Excluding lCN
N , and using lSal and lw−M SE
for r = 4 gives SSIM= 0.81, RMSE= 6.9, PSNR= 38.6 dB, and S3= 0.69. They
are slightly higher than SRGANLedig , indicating local saliency maps alone perSR
form better than lCN
N and MSE in preserving image information. Combining
local saliency information with CNN loss significantly improves SR image quality.
Using either curvature (SSIM= 0.82) or entropy (SSIM= 0.83) for r = 4 lowers
performance,thus highlighting their individual importance in the final saliency
map. Using saliency maps of [106] gave SSIM= 0.80, 0.74, 0.70 for r = 4, 8, 16.
Our proposed local saliency maps outperforms [106] (a global saliency map) as
it does not capture fine structural information in retinal fundus images.

7
HR
SRGANSal SRGANLedig SRCN N SR-RF
SSR
Acc Sen Acc Sen Acc
Sen
Acc Sen Acc Sen Acc Sen
DRIVE 0.98 0.79 0.96 0.77 0.92 0.74 0.89 0.73 0.87 0.70 0.85 0.69
STARE 0.98 0.90 0.96 0.87 0.91 0.84 0.89 0.81 0.86 0.77 0.83 0.72
CHASE DB 0.97 0.84 0.95 0.82 0.91 0.76 0.87 0.72 0.85 0.70 0.81 0.68
Table 2. Comparative vasculature segmentation results of different super resolution
methods. The values are for scaling factor 4 and 8.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Fig. 4. Results for retinal vessel segmentation; (a) retinal image; (b) manual ground
truth mask; results obtained when training on (c) orginal HR images; SR images by
(d) SRGANSal ; (e) SRGANLedig ; (f) SRCN N ; (g) SR-RF; (h) SSR. Yellow arrows
highlight regions of inaccurate segmentation.

4.2

Retinal Blood Vessel Segmentation Results

We present retinal vessel segmentation results on the DRIVE [123], STARE [31]
and CHASE DB1 [21] datasets with 40, 20 and 28 images respectively. Original
images and manual annotations (I HR ) are downsampled by r = 4, 8 to get I LR
and 5 sets of I SR from 5 methods trained on [4]. I HR and the 5 sets of I SR were
used to train 6 different state-of-the art U-Nets for vasculature segmentation [2].
The average accuracy (Acc) and sensitivity (Sen) for r = 4, 8 is summarized in
Table 2. Better ISR methods should give higher vessel segmentation accuracy and
performance of I HR gives a lower bound on the segmentation error. SRGANSal ’s
performance is closest to I HR , and establishes its superiority over all competing
methods. Figures 4 (a)-(h) show results of vessel segmentation on an example
image. SRGANSal ’s performance is most similar to I HR as is evident from
the areas of inaccurate segmentation highlighted by yellow arrows. Most of the
methods do not segment the finer vasculature structures, while SSR and SR-RF
are unable to segment some of the major arteries.

5

Conclusion

We have proposed a novel method for super resolution of retinal fundus images
based on GANs. Local saliency maps effectively quantify a pixel’s perceptual relevance, and are used to weight each pixel according to its importance and define a
novel saliency loss. When incorporated into the GAN loss function, the resulting
SR images are better than those obtained using CNN feature loss. Experimental results show combination of saliency and CNN loss significantly outperforms
current state of the art GANs and other competing ISR methods. The resulting

8

super resolved images can be used to increase the size and resolution of low
dimensional images, and then apply different image analysis algorithms. When
using our SR images for retinal vessel segmentation the results are close to those
obtained with the original high resolution images. Our method can be applied
for other medical images as well.

References
1.
2.
3.
4.
5.

6.

7.

8.
9.

10.
11.
12.
13.

14.

15.
16.
17.
18.
19.

E. Goldstein, Sensation and perception, Thomson Wadsworth, 2007
https://github.com/orobix/retina-unet
https://www.kaggle.com/c/diabetic-retinopathy-detection
http://www.eyepacs.com/
Abramoff, M.D., Suttorp-Schulten, M.S.A.: Web-based screening for diabetic
retinopathy in a primary care population: The Eyecheck project. Telemedicine
e-Health 11(6), 668–674 (2005)
Alexander, D., Zikic, D., Zhang, J., Zhang, H., Criminisi, A.: Image quality transfer via random forest regression: applications in diffusion mri. In: In Proc MICCAI. pp. 225–232 (2014)
Bar, Y., Diamant, I., Wolf, L., Lieberman, S., Konen, E., Greenspan, H.: Chest
pathology detection using deep learning with non-medical training. In: IEEE ISBI.
pp. 294–297 (2015)
Becker, C., Rigamonti, R., Lepetit, V., Fua, P.: Supervised feature learning for
curvilinear structure segmentation. In: MICCAI. pp. 526–533 (2013)
Bhatia, K., Price, A., Shi, W., Hajnal, J., Rueckert., D.: Super-resolution reconstruction of cardiac mri using coupled dictionary learning. In: In Proc ISBI. pp.
947–950 (2014)
Boykov, Y., Veksler, O.: Fast approximate energy minimization via graph cuts.
IEEE Trans. Pattern Anal. Mach. Intell. 23, 1222–1239 (2001)
Breiman, L.: Random forests. Machine Learning 45(1), 5–32 (2001)
Bruna, J., Sprechmann, P., Y, L.: Super-resolution with deep convolutional sufficient statistics. In: Proc ICLR (2016)
Decenciere, E., Cazuguel, G., Zhang, X., Thibault, G., Klein, J.C., Meyer, F.,
Marcotegui, B., Quellec, G., Lamard, M., Danno, R.: Teleophta: Machine learning and image processing methods for teleophthalmology. IRBM 34(2), 196–203
(2013)
Denton, E., Chintala, S., Szlam, A., Fergus., R.: Deep generative image models
using a laplacian pyramid of adversarial networks. In: Proc. NIPS. pp. 1486–1494
(2015)
Dong, C., Deng, Y., Loy, C., Tang., X.: Compression artifacts reduction by a deep
convolutional network. In: In Proc IEEE ICCV. pp. 576–584 (2015)
Dong, C., Loy, C., He, K.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)
Dong, C., Loy, C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Patt. Anal. Mach. Intell. 38(2), 295–307 (2016)
Dosovitskiy, A., Brox, T.: Generating images with perceptual similarity metrics
based on deep networks. In: arXiv:1602.02644 (2016)
Durand, T., Thome, N., Cord, M.: WELDON:weakly supervised learning of deep
convolutional neural networks. In: IEEE CVPR (2016)

9
20. Fraz, M., Remagnino, P., Hoppe, A., Uyyanonvara, B., Rudnicka, A., Owen, C.,
Barman, S.: Blood vessel segmentation methodologies in retinal imagesa survey.
Computer methods and programs in biomedicine 108(1), 407–433 (2012)
21. Fraz, M., Remagnino, P., Hoppe, A., Uyyanonvara, B., Rudnicka, A., Owen, C.,
Barman, S.: An ensemble classification-based approach applied to retinal blood
vessel segmentation. IEEE Trans. Biomed. Engg. 59(9), 2538–2548 (2012)
22. Freeman, W., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE
Comput. Graph. Appl. 22(2), 56–65 (2002)
23. Fu, H., Xu, Y., Lin, S., Wong, D., Liu, J.: Deepvessel: Retinal vessel segmentation
via deep learning and conditional random field. In: To appear MICCAI. pp. 1–8
(2016)
24. Ganin, Y., Lempitsky, V.: N4-fields: Neural network nearest neighbor fields for
image transforms. In: ACCV. pp. 536–551 (2014)
25. Goodfellow, I., Pouget-Abadiey, J., Mirza, M., Xu, B., Warde-Farley, D., Ozairz,
S., Courville, A., Bengio., Y.: Generative adversarial nets. In: Proc. NIPS. pp.
2672–2680 (2014)
26. Gu, L., Cheng, L.: Learning to boost filamentary structure segmentation. In: IEEE
ICCV. pp. 639–647 (2015)
27. Hariharan, B., Arbelaez, P., Girshick, R., Malik, J.: Hypercolumns for object
segmentation and fine-grained localization. In: IEEE CVPR. pp. 447–456 (2015)
28. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.
In: Proc. CVPR. pp. 770–778 (2016)
29. He, K., Zhang, X., Ren, S., Sun., J.: Identity mappings in deep residual networks.
In: Proc. ECCV. pp. 630–645 (2016)
30. Hong, S., Oh, J., Lee, H., Han, B.: Learning transferrable knowledge for semantic
segmentation with deep convolutional neural network. In: IEEE CVPR. pp. 3204–
3212 (2015)
31. Hoover, A., Kouznetsova, V., Goldbaum, M.: Locating blood vessels in retinal
images by piecewise threshold probing of a matched filter response. IEEE Trans.
Med. Imag. 19(3), 203–210 (2000)
32. http://www.cs.ucl.ac.uk/staff/S.Arridge/teaching/ndsp/.:
33. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by
reducing internal covariate shift. In: Proc. ICML. pp. 448–456 (2015)
34. Jog, A., aron Carass, A., Prince., J.: Self super-resolution for magnetic resonance
images. In: In Proc MICCAI. pp. 553–560 (2016)
35. Johnson, J., Alahi, A., Fei-Fei., L.: Perceptual losses for real-time style transfer
and super- resolution. In: Proc. ECCV. pp. 694–711 (2016)
36. Kim, J., Lee, J., Lee., K.: Deeply-recursive convolutional network for image superresolution. In: Proc. CVPR. pp. 1637–1645 (2016)
37. Konukoglu, E., van der Kouwe, A., Sabuncu, M., Fischl., B.: Example-based
restoration of high-resolution magnetic resonance image acquisitions. In: In Proc
MICCAI. pp. 131–138 (2013)
38. Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)
39. Kuang, H., Guthier, B., Saini, M., Mahapatra, D., Saddik, A.E.: A real-time smart
assistant for video surveillance through handheld devices. In: In Proc: ACM Intl.
Conf. Multimedia. pp. 917–920 (2014)
40. Ledig, C., et. al.: Photo-realistic single image super-resolution using a generative
adversarial network. CoRR abs/1609.04802 (2016)
41. Li, C., Wand, M.: Combining markov random fields and convolutional neural
networks for image synthesis. In: Proc. CVPR. pp. 2479–2486 (2016)

10
42. Li, Z., Mahapatra, D., J.Tielbeek, Stoker, J., van Vliet, L., Vos, F.: Image registration based on autocorrelation of local structure. IEEE Trans. Med. Imaging
35(1), 63–75 (2016)
43. Mahapatra, D.: Neonatal brain mri skull stripping using graph cuts and shape
priors. In: In Proc: MICCAI workshop on Image Analysis of Human Brain Development (IAHBD) (2011)
44. Mahapatra, D.: Cardiac lv and rv segmentation using mutual context information.
In: Proc. MICCAI-MLMI. pp. 201–209 (2012)
45. Mahapatra, D.: Groupwise registration of dynamic cardiac perfusion images using
temporal information and segmentation information. In: In Proc: SPIE Medical
Imaging (2012)
46. Mahapatra, D.: Landmark detection in cardiac mri using learned local image
statistics. In: Proc. MICCAI-Statistical Atlases and Computational Models of
the Heart. Imaging and Modelling Challenges (STACOM). pp. 115–124 (2012)
47. Mahapatra, D.: Skull stripping of neonatal brain mri: Using prior shape information with graphcuts. J. Digit. Imaging 25(6), 802–814 (2012)
48. Mahapatra, D.: Cardiac image segmentation from cine cardiac mri using graph
cuts and shape priors. J. Digit. Imaging 26(4), 721–730 (2013)
49. Mahapatra, D.: Cardiac mri segmentation using mutual context information from
left and right ventricle. In press J. Digit. Imaging 26(5), 898–908 (2013)
50. Mahapatra, D.: Graph cut based automatic prostate segmentation using learned
semantic information. In: Proc. IEEE ISBI. pp. 1304–1307 (2013)
51. Mahapatra, D.: Joint segmentation and groupwise registration of cardiac perfusion images using temporal information. J. Digit. Imaging 26(2), 173–182 (2013)
52. Mahapatra, D.: Automatic cardiac segmentation using semantic information from
random forests. J. Digit. Imaging. 27(6), 794–804 (2014)
53. Mahapatra, D.: Combining multiple expert annotations using semi-supervised
learning and graph cuts for medical image segmentation. Computer Vision and
Image Understanding 151(1), 114–123 (2016)
54. Mahapatra, D.: Semi-supervised learning and graph cuts for consensus based
medical image segmentation. Pattern Recognition 63(1), 700–709 (2017)
55. Mahapatra, D., Bozorgtabar, S., Hewavitahranage, S., Garnavi, R.: Image super
resolution using generative adversarial networks and local saliencymaps for retinal
image analysis,. In: In Proc. MICCAI. pp. 382–390 (2017)
56. Mahapatra, D., Buhmann, J.: Obtaining consensus annotations for retinal image
segmentation using random forest and graph cuts. In: In Proc. OMIA. pp. 41–48
(2015)
57. Mahapatra, D., Buhmann, J.: Visual saliency based active learning for prostate
mri segmentation. In: In Proc. MLMI. pp. 9–16 (2015)
58. Mahapatra, D., Buhmann, J.: Visual saliency based active learning for prostate
mri segmentation. SPIE Journal of Medical Imaging 3(1) (2016)
59. Mahapatra, D., Buhmann, J.: Automatic cardiac rv segmentation using semantic
information with graph cuts. In: Proc. IEEE ISBI. pp. 1094–1097 (2013)
60. Mahapatra, D., Buhmann, J.: Analyzing training information from random forests
for improved image segmentation. IEEE Trans. Imag. Proc. 23(4), 1504–1512
(2014)
61. Mahapatra, D., Buhmann, J.: Prostate mri segmentation using learned semantic
knowledge and graph cuts. IEEE Trans. Biomed. Engg. 61(3), 756–764 (2014)
62. Mahapatra, D., Buhmann, J.: A field of experts model for optic cup and disc
segmentation from retinal fundus images. In: In Proc. IEEE ISBI. pp. 218–221
(2015)

11
63. Mahapatra, D., Gilani, S., Saini., M.: Coherency based spatio-temporal saliency
detection for video object segmentation. IEEE Journal of Selected Topics in Signal
Processing. 8(3), 454–462 (2014)
64. Mahapatra, D., J.Tielbeek, Makanyanga, J., Stoker, J., Taylor, S., Vos, F., Buhmann, J.: Automatic detection and segmentation of crohn’s disease tissues from
abdominal mri. IEEE Trans. Med. Imaging 32(12), 1232–1248 (2013)
65. Mahapatra, D., J.Tielbeek, Makanyanga, J., Stoker, J., Taylor, S., Vos, F., Buhmann, J.: Active learning based segmentation of crohn’s disease using principles
of visual saliency. In: Proc. IEEE ISBI. pp. 226–229 (2014)
66. Mahapatra, D., J.Tielbeek, Makanyanga, J., Stoker, J., Taylor, S., Vos, F., Buhmann, J.: Combiningmultiple expert annotations using semi-supervised learning
and graph cuts for crohns disease segmentation. In: In Proc: MICCAI-ABD (2014)
67. Mahapatra, D., J.Tielbeek, Vos, F., Buhmann, J.: A supervised learning approach
for crohn’s disease detection using higher order image statistics and a novel shape
asymmetry measure. J. Digit. Imaging 26(5), 920–931 (2013)
68. Mahapatra, D., Li, Z., Vos, F., Buhmann, J.: Joint segmentation and groupwise
registration of cardiac dce mri using sparse data representations. In: In Proc.
IEEE ISBI. pp. 1312–1315 (2015)
69. Mahapatra, D., Routray, A., Mishra, C.: An active snake model for classification
of extreme emotions. In: IEEE International Conference on Industrial Technology
(ICIT). pp. 2195–2199 (2006)
70. Mahapatra, D., Roy, P., Sedai, S., Garnavi, R.: A cnn based neurobiology inspired
approach for retinal image quality assessment. In: In Proc. EMBC. pp. 1304–1307
(2016)
71. Mahapatra, D., Roy, P., Sedai, S., Garnavi, R.: Retinal image quality classification
using saliency maps and cnns. In: In Proc. MICCAI-MLMI. pp. 172–179 (2016)
72. Mahapatra, D., Roy, P., Sedai, S., Garnavi, R.: Retinal image quality classification
using saliency maps and cnns. In: In Proc. MICCAI-MLMI. pp. 172–179 (2016)
73. Mahapatra, D., Roy, S., Sun, Y.: Retrieval of mr kidney images by incorporating
spatial information in histogram of low level features. In: In 13th International
Conference on Biomedical Engineering (2008)
74. Mahapatra, D., Saini, M., Sun, Y.: Illumination invariant tracking in office environments using neurobiology-saliency based particle filter. In: IEEE ICME. pp.
953–956 (2008)
75. Mahapatra, D., Sun, Y.: Nonrigid registration of dynamic renal MR images using
a saliency based MRF model. In: Proc. MICCAI. pp. 771–779 (2008)
76. Mahapatra, D., Sun, Y.: Nonrigid registration of dynamic renal MR images using
a saliency based MRF model. In: Proc. MICCAI. pp. 771–779 (2008)
77. Mahapatra, D., Sun, Y.: Nonrigid registration of dynamic renal MR images using
a saliency based MRF model. In: Proc. MICCAI. pp. 771–779 (2008)
78. Mahapatra, D., Sun, Y.: Registration of dynamic renal mr images using neurobiological model of saliency. In: Proc. ISBI. pp. 1119–1122 (2008)
79. Mahapatra, D., Sun, Y.: Using saliency features for graphcut segmentation of
perfusion kidney images. In: In 13th International Conference on Biomedical Engineering (2008)
80. Mahapatra, D., Sun, Y.: Joint registration and segmentation of dynamic cardiac
perfusion images using mrfs. In: Proc. MICCAI. pp. 493–501 (2010)
81. Mahapatra, D., Sun, Y.: Joint registration and segmentation of dynamic cardiac
perfusion images using mrfs. In: Proc. MICCAI. pp. 493–501 (2010)
82. Mahapatra, D., Sun, Y.: Joint registration and segmentation of dynamic cardiac
perfusion images using mrfs. In: Proc. MICCAI. pp. 493–501 (2010)

12
83. Mahapatra, D., Sun., Y.: An mrf framework for joint registration and segmentation of natural and perfusion images. In: Proc. IEEE ICIP. pp. 1709–1712 (2010)
84. Mahapatra, D., Sun, Y.: Retrieval of perfusion images using cosegmentation and
shape context information. In: Proc. APSIPA Annual Summit and Conference
(ASC) (2010)
85. Mahapatra, D., Sun, Y.: Rigid registration of renal perfusion images using a
neurobiology based visual saliency model. EURASIP Journal on Image and Video
Processing. pp. 1–16 (2010)
86. Mahapatra, D., Sun, Y.: A saliency based mrf method for the joint registration
and segmentation of dynamic renal mr images. In: Proc. ICDIP (2010)
87. Mahapatra, D., Sun, Y.: Mrf based intensity invariant elastic registration of cardiac perfusion images using saliency information. IEEE Trans. Biomed. Engg.
58(4), 991–1000 (2011)
88. Mahapatra, D., Sun, Y.: Orientation histograms as shape priors for left ventricle
segmentation using graph cuts. In: In Proc: MICCAI. pp. 420–427 (2011)
89. Mahapatra, D., Sun, Y.: Orientation histograms as shape priors for left ventricle
segmentation using graph cuts. In: In Proc: MICCAI. pp. 420–427 (2011)
90. Mahapatra, D., Sun, Y.: Integrating segmentation information for improved mrfbased elastic image registration. IEEE Trans. Imag. Proc. 21(1), 170–183 (2012)
91. Mahapatra, D., Tielbeek, J., Buhmann, J., Vos, F.: A supervised learning
based approach to detect crohn’s disease in abdominal mr volumes. In: Proc.
MICCAI workshop Computational and Clinical Applications in Abdominal
Imaging(MICCAI-ABD). pp. 97–106 (2012)
92. Mahapatra, D., Tielbeek, J., Vos, F., ., J.B.: Crohn’s disease tissue segmentation
from abdominal mri using semantic information and graph cuts. In: Proc. IEEE
ISBI. pp. 358–361 (2013)
93. Mahapatra, D., Tielbeek, J., Vos, F., Buhmann, J.: Localizing and segmenting
crohn’s disease affected regions in abdominal mri using novel context features. In:
Proc. SPIE Medical Imaging (2013)
94. Mahapatra, D., Tielbeek, J., Vos, F., Buhmann, J.: Semi-supervised and active
learning for automatic segmentation of crohn’s disease. In: Proc. MICCAI, Part
2. pp. 214–221 (2013)
95. Mahapatra, D., Tielbeek, J., Vos, F., Buhmann, J.: Semi-supervised and active
learning for automatic segmentation of crohn’s disease. In: Proc. MICCAI, Part
2. pp. 214–221 (2013)
96. Mahapatra, D., Tielbeek, J., Vos, F., Buhmann, J.: Weakly supervised semantic
segmentation of crohn’s disease tissues from abdominal mri. In: Proc. IEEE ISBI.
pp. 832–835 (2013)
97. Mahapatra, D., Vos, F., Buhmann, J.: Crohn’s disease segmentation from mri
using learned image priors. In: In Proc. IEEE ISBI. pp. 625–628 (2015)
98. Mahapatra, D., Vos, F., Buhmann, J.: Active learning based segmentation
of crohns disease from abdominal mri. Computer Methods and Programs in
Biomedicine 128(1), 75–85 (2016)
99. Mahapatra, D., Winkler, S., Yen, S.: Motion saliency outweighs other low-level
features while watching videos. In: SPIE HVEI. pp. 1–10 (2008)
100. Maninis, K., Pont-Tuset, J., Arbelez, P., Gool, L.V.: Deep retinal image understanding. In: To appear MICCAI. pp. 1–8 (2016)
101. Manjon, J., Coupe, P., Buades, A., Fonov, V., Collins, D., Robles, M.: Non-local
mri upsampling. Med. Image Anal. 14(6), 784–792 (2010)
102. Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: International Conference on Machine Learning. pp. 807–814 (2010)

13
103. Niemeijer, M., et. al.: Retinopathy online challenge: Automatic detection of microaneurysms in digital color fundus photographs. IEEE Trans. Med. Imag. 29(1),
185–195 (2010)
104. Oktay, O., et. al.: Multi-input cardiac image super-resolution using convolutional
neural networks. In: In Proc MICCAI. pp. 246–254 (2016)
105. Orlando, J., Blaschko, M.: Learning fully-connected crfs for blood vessel segmentation in retinal images. In: MICCAI. pp. 634–641 (2014)
106. Perazzi, F., Kraehenbuhl, P., Pritch, Y., Hornung, A.: Saliency filters: Contrast
based filtering for salient region detection. In: Proc. IEEE Intl. Conf. Comp. Vis.
Pattern. Recog. pp. 733–740 (2012)
107. Radford, A., Metz, L., Chintala., S.: Unsupervised representation learning with
deep convolutional generative adversarial networks. In: Proc. ICLR. (2016)
108. Ricci, E., Perfetti., R.: Retinal blood vessel segmentation using line operators and
support vector classification. IEEE Trans. Med. Imag 26(10), 1357–1365 (2007)
109. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI Springer, LNCS, Vol.9351. pp. 234–241
(2015)
110. Rousseau, F.: Brain hallucination. In: ECCV. pp. 497–508 (2008)
111. Roy, P., Chakravorty, R., Sedai, S., Mahapatra, D., Garnavi, R.: Automatic
eye type detection in retinal fundus image using fusion of transfer learning and
anatomical features. In: In Proc. DICTA. pp. 1–7 (2016)
112. Roy, P., Tennakoon, R., Cao, K., Sedai, S., Mahapatra, D., Maetschke, S., Garnavi, R.: A novel hybrid approach for severity assessment of diabetic retinopathy
in colour fundus images,. In: In Proc. IEEE ISBI. pp. 1078–1082 (2017)
113. Rueda, A., Malpica, N., Romero., E.: Single-image super-resolution of brain mr
images using overcomplete dictionaries. Med. Image Anal. 17(1), 113–132 (2013)
114. Schulter, S., Leistner, C., Bischof., H.: Fast and accurate image upscaling with
super-resolution forests. In: In Proc CVPR. pp. 3791–3799 (2015)
115. Sedai, S., Mahapatra, D., Hewavitharanage, S., Maetschke, S., Garnavi, R.: Semisupervised segmentation of optic cup in retinal fundus images using variational
autoencoder,. In: In Proc. MICCAI. pp. 75–82 (2017)
116. Sedai, S., Roy, P., Mahapatra, D., Garnavi, R.: Segmentation of optic disc and
optic cup in retinal fundus images using shape regression. In: In Proc. EMBC.
pp. 3260–3264 (2016)
117. Sedai, S., Roy, P., Mahapatra, D., Garnavi, R.: Segmentation of optic disc and
optic cup in retinal images using coupled shape regression. In: In Proc. MICCAIOMIA. pp. 1–8 (2016)
118. Shi, W., et. al.: Cardiac image super-resolution with global correspondence using
multi-atlas patchmatch. In: In Proc MICCAI. pp. 9–16 (2013)
119. Shi, W., Caballero, J., Huszr, F., Totz, J., Aitken, A., Bishop, R., Rueckert, D.,
Wang., Z.: Real-time single image and video super-resolution using an efficient
sub-pixel convolutional neural network. In: Proc. CVPR. pp. 1874–1883 (2016)
120. Simonyan, K., Zisserman., A.: Very deep convolutional networks for large-scale
image recognition. CoRR abs/1409.1556 (2014)
121. Sironi, A., Lepetit, V., Fua, P.: Projection onto the manifold of elongated structures for accurate extraction. In: IEEE ICCV. pp. 316–324 (2015)
122. Soares, J., Leandro, J., Cesar, R., Jelinek, H., Cree, M.: Retinal vessel segmentation using the 2-d gabor wavelet and supervised classification. IEEE Trans. Med.
Imag 25(9), 1214–1222 (2006)

14
123. Staal, J., Abramoff, M., Niemeijer, M., Viergever, M., Ginneken, B.V.: Ridge
based vessel segmentation in color images of the retina. IEEE Trans. Med. Imag.
23(4), 501–509 (2004)
124. Tanno, R., Ghosh, A., Grussu, F., Kaden, E., Criminisi, A., Alexander., D.:
Bayesian image quality transfer. In: In Proc MICCAI. pp. 265–173 (2016)
125. Tennakoon, R., Mahapatra, D., Roy, P., Sedai, S., Garnavi, R.: Image quality
classification for dr screening using convolutional neural networks. In: In Proc.
MICCAI-OMIA. pp. 113–120 (2016)
126. Timofte, R., Smet, V.D., Gool., L.V.: Anchored neighborhood regression for fast
example-based superresolution. In: ICCV. pp. 1920–1927 (2013)
127. Vos, F.M., Tielbeek, J., Naziroglu, R., Li, Z., Mahapatra, D., Wiebel, A., Lavini,
C., Buhmann, J., Hege, H., Stoker, J., van Vliet, L.: Computational modeling for
assessment of IBD: to be or not to be? In: Proc. IEEE EMBC. pp. 3974–3977
(2012)
128. Vu, C., Phan, T., Chandler., D.: S3: A spectral and spatial measure of local
perceived sharpness in natural images. IEEE Trans. Imag. Proc. 21(3), 934–945
(2012)
129. Wang, Z., et. al.: Image quality assessment: from error visibility to structural
similarity. IEEE Trans. Imag. Proc. 13(4), 600–612 (2004)
130. Xie, S., Tu, Z.: Holistically nested edge detection. In: IEEE ICCV. pp. 1395–1403
(2015)
131. Yang, J., Wright, J., Huang, T., Ma., Y.: Image super-resolution via sparse representation. IEEE Trans. Image Proc. 19(11), 2861–2873 (2010)
132. Yeh, R., Chen, C., Lim, T., Hasegawa-Johnson, M., Do., M.: Semantic image inpainting with perceptual and contextual losses. In: arXiv preprint
arXiv:1607.07539 (2016)
133. Yu, X., Porikli., F.: Ultra-resolving face images by discriminative generative networks. In: Proc. ECCV. pp. 318–333 (2016)
134. Zilly, J., Buhmann, J., Mahapatra, D.: Boosting convolutional filters with entropy
sampling for optic cup and disc image segmentation from fundus images. In:
MLMI. pp. 136–143 (2015)
135. Zilly, J., Buhmann, J., Mahapatra, D.: Boosting convolutional filters with entropy
sampling for optic cup and disc image segmentation from fundus images. In: In
Proc. MLMI. pp. 136–143 (2015)
136. Zilly, J., Buhmann, J., Mahapatra, D.: Glaucoma detection using entropy sampling and ensemble learning for automatic optic cup and disc segmentation. In
Press Computerized Medical Imaging and Graphics 55(1), 28–41 (2017)

