JOINT CONVOLUTIONAL NEURAL PYRAMID FOR DEPTH MAP SUPER-RESOLUTION
Yi Xiao1 , Xiang Cao1 , Xianyi Zhu1 , Renzhi Yang1 , Yan Zheng2

arXiv:1801.00968v1 [cs.CV] 3 Jan 2018

1 College of Computer Science and Electronic Engineering
2 College of Electric and Information Engineering
Hunan University
Changsha, China
ABSTRACT
High-resolution depth map can be inferred from a lowresolution one with the guidance of an additional highresolution texture map of the same scene. Recently, deep neural networks with large receptive fields are shown to benefit
applications such as image completion. Our insight is that
super resolution is similar to image completion, where only
parts of the depth values are precisely known. In this paper,
we present a joint convolutional neural pyramid model with
large receptive fields for joint depth map super-resolution.
Our model consists of three sub-networks, two convolutional
neural pyramids concatenated by a normal convolutional neural network. The convolutional neural pyramids extract information from large receptive fields of the depth map and
guidance map, while the convolutional neural network effectively transfers useful structures of the guidance image to the
depth image. Experimental results show that our model outperforms existing state-of-the-art algorithms not only on data
pairs of RGB/depth images, but also on other data pairs like
color/saliency and color-scribbles/colorized images.

process is rather time-consuming and infeasible in dynamic
scenes. In contrast, range sensing based methods which use
depth sensors, such as time-of-fight(TOF) camera and Microsoft Kinect, can be used in a dynamic environment. However, the acquired depth maps are of low resolution (512×424
for Kinect 2.0).

1. INTRODUCTION

In order to solve this problem, researchers have made a
lot of efforts to improve the resolution of LR depth maps.
Joint or guided image filtering methods [2, 3], which take
an extra HR texture image as a reference or guidance, have
achieved great success in recent years. The main idea of
joint filtering is to transfer the structural information in the
HR reference image to the up-sampled depth image, so that
the missing information of the depth image are restored as
much as possible. However, it is ambiguous to determine
which parts should be preserved and which parts should be
smoothed. Ham et al took into account the structures in both
target and guidance image instead of unilaterally transferring
the structures of RGB image to the depth image [4]. However, the hand-crafted objective functions used in [4] may not
reflect natural image priors well. Recently, joint deep convolutional neural networks (JDCNNs) [5, 6] were proposed
by combining the joint filtering and deep convolutional neural networks (CNNs). They showed better performances than
classical methods in depth map super resolution.

Acquiring accurate, high quality and HR depth information is
especially important for many applications of vision related
tasks, such as 3D reconstruction, virtual reality, robot vision
and 3DTV. While high quality texture images can be easy acquired by a simple color camera, depth data of HR is hard to
acquire. Generally speaking, depth acquisition methods can
be divided into stereo matching based methods, laser scanning based methods and range sensing based methods.
Stereo matching based methods obtain depth information
by correspondence matching and triangulating two or multiple texture images. However, the performance of these methods are dramatically affected by the occlusion and distributions of textures [1]. Also, high resolution results require high
computational costs. Laser scanning based methods can acquire high quality depth maps, but its slice-by-slice scanning

As pointed out by Shen et al, large receipt fields in deep
CNN model can significantly benefit applications such as
noise suppression and image completion [7]. Our insight is
that super resolution is similar to noise suppression and image completion, where only parts of the depth values are precisely known. Therefore, increasing the receipt fields in the
JDCNN model can also improve the performance of the network. In this paper, we propose a novel model called joint
convolutional neural pyramid (JCNP), which enables large
receipt fields without significantly increasing the computational costs and memory costs. Our JCNP model main consists of three sub-networks, where two convolutional neural
pyramids (CNPs) are concatenated by a CNN. The two CNPs
are designed to extract information from large receptive fields,
while the CNN model is designed to transfer the useful struc-

Index Terms— deep neural network, super-resolution,
convolutional pyramid

tures of the guidance images to the up-sampled target image.
The flowchart of our JCNP model is shown in Figure 1. To
verify the performance of our model, we tested our model
with several types of data pairs, including RGB/depth images,
color/saliency images, and color-scribbles/colorized images.
Experimental results on several benchmark data sets show
that our method outperforms the existing state-of-the-art algorithms.

2.1. Local-based Methods
In local-based depth map SR methods, each value in the SR
image is given by the weighted average of its neighboring pixels in the LR depth map and guidance image [2, 8, 3, 9, 10].
Yang et al proposed to use a HR color map as a reference
to iteratively refine LR depth maps [8]. Kopf et al proposed
a joint bilateral upsampling (JBU) procedure that considered
both the depth map smoothness and the color map similarity [2], but it may cause unwanted gradient reversal artifacts.
To solve this problem, He et al used local linear transform
of the guidance image to product filtering output [3]. Liu
et al proposed utilizing geodesic distances to upsample LR
depth map with a registered HR color image [9]. Lu et al used
the relationship between image segmentation boundaries and
depth boundaries to reconstruct HR depth map [10].
2.2. Global-based Methods

(a) Joint Convolutional Neural Pyramid (JCNP)

(b) Convolutional Neural Pyramid (CNP)

Fig. 1. The network architecture of our model. (a) Joint
pyramid network model. Our model consists of three subnetwork:CNPT , CNPG and CNNF . CNPT and CNPG extract
feature maps from the target and guidance images, respectively. CNNF concatenate the feature maps from CNPG and
CNPT and reconstruct the desired output. (b) Convolution
neural pyramid. Each pyramid level includes feature extracting part, mapping part and reconstructing part. The CNP can
enlarge the receptive fields without sacrificing computational
efficiency.

2. BACK GROUND AND RELATED WORK
Generally speaking, depth map SR methods can be classified
into local-based, global-based and CNN-based methods.

In contrast, global-based depth map SR methods [11, 12, 13,
14, 15, 4, 16] restore the SR image by solving an optimizing problem with certain regression terms. Diebel et al combined LR depth maps with registered HR camera maps to
reconstruct HR depth maps, which may cause over-smooth
depth map [11]. To maintain sharp depth boundaries and prevent depth bleeding during propagation, Park et al combined
several weighting factors together with nonlocal means filtering [12]. Given a database of training patches [13], Aodha
et al inferred the HR map from a single LR map. Ferstl et al especially formulated SR as a global energy optimization problem using Total Generalized Variation (TGV)
regularization [14]. Yang et al proposed an adaptive colorguided autoregressive model to construct a unified depth recovery framework [15]. Ham et al fused appropriate structures of static and dynamic color maps to reconstruct HR
depth maps [4]. Lei et al considered view synthesis quality
to reconstruct SR depth maps [16].
2.3. CNN-based Methods
Convolutional neural networks (CNNs) have achieved great
success in high-level computer vision [17]. They were used
to solve some low-level vision, such image SR [18, 5, 6, 19].
Existing CNN-based methods take either one or two image
as inputs. Dong et al proposed an end-to-end SR convolution neural network (SRCNN) for single image SR [18]. Ren
et al fused different individual networks to construct a super
resolution system [19]. Joint deep convolutional neural networks [5, 6], taking the depth image and guidance images
as inputs, were proposed to simulate a joint filter. Hui et al
proposed a multi-scale guided convolutional network (MSGNet) for depth map super resolution. Li et al proposed a
deep convolution network to perform joint filtering [6]. The
model consists of three sub-networks. The first two CNNs extract informative features from both target and guidance im-

ages in parallel. The third CNN then concatenates the feature
responses to selectively transfer common structures and reconstruct the filtered output [6]. In contrast, our model uses
two convolution neural pyramid [7] with large receptive fields
as feature extractors. In addition, we concatenate the target
image at the end of our model to improve the flow of information and gradients throughout the network.
3. PROPOSED METHOD
3.1. Overview
As illustrated in Figure 1(a), our joint convolutional neural
pyramid (JCNP) is composed of three sub-networks, two convolutional neural pyramids (CNPT , CNPG ) followed by a
normal CNN (CNNF ). CNPT and CNPG concurrently extract the informative features from large receptive fields in
the target and guidance images. Their output feature maps
are then concatenated and feeded to CNNF , which transfers
useful structures from the guidance image to the target image. In order to improve the flow of information and gradients
throughout the network [20], the target image is concatenated
with the output of CNNF and then convolved to produce the
final SR image.
3.2. Convolutional Neural Pyramid
Convolution neural network with large receptive fields are
essential for many low-level vision tasks such as image SR
and suppressing noise. Pyramid structure can greatly enlarge the receptive fields without sacrificing computation efficiency. We adopt the CNP model [7] in our joint framework
to extract the informative features from large receptive fields.
Each CNP level includes feature extraction/representation
part, non-linear mapping part and reconstruction part, as illustrated in Figure 1(b). In the following, we will introduce
each part in more detail.
Feature Extraction Part: The feature extraction part in
level Li , consisting of two convolution-PReLU layers with
kernel size 3×3, takes the down-sampled feature maps from
the feature extraction part in Li−1 (expect L0 , whose input
is the target image or the guidance image) and outputs 64channel feature maps. The output feature maps are feeded to
not only the non-linear mapping part in the same level, but
also to the feature extraction part in level Li+1 . Since the
feature extraction in higher levels receives recursively downsampled feature maps, it can extract information from larger
recept fields with a fixed kernel size. The down-sampling operation is performed by a max pooling layer.
Non-linear Mapping Part: The non-linear mapping part
is designed to contain three convolution layers with kernel
size 3×3. The first and second convolution layers output 16channel feature maps; third layer outputs 64-channel feature
maps. And PReLU rectification is applied after each layer.

Reconstruction Part: The reconstruction part in level Li
fuses not only the feature maps from the nonlinear mapping
part in the same level, but also the up-sampled feature maps
from the reconstruction part in level Li+1 The up-sampling
operation is performed by a deconvolution layer of stride 2.
Since features from level Li+1 with larger recept fields are
recursively fused to level Li , the final reconstruction part in
level L0 will obtain features of a large recept field.
3.3. Joint SR Network
The features extracted from the target image and the guidance
image by CNPT and CNPG should be fused appropriately to
increase the resolution of the target image. To achieve this
task, a sub-network CNNF is designed to concatenate the output feature maps of CNPT and CNPG (128 channels in total).
The CNNF consists of three convolution layers of kernel size
3×3. The first convolution layer outputs 32-channel feature
maps; the second layer outputs a 1-channel map; third layer
outputs the final SR image. A PReLU layer is applied after the
second convolution layer. In order to better guide the training,
the target image is concatenated as the input of the third convolution layer, which is shown to improve the performance in
the experiments.
3.4. Loss Function
We train our network by minimizing the mean squared error
(MSE) for N training samples as
M SE =

1 X gt
I − φ(I G , I T )
N

2

,

where Igt , IG , IT denote ground truth depth image, guidance
image and target image, respectively. And φ denotes our joint
convolutional neural pyramid model.
4. EXPERIMENTS AND APPLICATIONS
4.1. Training Settings
We implement our network in Tensorflow on an NVIDIA
GeForce GTX 1080Ti graphics card. We collect 1449 RGB/D
image pairs from NYU data set [21], 1000 RGB/D image
pairs for training and 449 RGB/D image pairs for testing. We
augment the training data by clip, rotation and mirror and generate 180,000 training patch pairs of size 128×128. The batch
size is set to 36. The network is trained with 200,000 steps
and the initial learning rate is 1e-3, which decays to 0.8 times
per 10000 steps. We train our network for joint image SR. We
get the low-resolution target image from a ground-truth image using nearest-neighbor down-sampling. When our JCNP
model is trained with the augmented RGB/D data pairs, it can
be directly applied to several different joint image SR tasks,
including depth map SR, chromaticity map SR and saliency

map SR. In experiments, we provide the average root mean
squared errors (RMSEs) of test data set, except for the visual
results, to evaluate the results.
4.2. Pyramid Levels vs. Performance
Since the number of pyramid levels N in our JCNP model
will affect the performance of our network, we test the impact
of N on joint depth map SR using the data set of Lu [22].
As shown in Table 1, our method shows the best results when
N = 2. Therefore, we set N = 2 in all our experiments. We
also compare the recept field, runtime, memory consumption,
number of parameters and performance of our JCNP model
with those of the single-level CNN model [6] (similar to level
0 of our model) as shown in Table 1. To increase the recept
field of the model in [22], we increase the number of convolution layers. As shown in Table 1, our JCNP model can
enlarge the receptive fields and produce better results without
significantly sacrificing computational efficiency.
4.3. Depth Map SR
We compare our JCNP model with several state-of-art methods, including local-based methods [2, 3], global-based
method [4], and CNN-based method [6]. We first provide
the visual comparisons with several state-of-art SR methods
in Figure 2. It is observed that the results of He [3] and Li [6]
are over smoothed (Figure 2(d)(e)). The JBU [2] and Ham [4]
approaches transfer erroneous details (Figure 2(f)(g)). In contrast, the SR depth maps reconstructed by ours JCNP model
are sharper and has less artifacts than the other methods (Figure 2(h)).
We also provide a numerical comparison with the stateof-art SR methods using the average RMSEs in Table 2. The
four benchmark test data sets include: (1) 30 RGB/depth
hole-filled pairs from Middlebury data sets [23, 24], (2) Lu
data sets [22] contains 6 RGB/depth pairs captured by ASUS
Xtion Pro, (3) the 449 RGB/D testing pairs in NYU v2 data
sets [21], (4) 299 RGB-D image pairs from align kv2 subfolder obtained by the Kinect V2 sensor in SUN RGB/D data
sets [25].
The RMSE values are computed from the code packages
with suggested parameters provided by the authors. As shown
in the Table 2, our JCNP model achieves the state-of-the-art
performance, especially for Middlebury [23, 24], Lu [22] and
NYU [21] data sets. For the Sun data set, our results are comparable to the best results in the 2× and 4× SR case. Thanks
to the large recept field provided by our JCNP model, our results outperform the best results in 8× and 16× SR.
4.4. Chromaticity Map SR
We apply our model to chromaticity map SR. The test data
are six color-scribbles/colorized image pairs provided by the

authors of [26]. We first compute the LR map by downsampling the colorized image and use it as the target image.
We then use the HR color-scribbles map as guidance image to
construct the output HR Chromaticity map.
Figure 3 shows the visual comparisons with the method
of Bicubic, He [3] and Li [6]. Our results show more faithful
edges and less color bleeding artifacts (3(f)). We also provide
the qualitative evaluations in Table 3. We use the colorized
image as the ground truth and calculate the RMSE values.
The RMSE values in Table 3 show that our results best approximate the ground truth.
4.5. Saliency Map SR
We also apply our trained JCNP model in saliency map
SR. We random collect eight color/saliency map pairs from
MSRA10K salient object database [27] in the test. We use
nearest-neighbor down-sampling to get the LR saliency map
and use HR color map as the guidance image. We compare
our results with those of Bicubic, He [3] and Li [6]. Visual
comparison in Figure 4 and numerical comparison in 3 show
that our results can better reconstruct image content by better
transferring useful structure from the guidance image to the
target image.
5. CONCLUSION AND FUTURE WORK
We have proposed a joint convolution neural pyramid for joint
image SR. Our model can enable large receptive fields and
transfer useful structures from the guidance image to enhance
the resolution of the target image. Experimental results verified the performance of method on joint depth map SR, chromaticity map SR and saliency map SR. In the future, we
would like to apply our model on more applications. Also,
another future work is to study the impact of different loss
functions on our model.
6. REFERENCES
[1] Scharstein, Daniel, Szeliski, and Richard, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” IJCV, vol. 47, no. 1-3, pp. 7–42,
2002.
[2] Johannes Kopf, Michael F. Cohen, Dani Lischinski, and
Matt Uyttendaele, “Joint bilateral upsampling,” in SIGGRAPH, 2007, p. 96.
[3] Kaiming He, Jian Sun, and Xiaoou Tang, “Guided image filtering,” in ECCV, 2010, pp. 1–14.
[4] Bumsub Ham, Minsu Cho, and Jean Ponce, “Robust
image filtering using joint static and dynamic guidance,”
in CVPR, 2015, pp. 4823–4831.

Table 1. Time(second) of each 100 steps with batch size 1, network parameters(million), RMSEs and memory consumption(MB) analysis in terms of different sizes of receptive fields. The RMSE of 58 Layers is missing, because our GPU memory
does not allow 58 layers in training. The training batch size is 36.
Our CPN
Single-Level CNN
RF Levels Time Mem. Paras. RMSE Layers Time Mem. Paras. RMSE
17
0
1.47 38.50
0.15
7.42
8
1.47
38.50
0.15
7.42
26
1
2.27 63.50
0.41
6.64
13
3.37
84.50
0.74
7.37
56
2
2.90 69.75
0.67
6.28
28
8.13 204.50
1.85
7.05
116
3
3.42 71.31
0.92
6.39
58
17.68 444.50
4.07
−

(a) RGB

(b) GT

(c) Bicubic

(d) He [3]

(e) Li [6]

(f) JBU [2]

(g) Ham [4]

(h) Ours

Fig. 2. Visual comparisons of depth map SR results(4×).
[5] Tak Wai Hui, Change Loy Chen, and Xiaoou Tang,
“Depth map super-resolution by deep multi-scale guidance,” in ECCV, 2016, pp. 353–369.
[6] Yijun Li, Jia Bin Huang, Narendra Ahuja, and
Ming Hsuan Yang, “Deep joint image filtering,” in
ECCV, 2016, pp. 154–169.
[7] Xiaoyong Shen, Ying-Cong Chen, Xin Tao, and Jiaya
Jia, “Convolutional neural pyramid for image processing,” CoRR, vol. abs/1704.02071, 2017.
[8] Qingxiong Yang, Ruigang Yang, James Davis, and
David Nister, “Spatial-depth super resolution for range
images,” in CVPR, 2007, pp. 1–8.
[9] Ming Yu Liu, Oncel Tuzel, and Yuichi Taguchi, “Joint
geodesic upsampling of depth images,” in CVPR, 2013,
pp. 169–176.
[10] Jiajun Lu and David Forsyth, “Sparse depth super resolution,” in CVPR, 2015, pp. 2245–2253.
[11] James Diebel and Sebastian Thrun, “An application of
markov random fields to range sensing,” NIPS, pp. 291–
298, 2005.

[12] Jaesik Park, Hyeongwoo Kim, Yu Wing Tai, Michael S
Brown, and Inso Kweon, “High quality depth map upsampling for 3d-tof cameras,” in ICCV, 2011, pp. 1623–
1630.
[13] Oisin Mac Aodha, Neill D. F. Campbell, Arun Nair, and
Gabriel J. Brostow, “Patch based synthesis for single
depth image super-resolution,” in ECCV, 2012, pp. 71–
84.
[14] David Ferstl, Christian Reinbacher, Rene Ranftl,
Matthias Ruether, and Horst Bischof, “Image guided
depth upsampling using anisotropic total generalized
variation,” in ICCV, 2013, pp. 993–1000.
[15] J. Yang, X. Ye, K. Li, C. Hou, and Y. Wang, “Colorguided depth recovery from rgb-d data using an adaptive
autoregressive model,” TIP, vol. 23, no. 8, pp. 3443–
3458, 2014.
[16] Jianjun Lei, Lele Li, Huanjing Yue, Wu Feng, Nam
Ling, and Chunping Hou, “Depth map super-resolution
considering view synthesis quality,” TIP, vol. 26, no. 4,
pp. 1732, 2017.

Table 2. Average RMSE comparison of depth map SR on the four benchmark datesets. We map depth values into the
range[0,255] for each data set. Bold Values indicate the best performance and underscored values indicate the second best.
Methods
Middlebury [23, 24]
Lu [22]
NYU [21]
SUN [25]
2× 4× 8× 16× 2× 4× 8× 16× 2× 4× 8× 16× 2× 4× 8× 16×
Bicubic 1.97 4.45 7.59 11.88 2.16 5.08 9.22 14.27 2.11 4.51 7.85 12.36 2.44 3.89 5.62 8.05
Kopf [2] 1.41 2.44 3.82 6.12 1.39 2.99 5.06 7.51 1.22 2.30 4.60 7.23 1.74 2.73 3.85 5.72
He [3] 2.43 4.10 7.26 11.71 3.03 4.89 8.85 14.10 2.31 4.10 7.53 12.20 2.43 3.56 5.40 7.96
Li [6] 1.28 2.14 3.63 6.13 1.24 2.55 4.71 7.66 1.30 2.00 3.41 5.67 1.75 3.25 5.04 7.56
Ham [4] 1.91 3.15 5.01 8.81 2.40 4.61 7.50 11.49 1.63 3.01 7.20 11.00 2.13 3.35 5.19 7.75
Ours
1.16 1.79 3.22 5.54 0.83 1.86 4.04 6.28 0.93 1.59 3.17 5.09 1.81 2.75 3.64 5.16

(a) Scribbles

(b) GT [26]

(c) Bicubic

(d) He [3]

(e) Li [6]

(f) Ours

Fig. 3. Visual comparisons of chromaticity map SR results(2×).

Table 3. Average RMSE comparisons of different methods
for chromaticity map SR and saliency map SR.
Methods
Bicubic He[3] Li[6] Ours
2× Chromaticity
10.91
11.18 9.72
6.73
4× Chromaticity
20.54
17.99 14.18 13.41
2× Saliency
14.08
14.99 12.29 11.19
4× Saliency
23.05
21.08 18.27 15.46

[17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, “Imagenet classification with deep convolutional
neural networks,” in NIPS, 2012, pp. 1097–1105.
[18] Chao Dong, Change Loy Chen, Kaiming He, and Xiaoou Tang, “Learning a deep convolutional network for
image super-resolution,” in ECCV, 2014, pp. 184–199.
[19] Haoyu Ren, Mostafa Elkhamy, and Jungwon Lee, “Image super resolution based on fusing multiple convolution neural networks,” in CVPR, 2017, pp. 1050–1057.
[20] Gao Huang, Zhuang Liu, and Kilian Q. Weinberger,
“Densely connected convolutional networks,” in CVPR,
2017.
[21] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and

Rob Fergus, “Indoor segmentation and support inference from rgbd images,” in ECCV, 2012, pp. 746–760.
[22] Si Lu, Xiaofeng Ren, and Feng Liu, “Depth enhancement via low-rank matrix completion,” in CVPR, 2014,
pp. 3390–3397.
[23] D. Scharstein and C. Pal, “Learning conditional random
fields for stereo,” in CVPR, 2007, pp. 1–8.
[24] Heiko Hirschmuller and Daniel Scharstein, “Evaluation
of cost functions for stereo matching,” in CVPR, 2007,
pp. 1–8.
[25] Shuran Song, Samuel P. Lichtenberg, and Jianxiong
Xiao, “Sun rgb-d: A rgb-d scene understanding benchmark suite,” in CVPR, 2015, pp. 567–576.
[26] Anat Levin, Dani Lischinski, and Yair Weiss, “Colorization using optimization,” in SIGGRAPH, 2004, pp. 689–
694.
[27] Ming-Ming Cheng, Jonathan Warrell, Wen-Yan Lin,
Shuai Zheng, Vibhav Vineet, and Nigel Crook, “Efficient salient region detection with soft image abstraction,” in ICCV, 2013, pp. 1529–1536.

(a) Input(guidance)

(b) GT

(c) Bicubic

(d) He [3]

(e) Li [6]

Fig. 4. Visual comparisons of saliency map SR results(4×).

(f) Ours

