1

Semi-Parallel Deep Neural Network (SPDNN)
Hybrid Architecture, First Application on Depth
from Monocular Camera
Shabab Bazrafkan 1, Student, IEEE, Hossein Javidnia 1, Student, IEEE, Joseph Lemley 2, Student, IEEE,
Peter Corcoran3, Fellow, IEEE

Abstract—Convolutional Neural Network (CNN) techniques
are applied to the problem of determining the depth from a single
camera image (monocular depth). Fully connected CNN
topologies preserve all details of the input images, enabling the
detection of fine details, but miss larger features; networks that
employ 2×2, 4×4 and 8×8 max-pooling operators can determine
larger features at the expense of finer details. After designing,
training and optimising a set of topologies, these networks may
be combined into a single network topology using graph
optimization techniques. This “Semi Parallel Deep Neural
Network (SPDNN)” eliminates duplicate common network
layers, reducing network size and computational effort
significantly, and can be further optimized by retraining to
achieve an improved level of convergence over the individual
topologies. In this study four models are trained and have been
evaluated at 2 stages on the KITTI dataset. The ground truth
images in the first part of the experiment come from the
benchmark, and for the second part, the ground truth images are
the depth map results from applying a state-of-the-art stereo
matching method. The results of this evaluation demonstrate that
using post-processing techniques to refine the target of the
network increases the accuracy of depth estimation on individual
mono images. The second evaluation shows that using
segmentation data as the input can improve the depth estimation
results to a point where performance is comparable with stereo
depth estimation. The computational time is also discussed in this
study.
Index Terms— Depth Estimation, Deep Convolutional Neural
Networks (CNN), Monocular Image

The research work presented here was funded under the Strategic
Partnership Program of Science Foundation Ireland (SFI) and co-funded by
SFI and FotoNation Ltd. Project ID: 13/SPP/I2868 on “Next Generation
Imaging for Smartphone and Embedded Platforms”. This work is also
supported by an Irish Research Council Employment Based Programme
Award. Project ID: EBPPG/2016/280
1
S. Bazrafkan is with the Department of Electronic Engineering, College
of Engineering, National University of Ireland, Galway, University Road,
Galway, Ireland. (e-mail: s.bazrafkan1@nuigalway.ie).
1
H. Javidnia is with the Department of Electronic Engineering, College of
Engineering, National University of Ireland, Galway, University Road,
Galway, Ireland. (e-mail: h.javidnia1@nuigalway.ie).
2
J. Lemley is with the Department of Electronic Engineering, College of
Engineering, National University of Ireland, Galway, University Road,
Galway, Ireland. (e-mail: j.lemley2@nuigalway.ie).
3
P. Corcoran is with the Department of Electronic Engineering, College of
Engineering, National University of Ireland, Galway, University Road,
Galway, Ireland. (e-mail: peter.corcoran@nuigalway.ie).

C

I. INTRODUCTION

omputing pixel depth values provides a basis for
understanding the 3D geometrical structure of images. As
it has been presented in recent research [1], using stereo
images provides an accurate depth due to the advantage of
having local correspondences; however, the processing times
of these methods is still an open issue.
To solve this problem, it has been suggested to use single
images to compute the depth values, but extracting depth from
monocular images requires extracting a large number of cues
from the global and local information in the image. Using a
single camera is more convenient in industrial applications.
Stereo cameras require detailed calibration and many
industrial use cases already employ single cameras – e.g.
security monitoring, automotive & consumer vision systems,
and camera infrastructure for traffic and pedestrian
management in smart-cities. These and other smart-vision
applications can greatly benefit from accurate monocular
depth analysis.
This challenge has been studied for a decade and is still an
open research problem.
Recently the idea of using neural networks to solve this
problem has attracted attention. In this paper, we tackle this
problem by employing a Deep Neural Network (DNN)
equipped with semantic pixel-wise segmentation utilizing our
recently published disparity post-processing method.
This paper also introduces the use of Semi Parallel Deep
Neural Networks (SPDNN). A SPDNN is a semi-parallel
network topology developed using a graph theory optimization
of a set of independently optimized CNNs, each targeted at a
specific aspect of the more general classification problem. For
the depth from monocular vision problem a fully-connected
topology, optimized for fine features, is combined with a
series of max-pooled topologies (2×2, 4×4 and 8×8) each
optimised for coarser image features. The optimized SPDNN
topology is re-trained on the full training dataset and
converges to an improved set of network weights.
It is worth mentioning that this network design strategy is
not limited to the ‘depth from monocular vision’ problem, and
further application examples and refinements will be
developed in a series of future publications, currently in press.
A. Depth Map
Deriving the 3D structure of an object from a set of 2D
points is a fundamental problem in computer vision. Most of

2
these conversions from 2D to 3D space are based on the depth
values computed for each 2D point. In a depth map, each pixel
is defined not by color, but by the distance between an object
and the camera.
In general, depth computation methods are divided into two
categories:
1- Active methods
2- Passive methods
Active methods involve computing the depth in the scene by
interacting with the objects and the environment. There are
different types of active methods, such as light-based depth
estimation, which uses the active light illumination to estimate
the distance to different objects [2]. Ultrasound and time-offlight (ToF) are other examples of active methods. These
methods use the known speed of the wave to measure the time
an emitted pulse takes to arrive at an image sensor [3].
Passive methods utilize the optical features of captured
images. These methods involve extracting the depth
information by computational image processing. In the
category of passive methods there are two primary approaches
a) Multi-view depth estimation, such as depth from stereo, and
b) Monocular depth estimation.
B. Stereo Vision Depth
Stereo matching algorithms can be used to compute depth
information from multiple images. By using the calibration
information of the cameras, the depth images can be
generated. This depth information provides useful data to
identify and detect objects in the scene [4].
In recent years, many applications, including time-of-flight
[5, 6], structured light [7], and Kinect were introduced to
calculate depth from stereo images. Stereo vision algorithms
are generally divided into two categories: Local and Global.
Local algorithms were introduced as statistical methods that
use the local information around a pixel to determine the depth
value of the given pixel. These kinds of methods can be used
for real time applications if they are implemented efficiently.
Global algorithms try to optimize an energy function to satisfy
the depth estimation problem through various optimization
techniques [8].
In terms of computation, global methods are more complex
than local methods, and they are usually impractical for real
time applications. Despite these drawbacks, they have the
advantage in being more accurate than local methods. This
advantage recently attracted considerable attention in
academic literature [9, 10].
For example, the global stereo model proposed in [9] works
by converting the image into a set of 2D triangles with
adjacent vertices. Later, the 2D vertices are converted to a 3D
mesh by computing the disparity values. To solve the problem
of depth discontinuities, a two layer Markov Random Field
(MRF) is employed. The layers are fused with an energy
function allowing the method to handle the depth
discontinuities. The method has been evaluated on the new
Middlebury 3.0 benchmark [10] and it was ranked the most
accurate at the time of the paper's publication based on the
average weight on the bad 2.0 index.
Another global stereo matching algorithm, proposed in [11],
makes use of the texture and edge information of the image.

The problem of large disparity differences in small patches of
non-textured regions is addressed by utilizing the color
intensity. In addition, the main matching cost function
produced by a CNN is augmented using the same color-based
cost. The final results are post-processed using a 5×5 median
filter and a bilateral filter. This adaptive smoothness filtering
technique is the primary reason for the algorithm’s excellent
performance and placement in the top of the Middlebury 3.0
benchmark [10].
Many other methods have been proposed for stereo depth,
such as PMSC [10], GCSVR [10], INTS [12], MDP [13],
ICSG [14], which all aimed to improve the accuracy of the
depth estimated from stereo vision, or to introduce a new
method to estimate the depth from a stereo pair. However,
there is always a trade-off between accuracy and speed for
stereo vision algorithms.
Table I shows an overview of the average normalized time
by the number of pixels (sec/megapixels) of the most accurate
stereo matching algorithms as they are ranked by the
Middlebury 3.0 benchmark, based on the “bad 2.0” metric.
The ranking is on the test dense set. This comparison
illustrates that obtaining an accurate depth from a stereo pair
requires significant processing power. These results
demonstrate that, today, these methods are too resource
intensive for real-time applications like street sensing or
autonomous navigation due to their demand for processing
resources.
To decrease the processing power of stereo matching
algorithms, researchers recently began to work on depth from
monocular images. Such algorithms estimate depth from a
single camera while keeping the processing power low.
C. Deep Learning
DNN (Deep Neural Networks) are among the most recent
approaches in pattern recognition science that are able to
handle highly non-linear problems in classification and
regression. These models use consecutive non-linear signal
processing units in order to mix and re-orient their input data
to give the most representative results. The DNN structure
learns from the input and then it generalizes what it learns into
data samples it has never seen before [15]. The typical deep
neural network model is composed of one or more
convolutional, pooling, and fully connected layers
accompanied by different regularization tasks. Each of these
units is as follows:
Convolutional Layer: This layer typically convolves the
3D image “I” with the 4D kernel “W” and adds a 3D bias term
“b” to it. The output is given by:
𝑃𝑃 = 𝐼𝐼 ∗ 𝑊𝑊 + 𝑏𝑏 (1)
where * operator is nD convolution and P is the output of the
convolution. During the training process, the kernel and bias
parameters are updated in a way that optimizes the error
function of the network output.
Pooling Layer: The pooling layer applies a (usually) nonlinear transform (Note that the average pooling is a linear
transform, but the more popular max-pooling operation is nonlinear) on the input image which reduces the spatial size of the
data representation after the operation.

3
TABLE I. COMPARISON OF THE PERFORMANCE TIME BETWEEN THE MOST ACCURATE STEREO MATCHING ALGORITHMS
Algorithm

Time/MP (s)

PMSC [10]

453

MeshStereoExt [9]

121

APAP-Stereo [10]

97.2

NTDE [11]

114

MC-CNN-acrt [16]

112

MC-CNN+RBS [17]

140

SNP-RSM [10]

258

MCCNN_Layout [10]

262

MC-CNN-fst [16]

1.26

LPU [10]

3523

MDP [13]

58.5

MeshStereo [9]

54

SOU4P-net [10]

678

INTS [12]

127

GCSVR [10]

4731

JMR [10]

11.1

LCU [10]

9572

TMAP [18]

1796

SPS [10]

49.4

IDR [19]

0.36

Programming
Platform

Hardware

C++

i7-6700K, 4GHz-GTX TITAN X

C, C++

8 Cores-NVIDIA TITAN X

1500 × 1000 (<= 400)

Matlab+Mex

i7 Core 3.5GHz, 4 Cores

n/a

i7 Core, 2.2 GHz-Geforce GTX TITAN X

1500 × 1000 (<= 400)

n/a

NVIDIA GTX TITAN Black

C++

Intel(R) Xeon(R) CPU E5-1650 0, 3.20GHz, 6 Cores32 GB RAM-NVIDIA GTX TITAN X

1500 × 1000 (<= 400)

Matlab

i5, 4590 CPU, 3.3 GHz

1500 × 1000 (<= 400)

Matlab

i7 Core, 3.5GHz

1500 × 1000 (<= 400)

n/a

NVIDIA GTX TITAN X

1500 × 1000 (<= 400)

Matlab

Core i5, 4 Cores- 2xGTX 970

1500 × 1000 (<= 400)

n/a

4 i7 Cores, 3.4 GHz

1500 × 1000 (<= 400)

C++

i7-2600, 3.40GHz, 8 Cores

1500 × 1000 (<= 400)

n/a

i7 Core, 3.2GHz-GTX 980

1500 × 1000 (<= 400)

C, C++

i7 Core, 3.2 GHz

1500 × 1000 (<= 400)

C++

i7 Core, 2.8GHz-Nvidia GTX 660Ti

1500 × 1000 (<= 400)

C++

Core i7, 3.6 GHz-GTX 980

750 × 500 (<= 200)

Matlab, C++

1 Core Xeon CPU, E5-2690, 3.00 GHz

Matlab

i7 Core, 2.7GHz

3000 × 2000 (<= 800)

C, C++

1 i7 Core, 2.8GHz

CUDA C++

NVIDIA GeForce TITAN Black

W × H (ndisp)

1500 × 1000 (<= 400)
1500 × 1000 (<= 400)
1500 × 1000 (<= 400)
1500 × 1000 (<= 400)

1500 × 1000 (<= 400)
1500 × 1000 (<= 400)

It is common to put a pooling layer after each convolutional
layer. Reducing the spatial size leads to less computational
load and also prevents over-fitting. The reduced spatial size
also provides a certain amount of translation invariance.
Fully Connected Layer: Fully connected layers are the
same as classical Neural Network (NN) layers, where all the
neurons in a layer are connected to all the neurons in their
subsequent layer. The neurons give the summation of their
input, multiplied by their weights, passed through their
activation functions.
Regularization: Regularization is often used to prevent
overfitting of a neural network. One can train a more complex
network (more parameters) with regularization and prevent
over-fitting. Different kinds of regularization methods have
been proposed. The most important are weight regularization,
drop-out [20], and batch normalization [21]. Each
regularization technique is suitable for specific applications,
and no single technique works for every task.
D. Monocular Vision Depth
Depth estimation from a single image is a fundamental
problem in computer vision and has potential applications in
robotics, scene understanding, and 3D reconstruction. This
problem remains challenging because there are no reliable
cues for inferring depth from a single image. For example,

temporal information and stereo correspondences are missing
from such images.
As the result of the recent research, deep Convolutional
Neural Networks (CNN) are setting new records for various
vision applications. A deep convolutional neural field model
for estimating depths from a single image has been presented
in [22] by reformulating the depth estimation into a continuous
conditional random field (CRF) learning problem. The CNN
employed in this research was composed of 5 convolutional
and 4 fully-connected layers. At the first stage of the algorithm
the input image was over-segmented into superpixels. The
cropped image patch centered on its centroid was used as
input to the CNN. For a pair of neighboring superpixels, a
number of similarities were considered and were used as the
input to the fully connected layer. The output of these 2 parts
was then used as input to the CRF loss layer. As a result, the
time required for estimating the depth from a single image
using the trained model decreased to 1.1 seconds on a desktop
PC equipped with a NVIDIA GTX 780 GPU with 6GB
memory.
It has been found that the superpixelling technique of [22] is
not a good choice to initialize the disparity estimation from
mono images because of the lack of the monocular visual cues
such as texture variations and gradients, defocus, or color/haze
in some parts of the image. To solve this issue a MRF learning
algorithm has been implemented to capture some of these
monocular cues [23]. The captured cues were integrated with a
stereo system to obtain better depth estimation than the stereo

4
system alone. This method uses a fusion of stereo + mono
depth estimation.
Depth Map

Semantic
Segmentation

Visible Single
Image

+

Deep Neural Network

Figure 1. The overview of the trained models in this paper. The semantic
segmentation is just used in two experiments

At small distances, the algorithm relies more on stereo vision,
which is more accurate than monocular vision. However, at
further distances, the performance of stereo degrades; and the
algorithm relies more on monocular vision.
E. Paper Overview
In this paper, a hybrid DNN is presented to estimate depth
from monocular cameras. The depth map from the stereo sets
are estimated using the same approach as [24] and they are
used as the target to train the network while using information
from a single image (the left image in stereo set) as input. Four
models are trained and evaluated to estimate the depth from
single camera images. The network structure for all the
models is same. In the first case, the input is simply the
original image. In the second case, the first channel is the
original image and the second channel is its segmentation
map. For each of these two cases, one of two different targets
are used; specifically, these targets were the stereo depth maps
with or without post-processing as explained in [24]. The
technical details of each model are presented in the next
section. Fig. 1 shows the overview of the general approach
used in this paper.
F. Contributions
In this paper two major contributions are presented:
1- A novel method to mix and merge several deep
neural networks called “Semi Parallel Deep Neural
Network (SPDNN)”.
2- The application of deep neutral networks and
SPDNN on estimating depth from a monocular
camera.
The rest of the paper is organized as follows: In the next
section the network structure, database preparation, and the
training process are presented. Section 3 discusses the results
and evaluation of the proposed method. The conclusion and
discussions are presented in the last section.
II.

METHODOLOGY

A. Network Structure
1) Semi-Parallel Deep Neural Network (SPDNN)
This paper is the first to introduce the SPDNN concept,
inspired from graph optimization techniques. In this method,
several deep neural networks are parallelized and merged in a
novel way that facilitates the advantages of each. The final
model is trained for the problem. Our observations show that

using this method reduces overfitting and provides superior
convergence compared to alternatives.
The merging of multiple networks using SPDNN is described
in the context of the current depth mapping problem. In this
particular problem, eight different networks were designed for
the depth estimation task. These are illustrated in Fig. 2 – Fig.
9 and will be described in the next section. None of these
networks on their own gave useful results on the depth
analysis problem. However, it was noticed that each network
tended to perform well on certain aspects of this task, while
failing at others. This led to the idea that it would be
advantageous to combine multiple individual networks and
train them in a parallelized architecture. Our experiments
showed that better output could be achieved by merging the
networks and then training them concurrently.
2) Individual Networks for Depth Analysis
The network shown in Fig. 2 is a deep fully convolutional
neural network (A fully convolutional neural network is a
network wherein all the layers are convolutional layers) with
no pooling and no padding. Therefore, no information loss
occurs inside the network, as there is no bottleneck or data
compression; this network is able to preserve the details of the
input samples. But the main problem is that this model is
unable to find big objects and coarse features in the image. In
order to solve this problem three other networks have been
designed as shown in Fig. 3 – Fig. 5. These three networks
take advantage of the max-pooling layers to gain transition
invariance and also to recognize bigger objects and coarser
features inside the image. These networks use 2×2, 4×4, and
8×8 max-pooling operators, respectively. Larger pooling
kernels allow coarser features to be detected by the network.
The main problem with these networks was that the spatial
details vanished as a result of data compression in pooling
layers.
After several attempts of designing different networks, the
observations showed that in order to estimate the depth from
an image, the network needed to see the whole image as one
object. To do that it requires the kernel to be the same size as
the image in at least one layer that is equivalent to a fully
connected layer inside the network.
In fully connected layers each neuron is connected to all
neurons in the previous/next layer. Due to the computationally
prohibitive nature of training fully connected layers, and their
tendency to cause overfitting, it is desirable to reduce the
number of these connections. Adding fully connected layers
results in a very tight bottleneck, which seems to be crucial for
the depth estimation task, but also causes the majority of the
details in the image to be lost. In Fig. 6 – Fig. 9 the networks
with fully connected layers are shown. These networks
correspond to networks in Fig. 2 – Fig. 5 but with
convolutional layers replaced with fully connected layers in
the right hand side of the network. Using different pooling
sizes before the fully connected layer will cause the network
to extract different levels of features, but all these
configurations introduce loss of detail.
Each of these eight configurations has its own advantages
and shortcomings, from missing the coarse features to missing
the details. None of these designs converged to a reasonable
depth estimation model.

5

input

input

output

Conv2D

Conv2D

Conv2D

Conv2D

Conv2D

Conv2D

Conv2D

Conv2D

3x3

3x3

3x3

3x3

3x3

3x3

3x3

3x3

(3C,1)

(3C,2)

(3C,3)

(3C,4)

(3C,5)

(3C,6)

(3C,7)

(3C,8)

output

Figure 2. Top row: network 1, Bottom row: graph corresponds to network1.

output

3x3

input

(3C,1)

Conv2D

Conv2D

Conv2D

3x3

3x3

3x3

(3C2P,2)

(3C2P,3)

Un-pool
2x2

Conv2D

Maxpool
2x2

input

Conv2D

Conv2D

Conv2D

Conv2D

3x3

3x3

3x3

3x3

(3C,5)

(3C2P,4)

(3C,6)

(3C,7)

(3C,8)

output

Figure 3. Top row: network 2, Bottom row: graph corresponds to network2.

output

3x3

input

(3C,1)

Conv2D

Conv2D

Conv2D

3x3

3x3

3x3

(3C4P,2)

(3C4P,3)

Un-pool
4x4

Conv2D

Maxpool
4x4

input

(3C4P,4)

Conv2D

Conv2D

Conv2D

Conv2D

3x3

3x3

3x3

3x3

(3C,5)

(3C,6)

(3C,7)

(3C,8)

output

Figure 4. Top row: network 3, Bottom row: graph corresponds to network3.

output

3x3

input

(3C,1)

Conv2D

Conv2D

Conv2D

3x3

3x3

3x3

(3C8P,3)

(3C8P,4)

(3C8P,2)

Figure 5. Top row: network 4, Bottom row: graph corresponds to network4.

Un-pool
8x8

Conv2D

Maxpool
8x8

input

Conv2D

Conv2D

Conv2D

Conv2D

3x3

3x3

3x3

3x3

(3C,5)

(3C,6)

(3C,7)

(3C,8)

output

6
output

input

Conv2D

Conv2D

3x3

3x3

3x3

3x3

Unpool
8x8

Max-pool
8x8

Conv2D
3x3

...

Conv2D

...

Conv2D

Fully-connected layers
With drop-out

input

(3C,1)

(3C,2)

(3C,3)

(3C,4)

(3C,7)

(30F,5) (330F,6)

output

Figure 6. Top row: network 5, Bottom row: graph corresponds to network5.

output

input

Conv2D

Conv2D

3x3

3x3

3x3

Unpool
8x8

Max-pool
4x4

Conv2D
3x3

...

3x3

Conv2D

...

Conv2D

Fully-connected layers
With drop-out

input

(3C,1)

(3C2P,2)

(3C2P,4)

(3C2P,3)

(30F,5) (330F,6)

(3C,7)

output

Figure 7. Top row: network 5, Bottom row: graph corresponds to network6.

output

input

Conv2D

Conv2D

3x3

3x3

3x3

3x3

Unpool
8x8

Max-pool
4x4

Conv2D
3x3

...

Conv2D

...

Conv2D

Fully-connected layers
With drop-out

input

(3C,1)

(3C4P,2)

(3C4P,3)

(3C4P,4)

(30F,5) (330F,6)

(3C,7)

output

Figure 8. Top row: network 7, Bottom row: graph corresponds to network7.
Output

Conv2D

Conv2D

3x3

3x3

3x3

Unpool
8x8

Conv2D
3x3

...

Conv2D

...

Conv2D
3x3

Maxpool
8x8

Input

Fully-connected layers
With drop-out
Input

(3C,1)

(3C8P,2)

(3C8P,3)

(3C8P,4)

Figure 9. Top row: network 8, Bottom row: graph corresponds to network8.

The main idea of the SPDNN method is to mix and merge
these networks and generate a single model which includes all
the layers of the original models in order to be able to preserve
the details and also detect the bigger objects in the scene for
the depth estimation task.

(30F,5)

(3C,7)

(330F,6)

Output

3) The SPDNN Parallelization Methodology
a)

Graph Contraction

A consideration while parallelizing neural networks is that
having the same structure of layers with the same distance
from the input, might lead all the layers to converge to similar
values. For example the first layer in all of the networks

7
shown in Fig. 2 – Fig. 9 is a 2D convolutional layer with a
3×3 kernel.
The SPDNN idea uses graph contraction to merge several
neural networks. The first step is to turn each network into a
graph in which it is necessary to consider each layer of the
network as a node in the graph. Each graph starts with the
input node and ends with output node. The nodes in the graph
are connected based on the connections in the corresponding
layer of the network. Note that the pooling and un-pooling
layers are not represented as nodes in the graph, but their
properties will stay with the graph labels, which will be
explained later.
(3C,1)

(3C,2)

(3C,4)

(3C,5)

(3C,6)

(3C,7)

(3C,8)

F

J

N

P

R

S

(3C2P,2)

(3C2P,3)

(3C2P,4)

(3C,5)

(3C,6)

(3C,7)

(3C,8)

C

G

K

N

P

R

S

A

input

(3C,3)

B

A
(3C,1)

Fig. 2 – Fig. 9 presents the networks and their
corresponding compressed graphs. Two properties are
assigned to each node in the graph. The first property is the
layer structure, and the second one is the distance of the
current node to the input node. To convert the network into a
graph, a labelling scheme is required. The proposed labelling
scheme uses different signs for different layer structures, C for
convolutional layer (for example 3C mean a convolutional
layer with 3×3 kernel), F for fully connected layer (for
example 30F means a fully connected layer with 30 neurons)
and P for pooling property (for example 4P means that the
data has been pooled by the factor of 4 in this layer).

(3C,1)

(3C4P,2)

(3C4P,3)

(3C4P,4)

(3C,5)

(3C,6)

(3C,7)

(3C,8)

A

D

H

L

N

P

R

S

(3C,1)

(3C8P,2)

(3C8P,3)

(3C8P,4)

(3C,5)

(3C,6)

(3C,7)

(3C,8)

A

E

I

M

N

P

R

S

(3C,1)

(3C,2)

(3C,3)

(3C,4)

A

B

F

J

(3C,1)

(3C2P,2)

(3C2P,3)

(3C2P,4)

A

C

G

K

(3C,1)

(3C4P,2)

A

(3C,7)

(30F,5) (330F,6)

T

Q

O

(30F,5) (330F,6)

(3C4P,4)

H

L

(3C,7)

Q

O

(3C4P,3)

D

output

T
(3C,7)

(30F,5) (330F,6)

O

(3C,1)

(3C8P,2)

(3C8P,3)

(3C8P,4)

A

E

I

M

T

Q
(3C,7)

(30F,5) (330F,6)

O

Q

T

Figure 10. Parallelized version of the graphs shown in Fig. 2 – Fig. 9 sharing a single input node and single output node
(3C)

(3C)

(3C)

B

F

J

(3C2P)

input

(3C)

A

(3C2P)

(3C2P)

G

K

(3C4P)

(3C4P)

(3C4P)

D

H

L

C

(3C8P)

(3C8P)

(3C8P)

E

I

M

(3C)

(3C)

(3C)

(3C)

N

P

R

S

(30F)

(330F)

O

Q

output

(3C)

T

Figure 11. Contracted version of the big graph shown in Fig. 10

Some properties, like convolutional and fully connected
layers, occur in a specific node, but pooling and un-pooling
operations will stick with the data to the next layers. The
pooling property stays with the data except when an unpooling or a fully connected layer is reached. For example, a
node with the label (3C8P, 4) corresponds to a convolutional
layer with a 3×3 kernel, the 8P portion of this label indicates

that the data has undergone 8×8 pooling and the 4 at the end
indicates that this label is at a distance of 4 from the input
layer. The corresponding graphs, with assigned labels for each
network, are illustrated in Fig. 2 – Fig. 9.
The next step is to put all these graphs in a parallel format
sharing a single input and single output node. Fig. 10 shows
the graph in this step.

8
In order to merge layers with the same structure and the
same distance from the input node, nodes with the exact same
properties are labelled with the same letters. For example, all
the nodes with properties (3C, 1) are labelled with letter A,
and all the nodes with the properties (3C2P, 4) are labelled K,
and so on.
The next step is to apply graph contraction on the parallelized
graph. In the graph contraction procedure, the nodes with the
same label are merged to a single node while saving their
connections to the previous/next nodes. For instance, all the
nodes with label A are merged into one node, but its
connection to the input node and also nodes B, C, D and E are

preserved. The contracted version of the graph in Fig. 10 is
shown in Fig. 11.
Afterwards, the graph has to be converted back to the neural
network structure. In order to do this process, the preserved
structural properties of each node are used. For example node
C is a 3×3 convolutional layer which has experienced a
pooling operation. Note that the pooling quality will be
recalled from the original network.
The concatenation layer is used in the neural network in order
to implement the nodes wherein several other nodes lead to
one node. For example, in nodes N and O the outputs of nodes

Figure 13. The converted network from graph shown in Fig. 11

J, K, L and M are concatenated with the pooling qualities
taken from their original networks.
b)

The Combined Model/Architecture

In the final model presented in Fig. 13, the input image is
first processed in four, parallel fully convolutional subnetworks with different pooling sizes. This provides the
advantages of different networks with different pooling sizes
at the same time. The outputs of these four sub-networks are
concatenated in two different forms; one to pool the larger
images to be the same size as the smallest image in the
previous part, and the other one is to un-pool the smaller
images of the previous part to be the same size as the largest
image.
After merging these outputs, the data is led to 2 different
networks. One is the fully convolutional network to deepen
the learning and release more abstract features of the input,
and the other network is an auto-encoder network with
different architecture for encoder and decoder.
As previously mentioned, having a fully connected layer in
the network is crucial for the reasonable estimation of the
image’s depth which is provided in the bottleneck of the auto
encoder. The results from the auto encoder and the fully
convolutional sub-network are again merged in order to give a
single output after applying a one channel convolutional layer.
In order to regularize the network, prevent overfitting and
increase the convergence, batch normalization [21] is applied
after every convolutional layer, and the drop-out technique
[20] is used in fully connected layers. The experiments in this
paper show that using weight regularization in the fully
connected layers gives slower convergence; therefore, this
regularization was eliminated from the final design. All the

nonlinearities in the network are the ReLU non linearity,
which is widely used in deep neural networks, except the
output layer, which took advantage of the sigmoid
nonlinearity. The value repeating technique was used in the
un-pooling layer due to non-specificity of the corresponding
pooled layer in the decoder part of the auto-encoder subnetwork.
The value repeating technique, illustrated in Fig. 12, involves
repeating the value from the previous layer in order to obtain
the un-pooled image. The figure shows the 2×2 un-pooling,
and the process is the same for other un-pooling sizes.
4) SPDNN: How it Works and why it is Effective?
One might ask why the SPDNN approach is effective and
what the difference is between this approach and other mixing
approaches. Here the model designed by the SPDNN scheme
is investigated in the forward and back propagation steps. The
key component is in the back-propagation step where the
parameters in parallel layers influence each other. These two
steps are described below:
Forward propagation: Consider the network designed by the
SPDNN approach shown in Fig. 14. This exemplar network is
made of five sub-networks. Just the general view of the
network is shown in this figure and the layers’ details are
ignored since the main goal is to show the information flow
within the whole network.
When the input samples are fed into the network, the data
travels through the network along three different paths shown
in Fig. 15.

9

1

5

5

1 1
2 2

5
9

5

2 2

9

9

1
1 5
2 9

Unpool
2x2

shortcomings by reducing the final mixed error value. In the
voting approach, different classifiers are trained independently
of each other and they do not communicate to reduce their
total error value.

9

a)

Figure 12. Repeating technique used in un-pooling layers.

Sub-Net
1

Sub-Net
3

Input

Sub-Net
5
Sub-Net
2

Output

1.

Sub-Net
4

Figure 14. A network designed using the SPDNN approach. It contains 5 subnetworks placed in parallel and semi-parallel form.
path1
Input

path2

path3

Sub-Net
1

Sub-Net
3
Sub-Net
5

Sub-Net
2

Error

Output

Input

path2

path3

Sub-Net
1

3.

Sub-Net
3
Sub-Net
5

Sub-Net
2

2.

Sub-Net
4

Figure 15. Forward propagation inside the SPDNN. There are three different
paths on which the information can flow inside the network

path1

SPDNN vs Inception

One of the approaches that has superficial similarities to
SPDNN is the Inception technique [25]. For clarity, and to aid
the reader in understanding, the authors list four significant
points of difference between SPDNN and Inception with
regard to mixing networks.

Output
Error

Sub-Net
4

Figure 16. Back propagation for SPDNN. The mixed error is back propagated
throughout the network while updating parameters.

At this stage the parallel networks are blind to each other, i.e.,
the networks placed in parallel do not share any information
with each other. As shown in Fig. 15 the data traveling in SubNet 1 and Sub-Net 2 are not influenced by each other since
they do not share any path together, as in Sub-Net 3 and SubNet 4.
Back propagation: while training the network, the loss
function calculated based on the error value at the output of
the neural network is a mixed and merged function of the error
value corresponding to every data path in the network. In the
backpropagation step the parameters inside the network update
based on this mixed loss values. i.e., this value backpropagates throughout the whole network as it is shown in Fig.
16. Therefore, at this stage of training, each subnetwork is
influenced by the error value from every data path shown in
Fig. 16. This illustrates the way each subnetwork is trained to
reduce the error of its own path and also the error from the
mixture of all paths.
The main difference between the SPDNN approach and other
mixing approaches, like the voting approach, lies in the back
propagation step where different sub-nets are influenced by
the error of each other and try to compensate for each other’s

4.

The main idea in SPDNN is to maintain the overall
structure of the networks, but to mix them in a
reasonable way. For example, if there is a big kernel
such as 13x13 in one of the configurations, the
SPDNN method always preserves the structure
(13x13 kernel) inside the final network. This
contrasts with inception [25], which reduces larger
kernels into smaller ones.
In the inception method, all the layers are merged
into one final layer, which does not happen with the
SPDNN approach.
The number of the layers in the SPDNN architecture
is less than or equal to the number of the layers in the
original networks. In contrast, the inception idea aims
to increase the number of layers in the network by (it
breaks down each layer into several layers with
smaller kernels).
The SPDNN idea is to design a new network from
existing networks that perform well at some task or
subtask while the idea in inception is to design a
network from scratch.

B. Database
In this paper, the KITTI Stereo 2012, 2015 datasets [26] are
used for training and evaluation of the network. The database
is augmented by vertical and horizontal flipping to expand the
total size to 33,096 images. 70% of this dataset is used for
training, 20% for validation and 10% for testing. Each model
is trained for two sets of input samples and two sets of output
targets. The input and target preparation are explained in the
following sections.
1) Data Preparation
a)

Input Preparation

Two different sets have been used as the input of the
network. The first set includes the visible images given by the
left camera. The second set is the visible image + the semantic
segmentation of the corresponding input. This gives the
opportunity of investigating the segmentation influence on the
depth estimation problem. The segmentation map for each
image is calculated by employing the well-known model
“SegNet” [27, 28]. This model is one of the most successful
recent implementations of DNN for semantic pixel-wise image
segmentation and has surpassed other configurations of Fully

10

b)

Target Preparation

The targets for training the network are generated from the
stereo information using the Adaptive Random Walk with
Restart algorithm [31]. The output of the stereo matching
algorithm suffers from several artefacts which are addressed
and solved by a post-processing method in [24]. In the present
experiments, both depth maps (before post-processing and
after post-processing) are used independently as targets. The
post processing procedure is based on the mutual information
of the RGB image (used as a reference image) and the initial
estimated depth image. This approach has been used to
increase the accuracy of the depth estimation in stereo vision
by preserving the edges and corners in the depth map and
filling in the missing parts. The method was compared with
the top 8 depth estimation methods in the Middlebury
benchmark [10] at the time the paper was authored. Seven
metrics, including Mean Square Error (MSE), Root Mean
Square Error (RMSE), Peak Signal-to-Noise Ratio (PSNR),
Signal-to-Noise Ratio (SNR), Mean Absolute Error (MAE),
Structural Similarity Index (SSIM) and Structural

Dissimilarity Index (DSSIM) were used to evaluate the
performance of each method. The evaluation ranked the
method as 1st in 5 metrics and 2nd and 3rd in other metrics.
C. Training
As described in sections 2.2.1.1 and 2.2.1.2 there are two
separate sets as input and two separate sets as targets for the
training process. This will give four experiments in total as
follows:
1. Experiment 1: Input: Left Visible Image + Pixel-wise
Segmented Image. Target: Post-Processed Depth map.
2. Experiment 2: Input: Left Visible Image. Target: PostProcessed Depth map.
3. Experiment 3: Input: Left Visible Image + Pixel-wise
Segmented Image. Target: Depth map.
4. Experiment 4: Input: Left Visible Image. Target: Depth
map.
The images are resized to 80×264 pixels during the whole
process. Training is done on a standard desktop with an
NVIDIA GTX 1080 GPU with 8GB memory.
In the presented experiments, the mean square error value
between the output of the network and the target values have
been used as the loss function, and the Nestrov momentum
technique [32] with learning rate 0.01 and momentum 0.9 has
been used to train the network. The Training and Validation
Loss for each of these experiments are shown in Fig. 17 and
Train Loss for four different experiments

0.1

Experiment 1
0.09

Experiment 2

0.08

Experiment 4

Experiment 3

0.07

Mean Square Error

Convolutional Networks (FCN) both in accuracy and
simplicity of implementation. SegNet uses the convolutional
layers of the VGG16 network as the encoder of the network
and eliminates the fully connected layers, thus reducing the
number of trainable parameters from 134M to 14.7M, which
represents a reduction of 90% in the number of parameters to
be trained. The encoder portion of SegNet consists of 13
convolutional layers with ReLU nonlinearity followed by
max-pooling (2×2 window) and stride 2 in order to implement
a non-overlapping sliding window. This consecutive maxpooling and striding results in a network configuration that is
highly robust to translation in the input image but has the
drawback of losing spatial resolution of the data.
This loss of spatial resolution is not beneficial in
segmentation tasks where it is necessary to preserve the
boundaries from the input image in the segmented output. To
overcome this problem, the following solution is given in [27].
As most of the spatial resolution information is lost in the
max-pooling operation, saving the information of the maxpooling indices and using this information in the decoder part
of the network preserves the high frequency information.
Note that for each layer in the encoder portion of the
network there is a corresponding decoder layer. The idea of
SegNet is that wherever max-pooling is applied to the input
data, the index of the feature with the maximum value is
preserved. Later these indices will be employed to make a
sparse feature space before the de-convolution step, applying
the un-pooling step in the decoder part. A batch normalization
layer [21] is placed after each convolutional layer to avoid
overfitting and to promote faster convergence. Decoder filter
banks are not tied to corresponding encoder filters and are
trained independently in the SegNet architecture.
In our experiments, SegNet was trained using Stochastic
Gradient Descent (SGD) with learning rate 0.1 and momentum
0.9. In this paper, the Caffe implementation of SegNet has
been employed for training purposes [29]. The gray-scale
CamVid road scene database (360×480) [30] has been used in
the training step.

0.06

0.05

0.04

0.03

0.02

0.01

0

500

1000

Epoches

Figure 17. Train Loss for each experiment

1500

11
Validation Loss for four different experiments

0.065

Experiment 1

0.06

Experiment 2
Experiment 3

0.055

Experiment 4

Mean Square Error

0.05

0.045

0.04

0.035

0.03

0.025

500

0

1000

1500

Epoches

Figure 18. Validation Loss for each experiment

Fig. 18 respectively. These figures show that using the PostProcessed Depth map as the target results in lower loss values,
which means that the network was able to learn better features
in those experiments, while semantic segmentation decreases
the error only marginally.
III.

RESULTS AND EVALUATIONS

The evaluation in this paper has been done in 3 parts. In the
first two parts, the four experiments given in section 2.3 are
compared to each other given different ground truths, and the
last part compares the proposed method to a stereo matching
method. For evaluation purposes, 8 metrics including PSNR,
MSE (between 0 and 1), RMSE (between 0 and 1), SNR,
MAE (between 0 and 1), Structural Similarity Index
(SSIM)(between 0 and 1) [33], Universal Quality Index (UQI)
(between 0 and 1) [34] and Pearson Correlation Coefficient
(PCC) (between -1 and 1) [35] are used. For the metrics
PSNR, SNR, SSIM, UQI, and PCC the larger value indicates
better performance, and for MSE, RMSE and MAE, the lower
value indicates better performance. PSNR, MSE, RMSE,
MAE and SNR represent the general similarities between two
objects. UQI and SSIM are structural similarity indicators and
PCC represents the correlation between two samples. To the
best of our knowledge there have been no other attempts at
estimating depth from a mono camera on the KITTI
benchmark.
TABLE II. NUMERICAL COMPARISON OF THE MODELS GIVEN THE
BENCHMARK’S GROUND TRUTH
Experiment
Experiment
Experiment
Experiment
1
2
3
4
PSNR
13.7677
13.8333
13.8179
14.3424
MSE
0.0436
0.0435
0.0439
0.0382
RMSE
0.2069
0.206
0.2066
0.1937
SNR
4.4026
3.8279
6.1798
6.1952
MAE
0.1212
0.1236
0.1234
0.1107
SSIM
0.9955
0.9955
0.9955
0.9959
UQI
0.9234
0.9053
0.9064
0.9252
PCC
0.7687
0.7702
0.7729
0.8485

A. Comparing Experiments Given Benchmark Ground Truth
The KITTI database came with a depth map ground truth
generated by a LIDAR scanner. The test set has been forward
propagated through the four different models trained in the
four experiments, and the output of the networks has been

compared to the benchmark ground truth. The results are
shown in Table II.
The best value for each metric is presented in bold. Fig. 19 –
Fig. 21 represent the color coded depth maps computed by the
trained models using the proposed DNN, where the dark red
and dark blue parts represent closest and furthest points to the
camera respectively. On the top right of each figure, the
ground truth given by the benchmark is illustrated. For
visualization purposes, all of the images presented in this
section are upsampled using Joint Bilateral Upsampling [36].
The results show that using semantic segmentation along with
the visible image as input will improve the model marginally.
Using the post-processed target in the training stage helps the
model to converge to more realistic results. As it is illustrated
in Fig. 19 – Fig. 21, the depth map generated in experiment 1
contains more structural details, and more precise, less faulty
depth levels compared with the other experiments. In general,
the presented models in this paper are able to handle
occlusions and discontinuities on different depth levels.

RGB Reference Frame

Ground Truth

Model 1 (Experiment 1)

Model 2 (Experiment 2)

Model 3 (Experiment 3)
Model 4 (Experiment 4)
Figure19. Estimated Depth Maps from the Trained Models – Example 1

RGB Reference Frame

Ground Truth

Model 1 (Experiment 1)

Model 2 (Experiment 2)

Model 3 (Experiment 3)
Model 4 (Experiment 4)
Figure 20. Estimated Depth Maps from the Trained Models – Example 2

RGB Reference Frame

Ground Truth

Model 1 (Experiment 1)

Model 2 (Experiment 2)

Model 3 (Experiment 3)

Model 4 (Experiment 4)

12
Figure 21. Estimated Depth Maps from the Trained Models – Example 3

B. Comparing Experiments Given the Ground Truth from
Stereo Matching
In this section, proposed models are compared to see which
one produces closer results to the target value. This gives an
idea whether using deep learning techniques on the mono
camera can produce reasonable results or not. Images in the
test set have been forward propagated through the models
trained in section 2.3, and the outputs are compared with the
depth map generated by [24]. The numerical results are shown
in Table III.
The best value for each metric is presented in bold. Fig. 22 –
Fig. 24 represent the color coded depth maps computed by the
trained models using the proposed DNN, where the dark red
and dark blue parts represent closest and furthest points to the
camera respectively. On the top right of each figure, the
ground truth calculated by [24] is illustrated. For visualization
purposes, all of the images presented in this section are
upsampled using Joint Bilateral Upsampling [36]. The results
show that using semantic segmentation along with the visible
image as input will improve the model marginally. Using the
post-processed target in the training stage helps the model to
converge to more realistic results. Fig. 22 – Fig. 24 indicate
that the trained models in this paper are able to estimate depth
maps comparable to state-of-the-art stereo matching with
structural accuracy and precise depth levels. This is also a
result of using the semantic segmentation data and injecting
the structural information into the network.
TABLE III. NUMERICAL COMPARISON OF THE MODELS –
EXPERIMENT 2
Experiment
Experiment
Experiment
Experiment
1
2
3
4
PSNR
14.1895
13.3819
14.0491
15.0418
MSE
0.0447
0.0535
0.0441
0.0378
RMSE
0.203
0.2223
0.2039
0.1854
SNR
7.9696
5.4271
6.0943
8.822
MAE
0.1581
0.1673
0.153
0.1442
SSIM
0.9943
0.994
0.9951
0.9952
UQI
0.8369
0.7951
0.8178
0.8401
PCC
0.795
0.704
0.6919
0.8082

RGB Reference Frame

Ground Truth Computed by Stereo
Matching

Model 1 (Experiment 1)

Model 2 (Experiment 2)

Model 3 (Experiment 3)
Model 4 (Experiment 4)
Figure 22. Estimated Depth Maps from the Trained Models – Example 1

RGB Reference Frame

Ground Truth Computed by Stereo
Matching

Model 1 (Experiment 1)

Model 2 (Experiment 2)

Model 3 (Experiment 3)
Model 4 (Experiment 4)
Figure 23. Estimated Depth Maps from the Trained Models – Example 2

RGB Reference Frame

Ground Truth Computed by Stereo
Matching

Model 1 (Experiment 1)

Model 2 (Experiment 2)

Model 3 (Experiment 3)
Model 4 (Experiment 4)
Figure 24. Estimated Depth Maps from the Trained Models – Example 3

C. Comparing Mono Camera Results with Stereo Matching
In this section the results from the mono camera depth
estimation given by the proposed method is compared with
one of the top ranked stereo matching methods given in [24].
The ground truth for this comparison is the set of depth maps
provided by the KITTI benchmark.
The test images have been forward propagated through the
models trained in section 2.3 and the best results are compared
with the stereo matching technique. The results are shown in
Table IV.
The results indicate that using mono camera images and deep
learning techniques can provide results which are comparable
to stereo matching techniques. As shown in Table IV, the
mono camera DNN method was able to provide depth maps
similar to the stereo matching methods, represented by PSNR,
MSE, MAE, RMSE and SNR.
TABLE IV. NUMERICAL COMPARISON BETWEEN STEREO
MATCHING AND THE PROPOSED MONO CAMERA MODEL
Stereo Matching [24] Mono Camera DNN
PSNR
14.8234
14.3424
MSE
0.0351
0.0382
RMSE
0.1845
0.1937
SNR
4.8836
4.4026
MAE
0.1017
0.1107
SSIM
0.9966
0.9959
UQI
0.9353
0.9234
PCC
0.823
0.7687

13
Calculation Time Comparison (logarithmic scale)

2
1
0

M

es
h
P
A Ste MS
PA re
C
P- oE [1
St xt 0]
er [9
eo ]
M
C
[
M -CN N 10
T ]
C
-C N- DE
N ac
[
N
+R rt [ 11]
M
1
C
C SN BS 6]
N
N P-R [17
_L
S ]
M ay M
C o
-C ut [10
N [1 ]
N 0
-fs ]
t[
1
LP 6]
M
U
es M [1
SO hS DP 0]
U tere [13
4P o
]
-n [9
et ]
[1
0
I
G NT ]
C S
SV [1
R 2]
[
JM 10]
R
LC [10
TM U ]
A [10
P ]
[1
Pr
S
op
PS 8]
os
[
ed ID 10]
M R[
et
1
ho 9]
d

log10(Time/MP (s))

3

Methods

Figure 25. Comparison of computational time in logarithmic scale

Having close values for SSIM (0.9966 and 0.9959 in the range
[0,1]) and UQI (0.9353 and 0.9234 in the range [0,1]) shows
how the mono camera DNN method is able to preserve the
structural information, as compared to the Stereo Matching
method.
D. Comparing Running Times
In this section the computational time of the proposed method
is compared against the stereo matching methods provided in
Table I. The evaluations indicate that the proposed method is
able to perform at a rate of ~1.23 sec/MP on a desktop
computer equipped with i7 2600 CPU @ 3.4 GHz and 16GB
of RAM.
Fig. 25 shows the comparison of the computational times. The
comparison is done in a logarithmic scale due to the large
range of computational times between different methods.
IV. CONCLUSION AND DISCUSSION
In this paper we have introduced the use of the Semi Parallel
Deep Neural Networks (SPDNN) method. An SPDNN is a
semi-parallel network topology developed using a graph
theory optimization of a set of independently optimized
CNNs, each targeted at a specific aspect of the more general
classification problem. For depth estimation from a monocular
set up, a model including fully-connected topology optimized
for fine features is combined with a series of max-pooled
topologies. The optimized SPDNN topology is re-trained on
the full training dataset and converges to an improved set of
network weights. Here we used this design strategy to train an
accurate model for estimating depth from monocular images.
In this work, 8 different deep neural networks have been
mixed and merged using the SPDNN method in order to take
advantage of each network’s qualities. The mixed network
architecture was then trained in four separate scenarios
wherein each scenario uses a different set of inputs and targets
during training. Four distinct models have been trained. The
pixel-wise segmentation and depth estimations given in [24]
were used to provide samples for use in the training stage. The
KITTI benchmark was used for training and experimental
purposes.
Each model was evaluated in two sections, first against the
ground truth provided by the benchmark, and secondly against

the disparity maps computed by the stereo matching method
(section 3.1 and 3.2). The results show that using the postprocessed depth map presented in [24] for training the network
results in more precise models, and adding the semantic
segmentation of the input frame to the input helps the network
preserve the structural information in the output depth map.
The results in section 3.2 show how close the proposed depth
estimation using mono camera can be to the stereo matching
method. The semantic segmentation information helps the
network converge to the stereo matching results, although the
improvement is marginal in this case. The results of the third
comparisons in section 3.3 show a slightly higher accuracy
obtained by employing the stereo matching technique, but our
results demonstrate that there is not a big difference between
the depths from the models trained by proposed DNN and the
values computed by stereo matching. The numerical results for
this evaluation show the similarity between the mono camera
using DNN method and the stereo matching method, and also
the power of the presented method in preserving the structural
information in the output depth map.
An important advantage of these models is the processing time
of ~1.23 sec/MP. This is equal to 38 fps for an input image of
size (80×264) on an i7 2600 CPU @ 3.4 GHz and 16GB of
RAM. This makes the model suitable for providing depth
estimation in real time. This performance is comparable to the
stereo methods MC-CNN-fst [16] and JMR [10], which are 37
fps and 4 fps respectively for the same size of image, taking
advantage of GPU computation power (NVIDIA GTX TITAN
X and GTX 980 respectively). The IDR method [19] can give
up to 131 fps for the same image size by using an NVIDIA
GeForce TITAN Black GPU and CUDA C++ implementation,
but the performance on CPU is not given by the authors, so
any comparisons with this method would be unfair.
Using pixel-wise segmentation as one of the inputs of the
network slightly increased the accuracy of the models, and
also helped the model preserve the structural details of the
input image. However, it also brought some artefacts, such as
wrong depth patches on the surfaces. The evaluation results
also illustrate the higher accuracy of the models where a postprocessed depth map was used as the target in the training
procedure.
A. Future Works and Improvements
The model presented in this work is still a big model to
implement in low power consumer electronic devices (e.g.,
hand held devices). Future work will include a smaller design
which is able to perform as well as the presented model. The
other consideration for the current method is the training data
size (which is always the biggest consideration with deep
learning approaches). The amount of stereo data available in
the databases is usually not big enough to train a deep neural
network. The augmentation techniques can help to expand
databases, but the amount of extra information they provide is
limited. Providing a larger set with accurate depth maps will
improve the results significantly.
The SPDNN approach is currently being to other problems
and is giving promising results on both classification and
regression problems. Those results will be presented in future
publications.

14
REFERENCE
[1]

[2]
[3]
[4]
[5]

[6]

[7]

[8]

[9]

[10]
[11]

[12]

[13]

[14]

[15]
[16]
[17]
[18]

[19]

[20]

M. Weber, M. Humenberger, and W. Kubinger, "A very fast
census-based stereo matching implementation on a graphics
processing unit," in 2009 IEEE 12th International Conference on
Computer Vision Workshops, ICCV Workshops, 2009, pp. 786793.
B. Freedman, A. Shpunt, M. Machline, and Y. Arieli, "Depth
mapping using projected patterns," ed: Google Patents, 2013.
A. Govari, A. C. Altmann, Y. Ephrath, and V. Gliner, "Tissue
depth estimation using gated ultrasound and force measurements,"
ed: Google Patents, 2016.
D. Nair. (Aug 24, 2016). 3D Imaging with NI LabVIEW.
Available: http://www.ni.com/white-paper/14103/en/
C. Niclass, M. Soga, H. Matsubara, and S. Kato, "A 100m-Range
10-Frame/s 340×96-Pixel Time-of-Flight Depth Sensor in 0.18-μm
CMOS," in ESSCIRC (ESSCIRC), 2011 Proceedings of the, 2011,
pp. 107-110.
C. Niclass, K. Ito, M. Soga, H. Matsubara, I. Aoyagi, S. Kato, et
al., "Design and characterization of a 256x64-pixel single-photon
imager in CMOS for a MEMS-based laser scanning time-of-flight
sensor," Optics Express, vol. 20, pp. 11863-11881, 2012/05/21
2012.
D. Scharstein and R. Szeliski, "High-accuracy stereo depth maps
using structured light," in Computer Vision and Pattern
Recognition, 2003. Proceedings. 2003 IEEE Computer Society
Conference on, 2003, pp. I-195-I-202 vol.1.
R. K. Gupta and S.-Y. Cho, "A Correlation-Based Approach for
Real-Time Stereo Matching," in Advances in Visual Computing:
6th International Symposium, ISVC 2010, Las Vegas, NV, USA,
November 29 – December 1, 2010, Proceedings, Part II, G. Bebis,
R. Boyle, B. Parvin, D. Koracin, R. Chung, R. Hammound, et al.,
Eds., ed Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp.
129-138.
C. Zhang, Z. Li, Y. Cheng, R. Cai, H. Chao, and Y. Rui,
"MeshStereo: A Global Stereo Model with Mesh Alignment
Regularization for View Interpolation," in 2015 IEEE
International Conference on Computer Vision (ICCV), 2015, pp.
2057-2065.
Middlebury Stereo Evaluation - Version 3. Available:
http://vision.middlebury.edu/stereo/eval3/
K. R. Kim and C. S. Kim, "Adaptive smoothness constraints for
efficient stereo matching using texture and edge information," in
2016 IEEE International Conference on Image Processing (ICIP),
2016, pp. 3429-3433.
X. Huang, Y. Zhang, and Z. Yue, "Image-Guided Non-Local
Dense Matching with Three-Steps Optimization," ISPRS Ann.
Photogramm. Remote Sens. Spatial Inf. Sci., vol. III-3, pp. 67-74,
2016.
A. Li, D. Chen, Y. Liu, and Z. Yuan, "Coordinating Multiple
Disparity Proposals for Stereo Computation," in IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), Las Vegas,
USA, 2016, pp. 4022-4030.
M. Shahbazi, G. Sohn, J. Théau, and P. Ménard, "Revisiting
Intrinsic Curves for Efficient Dense Stereo Matching," ISPRS Ann.
Photogramm. Remote Sens. Spatial Inf. Sci., vol. III-3, pp. 123130, 2016.
B. Yegnanarayana, Artificial neural networks: PHI Learning Pvt.
Ltd., 2009.
J. Žbontar and Y. LeCun, "Stereo matching by training a
convolutional neural network to compare image patches," J. Mach.
Learn. Res., vol. 17, pp. 2287-2318, 2016.
J. T. Barron and B. Poole, "The Fast Bilateral Solver,"
arXiv:1511.03296, vol. abs/1511.03296, 2016.
E. T. Psota, J. Kowalczuk, M. Mittek, P. L. C, and rez, "MAP
Disparity Estimation Using Hidden Markov Trees," in 2015 IEEE
International Conference on Computer Vision (ICCV), 2015, pp.
2219-2227.
J. Kowalczuk, E. T. Psota, and L. C. Perez, "Real-Time Stereo
Matching on CUDA Using an Iterative Refinement Method for
Adaptive Support-Weight Correspondences," IEEE Transactions
on Circuits and Systems for Video Technology, vol. 23, pp. 94-104,
2013.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R.
Salakhutdinov, "Dropout: a simple way to prevent neural networks

[21]
[22]

[23]

[24]
[25]

[26]
[27]
[28]

[29]
[30]

[31]
[32]
[33]

[34]
[35]
[36]

from overfitting," J. Mach. Learn. Res., vol. 15, pp. 1929-1958,
2014.
S. Ioffe and C. Szegedy, "Batch Normalization: Accelerating Deep
Network Training by Reducing Internal Covariate Shift,"
arXiv:1502.03167, vol. abs/1502.03167, 2015.
F. Liu, C. Shen, G. Lin, and I. Reid, "Learning Depth from Single
Monocular Images Using Deep Convolutional Neural Fields,"
IEEE Transactions on Pattern Analysis and Machine Intelligence,
vol. 38, pp. 2024-2039, 2016.
A. Saxena, J. Schulte, and A. Y. Ng, "Depth estimation using
monocular and stereo cues," in Proceedings of the 20th
international joint conference on Artifical intelligence, Hyderabad,
India, 2007, pp. 2197-2203.
H. Javidnia and P. Corcoran, "A Depth Map Post-Processing
Approach Based on Adaptive Random Walk With Restart," IEEE
Access, vol. 4, pp. 5509-5519, 2016.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, et
al., "Going deeper with convolutions," in Proceedings of the IEEE
conference on computer vision and pattern recognition, 2015, pp.
1-9.
M. Menze and A. Geiger, "Object scene flow for autonomous
vehicles," in 2015 IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2015, pp. 3061-3070.
V. Badrinarayanan, A. Kendall, and R. Cipolla, " SegNet: A Deep
Convolutional Encoder-Decoder Architecture for Image
Segmentation," arXiv:1511.00561, vol. abs/1511.00561, 2015.
A. Kendall, V. Badrinarayanan, and R. Cipolla, "Bayesian segnet:
Model uncertainty in deep convolutional encoder-decoder
architectures for scene understanding," arXiv preprint
arXiv:1511.02680, 2015.
A. Kendall, V. Badrinarayanan, and R. Cipolla. Caffe
Implementation
of
SegNet.
Available:
https://github.com/alexgkendall/caffe-segnet
G. J. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla,
"Segmentation and Recognition Using Structure from Motion
Point Clouds," in Computer Vision – ECCV 2008: 10th European
Conference on Computer Vision, Marseille, France, October 1218, 2008, Proceedings, Part I, D. Forsyth, P. Torr, and A.
Zisserman, Eds., ed Berlin, Heidelberg: Springer Berlin
Heidelberg, 2008, pp. 44-57.
S. Lee, J. H. Lee, J. Lim, and I. H. Suh, "Robust stereo matching
using adaptive random walk with restart algorithm," Image and
Vision Computing, vol. 37, pp. 1-11, 2015/05/01/ 2015.
I. Sutskever, J. Martens, G. E. Dahl, and G. E. Hinton, "On the
importance of initialization and momentum in deep learning,"
ICML (3), vol. 28, pp. 1139-1147, 2013.
W. Zhou, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image
quality assessment: from error visibility to structural similarity,"
IEEE Transactions on Image Processing, vol. 13, pp. 600-612,
2004.
W. Zhou and A. C. Bovik, "A universal image quality index,"
IEEE Signal Processing Letters, vol. 9, pp. 81-84, 2002.
K. Pearson, "Note on regression and inheritance in the case of two
parents," Proceedings of the Royal Society of London, vol. 58, pp.
240-242, 1895.
J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, "Joint
bilateral upsampling," ACM Trans. Graph., vol. 26, p. 96, 2007.

