Learning Deeply Supervised Visual Descriptors for Dense Monocular
Reconstruction

arXiv:1711.05919v1 [] 16 Nov 2017

Chamara Saroj Weerasekera, Ravi Garg, Ian Reid
Abstract— Visual SLAM (Simultaneous Localization and
Mapping) methods typically rely on handcrafted visual features
or raw RGB values for establishing correspondences between
images. These features, while suitable for sparse mapping,
often lead to ambiguous matches at texture-less regions when
performing dense reconstruction due to the aperture problem.
In this work, we explore the use of learned features for the
matching task in dense monocular reconstruction. We propose
a novel convolutional neural network (CNN) architecture along
with a deeply supervised feature learning scheme for pixelwise regression of visual descriptors from an image which
are best suited for dense monocular SLAM. In particular, our
learning scheme minimizes a multi-view matching cost-volume
loss with respect to the regressed features at multiple stages
within the network, for explicitly learning contextual features
that are suitable for dense matching between images captured
by a moving monocular camera along the epipolar line. We
utilize the learned features from our model for depth estimation
inside a real-time dense monocular SLAM framework, where
photometric error is replaced by our learned descriptor error.
Our evaluation on several challenging indoor scenes demonstrate greatly improved accuracy in dense reconstructions of
the well celebrated dense SLAM systems like DTAM, without
compromising their real-time performance.

I. INTRODUCTION
Visual 3D reconstruction finds it uses in many visionbased autonomous systems, including robotic navigation and
interaction. It is a core component of structure from motion
and visual SLAM systems that work by establishing correspondences between multi-view images, either explicitly or
implicitly, and use this information to solve for the locations
of both the camera and 3D landmarks. Depending on the
method used, and also the application, the reconstructed 3D
map varies from being sparse to dense. While sparse mapping
is usually sufficient in cases where the desired output is the
trajectory of a moving robot, a dense 3D map is useful
when a robot is required to reliably interact and move in
a given environment. Success of both dense mapping and
direct dense tracking given a map are crucially depends on
ability to generate accurate dense correspondences between
images. Reliance on hand-crafted features or raw RGB pixel
values is the crucial bottleneck of dense SLAM as matching
ambiguities arise due lack of unique local texture, repetitive
texture in the image, appearance distortion due to change in
perspective, change in lightning, motion blur and occlusions.
Strong priors are often used to “fill in” or rectify parts of
the map, but they are not always sufficient for accurate
reconstructions.
*All authors are with the School of Computer Science, at the University
of Adelaide, and Australian Centre for Robotic Vision

Input Image

RGB Features with TV

Learned Features with TV

Learned Features with [20]

Fig. 1. Replacing color features with learned deep features and hand crafted
priors with learned priors improves the reconstruction quality. Notice the
dent in table, the very structured error in fusing the depths of the monitor and
wrongly reconstructed notebook in the left corner of top left reconstruction
which is obtained by using RGB cost volume with TV like handcrafted
smoothness prior in SLAM framework of [14]. Most of these errors can be
fixed by learning deep-features to generate cost-volume and learned priors
to replace TV.

Recently, the resurgence of CNNs and their capacity
to capture full context of the scenes directly from data
have allowed relatively accurate predictions surface normals
and depth to be made just from a single image [2], [9],
[5]. CNNs have shown to be capable of accurately solving low-level vision problems like predicting dense image
correspondence[15], [7] or depth and motion from two-views
[19] outperforming traditional vision methods which rely on
handcrafted features and loss functions. However they have
mainly been used for solving single or two-frame problems.
It is not straightforward to extend standard end-to-end learning frameworks to use data from arbitrary number of views
without increasing memory and computational requirements
while training and testing.
In a typical visual SLAM system where the camera
captures images at 30Hz, it is always beneficial to make
use of all the information available. In this work, we take
a different approach of learning dense features good for
matching which are fast to compute and can directly be used
in the existing dense monocular reconstruction systems for
reliable correspondence estimation.

Fig. 2. Overview of the proposed multi-scale network architecture and training setup. Blocks B1-B5 each consist of 3 convolutional layers (kernel size=3,
pad = 1, output filters=32) with the first layer in each block having a stride=2, and the rest having stride=1, so that the features are downsampled by
a factor of two after each block. The upsampled course features are summed up with the fine features to produce a feature comprising of contextual
information as well as fine image details. At train-time both the reference and live images (Iref and Ilive ) are passed into the network as a batch in two
independent streams that share weights. The final 32 dimensional output features (fref and flive ) for the reference and live image respectively as well as
their intermediate output features are used as input to the cost volume loss layers L,L1-L5 for deep feature supervision at every image scale. Note that
bold lines do not intersect with thin lines. The forward time for the network for a single image to produce a dense feature map is only ≈ 8ms on a Nvidia
GTX 980 GPU.

More specifically we want to automatically learn features
via a CNN to deal with the ambiguities that occur when
matching along epipolar lines for per-pixel depth estimation,
especially when presented with images captured with a handheld monocular camera. To this end we introduce a novel
CNN-based deeply supervised training scheme for feature
descriptor learning, and a fully convolutional multi-scale
network design that can help efficiently capture both local
and global pixelwise context, and simultaneously regress
for a high dimensional feature vector for each pixel in the
image. In contrast to previous feature learning approaches,
we propose to construct and minimize a multi-view cost
volume loss during training where the predicted feature for
each pixel in the reference image is compared with those
for a range of pixels along the corresponding epipolar line
in the live frame (one of which will be the right match).
The minimization is done with respect to predicted features
over the entire cost volume during training and thus millions
of training instances are incorporated into a single training

batch which makes the training very efficient. The efficient
computation of our loss is enabled by the large parallel
compute capability of GPUs. We apply our cost volume
loss after every feature map downsampling stage in order
to regularize the network to learn good features to match
at every image scale. Moreover since our training loss is
based on the same cost volume loss minimised at test-time
for estimating depth, our belief is that it will help the network
learn the optimal features suited for dense correspondence
in a monocular reconstruction setting given the training data
and model capacity. Similar to [17] our training framework is
self-supervised, requiring only raw RGB-D data (for example
as captured by Kinect sensor) and the camera trajectory that
can be estimated from the data itself using a method like
RGB-D ORB-SLAM [11] or Kinect Fusion [13] for rigid
scenes.
We also integrate our feature descriptors into the real-time
dense monocular reconstruction system in [20] which uses
in addition a CNN-based learned surface normal prior for

depth regularization (Fig. 1). We show that the combined
system leads to further improvement in dense monocular reconstructions in challenging indoor environments. To the best
of our knowledge this combined system is the first framework
for real-time monocular dense reconstruction to harnesses
the feature-learning capabilities of CNNs in terms of both
geometric scene understanding and dense correspondence
given arbitrary number of input images.
II. BACKGROUND
SLAM methods with hand crafted features: Most popular
SLAM methods like PTAM[8] and ORB-SLAM[10] use
handcrafted features successfully for sparse matching and
solving for sparse map and camera trajectory. LSD-SLAM
[3] uses direct RGB alignment for solving for both pose and
map but results in a semi-dense map. DTAM[14] is able to
generate dense depth maps by utilizing smoothness prior to
help resolve matching ambiguities. While recent work have
begun exploring the use of learned priors for reconstruction
[20], [4], [18], these methods still rely on low-level feature
matching for dense correspondence.
End-to-end CNNs for stereo: Ummenhofer et al. [19] propose to train a network for depth, camera motion, and optical
flow prediction from two frames for indoor scenes whereas
Kendel et.al. [7] propose to train a deep network end-toend for disparity prediction outdoors, to give state-of-the-art
results for stereo reconstruction in NYU and KITTI datasets
respectively. These two frame reconstruction architectures
however are not very easily extensible to use more then
a couple of frames – a limitation we want to address in
this work. In particular, our work is inspired by [7], which
constructs a cost-volume with learned features for two frames
by simply concatenating the features of these images for
all possible candidate matches. This stereo cost-volume is
further passed to a deep 3D convolutional network which
implicitly acts as a regularizer to finally select the correct
matches out of these candidates. Instead of learning this
regularization network, in this work we propose to make the
matching decision on per-pixel basis directly on the costvolume by explicitly forcing the correctly matched features
to have small distances. This allows us to manually define a
distance in the feature space which can be summed over
multiple frames which end-to end stereo networks listed
above does not allow.
Feature learning with CNNs for low-level vision: Feature
learning methods designed for good matching e.g. MCCNN [23] or LIFT [22] are often patch based Siamese
like architectures. Unlike these methods, our architecture is
fully convolutional allowing for efficient training and testing
and flexibility to match each pixel relying potentially on a
receptive field of as large as the entire image. Loosely related
to our work is recently published self-supervised visual descriptor learning method of [17] which by taking advantage
of the model to frame alignment provides very accurate dense
matching under extreme seen variations. Another related
work is Universal Correspondence Network [1] that uses a
contrastive loss to learn a network in fully supervised manner

for feature matching. However, these methods don’t use deep
supervision, rely on randomly selected negative samples, and
have not been extensively analyzed or integrated in a SLAM
framework.
III. METHOD
In this section we describe our network architecture and
training and inference time setup, and discuss the motivations
behind the design choices.
A. Network architecture
Our network consists of five blocks B1-B5 (ref. figure
2), each block consisting of 3 convolutional layers. All
convolutional layers have a kernel size of 3x3 and 32
output filters, which we found to be a good middle ground
for computational speed and matching accuracy. The first
layer of each block has a stride of 2 for 2x downsampling
after each subsequent block. The progressive down sampling
rapidly increases the receptive field for a pixel without the
need for a deeper network and helps the network capture
contextual information which is key for successful matching.
Not having a very deep network also prevents the network
from overfitting to a particular dataset, and helps it learn
more generic descriptors that are useful for matching even in
completely different types of environments (ref. figure 6). In
our network, the maximum potential receptive for a pixel is
roughly half the input image resolution which is significantly
larger than in work like [23] which limit the receptive field
to 9x9 patches.
The downsampling however comes at a cost of reduced
precision in the matching. This is because course features,
although useful for matching large textureless regions, are
too abstract for fine-grain pixel level matching which is
required for precise depth estimation. Inspired by the U-Net
architecture [16] we upsample each course prediction and
sum it up with the preceding fine predictions (as shown in
figure 2) in order to produce our final feature output. Doing
so explicitly embeds local information about a pixel into
its contextual representation enabling precise sub-pixel level
matching.
The final output of our network is therefore a 32 dimensional feature vector for each pixel in the image, encoding
both contextual and low-level image information. Each upsampling operation on the output of each course block is
done through a single learnable deconvolution layer with
a kernel size of 5x5 and stride of 2. Making this layer
learnable resulted in improved results over a simple bilinear
upsampling, showing that in this case it is better to allow the
network to learn a more appropriate upsampling method. Furthermore, concatenating a bilinearly downsampled version of
the input image with the output features from each block
before passing them as input into the subsequent courser
blocks also improved the matching result. This shows that
there is still information present in the low-level features
that can complement the abstract ones in order to benefit the
pixelwise matching at lower resolutions.

Two images

CV minima with two images
Learned — RBG

CV minima with 30 imges
Learned — RBG

Depth GT

Fig. 3. Figure compares RGB and learned features for keyframe reconstruction purely based on feature matching for: a bathroom, a kitchen and a bedroom
sequence from the NYU dataset [12] in rows 1-3 respectively. The red square in the first image in each row is a point of interest, which has to be mapped
to the corresponding image in the second column on the epipolar line (denoted in red). The red square in the second image correspond to the predicted
match using deep features which aligns correctly with the true match (green square) while the blue square represents the best RGB match. Columns 3,4
show the depth maps obtained by matching these two frames using learned features and color respectively. It is evident that, the learned feature’s are more
reliable for matching two-frames and successfully match far more points correctly despite absence of local texture. Columns 4,5 show the depth maps
obtained by minimizing learned feature-based and RGB-based cost volumes respectively for 30 frames. Matching evidence gets accumulated over frames
for both cases but the cost-volume constructed using the learned features gives considerably better results. Last column is the ground truth depth map for
each case.
TABLE I
Q UANTITATIVE RESULTS ON 25 RAW NYU-D V2 DATASET [12] TEST SEQUENCES . I N THE TABLE P.E. = P HOTOMETRIC E RROR . L.F.E. = L EARNED
F EATURE E RROR . L. N ORMALS = L EARNED N ORMALS . T HE AVERAGE ERRORS AND ACCURACY ARE FOR KEYFRAME RECONSTRUCTIONS AGAINST
K INECT DEPTH MAPS ( WHERE VALID DEPTHS ARE AVAILABLE ). T HE RESULTS HERE ARE SHOWN FOR THE OPTIMAL LAMBDA VALUES FOR EACH
CASE BASED ON F IG . 5.

NYU-D V2 Raw 25 Test Scenes

CNN Depth [2]
P.E. + Smoothness [14]
P.E. + L. Normals [20]
L.F.E. + Smoothness (Ours)
L.F.E. + L. Normals (Ours)

Error (lower
rms (m)
log
0.637
0.226
0.489
0.193
0.444
0.174
0.372
0.143
0.356
0.135

Although the multiscale features from our network can
be used independently for course-to-fine matching at test
time, we perform matching using just the final aggregated
high resolution feature output of our network instead as the
contextual information contained in it allows for a large
basin of convergence and makes the solution less sensitive to
initialization (see section IV). Our network is fast to evaluate
(≈ 8ms on a Nvidia GTX 980 GPU) making it suitable
for use as a feature extractor in real-time SLAM. Another
advantage of our architecture is that it provides the flexibility
for one to discard the courser blocks if needed, as the features
that are generated by each block are explicitly trained to be
suitable for matching through deep supervision (see section
III-B). For example one can choose to remove Blocks B2-B5
and simply use features from Block B1 if further efficiency is
desired without compromising too much on accuracy (refer
figure 6).

is better)
abs.rel
0.163
0.105
0.083
0.067
0.058

sq.rel
0.135
0.091
0.073
0.054
0.049

Accuracy (higher is better)
δ < 1.25 δ < 1.252
δ < 1.253
0.738
0.937
0.982
0.865
0.958
0.982
0.897
0.965
0.985
0.936
0.978
0.990
0.948
0.981
0.991

B. Training
The output of our network are dense pixelwise features
for any image. Using these features for multiple images,
our task is to search for the best inverse depth ρp of a
pixel up in the reference image – which matches this pixel
in other images on the corresponding epipolar line defined
by the camera motion. To do this, first we discretize the
(inverse) depth ρp into N bins {ρ1p ... ρN
p } and convert
this continuous matching problem into N class classification
problem. We create the discrete multi-view cost-volume
[14] and following [7] minimize the softmax loss in order
to regress the optimal inverse-depth ρL
p that matches the
groundtruth inverse depth. In this work we discretize the
inverse depth into 256 bins, with labels uniformly sampled
similar to [14] (we set maximum inverse depth as 4 and
minimum as 0). This gives a coarser match depending on the
number of the bins used for discretization and the baseline
which in the general case vary. To get a more refined match
we further minimize the regression loss in the same manner
as [7].

Referece Image

Point A: on the chair

Image 2

Image 15

Point B: on the wall

Image 30

Point C: on the ground

Fig. 4. Figure shows the cost-volume as it evolves after accumulating
evidences from multiple frames for 3 different points, using learned features
(red) and rgb features (blue). The red and blue vertical lines in the plots
indicate the location of the minimum using learned features and rgb
features respectively, and the green vertical line indicates the location of
the groundtruth minimum. From top to bottom, the plots correspond to cost
volumes accumulated over 2 frames, 15 frames and 30 frames respectively.
In the top images, the green squares show the location of the 3 points
of interest in the reference image (left), and the corresponding groundtruth
locations in 3 of the live images that were used to generate the cost volume.
The red and blue squares show the matched locations of the same 3 points
after searching along the corresponding epipolar lines in the live frames
using learned features and rgb features respectively.

Fig. 5. The effect of changing regularization strength on reconstruction
accuracy. The reconstruction errors are significantly lower when the features
are learned than that of using RGB values as the feature. Using learned
normal based prior of [20] with our learned features gives best results.
Please note that as the cost-volume magnitude depends on the length and
range of the used features, for each of the 4 plots shown the regularization
strength λ is renormalized by a constant factor for better visualization.

E(ρ) =

X1
Eφ (ρp ) + Esmooth (ρp ),
λ

(1)

p∈P

where Eφ is the dataterm that computes the descriptor
matching error for a keyframe Iref accumulated over N
overlapping frames:
Eφ (ρp ) =

N
1 X
fref (up ) − fn (π(Tnr π −1 (up , ρp )))
N n=1

1

Since our network regresses multi-scale features, similar to
[21], we explicitly perform deep supervision by minimizing
both classification and regression losses with respect to the
features at each scale independently (L1-L5), as well as the
aggregated feature (L in figure 2). In comparison, similar
works to ours like [17], [1] have relied upon a single contrastive loss function at the end of the network. In addition to
providing the flexibility of discarding blocks of the network
in trade for computation speed, this deep supervision acts as a
good regularizer for the network resulting in fast convergence
during training. Our network is trained from scratch with
Xavier initialization using the Caffe framework [6].

(2)
Esmooth is the local smoothness prior. We experiment with
the TV and normal based regularizer as two variations where
RGB cost-volume replacing f in (2) forms [14] and [20]
as our baselines. λ in each case controls the regularization
strength.

C. Keyframe reconstruction using deep features

A. A detailed analysis of learned deep features

Once we have learned the multi-scale features as specified
above, we can use then in the mapping frameworks [14],
[20] by simply replacing the RGB cost-volume with the
learned deep-features based cost-volume. Our inference loss
is defined as:

Figure 3 compares the cost volume generated using RGB
features as well as the learned feature in some of the challenging scenarios of large baseline matching along epipolar
line. Even when only two frames are used for matching, the
learned features – consisting contextual information from
large receptive field – can be successful matched whereas

IV. EXPERIMENTS
In this section, we extensively analyze different aspects
of the learned deep features particularly advantageous for
dense tracking and mapping over RGB features and quantify
this advantage benchmarking the learned network on a large
dataset.

minima’s due to repetitive texture in the images leading to
mismatching even a well textured region (See the corner of
the chair) in figure 4. Additionally, the cost volume also
remains flat in homogeneous regions of the images due
to aperture problem for example the pixel on the floor.
Due to multiple local minima gradient based optimization
methods suffers from incorrect initialization forcing many
direct tracking/mapping method to resort to coarse-to-fine
strategy whereas the flatness of the cost-volume makes
SLAM systems heavily reliant on strong priors.
B. Evaluation

Fig. 6. Generalizability of learned features trained on NYU dataset to
a completely different type of scene on the KITTI stereo dataset without
any finetuning. Note that no regularization was added on top, so this is
just the pure matching result. From Top to Bottom: Reference Image, Live
Image, Disparity map obtained by matching using rgb features, Disparity
map obtained by matching using learned features from Block B1, Disparity
map obtained by matching using all learned features combined, Groundtruth.
Notice how explicitly combining course features with the fine features
provides the necessary context to match large textureless regions while
maintaining precision.

RGB cost volume struggles. As the number of frames for
matching grows the deep feature based cost-volume gives
very realistic and smooth depth maps without any prior.
In Figure 4, we further explore the advantages deep
feature learning posses. Our learned features when used
in the SLAM framework of [14] for mapping give much
sharper local minimas for cost-volume – often the correct
depth – compared to that of the RGB. Deep features based
cost-volume often has very sharp global minima even with
very less number of frames. This sharpness of the global
minima is indicative of large basin of convergence. As the
decision can be made based on the image evidence alone,
both tracking and mapping becomes less reliant on priors
which often are inaccurate and computationally expansive
part of SLAM. Interestingly, given enough number of frames
for matching, the deep feature’s cost-volume often have
much fewer local minima. This is again another desirable
property as this suggests insensitivity to the initialization for
tracking and mapping methods which use’s gradient based
optimization.
On the other hand, RGB cost volume even after gathering
evidence from 10’s of frames, often consists of multiple local

We extensively evaluate the learned deep features as
explained in the section III on a large subset of NYU v2
dataset [12]. We follow the same experimental set-up and
the train/test splits of NYU dataset as that of [20]. Camera
motion (in metric units) for all the NYU train and test sequences were precomputed using [11] to isolate the mapping
process for which we use our own implementation of [14]
and [20] as two very strong baselines for this work. Small
sub-sequences of 61 frames (30 past and 30 future frames)
were used to reconstruct all the key-frames using both the
baselines. We show the improvement in the depth estimation
by replacing the RGB features by our learned features for
cost-volume creation in both the baselines mentioned above
and compared the results to quantify the improvements. We
evaluate the accuracy of the depth-maps on the standard
evaluation criteria used on NYU datasets by [2], [20].
To find optimal hyper parameter λ, we grid-search in a
sensible range for each of the approach individually. Figure 5
visualizes the sensitivity of all the reconstruction approaches
to the choice of regularization strength λ. After a constant
normalization factor, it can be seen that all algorithms
degrade gracefully when we deviate from the optimal choice
of λ while the learned features make this choice less critical.
In Table I we report the best reconstruction accuracies
obtained on NYU dataset with and without using our learned
features. Using our learned feature to create cost-volume
gives a significant boost to the performance (evaluated on
all the error measures) to that of RGB cost volume based
reconstruction.
Figure 7 and 8 show the visual comparison of the proposed
features used in the mapping systems [14], [20] working in
real-time with ORBSLAM monocular visual tracking [10]
against that of color features. A significant advantage of the
proposed deep features can be easily seen in these results
visually. More live qualitative results are available here:
https://goo.gl/jUqiBN.
V. ACKNOWLEDGEMENTS
This work was supported by the ARC Laureate Fellowship
FL130100102 to IR and the Australian Centre of Excellence
for Robotic Vision CE140100016.
R EFERENCES
[1] C. B. Choy, J. Gwak, S. Savarese, and M. Chandraker, “Universal correspondence network,” in Advances in Neural Information Processing
Systems 30, 2016.

Keyframe and its Learned Normals

RGB Features (TV / Learned Prior [20])

Learned Features (TV / Learned Prior)

Input Image

RGB Features, TV

Learned Features, TV

RGB Features, Learned Prior [20]

Learned Features, Learned Prior

Fig. 7. A snapshot of our results on the ’living room 0075’ sequence from NYU-D V2 raw dataset. Top part of the figure shows a sample frame
from the sequence selected as a keyframe in our SLAM framework and the depths estimated for these frames with different mapping algorithms. From
left to right are: (1) Keyframe and (2) the estimated normal map for this keyframe (by [2]). From the third column onwards we show the depth-maps
generated using: (3) RGB features and hand-crafted smoothness prior [14], (4) RGB features and the predicted normals-based smoothness prior [20], (5)
learned features and hand crafted smoothness prior and (6) learned features and learned normals-based smoothness prior. Note how using learned features
drastically improves the depth estimates for the keyframes. In particular notice the back of fully visible sofa, the wall and glass behind the sofa and the
floor underneath. All of it gradually becomes smoother as the wrongly estimated discontinuities in depths are fixed when the learned features replaces RGB
in cost volume estimation. In the Bottom part of the figure, the first column shows a few sample images from of the room. The next two columns in the
bottom part show the results obtained by fusing the dense maps for the key-frame using: RGB and learned features respectively. As we expect, learned
features gives consistent accurate depth maps for multiple keyframes allowing easy fusion to get high quality dense volumetric-maps. The normal-based
prior additionally removes the noise in the reconstructions.

[2] D. Eigen and R. Fergus, “Predicting depth, surface normals
and semantic labels with a common multi-scale convolutional
architecture,” CoRR, vol. abs/1411.4734, 2014. [Online]. Available:
http://arxiv.org/abs/1411.4734
[3] J. Engel, T. Schöps, and D. Cremers, “Lsd-slam: Large-scale direct monocular slam,” in European Conference on Computer Vision.
Springer, 2014, pp. 834–849.
[4] J. M. Fácil, A. Concha, L. Montesano, and J. Civera, “Deep single
and direct multi-view depth fusion,” CoRR, vol. abs/1611.07245,
2016. [Online]. Available: http://arxiv.org/abs/1611.07245
[5] R. Garg, V. Kumar B, G. Carneiro, and I. Reid, “Unsupervised CNN
for Single View Depth Estimation: Geometry to the Rescue,” in
European Conference on Computer Vision, (ECCV), 2016.
[6] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, and T. Darrell, “Caffe: Convolutional Architecture for
Fast Feature Embedding,” arXiv preprint arXiv:1408.5093, 2014.
[7] A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy,
A. Bachrach, and A. Bry, “End-to-end learning of geometry and
context for deep stereo regression,” CoRR, vol. abs/1703.04309, 2017.
[Online]. Available: http://arxiv.org/abs/1703.04309
[8] G. Klein and D. Murray, “Parallel tracking and mapping for small ar
workspaces,” in Mixed and Augmented Reality, 2007. ISMAR 2007.
6th IEEE and ACM International Symposium on. IEEE, 2007, pp.
225–234.

[9] I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab,
“Deeper depth prediction with fully convolutional residual networks,”
in 3D Vision (3DV), 2016 Fourth International Conference on. IEEE,
2016, pp. 239–248.
[10] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós, “ORB-SLAM: A
Versatile and Accurate Monocular SLAM System,” IEEE Transactions
on Robotics, vol. 31, no. 5, pp. 1147–1163, oct 2015.
[11] R. Mur-Artal and J. D. Tardós, “ORB-SLAM2: an open-source
SLAM system for monocular, stereo and RGB-D cameras,” CoRR,
vol. abs/1610.06475, 2016. [Online]. Available: http://arxiv.org/abs/
1610.06475
[12] P. K. Nathan Silberman, Derek Hoiem and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in ECCV, 2012.
[13] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim,
A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon,
“Kinectfusion: Real-time dense surface mapping and tracking,” in
Proceedings of the 2011 10th IEEE International Symposium on
Mixed and Augmented Reality, ser. ISMAR ’11. Washington,
DC, USA: IEEE Computer Society, 2011, pp. 127–136. [Online].
Available: http://dx.doi.org/10.1109/ISMAR.2011.6092378
[14] R. A. Newcombe, S. J. Lovegrove, and A. J. Davison, “DTAM:
Dense tracking and mapping in real-time,” Proceedings of the IEEE
International Conference on Computer Vision, pp. 2320–2327, 2011.
[15] A. Ranjan and M. J. Black, “Optical flow estimation using a spatial

Input Image

RGB Features, TV

Learned Features, TV

Learned Features, TV

RGB Features, Learned Prior [20]

Learned Features, Learned Prior

Fig. 8. Qualitative comparison of reconstructions obtained by our monoslam framework for sequence ’living room 0080’. While a standard SLAM
framework like [14] gives inconsistent depth maps across keyframes which cannot be fused and thus leave holes in the reconstruction (top-middle), learned
features are more accurately matched. Bottom part shows how both deep-features proposed in this work and the learned normals-based smoothing of [20]
improves the quality of fused maps.

[16]

[17]
[18]

[19]

[20]

[21]
[22]
[23]

pyramid network,” CoRR, vol. abs/1611.00850, 2016. [Online].
Available: http://arxiv.org/abs/1611.00850
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional
networks for biomedical image segmentation,” CoRR, vol.
abs/1505.04597, 2015. [Online]. Available: http://arxiv.org/abs/1505.
04597
T. Schmidt, R. Newcombe, and D. Fox, “Self-supervised visual
descriptor learning for dense correspondence,” IEEE Robotics and
Automation Letters, vol. 2, no. 2, pp. 420–427, April 2017.
K. Tateno, F. Tombari, I. Laina, and N. Navab, “CNN-SLAM:
real-time dense monocular SLAM with learned depth prediction,”
CoRR, vol. abs/1704.03489, 2017. [Online]. Available: http://arxiv.
org/abs/1704.03489
B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy,
and T. Brox, “Demon: Depth and motion network for learning
monocular stereo,” CoRR, vol. abs/1612.02401, 2016. [Online].
Available: http://arxiv.org/abs/1612.02401
C. S. Weerasekera, Y. Latif, R. Garg, and I. Reid, “Dense monocular
reconstruction using surface normals,” in 2017 IEEE International
Conference on Robotics and Automation (ICRA), May 2017, pp. 2524–
2531.
S. Xie and Z. Tu, “Holistically-nested edge detection,” CoRR, vol.
abs/1504.06375, 2015. [Online]. Available: http://arxiv.org/abs/1504.
06375
K. M. Yi, E. Trulls, V. Lepetit, and P. Fua, “LIFT: learned invariant
feature transform,” CoRR, vol. abs/1603.09114, 2016. [Online].
Available: http://arxiv.org/abs/1603.09114
J. Zbontar and Y. LeCun, “Stereo matching by training a
convolutional neural network to compare image patches,” CoRR, vol.
abs/1510.05970, 2015. [Online]. Available: http://arxiv.org/abs/1510.
05970

