Loosely-Coupled Semi-Direct Monocular SLAM
Seong Hun Lee Javier Civera






 

This paper has been accepted for publication in IEEE Robotics and Automation Letters.




DOI: http://doi.org/10.1109/LRA.2018.288915610.1109/LRA.2018.2889156

IEEE Xplore: <http://ieeexplore.ieee.org/document/8584894/>






©2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.



Pseudo-scalar qq̅ bound states at finite temperatures within a Dyson-Schwinger–Bethe-Salpeter approach 
     B. Kämpfer
    
=======================================================================================================

empty
empty





    We propose a novel semi-direct approach for monocular simultaneous localization and mapping (SLAM) that combines the complementary strengths of direct and feature-based methods.  

    The proposed pipeline loosely couples direct odometry and feature-based SLAM to perform three levels of parallel optimizations: (1) photometric bundle adjustment (BA) that jointly optimizes the local structure and motion, (2) geometric BA that refines keyframe poses and associated feature map points, and (3) pose graph optimization to achieve global map consistency in the presence of loop closures.
    This is achieved in real-time by limiting the feature-based operations to marginalized keyframes from the direct odometry module.


    Exhaustive evaluation on two benchmark datasets demonstrates that our system outperforms the state-of-the-art monocular odometry and SLAM systems in terms of overall accuracy and robustness.






§ INTRODUCTION



Real-time visual odometry (VO) and simultaneous localization and mapping (SLAM) play an important role in many emerging technologies, such as autonomous ground/air vehicles <cit.> and virtual/augmented reality <cit.>. 
In particular, monocular methods have drawn significant attention due to their minimal hardware constraints.




Traditional algorithms relied heavily on feature extraction and matching to estimate structure and motion <cit.>.
In recent years, however, direct methods have gained rapidly increasing popularity <cit.>. 
In contrast to feature-based ones, direct methods are capable of leveraging raw photometric information from a chosen set of pixels in the image.
This removes the need for costly per-frame feature extraction and matching.
Also, they are shown to be relatively more robust in low-texture scenes <cit.>.

 
Although direct methods have their own merits in several aspects, they inevitably miss certain benefits of salient features.
For example, feature descriptors such as SIFT <cit.> or ORB <cit.> have a high degree of invariance to viewpoint and illumination changes, and they can be matched over wide baselines. 
Such properties are favorable for tracking large inter-frame motions and recognizing revisited places.
Recent studies indeed confirm that direct and feature-based methods have their own strengths and weaknesses in respective areas <cit.>. 
Semi-direct methods such as <cit.> attempt to take advantage of such complementary characteristics by incorporating ideas from both direct and feature-based methods.


In this paper, we propose a novel semi-direct approach for monocular SLAM that inherits both the robustness of direct VO and the map-reusing capability (e.g., loop closure) of feature-based SLAM. 

Our contribution is a loose coupling between direct and feature-based algorithms such that:

0em
    
  * Locally, a direct method is used to track the camera pose rapidly and robustly with respect to a locally accurate, short-term, semi-dense map.
    
  * Globally, a feature-based method is used to refine the keyframe poses, perform loop closures, and build a globally consistent, long-term, sparse feature map that can be reused. 

This strategy allows us to complement the weaknesses of each method without compromising their real-time efficiency and performance.
We implement our approach on top of DSO <cit.> and ORB-SLAM <cit.>, respectively the state-of-the-art in direct and feature-based methods, and demonstrate that our system outperforms both of them on two public benchmark datasets.

Fig. <ref> shows an example snapshot of the estimated camera trajectory and associated scene reconstruction using our method.
The full reconstruction process is demonstrated in the accompanying video:




 <https://youtu.be/j7WnU7ZpZ8c>





We make our implementation publicly available at:




 <https://github.com/sunghoon031/LCSD_SLAM>


















§ RELATED WORK




Modern keyframe-based VO/SLAM systems can be categorized into three classes:

(1) Feature-based: Feature-based (or indirect) methods recover both camera pose and scene structure by matching features and performing geometric bundle adjustment (BA) that minimizes the reprojection error.
PTAM <cit.> is one of the most representative systems of this type, where it was first proposed to split tracking and mapping into two parallel threads. 


At present, ORB-SLAM <cit.> is arguably the best-performing feature-based system.
Based on multiple successful ideas of PTAM and others <cit.>, ORB-SLAM uses ORB features <cit.> to perform tracking, mapping, relocalization and loop closing in a scalable manner. 
In <cit.>, an extension of ORB-SLAM was proposed to generate a semi-dense reconstruction of the scene.
This last method, however, does not use the resulting semi-dense map for tracking.

(2) Direct: Direct methods estimate structure and motion by minimizing the photometric error (i.e., intensity difference) between corresponding pixels in images <cit.>.

Unlike most feature-based methods, direct methods are not limited to a sparse map and can maintain either a sparse <cit.>, semi-dense <cit.> or dense <cit.> map in real-time. 
Currently, the best-performing odometry system is DSO <cit.> which performs photometric BA to jointly optimize camera intrinsics, extrinsics and inverse depths of sparse (or semi-dense) points in a sliding window fashion.
In <cit.>, it was found that, for a small number of map points (e.g., < 1000), such joint optimization tends to be more accurate than alternating tracking and mapping as in, for example, LSD-SLAM <cit.>.

DSO is also the first work to demonstrate the benefits of photometric calibration <cit.> in direct methods.
However, it is subject to drift over time as it is a pure odometry method and does not reuse the map points that leave the field of view (FOV).
Gao et al. <cit.> recently presented a modified DSO with the loop-closing capability similar to LSD-SLAM.
Unlike our method, however, their system does not produce a reusable sparse feature map which is useful for applications such as global relocalization, fixed-map tracking and collaborative mapping.

(3) Semi-Direct: Semi-direct (or hybrid) methods estimate camera poses using both direct and feature-based methods. 
For example, SVO <cit.> performs direct sparse image alignment to estimate the initial guess of the camera pose and feature correspondences. 
Afterwards, it performs geometric BA to refine the pose and structure. 
It was shown in <cit.> that SVO could also be used for dense mapping.
In <cit.>, several improvements to the original SVO were proposed. 
Although this system is highly efficient, it was shown to be less robust <cit.> than ORB-SLAM <cit.> and DSO <cit.>.


In <cit.> and <cit.>, similar approaches to SVO were proposed for monocular visual(-inertial) and RGB-D SLAM, respectively.
Both methods adopt a direct method for tracking and a feature-based method for keyframe pose refinement, mapping and loop closing.
At the end of Section <ref>, we briefly discuss how our method differs from these existing semi-direct methods.
In <cit.> and <cit.>, different semi-direct approaches were proposed for stereo odometry.
Both methods use feature-based tracking to obtain a motion prior, and then perform direct semi-dense or sparse alignment to refine the camera pose.
While they were respectively shown to perform well against large inter-frame motions and illumination changes, they do not utilize the robustness of direct tracking in low-texture scenes. 




§ SYSTEM OVERVIEW


Fig. <ref> illustrates our proposed semi-direct pipeline. 
The idea is to combine the currently best-performing feature-based and direct algorithms, namely ORB-SLAM <cit.> and DSO <cit.>, with some modifications.



To achieve real-time performance, we take inspiration from SVO <cit.> and apply a direct method to quickly track each frame and provide an initial seed for feature-based map optimization. 
Specifically, we use DSO to achieve real-time tracking and a modified version of ORB-SLAM to build a globally consistent map at a slower rate with marginalized keyframes from DSO. 
This is shown in Fig. <ref> as a direct and a feature-based module, respectively.

As the two separate asynchronous modules exchange information without sharing states, this approach is considered loosely coupled.

Our system architecture involves three different layers of optimization windows. 
At the most local level, a sliding window of keyframes and map points are photometrically bundle adjusted to obtain an accurate representation of the surrounding environment.
New frames are tracked using direct image alignment <cit.> with respect to the last keyframe and its depth map created by projecting active points in the window (see Fig. <ref>). 


When a keyframe is marginalized from the direct module, its image and pose information is sent to the feature-based module, along with the map points within its FOV.
The feature-based module extracts ORB descriptors from these keyframes and refines their poses with respect to the local feature map using motion-only BA.


Some of these keyframes and map points are added to the local map and geometrically bundle adjusted for optimal accuracy.

This process of feature-based mapping corresponds to the mid-level optimization.


Finally, at the most global level, a pose graph optimization <cit.> is performed over Sim(3) constraints after each loop detection.
Afterwards, a full BA <cit.> optimizes all keyframes and feature points in the map for global consistency.







The key difference between our approach and SVO <cit.> (or the semi-direct methods in <cit.>) is that we maintain in parallel two separate maps, each in the direct and the feature-based module.
This allows us to utilize a locally accurate semi-dense map for fast and robust tracking, as well as a globally consistent sparse feature map for long-term reuse (e.g., loop detection and closure).




§ NOTATION


Throughout the paper, we use bold lower- and upper-case letters for vectors (𝐯) and matrices (𝐌), and light lower- and upper-case letters for scalars (s) and scalar functions (F), respectively.
The intensity image is denoted by I:Ω↦ℝ where Ω⊂ℝ^2 is the image domain. 
We denote the camera intrinsic parameters with 𝐜, and corresponding camera projection and back-projection functions with Π_𝐜:ℝ^3↦Ω and Π^-1_𝐜:Ω×ℝ↦ℝ^3, respectively.
Camera poses are represented as either rigid body or similarity transformation matrices 𝐓_iw∈SE(3) or 𝐒_iw∈Sim(3) that transform a point from the world frame to frame i.
We use 𝒫_i to denote the set of map points belonging to keyframe i and obs(𝐩) to denote the set of keyframes in which the point 𝐩 is visible.
The Euclidean and Huber norms <cit.> are denoted by ‖·‖_2 and ‖·‖_γ respectively.
The operator ⊞ is defined as a simple addition for Euclidean parameters and a left multiplication for the pose, i.e., for Lie-algebra 𝔰𝔢(3) elements in twist coordinates 𝐱∈ℝ^6, 𝐱⊞𝐓:= exp_SE(3)(𝐱)𝐓.
We use the same notation as in <cit.> for the exponential and logarithmic mapping for SE(3) and Sim(3).




§ DIRECT MODULE


We use the original implementation of DSO <cit.> as our direct module, which is responsible for initial map bootstrapping, real-time camera tracking and local mapping.

In this section, we describe the windowed optimization and marginalization scheme of DSO.
The reader is referred to the original work <cit.> for details on direct tracking and other front-end operations.




 §.§ Windowed Photometric Bundle Adjustment


When a point 𝐩 in a reference frame I_i is observed in another frame I_j, the photometric error is defined as the weighted SSD over the 8-point neighborhood pixels 𝒩_𝐩 as proposed in <cit.>: 

    E^𝐩_ij := _𝐩∈𝒩_𝐩ω_𝐩‖ I_j[𝐩'] -b_j-t_je^a_j/t_ie^a_i(I_i[𝐩] -b_i)‖_γ




    with  ω_𝐩 := c^2/c^2+‖∇ I_i(𝐩)‖^2_2,




    𝐩'=Π_𝐜(𝐓_jw^-1𝐓_iwΠ^-1_𝐜(𝐩,d_𝐩))

where t, a and b are exposure time and affine brightness function parameters, and d_𝐩 is the inverse depth <cit.> of 𝐩 in the reference frame I_i.
The weight w_𝐩 down-weights high-gradient pixels with some constant c.
The total energy function to be minimized is given by the full photometric error plus a prior pulling the affine brightness parameters to zero:



    E_photo := _i ∈ℱ_𝐩∈𝒫_i_j∈obs(𝐩) E^𝐩_ij+_i∈ℱ(λ_aa^2_i+λ_bb_i^2 ),

where ℱ denotes the set of all frames in the window.

When exposure times are known, we set λ_a and λ_b to some constant values.
Otherwise, we set λ_a=λ_b=0 and t_i=t_j=1 in (<ref>).
The optimization is performed using an iteratively reweighted Gauss-Newton or Levenberg-Marquardt algorithm in a coarse-to-fine scheme.
The update equation is given by



    δξ =-(𝐉^T𝐖𝐉)^-1𝐉^T𝐖𝐫  and  ξ^new←δξ⊞ξ,

where 𝐫 is the stacked vector of residuals, 𝐉 is its Jacobian and 𝐖 is the diagonal weight matrix.
The state variable ξ contains all the active variables in the window, i.e., camera poses, affine brightness parameters, inverse depths and camera intrinsics.




 §.§ Marginalization


The size of the optimization window is kept bounded by marginalizing the least useful keyframes and points using the Schur complement <cit.>.

Points are marginalized if they are not observed in the latest two keyframes or their host keyframe is marginalized.
Keyframes (other than the latest two) are marginalized if either less than 5% of its points are visible in the latest keyframe, or if it has the highest “distance score" when the window contains more than a certain number of keyframes.
We refer to the original work <cit.> for the computation of this heuristic score.




§ FEATURE-BASED MODULE


When a keyframe is marginalized from the direct module, the feature-based module receives its image and pose information, as well as the 3D locations of both active and marginalized map points within its FOV.
This information is then used for feature-based pose refinement, mapping and loop closing.
Note that the marginalization strategy in Section <ref> does not necessarily marginalize the oldest keyframe in the window.
To avoid temporal inconsistency in such cases, we store the marginalized keyframes and points in a queue and wait until the next oldest keyframe is marginalized.

If more than five keyframes are queued and the next oldest keyframe is still active, we take its latest pose and points to proceed further instead of waiting. 
All optimizations are performed using the original implementation in ORB-SLAM <cit.>, which is based on the Levenberg-Marquardt algorithm in g2o <cit.>.




 §.§ Relative Scale and Initial Pose Estimation


In our loosely-coupled approach, the direct and the feature-based modules maintain two separate maps.
Due to the scale ambiguity of a monocular system, the scales of these two maps drift over time and do not converge to the same value. 
Therefore, we continuously compute the relative scale using Sim(3) alignment <cit.> between the 30 most recent keyframes in the feature-based module and their counterparts in the direct module.

Once the relative scale s is known, the incremental transformation in the direct module can be scaled appropriately and used as an initial pose guess in the feature-based module:
Let i and j denote the previous and current keyframe.
Then,

    𝐓_jw|F
    =
    [ 𝐑_ji|F 𝐭_ji|F;  0_1×3      1 ]𝐓_iw|F




    with  𝐑_ji|F   = 𝐑_ji|D=𝐑_jw|D(𝐑_iw|D)^T , 
    𝐭_ji|F   = s𝐭_ji|D = s(-𝐑_ji|D𝐭_iw|D+𝐭_jw|D),

where the subscripts D and F indicate the direct and the feature-based module, respectively.
For the derivation of (<ref>) and (<ref>), please refer to the supplementary material available at our public source-code repository (see Section <ref>).







 §.§ 3D Keypoints Generation


The map points from the direct module are used in two ways: (1) for creating an initial set of 3D keypoints to bootstrap the feature-based module, or (2) for adding more local map points to improve the tracking robustness.
Given the 2D position 𝐩 of an ORB feature in frame i, we generate a 3D keypoint 𝐱_w in the world frame as

    𝐱_w=𝐓_iw^-1Π^-1_𝐜(𝐩,d_𝐩/s)   with    d_𝐩 = _k∈𝒫_𝐩 d_k /σ_k^2/_k 1/σ_k^2,

where s is the relative scale between the direct and the feature-based module (Section <ref>), and 𝒫_𝐩 and d_k∈𝒫_𝐩 respectively denote the set of all map points whose projection in frame i is equal to 𝐩 and their inverse depths in frame i. 
We dilate the projection by two pixels to ensure a sufficient number of valid depths for the keypoints.
Note that the inverse depth d_𝐩 is computed as the inverse-variance weighted average, which is equivalent to the Kalman filter update with multiple measurements <cit.>.

We found that extracting more features during slower camera motions often increases the number of covisibility links <cit.> between the keyframes and improves mapping accuracy.

Therefore, we vary the number of features to extract per image as follows:

Let f_kf be the keyframe addition frequency in the direct module.
Using this as the indicator of the relative camera speed, we set N_features=2500 if f_kf< 4 Hz and N_features=1500 if f_kf> 7 Hz. 
Otherwise, we interpolate between the two bounds based on f_kf.





 §.§ Keyframe Pose Refinement and Failure Recovery


Once the direct module provides an initial pose estimate of a new keyframe (Section <ref>), we refine it using motion-only geometric BA with respect to the local feature map.
The total energy function is composed of the variance-normalized reprojection errors of the local map points:

    E_reproj= _i∈ℱ_local_𝐱∈𝒫_i_j ∈obs(𝐱)‖𝐩_j, 𝐱-Π_𝐜( 𝐓_iw𝐱_w)/σ^2_𝐱‖_γ
    with  σ^2_𝐱:=(λ_pyr)^2L_pyr, 𝐱,

where ℱ_local denotes the set of all local keyframes, i.e., all keyframes sharing map points with the new keyframe and their neighbors in the covisibility graph <cit.>, 𝐩_j,𝐱∈ℝ^2 the match to the keypoint 𝐱 in frame j, and σ^2_𝐱 the variance of the feature location in frame i.
This variance depends on the constant scale factor of the image pyramid  λ_pyr (>1) and the pyramid level L_pyr, 𝐱 at which the keypoint was detected.





When the feature-based module fails due to insufficient matches, we reinitialize the module, following the procedure explained in Section <ref>.
This is illustrated in Fig. <ref>.
Since our direct module is robust to low-texture scenes, we can rely on its tracking while the feature-based module is lost.
Then, as soon as we detect more features, we reinitialize the local map points using the depths from the direct module. 







 §.§ Feature-based Local Mapping and Loop Closing


After generating 3D keypoints and refining the keyframe pose, we insert them in the feature-based map if the number of matches falls below 150 or more than three keyframes passed from the last insertion.
Once the keyframe and the points are added to the map, they are processed by the local mapping, as outlined in <cit.>.
This includes the local geometric BA that minimizes (<ref>) to jointly optimize the current keyframe, its neighbors in the covisibility graph, and all the map points belonging to those keyframes.

In the loop closing thread, the place recognition module <cit.> based on DBoW2 <cit.> detects large loops by querying the keyframe database.
Once a loop is detected, the keyframes and map points on each side of the loop are aligned and fused.
To correct the scale drift, pose graph optimization <cit.> is performed over the essential graph <cit.>, minimizing

    E_graph = _(i,j) ∈ℰ_edge‖log_Sim(3)(𝐒_ij,0 𝐒_jw 𝐒^-1_iw)‖^2_2 ,

where ℰ_edge denotes the set of edges in the essential graph, and 𝐒_ij,0=𝐒_iw,0𝐒_jw,0^-1 is the fixed similarity transformation (with the scale 1) between the frame i and j just prior to the pose graph optimization.
If the edge is created from a loop closure, this transformation is instead computed using the method of Horn <cit.>.
Finally, a full BA <cit.> is performed afterwards.





 §.§ Does Feature-based Mapping Always Improve Accuracy?


The answer is no. 
In general, the feature-based mapping described in the previous sections improves the accuracy if there is a loop closure or when the camera motion is mostly loopy, which enhances the covisibility of features in multiple keyframes and the reuse of map points.
However, we found that it actually causes more drift when the camera motion is mostly exploratory without loop closures (a similar finding was also reported in <cit.>). 

We solve this by keeping two versions of the keyframe trajectory: one that is initially given by the direct module and the other modified by the feature-based module. 
We assume that the latter is more accurate if there is a loop closure or less than a quarter of all past keyframes have collinear covisibility links at a given point in time. 
The covisibility links (i.e., 3D lines connecting the keyframe to its neighbors in the covisibility graph) are considered collinear if none of them form an angle between 30^∘ and 150^∘.
This is illustrated in Fig. <ref>.
We found that this method is especially effective for detecting exploratory translational motions.

While this strategy allows us to mitigate the odometry drift in exploratory situations, a more elegant solution would be to deal with the source of inaccuracy in the feature-based mapping that causes drift.
Yang et al. <cit.> suggests a few hints on how this could be done (e.g., careful point management and sub-pixel matching refinement), but it is still an open problem and remains for future work.





§ EVALUATION




 §.§ Evaluated Settings, Datasets and Methodology

We implement our method using ROS[Robot Operating System, <http://www.ros.org/>.] and compare it against ORB-SLAM <cit.> and DSO <cit.>.
We evaluate each algorithm in two different settings:

0em
    
  * ORB-VO and ORB-SLAM: 
    For the VO setting, we disable the loop closing thread.
    Relocalization is disabled for both settings to evaluate the tracking robustness.
    We do not apply photometric calibration <cit.>, as we found that it worsens feature extraction and matching (similar findings were also reported in <cit.>).
    
  * DSO-default and DSO-reduced: 
    We use the default and reduced settings provided in the original DSO implementation. 
    The only difference is that, for the reduced setting, we always resize the input images by half.
    Photometric calibration <cit.> is applied when available.
    
  * Ours-VO and Ours-SLAM:
    The VO setting uses DSO-reduced and ORB-VO settings, whereas the SLAM setting uses DSO-reduced and ORB-SLAM settings.
    This means that the direct module processes photometrically calibrated images (if available) at half resolution, while the feature-based module processes photometrically non-calibrated images at full resolution.
    We found that using half resolution in the feature-based module significantly worsens the performance, which is in line with the observation in  <cit.>.
    For efficiency, we reduce the number of iterations in the local geometric BA by half.

We use two public benchmark datasets for evaluation:

0em

  * EuRoC MAV dataset <cit.>, which contains 11 indoor stereo sequences with 752×480 pixel resolution at 20 fps.
As in <cit.>, we crop the beginning and end of each sequence to disregard large occlusions due to the ground and aggressive motions meant for IMU initialization.
We evaluate the tracking accuracy using the absolute trajectory RMSE (e_ate) of keyframe poses after Sim(3) alignment with the ground truth.
Photometric calibration and exposure times are not available for this dataset.

  * TUM monoVO dataset <cit.>, which contains 50 in- and outdoor monocular sequences recorded at 20–50 fps.
We use 640×480 pixel resolution for undistorted images.
Since the full ground-truth data is not available for this dataset, the tracking accuracy is evaluated in terms of the alignment error (e_align) proposed in <cit.>.







The dataset also provides photometric calibration and exposure times.



To account for non-deterministic behaviour, we run each method 10 times.
This amounts to 220 runs for the EuRoC MAV dataset (i.e., 110 runs for each left and right camera) and 500 runs for the TUM monoVO dataset.
On the EuRoC MAV dataset, we consider that runs were unsuccessful if more than 20% of the total frames could not be tracked either due to the delayed map initialization or complete tracking failures.

On the TUM monoVO dataset, we disable the keyframe culling of ORB-VO/SLAM and our systems within the start- and end-segment of each trajectory where the ground-truth data is available.
This prevents the lack of keyframes when computing the alignment error on these segments.
All images were preloaded into memory, but not rectified or photometrically calibrated beforehand.
All results were obtained in real-time on a laptop CPU (Intel Core i7-4810MQ, quad-core at 2.8 GHz with 15GB RAM).






 §.§ Results


Fig. <ref> shows the error values for each sequence of both datasets, and their median values are reported in the supplementary material available at our public source-code repository.
The aggregated results are given in Fig. <ref> in the form of cumulative error plots indicating how many runs have yielded error values below a certain level.
On both datasets, we make three common observations:
First, DSO-reduced is more robust than DSO-default, which is most likely due to the faster tracking speed, as shown in Tab. <ref>.
We observe similar accuracy between the two settings on the EuRoC MAV dataset, but higher accuracy with DSO-default on the TUM monoVO dataset.
Second, loop closing in ORB-SLAM improves the performance.
It increases accuracy on the TUM monoVO dataset and robustness on both datasets.
Third, due to the non-deterministic nature of real-time multi-threading, the occurrence of loop closure is not necessarily consistent on each sequence (see Fig. <ref>).

On the EuRoC MAV dataset, DSO (both default and reduced) yields the lowest accuracy.
It was also reported in <cit.> that DSO was less accurate than ORB-VO on the same dataset.
However, we were unable to reproduce their exact results, in particular those showing that DSO was more robust in real-time.
Our systems (both VO and SLAM) and ORB-SLAM show very similar performance on this dataset,   except for V1_03_difficult where ORB-SLAM outperforms our SLAM system.
This is because in ORB-SLAM, a loop closure reintroduces the detected map points into the local map and robustifies the tracking.
On the other hand, our direct tracking does not reuse the detected map points from the loop closures, so its robustness is unaffected.

The comparison between DSO-reduced and our VO system indicates that the feature-based pose refinement (Section <ref>) and local mapping (Section <ref>) can improve the accuracy, even without loop closure.
Fig. <ref> shows this effect over time on two of the sequences of the EuRoC MAV dataset. 
Notice the increased amount of drift when we do not reuse the map points that leave the FOV.
We also note that even with loop closure, the local geometric BA is still important because the pose graph optimization alone does not guarantee the optimal reconstruction of the local environment.



On the TUM monoVO dataset, DSO (both default and reduced) is significantly more robust than ORB-SLAM/VO, which is similar to the results reported in <cit.>.
Our VO system achieves very similar performance to DSO-reduced, as none of the final trajectories are affected by the feature-based module.
This is because more than 75% of the total keyframes have collinear covisibility links in all sequences (see Section. <ref>), which is in contrast to the EuRoC MAV dataset where the opposite is true.
Fig. <ref> and <ref> show that the loop closure significantly reduces the alignment errors for our SLAM system in the majority of the sequences.
As a result, our SLAM system achieves the best overall performance across both datasets.
Fig. <ref> compares the estimated trajectories on some of the TUM monoVO sequences. 
Note how ORB-SLAM fails in the middle of some sequences and DSO accumulates drift, while our SLAM system completes the whole sequences and closes loops. 







Tab. <ref> summarizes the statistics of dropped frames and tracking times[This is defined as t_i-t_i-1 where t_i and t_i-1 are the two timestamps at which the tracking thread first processes frame i and i-1, respectively.] on the EuRoC MAV dataset.
It can be seen that DSO-reduced and both our systems have lower frame drops and faster tracking speed than the rest.
This shows the advantage of direct tracking, which eliminates the need to perform feature extraction and matching for every frame. 
In Fig. <ref>,  we further show how much percentage of keyframes and map points are reduced in our systems compared to ORB-VO/SLAM. 
We observe average 27% (up to 46%) keyframes reduction and average 6% (up to 27%) map points reduction for the SLAM system.
This suggests that our system is relatively more scalable than ORB-SLAM.
Note that the sparsity of the feature-based keyframes and map points is what enabled the efficient map reuse on both local and global scale.
Without relying on the sparse keypoints, it would require prohibitively more computation to maintain a global covisibility graph with the large number of map points in the direct module.




§ CONCLUSIONS


In this paper, we proposed a loosely-coupled semi-direct method for real-time monocular SLAM.
Our system consists of two modules running in parallel.
One module uses a direct method to track new frames fast and robustly with respect to a local semi-dense map.
The other module uses the resulting points and pose estimates as prior to build a globally consistent sparse feature-based map.
We have shown on two public datasets that our method outperforms the state-of-the-art in terms of tracking accuracy and robustness.



























































































































































































































































                                  
                                  
                                  
                                  
                                  



















IEEEtran 
10

url@rmstyle






autonomous_car
D. Nistér, O. Naroditsky, and J. Bergen, “Visual odometry for ground
  vehicle applications,” Journal of Field Robotics, vol. 23, no. 1, pp.
  3–20, 2006.

stability_scale
S. Lee and G. C. H. E. de Croon, “Stability-based scale estimation for
  monocular SLAM,” IEEE Robotics and Automation Letters, vol. 3,
  no. 2, pp. 780–787, 2018.

ptam
G. Klein and D. Murray, “Parallel tracking and mapping for small AR
  workspaces,” in Proc. Sixth IEEE and ACM International Symposium
  on Mixed and Augmented Reality (ISMAR), 2007.

mono-slam
A. J. Davison, I. D. Reid, N. D. Molton, and O. Stasse, “MonoSLAM: Real-time
  single camera SLAM,” IEEE Trans. Pattern Anal. Mach. Intell.,
  vol. 29, no. 6, pp. 1052–1067, 2007.

lsd-slam
J. Engel, T. Schöps, and D. Cremers, “LSD-SLAM: Large-scale direct
  monocular SLAM,” in Eur. Conf. on Computer Vision (ECCV), 2014, pp.
  834–849.

dso
J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,” IEEE
  Trans. Pattern Anal. Mach. Intell., vol. 40, no. 3, pp. 611–625, 2018.

sift
D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”
  International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110,
  2004.

orb
E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “ORB: An efficient
  alternative to SIFT or SURF,” in IEEE Int. Conf. on Computer
  Vision (ICCV), 2011, pp. 2564–2571.

dso-vs-orb
N. Yang, R. Wang, X. Gao, and D. Cremers, “Challenges in monocular visual
  odometry: Photometric calibration, motion bias, and rolling shutter effect,”
  IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 2878–2885,
  2018.

svo
C. Forster, M. Pizzoli, and D. Scaramuzza, “SVO: fast semi-direct monocular
  visual odometry,” in IEEE Intl. Conf. on Robotics and Automation,
  (ICRA), 2014, pp. 15–22.

orb-slam
R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós, “ORB-SLAM: A
  versatile and accurate monocular SLAM system,” IEEE Trans. Robot.,
  vol. 31, no. 5, pp. 1147–1163, 2015.

bags_of_binary_words
D. Galvez-López and J. D. Tardos, “Bags of binary words for fast place
  recognition in image sequences,” IEEE Trans. Robot., vol. 28, no. 5,
  pp. 1188–1197, 2012.

scale-drift-aware
H. Strasdat, J. M. M. Montiel, and A. J. Davison, “Scale drift-aware large
  scale monocular SLAM,” in Proc. Robotics: Science and Systems,
  2010.

double_window
H. Strasdat, A. J. Davison, J. M. M. Montiel, and K. Konolige, “Double window
  optimisation for constant time visual SLAM,” in IEEE Int. Conf. on
  Computer Vision (ICCV), 2011, pp. 2352–2359.

orb-semidense
R. Mur-Artal and J. D. Tardos, “Probabilistic semi-dense mapping from highly
  accurate feature-based monocular SLAM,” in Proc. Robotics: Science
  and Systems, 2015.

about_direct_methods
M. Irani and P. Anandan, “About direct methods,” in Proc. Workshop
  Vision Algorithms, 1999, pp. 267–277.

semidense
J. Engel, J. Sturm, and D. Cremers, “Semi-dense visual odometry for a
  monocular camera,” in IEEE Int. Conf. on Computer Vision (ICCV),
  2013, pp. 1449–1456.

dtam
R. A. Newcombe, S. Lovegrove, and A. J. Davison, “DTAM: dense tracking and
  mapping in real-time,” in IEEE Int. Conf. on Computer Vision
  (ICCV), 2011, pp. 2320–2327.

multilevel
W. N. Greene, K. Ok, P. Lommel, and N. Roy, “Multi-level mapping: Real-time
  dense monocular SLAM,” in IEEE Intl. Conf. on Robotics and
  Automation (ICRA), 2016, pp. 833–840.

sparse_vs_dense
L. Platinsky, A. J. Davison, and S. Leutenegger, “Monocular visual odometry:
  Sparse joint optimisation or dense alternation?” in IEEE Intl. Conf.
  on Robotics and Automation (ICRA), 2017, pp. 5126–5133.

photo_calibration
J. Engel, V. Koltun, and D. Cremers, “A photometrically calibrated benchmark
  for monocular visual odometry,” CoRR, vol. abs/1607.02555, 2016.

ldso
X. Gao, R. Wang, N. Demmel, and D. Cremers, “LDSO: Direct sparse odometry
  with loop closure,” in IEEE/RSJ Intl. Conf. on Intelligent Robots and
  Systems (IROS), 2018.

remode
M. Pizzoli, C. Forster, and D. Scaramuzza, “REMODE: Probabilistic, monocular
  dense reconstruction in real time,” in IEEE Intl. Conf. on Robotics
  and Automation (ICRA), 2014, pp. 2609–2616.

svo2
C. Forster, Z. Zhang, M. Gassner, M. Werlberger, and D. Scaramuzza, “SVO:
  semidirect visual odometry for monocular and multicamera systems,”
  IEEE Trans. Robot., vol. 33, no. 2, pp. 249–265, 2017.

li_semidirect
S. Li, T. Zhang, X. Gao, D. Wang, and Y. Xian, “Semi-direct monocular visual
  and visual-inertial SLAM with loop closure detection,” Robotics and
  Autonomous Systems, vol. 112, pp. 201 – 210, 2019.

semidirect_rgbd
S. Bu, Y. Zhao, G. Wan, K. Li, G. Cheng, and Z. Liu, “Semi-direct tracking and
  mapping with RGB-D camera for MAV,” Multimedia Tools and
  Applications, vol. 76, no. 3, pp. 4445–4469, 2017.

krombach
N. Krombach, D. Droeschel, and S. Behnke, “Combining feature-based and direct
  methods for semi-dense real-time stereo visual odometry,” in Int.
  Conf. on Intelligent Autonomous Systems, 2016, pp. 855–868.

kim_hybrid
P. Kim, H. Lee, and H. J. Kim, “Autonomous flight with robust visual odometry
  under dynamic lighting conditions,” Autonomous Robots, 2018.

lucas-kanade-unifying1
S. Baker and I. Matthews, “Lucas-kanade 20 years on: A unifying framework,”
  International Journal of Computer Vision, vol. 56, no. 3, pp.
  221–255, 2004.

orb-slam2
R. Mur-Artal and J. D. Tardós, “ORB-SLAM2: an open-source SLAM system
  for monocular, stereo and RGB-D cameras,” IEEE Transactions on
  Robotics, vol. 33, no. 5, pp. 1255–1262, 2017.

huber
P. J. Huber, “Robust estimation of a location parameter,” Annals of
  Mathematical Statistics, vol. 35, no. 1, pp. 73–101, 1964.

inverse_depth
J. Civera, A. J. Davison, and J. M. M. Montiel, “Inverse depth parametrization
  for monocular SLAM,” IEEE Trans. Robot., vol. 24, no. 5, pp.
  932–945, 2008.

keyframe-based-leutenegger
S. Leutenegger, S. Lynen, M. Bosse, R. Siegwart, and P. Furgale,
  “Keyframe-based visual-inertial odometry using nonlinear optimization,”
  The International Journal of Robotics Research, vol. 34, no. 3, pp.
  314–334, 2015.

g2o
R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard,
  “g2o: A general framework for graph optimization,” in IEEE Intl.
  Conf. on Robotics and Automation (ICRA), 2011, pp. 3607–3613.

arun
K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two
  3-D point sets,” IEEE Trans. Pattern Anal. Mach. Intell., vol.
  PAMI-9, no. 5, pp. 698–700, 1987.

horn
B. K. P. Horn, “Closed-form solution of absolute orientation using unit
  quaternions,” Journal of the Optical Society of America A, vol. 4,
  no. 4, pp. 629–642, 1987.

fast-relocal
R. Mur-Artal and J. D. Tardós, “Fast relocalisation and loop closing
  in keyframe-based SLAM,” in IEEE Intl. Conf. on Robotics and
  Automation (ICRA), 2014, pp. 846–853.

euroc_mav_dataset
M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W.
  Achtelik, and R. Siegwart, “The EuRoC micro aerial vehicle datasets,”
  The International Journal of Robotics Research, 2016.




