Multi-Session SLAM with Differentiable Wide-Baseline Pose Optimization
Lahav Lipson Jia Deng



Quantum optical classifier with superexponential speedup
    Seth Lloyd 0000-0003-0353-4529
    April 29, 2024
========================================================




We introduce a new system for Multi-Session SLAM, which tracks camera motion across multiple disjoint videos under a single global reference. Our approach couples the prediction of optical flow with solver layers to estimate camera pose. The backbone is trained end-to-end using a novel differentiable solver for wide-baseline two-view pose. The full system can connect disjoint sequences, perform visual odometry, and global optimization. Compared to existing approaches, our design is accurate and robust to catastrophic failures. Code is available at <https://github.com/princeton-vl/MultiSlam_DiffPose>








§ INTRODUCTION


Simultaneous Localization and Mapping (SLAM) is the task of estimating camera motion and a 3D map from video. The standard setup assumes a single continuous video. However,  video data in the wild often consists of not a single continuous stream, but rather multiple disjoint sessions, either deliberately such as in collaborative mapping when multiple robots perform joint rapid 3D reconstruction, or inadvertently due to visual discontinuities in the video stream which can result from camera failures, extreme parallax, rapid turns, auto-exposure lag, dark areas, or extreme occlusion by dynamic objects. Handling such disjoint videos is important for many applications in AR and robotics, and gives rise to the task of Multi-Session SLAM.=-1

In Multi-Session SLAM, the input consists of multiple disjoint video sequences and the goal is to estimate camera poses for all video frames under a single global reference. This is in contrast to “single-video SLAM”, whose input is a single continuous video. In this work, we focus on the monocular, RGB-only Multi-Session SLAM setting. 

Several approaches have been proposed to deal with Multi-Session SLAM, however existing solutions typically require additional sensor data in order to remove gauge freedoms and make tracking easier <cit.>. Only a small number of methods, notably CCM-SLAM <cit.> and ORB-SLAM3 <cit.>, support Multi-Session SLAM from monocular video alone, due to the difficulty of aligning disjoint sequences under the 7-DOF gauge freedoms in monocular video. However, these approaches are based on classical feature descriptors, making them less accurate on average compared to recent designs based on deep networks. 

In the standard SLAM setting, Teed and Deng <cit.> proposed to use a deep optical flow network (RAFT <cit.>) to track 2D motion, while jointly updating camera poses with a bundle adjustment layer. The method, DROID-SLAM, is accurate and avoids tracking failures, but the design assumes a continuous video stream and is not capable of the wide-baseline matching and non-local optimization necessary for Multi-Session SLAM. Deep Patch Visual Odometry <cit.> (DPVO) introduced a sparse visual-odometry-only analog of DROID-SLAM which achieves similar accuracy on single-video VO, but at much lower cost. However, DPVO also does not support Multi-Session SLAM for the same reasons as DROID-SLAM.=-1 


We propose a method for Multi-Session SLAM, capable of both wide-baseline relative pose and visual odometry using a single backbone architecture inspired by <cit.>. We introduce a differentiable solver layer which minimizes the symmetric epipolar distance (SED) from bi-directional optical flow. From this we construct a method for two-view pose which is capable of matching from far-apart views. This same design can be repurposed for visual odometry by swapping out the solver for bundle adjustment. By employing a unified backbone architecture for both tasks, we enable a simple approach to Multi-Session SLAM.

We evaluate our approach on challenging real-world datasets: EuRoC-MAV <cit.> and ETH3D <cit.>. Our system is more accurate than prior approaches, and is robust to catastrophic failures. We also evaluate our two-view pose method in isolation on the Scannet and Megadepth datasets, and show that it is competitive with transformer-based matching networks. For pairs of far-apart views, our method is capable estimating accurate relative pose.


Our backbone predicts iterative updates to optical flow coupled with a differentiable solver layer for estimating camera pose. This framework, based on RAFT <cit.>, has worked exceptionally well for Visual Odometry <cit.> and SLAM <cit.>. By leveraging this idea for wide-baseline matching, we can extend these methods to the Multi-Session setting without introducing substantial complexity.=-1








§ RELATED WORK





Most prior works treat Visual Simultaneous Localization and Mapping (Visual SLAM) as an optimization problem solving for a 3D scene model and camera trajectory which best explains the visual measurements <cit.>.


Indirect approaches to SLAM perform keypoint matching as a pre-processing step, then estimate a 3D point cloud and camera poses by optimizing the 2D reprojection error over all matches <cit.>. Reprojection-error is easier to optimize than photometric alignment, making indirect methods robust to lower camera hz <cit.>. Keypoint matching also enables a straightfoward approach to estimating two-view relative pose, a necessary step for Visual Multi-Session SLAM <cit.>. However, keypoint-based SLAM is less robust to low-texture environments compared to those which use photometric alignment <cit.> or optical flow <cit.>.=-1 


Semi-Indirect approaches similarly optimize 2D reprojection error like indirect methods, but without requiring matches as input <cit.>. Instead, these approaches alternate between predicting optical flow residuals and performing bundle adjustment. Semi-indirect methods do not require repeatable keypoints across images, making them robust to low-texture settings while retaining the easier reprojection-error objective.

Our approach is most similar to Deep Patch Visual Odometry <cit.> (DPVO), which is a sparse analog of DROID-SLAM. DPVO predicts sparse optical flow instead of dense, performing similarly to DROID-SLAM while running faster and using half the memory.


Differentiable solver layers for camera pose estimation have been used in order to learn outlier rejection with data-driven training. For two-view relative pose, Ranftl and Koltun <cit.> used a deep network to learn an iteratively reweighted least-squares algorithm, which solved a weighted variant of the 8-point-algorithm <cit.> using confidences predicted by the network. <cit.> required matched points as input, wheras our approach can work from images alone. Roessle and Nießner <cit.> proposed an end-to-end architecture which used the weighted 8-point-algorithm from  <cit.> on top of matches produced using Superglue <cit.>, and supervised directly on the predicted pose. <cit.> works well, but is unable to outperform existing methods that use minimal solvers with LO-RANSAC <cit.> instead of differentiable solvers. In contrast, our approach is not built on top of an existing SOTA 2-view matcher; our design is also iterative, applying a recurrent module and solver layers multiple times to refine the prediction.


Multi-Session SLAM is the task of performing SLAM on multiple trajectories of the same scene. Like SLAM, Multi-Session SLAM is an online task where camera motion is estimated from a stream of images. However, in the Multi-Session setting, there are known breaks in the data-stream over which the small-baseline assumption no longer holds. While local optimization is sufficient for motion tracking from video, estimating wide-baseline camera pose is often non-convex <cit.> and more challenging due to large viewpoint changes. In monocular visual SLAM, the scale of each sequence is also ambiguous; aligning two trajectories requires estimating 7 degrees-of-freedom (translation, rotation, scale) for all sequences, excluding the first which can be considered the reference.=-1

ORB-SLAM3 <cit.> performs Multi-Session SLAM by matching between ORB <cit.> descriptors. While tracking camera motion over each sequence, ORB-SLAM3 continually updates an Atlas of the scene - a lookup table between 2D keypoints and their 3D map point in the reference frame of their original sequence. To align disjoint sequences, candidate cross-sequence image pairs are identified using image-retrieval <cit.>, the 3D map points are obtained from the Atlas, and the relative transformation is found using the Umeyama algorithm <cit.>. Several methods perform Multi-Session Visual Inertial SLAM <cit.>, however we focus on the visual-only monocular setting where the scale of each session is ambiguous and must be estimated when joining sequences.=-1

CCM-SLAM <cit.> and SLAMM <cit.> perform Multi-Session visual SLAM and are built on top of ORB-SLAM <cit.> as well, but are optimized for limited bandwidth and distributed processing, while ORB-SLAM3 performs better and is optimized for accuracy and speed.






§ APPROACH


Overview: We propose a backbone for matching between ≥ 2 views, and then construct a method for multi-session SLAM upon it. Our backbone approaches matching as optical flow; it borrows several ideas from RAFT <cit.>, such as iteratively predicting flow residuals using a recurrent network, and the correlation feature pyramid. Our method maintains a running estimate of both camera pose and bi-directional optical flow, and uses the update operator to refine them both. We provide an overview in Fig. <ref>. An invariant of our backbone is that the matches are always clamped to plausible values given the pose estimates and an assumption of rigid scene geometry, therefore WLOG the matches can be considered a depth estimate.=-1

At initialization, our backbone produces dense correlation feature pyramids for each image and context features which remain fixed throughout the forward pass. The update operator uses an RNN to predict an update to the matches, and a differentiable solver to update poses. Our backbone treats the two-view and multi-view settings differently: In the two-view setting, the solver updates the poses to minimize the symmetric epipolar distance (SED). In the multi-view setting, the solver updates both poses and depth to minimize the reprojection error. After the solver layer, the matches are adjusted to agree with the poses/depth.



 §.§ Initialization



Feature Extraction and Feature Pyramid: Similar to RAFT, our method separately extracts context and correlation feature for each image. The context features are provided as input to the recurrent update operator, whereas the correlation features are used to evaluate the visual similarity between any two pixels using a dot product. The context feature maps are produced at 1/8 resolution using a residual network. We then apply a linear-self-attention residual for long-range feature sharing. The correlation feature maps are also produced using a residual network, but with several exit ramps to produce features at 1/2, 1/4, and 1/8 of the input image resolution. We also average-pool the last one three additional times to produce a correlation feature pyramid with 6 levels. We depict their architecture in the Appendix.=-1

Anchor-Point Selection: Our method predicts sparse optical flow, where matches have one end anchored and the other end free to move in ℝ^2. We use a mix of detector <cit.>-chosen and randomly-chosen anchor points. Each point is assigned an initial match in the other image(s) at the same pixel coordinate. For the rest of the paper, we will refer to the k^th anchor point as a_k, and its match in image j as m_kj. 

We index the context feature map at each a_k to produce a unique context feature vector ctx_k ∈ℝ^384, which is used by the update operator. Beyond this point, the full context feature map is not used and is discarded to save memory.

Correlation: Similar to other RAFT-based methods, we use correlation features to assess the visual alignment/similarity given by the current matching estimate. For each (a_k,m_kj) pair, we bilinearly sample the feature pyramids f and g for their respective frames at locations a_k and m_kj and take their inner product at each level, producing


    ⟨ f_k^i, g_kj^i ⟩∈ℝ : i=1...6

To provide additional spatial context, we also perturb both a_k and m_kj in 3× 3 and 7 × 7 grids, respectively, and calculate eq. <ref> for all pairs. The resulting feature vector 𝐂_kj∈ℝ^(6 × 3  3  7  7) is then passed to the update operator. 




 §.§ Update Operator

The update operator produces a revision to all m_kj and the estimated relative pose. It consists of three stages, depicted in Fig. <ref>: (1) An RNN predicts a 2D update to m_kj and an associated confidence weight w_kj. (2) We solve for a pose estimate which is consistent with the newly predicted matches and confidence. (3) We adjust the matches to be physically plausible assuming rigid scene geometry. In the two-view setting, we clamp the matches to the epipolar lines, and in the multi-view setting we reproject the anchor points using the depth and poses from the solver. 

Through the recurrent iterations, our method maintains a running state for each anchor-match (a_k,m_kj) pair, consisting of a hidden state vector h_kj, a confidence weight w_kj, and a match location m_kj in frame j. This state is continually updated, during which m_kj should ideally approach the true match location of a_k in frame j, and w_kj should approach 1. This is what we observe empirically.


RNN Module: The learnable component of the update operator is the recurrent network which predicts updates to h_kj, m_kj and w_kj for all anchor points. We visualize this operator in Fig. <ref>.

The input to the RNN are the context features, the previous hidden state, and the correlation features generated from the current matching estimate, all of which are added together and normalized with layernorm. A self-attention residual is also applied to edges with the same source and destination frame, followed by three gated-residual-units, whose architecture is depicted in the Appendix. The output is an updated hidden state, from which we predict an updated match and confidence using the flow head and confidence head, respectively, which are implemented as two-layer MLPs. The confidence head includes a softmax to restrict w_kj∈ (0,1).





 §.§ Two-View Solver and Pre-Conditioning


Our two-view solver aims to align the relative pose between two frames to the predicted matches by minimizing the distance between each match and its epipolar line. This strategy is accurate, however the objective function is non-convex, meaning it will only converge to the global minimum if it is initialized close to it. In contrast, the 8-point-algorithm, a common approach to this task, has the opposite problem; it does not suffer from local minimum since it is a homogeneous least squares, but the solution is sub-optimal. In Fig. <ref>, we visualize how our solver either converges to within a fraction of a degree, or not at all, meanwhile the 8-point-algorithm is more robust but less accurate.

The approach our method uses is to pre-condition the pose estimate using a weighted, dense variant of the 8-point algorithm <cit.>, and then run our solver layer to refine the pose. This combined strategy obtains the best of both worlds, since the pre-conditioning will typically initialize the pose within the basin of convergence of our solver, which then refines the prediction. Fig. <ref> visualizes this basin on the TartanAir <cit.> dataset using ground-truth flow.

Pre-conditioning: We solve the homogeneous least squares problem <cit.>


    _F ||diag(w⃗)𝐌F⃗||^2    s.t.  ||F||^2 = 1

where 𝐌 and w⃗ are constructed from the anchor-match pairs and confidence predictions. The points are normalized to [-1,1] beforehand. Afterwards, we reshape and uncalibrate 𝐅 to obtain the essential matrix.

We obtain the four relative pose candidates following the procedure detailed in  <cit.> and in the Appendix. During training, we select the pose candidate closest to the ground-truth on the SE(3) manifold. During inference, we select the candidate by testing for chirality <cit.>.


SED Solver Layer: The solver layer seeks to minimize the distance between each match m_kj, and the epipolar line induced by the current pose estimate and its respective anchor point a_k. The solver objective is bi-directional, since every image contains a unique set of anchor points; the objective is also known as the titular symmetric epipolar distance (SED) <cit.>. We parameterize the pose update as small rotations to the translation direction and orientation: ξ_𝐭, ξ_𝐑∈𝔰𝔬(3). These are the free variables. Let K(i) be the set of anchor indices for frame i, 𝐑 and 𝐭 be the output of the pre-conditioning stage, and epi(x,R,t) compute the epipolar line for a point x given the relative pose (R,t). Formally, the minimization objective in our solver is

    l_kj = epi(a_k, (e^ξ_𝐑𝐑)_i→ j, (e^ξ_𝐭𝐭)_i→ j) : k∈ K(i)
    
        E_i→ j = ∑_k ∈ K(i) w_kj·err(m_kj, l_kj)_2^2
    
        SED = _ξ_𝐑, ξ_𝐭(E_i→ j + E_j→ i)

where err computes the 2D point-to-line error. The predicted weights w_kj are included to allow the network to down-weight the contribution of any match which it deems unreliable. This solver layer is implemented in Pytorch, enabling gradients to propagate backward from the pose loss to the confidence head using autograd. We depict the gradient flow in Fig. <ref>.

To minimize eq. <ref>, we employ the Levenberg Marquardt algorithm. This requires computing the Jacobian for all terms. We provide the full derivations in the Appendix, but define the residual function err in its entirety here. 

Let l be the epipolar line produced by (a, e^ξ_𝐑𝐑, e^ξ_𝐭𝐭). We can express each term of eq. <ref> as:

    err(m,l) = (l_xm_x + l_ym_y + l_z/l_x^2 + l_y^2)[ l_x; l_y ]

and the epipolar line as

    𝐄 = (e^ξ_𝐑𝐑)^⊤ [ e^ξ_𝐭𝐭]_
    
        l = 𝐅[ a_x; a_y;   1 ] = [(K_2^⊤)^-1𝐄K_1^⊤] [ a_x; a_y;   1 ]

where 𝐅 and 𝐄 are the fundamental and essential matrices.


 §.§ Adapting our backbone to Visual Odometry


To adapt our backbone to VO, we make several changes:

Multi-view Solver (BA): Our local optimizer minimizes reprojection error and treats depth as a separate variable. This is identical to the bundle adjustment from DPVO. Formally, let F be the set of connected frames, G be the global poses, d_k be the depth estimate for anchor k, and Π(·) be the 3D→ 2D projection function. The bundle adjustment objective is:

    _G, 𝐝∑_(i,j)∈ F∑_k∈ K(i)  w_kj·Π[G_j^-1 G_i Π^-1(a_k,d_k)] - m_kj^2

The preconditioning stage is not necessary as we can linearly-extrapolate the camera pose estimates from previous frames to achieve good initialization.



Clamping: Since epipolar lines do not make sense in the multi-view setting, we reset the optical flow using the pose and depth from the bundle adjustment. 

RNN changes: We include mechanisms in the RNN to share latent features between updates if they stem from the same anchor point. Specifically, we use the message passing and temporal convolutions from DPVO <cit.>.


 §.§ Trajectory Alignment
 In the Multi-Session SLAM problem, we estimate a relative  between pairs of disjoint trajectories in order to align them. All anchor points already have a depth estimate d_k as result of the bundle adjustment (eq. <ref>) from the VO system. We use our two-view method to estimate the relative rotation and translation direction, and compute the translation magnitude and relative scaling by comparing the d_k's to the depth from the triangulated two-view matches. 


1) Estimate relative rotation, translation direction, and matches. We first retrieve a candidate image pair (i, j) using NetVLAD <cit.>, one from each trajectory, and apply our two-view model to estimate their relative rotation and translation direction.


2) Align the depth from the two-view and VO operators. We previously defined d_k as the depth output of the VO system. Let d^'_k be the triangulated depth obtained from our two-view matches:


    d^'_k =  
        triang(𝐑_i→ j, 𝐭_i→ j, a_k, m_kj)     : k ∈ K(i)
    
        triang(𝐑_j→ i, 𝐭_j→ ia_k, m_ki)     : k ∈ K(j)

We solve


    _s_i∈ℝ > 0∑_k∈ K(i)1[1/λ < d_k/s_i· d^'_k < λ]

to recover the translation magnitude, where the hyperparameter λ = 1.05. In layman terms, eq. <ref> seeks what translation magnitude would align the most triangulated points to the existing 3D map? We can obtain a reasonably good solution to eq <ref> by brute-force checking s_i=d_k / d_k^'  ∀ k∈ K(i), similar to RANSAC. If the number of inliers is too small, we retry on a new candidate pair. Conversely, the scale-difference is s_i / s_j, where s_j is estimated the same way, but for anchor points in K(j):


    _s_j ∈ℝ > 0∑_k ∈ K(j)1[1/λ < d_k/s_j· d_k^' < λ]

Given the rotation 𝐑_j→ i and translation 𝐭_j→ i from the two-view solver layers, the resulting transformation can be used to align the two trajectories:


    𝒮_j→ i = [ s_i/s_j𝐑     s_j𝐭;        0        1 ]∈

The graphs are then merged by concatenating all buffers.



 §.§ Training


We train our backbone separately for VO and for two-view pose. It is trained using a matching loss and a pose loss. The pose loss for the two-view training

    ℒ_pose2V = ∑_t=1^12 cos^-1(t̅^̅t̅·t̅^̅g̅t̅) + α[(R^t)^T R^gt]_SO(3)

penalizes angle error for both the predicted translation direction and predicted orientation. The pose loss for the VO training is the SE3 manifold distance between the predicted and ground-truth poses:


    ℒ_poseVO = ∑_t=1^T[(G^t)^-1G^gt]_SE(3)

The matching loss


    ℒ_matching = 1/|N|∑_a^N∑_t=1^Tm_a^t - m_a^gt_2

is a standard endpoint-error <cit.>, applied only on pixels which have valid depth and are verified to be visible in both images. Our update operator is applied 12 times per training example; supervision is applied after every update (See Fig. <ref>). The final loss is ℒ=ℒ_pose + β·ℒ_matching.

Following prior work <cit.>, we pre-train our two-view method on synthetic homographies (without pose loss) for two epochs on the Oxford-Paris 1M Distractors dataset <cit.>. We then tune our model on a 50/50 mixture of Scannet <cit.> and Megadepth <cit.> for 100,000 steps and a batch size of 120, using 10 A6000 GPUs for 5 days. The VO backbone is trained using the procedure from <cit.> on TartanAir.











 §.§ Multi-Session SLAM System



Overview: To perform Multi-Session SLAM, we use our VO-trained backbone to perform visual odometry and global optimization; our two-view backbone connects disjoint trajectories following the procedure in Sec <ref>. Our full system builds the pose graph incrementally: new video frames inserted into the factor graph as they are received, and unjoined trajectories are aligned and merged as soon as a connection is found.


VO Frontend: Our VO frontend extracts features and estimates camera poses and depth for incoming frames, operating on a sliding window covering the most recently observed 22 keyframes. Keyframing occurs retroactively on the 4^th oldest keyframe if it is found to be redundant. New poses are initialized using a linear-motion model, and depth is copied from the previous frame. Like DPVO <cit.>, we initialize after observing 8 frames with significant motion.


Global Optimization: We perform global optimization by introducing proximity factors using the existing pose and depth measurements, following the approach from DROID-SLAM <cit.>, and update the matches and poses/depth using the VO backbone. We only run the backend periodically, and after joining trajectories.


Trajectory Joining: We query the database of NetVLAD descriptors to find co-visible frames. If several pairs are retrieved, we compute relative poses in a mini-batch and use the estimate with the highest inlier-ratio.











§ EXPERIMENTS


We evaluate the performance of our two-view method in isolation, and our Multi-Session SLAM system as a whole.




 §.§ Two-View Evaluation







We evaluate our two-view method on two popular relative pose benchmarks: The Scannet <cit.> 1500 and MegaDepth <cit.> 1500 test datasets. We compare to existing matching networks <cit.>, which are typically in service of improving COLMAP <cit.>-based SfM pipelines <cit.>. In contrast, our approach is in service of Multi-Session SLAM. We report pose error AUC, as is done in prior matching work (higher is better). We use the same configuration and model weights on both datasets.

Most existing methods perform matching as a pre-processing step using a deep network, then estimate pose with a minimal solver using RANSAC <cit.>. Roessle and Nießner <cit.> replace the RANSAC optimizer with a weighted-8-point algorithm and bundle adjustment. In contrast, our two-view method couples the prediction of pose and matches, refining both over many iterations, and regresses optical flow instead of keypoint affinities.


Scannet: We report two-view results on Scannet <cit.> in Tab. <ref>. Our approach outperforms existing methods on Scannet, which contains fewer salient keypoints and many texture-less surfaces and motion blur.


MegaDepth: We report two-view results on Megadepth <cit.> in Tab. <ref>. On Megadepth, the matching problem is substantially easier since photo-tourism images contain many salient keypoints, resulting in better performance across all methods; our approach is on-par with existing approaches using the OpenCV RANSAC pose estimator <cit.>, but not the LO-RANSAC <cit.> estimator.


Ablations: We ablate on various aspects of our proposed architecture in Tab. <ref> and show that they lead to improved performance. Specifically, we show that pre-conditioning and the SED solver lead to better accuracy when used together, and that clamping the matches to the epipolar lines is also important.


 §.§ Multi-Session SLAM evaluation









We evaluate our approach to Multi-Session SLAM on the EuRoC-MAV <cit.> and ETH3D <cit.> datasets, since they provide camera poses under a single global reference. Following the evaluation for single-video SLAM, we align the final predictions to the ground truth by computing a global 7-DOF alignment to account for the guage freedoms. We report RMSE ATE[m] in metric units. We sample 96 anchors in each video frame using a mix of random and Superpoint <cit.> keypoints. Our two-view method uses those same anchors to join pairs of trajectories. We use the same weights and configuration on both datasets.



EuRoC-MAV: In Tab. <ref> and Fig. <ref>, we report results on the EuRoC MAV dataset <cit.>. Following the evaluation from  <cit.>, we compare on four groups of disjoint trajectories across the three distinct environments: 3 in the Machine Hall, 5 in the Machine Hall, 3 in Vicon 1, and 3 in Vicon 2. Ground-truth poses are obtained using a laser tracker and Vicon cameras. These video sequences are long, with the entirety of the Machine-Hall group being 11.5 minutes of video (13.7K frames) spread across 5 sequences. Our approach achieves significantly lower error than ORB-SLAM3 on all groups, including 11x lower error on the Vicon 2 sequences (0.024 vs 0.284). Video is recorded at 20-FPS; all methods run in real-time. We also compare to CCM-SLAM <cit.>, which only provides results on the on MH01-03 sequence, the mono-inertial ORB-SLAM3, and the mono-inertial VINS <cit.>. We outperform the other approaches on all trajectory groups.


ETH3D: In Tab. <ref>, we report results on the training set from ETH3D <cit.>. We compare across 5 unique scenes composed of multiple trajectories: 1-4 from Sofa, 3&4 from Table, 1-3 from Plant Scene, 1&2 from Einstein, and 2&3 from Planar. Sequences are excluded if they belong to the official ETH3D test set (with no ground-truth) or are from a different scene altogether. The sequences are trimmed from a single, longer video; to make joining the trajectories non-trivial, we reverse every other sequence, ensuring that there is a sufficient disconnect between subsequent videos. This is a novel benchmark, so <cit.> do not report results. =-1

ORB-SLAM3 fails on Sofa, Plant Scene, and Einstein for various reasons (couldn't initialize, or lost the feature tracks). Our approach outperforms ORB-SLAM3 on 4/5 groups, and does not fail on any. ORB-SLAM3 only succeeds on 2/5 groups.























§ CONCLUSION



We introduce a new method for mono-visual Multi-Session SLAM. Our system utilizes a novel backbone which can estimate two-view pose and perform visual odometry. We leverage a novel differentiable solver which minimizes the symmetric epipolar distance. We compare against existing approaches and show strong performance across several datasets. This work was partially supported by IARPA and the National Science Foundation. 



    ieeenat_fullname
    





Appendix

















§ GRADIENT COMPUTATION FOR THE SED SOLVER


    d     l_x^2 + l_y^2
    ζ    l_xm_x + l_ym_y + l_z
    ∂ err(m,l)/∂ l   = [ (-2l_x^2ζ/d^2 + l_xm_x/d + ζ/d)      (-2l_xl_yζ/d^2 + l_xm_y/d)                         (l_x/d);      (-2l_yl_xζ/d^2 + l_ym_x/d) (-2l_y^2ζ/d^2 + l_ym_y/d + ζ/d)                         (l_y/d);                                 ]



    Let a be the anchor location in homogenous coordinates. The partials for l w.r.t. 𝐅 are



    ∂ l/∂𝐅 = ∂𝐅a/∂𝐅 = a^⊤⊗𝕀∈ℝ^3 × (3 × 3)



    The partials for the fundamental matrix w.r.t. the essential matrix, given the known calibration matrix K:



    ∂𝐅/∂𝐄 = ∂[(K^-1)^⊤𝐄 K^-1]/∂𝐄 = (K^-1)^⊤⊗ K^-1∈ℝ^(3 × 3) × (3 × 3)



The 3 × 3 essential matrix in terms of the input rotation 𝐑 and translation 𝐭, and the local updates ξ_𝐑 and ξ_𝐭, is



    𝐄 = (𝐞^ξ_𝐑𝐑)^⊤[𝐞^ξ_𝐭𝐭]_×



The derivatives of ξ_𝐑 and ξ_𝐭 are taken at the identity, so they are equal to 0 when treated as a constant. To compute the partial of this essential matrix w.r.t. ξ_𝐑 at 0:


.3

    tx [𝐭]_×∈ℝ^3 × 3


.7

    ∂𝐄/∂ξ_𝐑 = ∂ (e^ξ_𝐑𝐑)^⊤[𝐭]_×/ξ_𝐑 = [ 𝐑^⊤[tx_c1]_×; 𝐑^⊤[tx_c2]_×; 𝐑^⊤[tx_c3]_×;              ]^⊤∈ℝ^(3 × 3) × 3




Similarly, the partial w.r.t. ξ_𝐭 at 0:



    ∂𝐄/∂ξ_𝐭 = ∂𝐑^⊤[𝐞^ξ_𝐭𝐭]_×/∂ξ_𝐭 = ∂𝐑^⊤[𝐞^ξ_𝐭𝐭]_×/∂ (e^ξ_𝐭𝐭) ∂ (e^ξ_𝐭𝐭)/∂ξ_𝐭 = 𝐑^⊤ ∂ [n⃗]_×/∂n⃗  (-tx) ∈ℝ^(3 × 3) × 3



    Putting it together with the chain rule:



    ∂ err(m,l)/∂ξ_𝐑   = ∂ err(m,l)/∂ l∂ l/∂𝐅 ∂𝐅/∂𝐄 ∂𝐄(e^ξ_𝐑𝐑, e^ξ_𝐭𝐭)/∂ξ_𝐑∈ℝ^2 × 3
    ∂ err(m,l)/∂ξ_𝐭   = ∂ err(m,l)/∂ l∂ l/∂𝐅 ∂𝐅/∂𝐄 ∂𝐄(e^ξ_𝐑𝐑, e^ξ_𝐭𝐭)/∂ξ_𝐭∈ℝ^2 × 3







§ EXTRACTING ROTATION AND TRANSLATION FROM THE ESSENTIAL MATRIX


To obtain the relative pose given the essential matrix 𝐄, we follow the procedure prescribed in  <cit.>.


.3

    W := [  0 -1  0;  1  0  0;  0  0  1;    ]


.3

    Z := [  0  1  0; -1  0  0;  0  0  0;    ]


.3

    U, Σ, V^⊤ = SVD(𝐄)







.3

    t = UZU^⊤


.3

    R1 = UWV^⊤


.3

    R2 = UW^⊤V^⊤




The four plausible solutions are

    [(t, R1), (t, R2), (-t, R1), (-t, R2)]

During training, we choose the solution closest to the ground-truth. During inference, we choose the pose which triangulates the most points in front of the camera.


§ ARCHITECTURE DETAILS









MLP-GRU We depict the gated residual unit in Fig. <ref>. This is the same design from DPVO <cit.>.


Feature Extractors We visualize the feature extractors in Fig. <ref>. The correlation features are produced at 1/2, 1/4 and 1/8 the image resolution. The context features are extracted only at 1/8 resolution, and have additional self-attention layers to propagate information over the image.
































§ ADDITIONAL QUALITATIVE RESULTS





