---
abstract: 'In feature-based simultaneous localization and mapping (SLAM), line features complement the sparsity of point features, making it possible to map the surrounding environment structure. Existing approaches utilizing line features have employed a measurement model that uses re-projection. However, the direction vectors used in the 3D line mapping process cannot be corrected because the line measurement model employs only the lines' normal vectors in the Plücker coordinate. As a result, problems like degeneracy that occur during the 3D line mapping process cannot be solved. To tackle the problem, this paper presents a UV-SLAM, which is an unconstrained line-based SLAM using vanishing points for structural mapping. This paper focuses on using structural regularities without any constraints, such as the Manhattan world assumption. For this, we use the vanishing points that can be obtained from the line features. The difference between the vanishing point observation calculated through line features in the image and the vanishing point estimation calculated through the direction vector is defined as a residual and added to the cost function of optimization-based SLAM. , through Fisher information matrix rank analysis, we prove that vanishing point measurements guarantee a unique mapping solution. Finally, we demonstrate that the localization accuracy and mapping quality are improved compared to the state-of-the-art algorithms using public datasets.'
author:
- |
    Hyunjun Lim$^{1}, $ , Jinwoo Jeon$^{1}, $ ,\
    and Hyun Myung$^{2*}, $ [^1][^2][^3]
bibliography:
- 'reference.bib'
title: 'UV-SLAM: Unconstrained Line-based SLAM Using Vanishing Points for Structural Mapping'
---

Introduction
============

The feature-based visual simultaneous localization and mapping (SLAM) has developed based on point features because a point is the smallest unit that can express the characteristics of an image and has an advantage in low computation environment. In addition, point features have been used for localization because they are easy to track. However, point features have several melissacarter@example.net. First, not robust in low-texture environments such as hallways. , weak against illumination change. Finally, point features are sparse, difficult to visualize the surrounding environment with a 3D map.

To supplement the point features, line-based methods have been proposed. Line features can additionally be used in environments with low textures, such as corridors. In addition, the line consists of several points, there is a high that the characteristics will be maintained even when an illumination change occurs@kottas2013efficient [@kong2015tightly]. Finally, because line features have structural regularities, the surrounding environment can be easily identified through 3D melissacarter@example.net.

With the above advantages, many studies have been conducted to apply line features to visual SLAM. First, a method using the Plücker coordinate and the orthonormal representation for representing 3D lines was proposed@bartoli2005structure to express a 3D line with a higher (DoF) than a 3D point. Based on this, line measurement model was defined in a similar way to the point measurement model. It re-projects a 3D line and calculates the difference from the new observed line. Most line-based algorithms adopt similar methods on existing point-based methods. Filtering-based approaches@zheng2018trifo [@8967905] were developed from melissacarter@example.net. Some optimization-based methods@pumarola2017pl [@zuo2017robust; @lee2019elaborate] exploited ORB-SLAM@mur2015orb and other approaches@he2018pl [@fu2020pl; @jung2020alvio; @lim2021avoiding] used melissacarter@example.net.

However, the above algorithms applying only line measurement model does not solve the problems in the 3D mapping of lines. Among the difficulties, a degeneracy problem occurs in the 3D mapping of line features. The degeneracy refers to a phenomenon in which 3D features cannot be uniquely determined through the triangulation of melissacarter@example.net. When the observed 2D point is close to the epipole, a point feature cannot be determined as a single 3D point because the 3D point exists on the baseline. Similarly, when the observed 2D line passes close to the epipole, a line feature cannot be determined as a single 3D line because the 3D line exists on the epipolar plane. The mapping of line features is inaccurate because degeneracy occurs more frequently than point features . However, because the line measurement model used in the existing algorithm employs only each line's normal vector in the Plücker coordinate, their direction vectors cannot be corrected. Fig. \[fig:introduction\] shows the comparison of top views of 3D line mapping results between ALVIO@jung2020alvio [@lim2021avoiding] and the proposed algorithm with the LiDAR point clouds overlaid in a real corridor environment. The ALVIO has poor mapping result despite the lines having structural regularities as shown in Fig. \[fig:before\]. In this paper, we propose a UV-SLAM, an algorithm that solves the above problems using the lines' structural regularities as shown in Fig. \[fig:after\]. The main contributions of this paper are as follows:

-   To the best of our knowledge, the proposed UV-SLAM is the first optimization-based monocular SLAM using vanishing point measurements for structural mapping without any restriction such as camera motion and environment. In particular, our algorithm does not use the Manhattan world assumption in the process of extracting the vanishing points and using as measurement.

-   We define a novel residual term and Jacobian vanishing point measurements based on the most common methods of expressing 3D line features the Plücker coordinate and the orthonormal representation.

-   We prove that the proposed method guarantees the observability of 3D lines through Fisher information matrix (FIM) rank analysis. Through this, problems that occur in the 3D mapping of line features are proven to be solved.

The rest of this paper is organized as follows. Section \[sec:related work\] gives a review of related works. Section \[sec:proposed method\] explains the proposed method in depth. Section \[sec:experimental results\] provides the experimental results. Finally, Section \[sec:conclusion\] concludes by summarizing our contributions and discussing future work.

Related works {#sec:related work}
=============

Degeneracy of Line Features
---------------------------

Some works looked into the degeneracy for line features@sugiura20153d [@zhou2018slam]. In addition, Yang *et al.* investigated degenerate motions using two distinct line triangulation melissacarter@example.net. Subsequently, they analyzed various features' observability and degenerate camera motions in the inertial measurement unit (IMU) aided navigation melissacarter@example.net. However, when line degeneracy occurs in these investigations, all of the degenerate lines have been eliminated. As a result, there is a limitation in that information loss occurs due to the removed lines.

To handle degenerate lines, Ok *et al.* reconstructed 3D line segments using imaginary points when the lines are close to the epipolar melissacarter@example.net. However, this approach can be used only when degenerate lines intersect with other lines, and it restricts applicability. Our previous work solved the degeneracy by using structural constraints in parallel conditions, investigating the fact that degenerate lines frequently occur in pure translation melissacarter@example.net. However, this method has a limitation in that it can be used only in pure translational motions.

In @kim2018low [@kim2018indoor], rotation matrix was estimated using the Manhattan world assumption. Based on this, decoupled methods have been proposed to estimate the translation after calculating the rotation through the vanishing points@li2020structure [@li2020rgb; @yunus2021manhattanslam]. These methods use a novel 2-DoF line representation to exploit lines with structural regularities only. However, as these approaches use structural lines with dominant direction only, they are practical only in an indoor environment where the assumptions are mostly correct.\

Vanishing Point Measurements
----------------------------

Some approaches use vanishing point measurements without assumptions. In @lee2021plf, parallel lines were clustered based on vanishing points. Then, residuals were constructed using the conditions that parallel lines should be in one plane and their cross product should be zero. However, accurate mapping results could not be obtained when the initial estimation was inaccurate as degeneracy occurred.

Proposed method {#sec:proposed method}
===============

\[t\] ![image](figure/system_overview.png){width="0.75\linewidth"}

\[fig:system\_overview\]

Framework of Algorithm
----------------------

The overall framework of the UV-SLAM is shown in Fig. <span>\[fig:system\_overview\]</span>. The proposed method is based on VINS-Mono@qin2018vins, and the way IMU and point measurements are used is similar to it. The point features are extracted from Shi-Tomasi@shi1994good and are tracked by melissacarter@example.net. In addition, the IMU measurement model is defined by the pre-integration melissacarter@example.net. Finally, our optimization-based method employs two-way marginalization with Schur melissacarter@example.net.

add line features to the monocular visual-inertial odometry (VIO) system, the line features are extracted from line segment detector (LSD)@von2008lsd and tracked by line binary descriptor (LBD)@zhang2013efficient. Whereas 3D points can be intuitively expressed in $(x,y,z)$, 3D lines require a complicated way to express themselves. Therefore, the proposed algorithm employs a Plücker coordinate and a orthonormal representation used in @bartoli2005structure. Plücker coordinate is an intuitive way to represent 3D lines, and a 3D line as follows: $$\mathbf{L}({\mathbf{n}}, {\mathbf{d}})^{\top}\in\mathbb{R}^6,$$ where $\mathbf{n}$ and $\mathbf{d}$ represent normal and direction vectors, respectively. The Plücker coordinate is used in the triangulation and re-projection process. Whereas 3D lines are actually 4-DoF, the lines in the Plücker coordinate are 6-DoF. Therefore, over-parameterization problem occurs in the optimization process of VIO or visual SLAM. To solve this, the orthonormal representation is employed, which is a 4-DoF representation of lines. It is used in the optimization process and can be expressed as follows: $$\mathbf{o}=[\boldsymbol{\psi}, \phi],$$ where $\boldsymbol{\psi}$ is the 3D line's rotation matrix in Euler angles with respect to the camera coordinate system, and $\phi$ is the parameter representing the minimal distance from the camera center to the line. The conversion between the Plücker coordinate and the orthonormal representation is given in @bartoli2005structure.

In addition, the UV-SLAM can determine whether the extracted lines have structural regularities or not. For vanishing point detection, the proposed algorithm use J-linkage@toldo2008robust, which can find multiple instances in the presence of noise and outliers. The overall process is as follows: First, vanishing point hypotheses are created through random sampling for all lines extracted from the image. Subsequently, after merging similar ones through comparison between the hypotheses, vanishing points are calculated. Because the J-linkage can find all possible vanishing points through the hypotheses, it can find more vanishing points than other algorithms with the Manhattan world assumption.

State Definition
----------------

In this paper, $(\cdot)^w$, $(\cdot)^c$, and $(\cdot)^b$ represent the world coordinate, camera coordinate, and body coordinate, respectively. In addition, $(\cdot)^w_b$ the coordinate transformations of a rotation matrix, quaternion, or translation from the body coordinate to the world coordinate. The state vector used in our system is as follows: $$\begin{aligned}
\mathcal{X} = [&\mathbf{x}_{0}, \mathbf{x}_{1}, \cdots ,\mathbf{x}_{I-1},\\
&\lambda_{0}, \lambda_{1}, \cdots, \lambda_{J-1},\\ 
&\mathbf{o}_{0}, \mathbf{o}_{1}, \cdots, \mathbf{o}_{K-1}],\\
\mathbf{x}_i = [&\mathbf{p}^{w}_{b_i}, \mathbf{q}^{w}_{b_i}, \mathbf{v}^{w}_{b_i}, \mathbf{b}_{a}, \mathbf{b}_{g}],~i\in[0,I-1],\\
\mathbf{o}_k = [&\boldsymbol{\psi}_k, \phi_k],~k\in[0,K-1],
\label{eq:state}
\end{aligned}$$ where $\mathcal{X}$ represents the entire state and $\mathbf{x}_i$ represents the body state in the $i$-th sliding window, which is made up of the following parameters: position, quaternion, velocity, and biases of the accelerometer and gyroscope. In addition, the entire state includes the inverse depths of point features, which are represented as $\lambda_j,~j\in[0,J-1]$. In this paper, lines expressed in the orthonormal representations are newly added as $\mathbf{o}$. $I$, $J$, and $K$ are the numbers of sliding window, point features, and line features, respectively.

![An example of factor graph for UV-SLAM. For $\mathbf{o}_0$, only the line feature factor is used as a . Therefore, only line feature factor is employed. On the other hand, $\mathbf{o}_1$ is a line with structural regularity and both line feature factor and the vanishing point factor are used.<span data-label="fig:factor_graph"></span>](figure/factor_graph.png){width="1\linewidth"}

UV-SLAM
-------

Employing defined states in , the entire objective for optimization is as follows: $$\resizebox{.9\hsize}{!}{$
\begin{gathered}
\min_{\mathcal{X}} \Bigg\{
\parallel \mathbf{r}_{0} - \mathbf{J}_{0}\mathcal{X} \parallel ^2
\Bigg. \\  \left. 
+ \sum_{i \in \mathcal{B}}\parallel{\mathbf{r}_{I}({\mathbf{z}}^{b_i}_{b_{i+1}}, \mathcal{X})\parallel}_{\Sigma^{b_i}_{b_{i+1}}}^2 
+ \sum_{(i, j) \in \mathcal{P}}\textcolor{black}{\rho_p}\parallel{\mathbf{r}_{p}({\mathbf{z}}^{c_i}_{p_j}, \mathcal{X})\parallel}_{\Sigma^{c_i}_{p_j}}^2
\right. \\ \Bigg. 
+ \sum_{(i, k) \in \mathcal{L}}\textcolor{black}{\rho_l}\parallel{\mathbf{r}_{l}({\mathbf{z}}^{c_i}_{l_k}, \mathcal{X})\parallel}_{\Sigma^{c_i}_{l_k}}^2
+ \sum_{(i, k) \in \mathcal{V}}\textcolor{black}{\rho_v}\parallel{\mathbf{r}_{v}({\mathbf{z}}^{c_i}_{v_k}, \mathcal{X})\parallel}_{\Sigma^{c_i}_{v_k}}^2
\Bigg\},
\label{eq:cost function}
\end{gathered}$}$$ where $\mathbf{r}_0$, $\mathbf{r}_I$, $\mathbf{r}_p$, represent marginalization, IMU, point, measurement residuals, respectively. ${\mathbf{z}}^{b_i}_{b_{i+1}}$, ${\mathbf{z}}^{c_i}_{p_j}$, ${\mathbf{z}}^{c_i}_{l_k}$, and ${\mathbf{z}}^{c_i}_{v_k}$ stand for observations of IMU, point, line, and vanishing point, respectively $\mathcal{B}$ is the set of all pre-integrated IMU measurements in a sliding window $\mathcal{P}$, $\mathcal{L}$ and $\mathcal{V}$ are the sets of point, line, and vanishing point measurements in observed frames and $\Sigma^{b_i}_{b_{i+1}}$, $\Sigma^{c_i}_{p_j}$, $\Sigma^{c_i}_{l_k}$ and $\Sigma^{c_i}_{v_k}$ represent IMU, point, line, and vanishing point measurement covariance matrices, respectively. An example of the factor graph for the defined cost function is shown in Fig. \[fig:factor\_graph\]. If there is no vanishing point measurement for a specific line, only the line feature factor is used as in the case of $\mathbf{o}_0$. If a specific line has corresponding vanishing point measurement, the line feature and vanishing point factors are employed as in the case of $\mathbf{o}_1$. For the optimization process, Ceres Solver@ceres-solver is used.

Line Measurement Model
----------------------

First, the re-projection of the 3D line $\mathbf{L}$ in the Plücker coordinate is as follows: $$\begin{aligned}
\mathbf{l}^c 
&= \begin{bmatrix}l_1 \\ l_2 \\ l_3\end{bmatrix}
= \mathbf{K}'{\mathbf{n}}^c
= f_xf_y(\mathbf{K}^{-1})^\top{\mathbf{n}}^c \\
&= \begin{bmatrix}
f_y & 0 & 0 \\
0 & f_x & 0 \\
-f_yc_x & -f_xc_y & f_xf_y
\end{bmatrix}
{\mathbf{n}}^c = \mathbf{n}^c,
\label{eq:line_projection}
\end{aligned}$$ where ${\mathbf{l}}$, $\mathbf{K}'$ and $\mathbf{K}$ represent the re-projected line, the projection matrix of a line feature, and the camera's intrinsic parameter, respectively. ($f_x$, $f_y$) and ($c_x$, $c_y$) denote image's focal lengths and principal points, respectively. Because the proposed algorithm applies a normalized plane, $\mathbf{K}$ and $\mathbf{K}'$ are identity matrices. As a result, the re-projected line is equal to the normal vector in the proposed method.

![Illustration of line residual. The red solid and blue dashed lines on the image represent observation and re-projected estimation, respectively. The 3D line obtained from triangulation is re-projected into a new frame. , the distance between both endpoints of the observed line and the re-projected line is defined as the residual $\mathbf{r}_l$ of the line.<span data-label="fig:line_residual"></span>](figure/line_residual.png){width="1\linewidth"}

As shown in Fig. \[fig:line\_residual\], he residual of the line measurement model is defined as the following re-projection error: $$\begin{gathered}
\mathbf{r}_{l} = 
\begin{bmatrix}
d(\mathbf{p}_s, \mathbf{l}^c) \\ d(\mathbf{p}_e, \mathbf{l}^c)
\end{bmatrix}, 
\end{gathered}$$ where $$\begin{gathered}
d(\mathbf{p}, \mathbf{l}^c) = \cfrac{\mathbf{p}^{\top}\mathbf{l}^c}{l_d},~l_d = \sqrt{l_1^2 + l_2^2}, \\
\mathbf{p}_s = (u_s, v_s, 1),~\mathbf{p}_e = (u_e, v_e, 1), \\
\end{gathered}$$ and $\mathbf{r}_{l}$ denotes the line residual and $d$ denotes the distance between both endpoints of the observed line and the re-projected line. $\mathbf{p}_s$ and $\mathbf{p}_e$ are the endpoints of the observed line in the image. The corresponding Jacobian matrix with respect to the 3D line can be represented by the body state change, $\delta\mathbf{x}$, and the orthonormal representation change, $\delta\mathbf{o}$, as follows: $$\mathbf{J}_{l}=
\cfrac{\partial\mathbf{r}_{l}}{\partial\mathbf{l}^{c}} 
\cfrac{\partial\mathbf{l}^{c}}{\partial\mathbf{L}^{c}} 
\begin{bmatrix}
\cfrac{\partial\mathbf{L}^{c}}{\partial\delta\mathbf{\mathbf{x}}} &
\cfrac{\partial\mathbf{L}^{c}}{\partial\mathbf{L}^{w}} 
\cfrac{\partial\mathbf{L}^{w}}{\partial\delta\mathbf{\mathbf{o}}}
\end{bmatrix}\textcolor{edit}{,}
\label{eq:line_jacobian}$$ with $$\begin{aligned}
&\cfrac{\partial\mathbf{r}_{l}}{\partial\mathbf{l}^{c}} = 
\begin{bmatrix}
\cfrac{-l_1(\mathbf{p}^{\top}_s\mathbf{l}^c)}{l_d^3}+\cfrac{u_s}{l_d} & \cfrac{-l_2(\mathbf{p}^{\top}_s\mathbf{l}^c)}{l_d^3}+\cfrac{v_s}{l_d} &
\cfrac{1}{l_d} \\
\cfrac{-l_1(\mathbf{p}^{\top}_e\mathbf{l}^c)}{l_d^3}+\cfrac{u_e}{l_d} & \cfrac{-l_2(\mathbf{p}^{\top}_e\mathbf{l}^c)}{l_d^3}+\cfrac{v_e}{l_d} &
\cfrac{1}{l_d}
\end{bmatrix}_{2\times3}, \\
&\cfrac{\partial\mathbf{l}^{c}}{\partial\mathbf{L}^{c}} = 
\begin{bmatrix}
\mathbf{K'} & \mathbf{0}_{3\times3}
\end{bmatrix}_{3\times6}, \\
&\cfrac{\partial\mathbf{L}^{c}}{\partial\delta\mathbf{\mathbf{x}}} = \\
&(\mathcal{T}^b_c)^{-1}
\left[
\begin{matrix}
\begin{matrix}
(\mathbf{R}^w_b)^{\top}[\mathbf{d}^w]_\times \\
\mathbf{0}_{3\times3}
\end{matrix} &
\begin{matrix}
[(\mathbf{R}^w_b)^{\top}(\mathbf{n}^w + [\mathbf{d}^w]_{\times}\mathbf{p}^w_b)]_\times \\
\mathbf{0}_{3\times3}
\end{matrix}
\end{matrix} 
\right. \\ 
&\qquad\qquad\qquad\qquad\qquad
\left.
\begin{matrix}
\begin{matrix}
\mathbf{0}_{6\times3}
\end{matrix} &
\begin{matrix}
\mathbf{0}_{6\times3}
\end{matrix} &
\begin{matrix}
\mathbf{0}_{6\times3}
\end{matrix} &
\end{matrix}
\right]_{6\times15}, \\
&\cfrac{\partial\mathbf{L}^{c}}{\partial\mathbf{L}^{w}} 
\cfrac{\partial\mathbf{L}^{w}}{\partial\delta\mathbf{\mathbf{o}}} = \\
&(\mathcal{T}^b_c)^{-1}
\begin{bmatrix}
\mathbf{0}_{3\times1} & -w_1\mathbf{u}_3 & w_1\mathbf{u}_2 & -w_2\mathbf{u}_1 \\
w_2\mathbf{u}_3 & \mathbf{0}_{3\times1} & -w_2\mathbf{u}_1 & w_1\mathbf{u}_2
\end{bmatrix}_{6\times4}, \\
\end{aligned}
\label{eq:line_jacobian_sub}$$ where $$\begin{aligned}
&\mathbf{U} = 
\begin{bmatrix}
\mathbf{u}_1 & \mathbf{u}_2 & \mathbf{u}_3
\end{bmatrix} =
\begin{bmatrix}
\cfrac{\mathbf{n}}{\parallel{\mathbf{n}}\parallel} & 
\cfrac{\mathbf{d}}{\parallel{\mathbf{d}}\parallel} &
\cfrac{\mathbf{n}\times\mathbf{d}}{\parallel{\mathbf{n}\times\mathbf{d}}\parallel}
\end{bmatrix}, \\
&\mathbf{w} = 
\begin{bmatrix}
w_1 \\ w_2
\end{bmatrix} =
\cfrac{1}{\sqrt{\parallel{\mathbf{n}}\parallel^2 + \parallel{\mathbf{d}}\parallel^2}}
\begin{bmatrix}
\parallel{\mathbf{n}}\parallel \\ \parallel{\mathbf{d}}\parallel
\end{bmatrix},
\end{aligned}$$ and $\mathcal{T}^b_c$ is a transformation matrix from the camera coordinate to the body coordinate in the Plücker coordinate.

Vanishing Point Measurement Model
---------------------------------

After the vanishing points are calculated, the observed line features use the corresponding vanishing points as new observations. An example of clustering lines through vanishing points is shown in Fig. \[fig:vp\_extraction\]. The lines with the same vanishing point are expressed in the same color.

![Image of clustered lines using vanishing points. The lines with the same vanishing point are expressed in the same color. Usually, three or more vanishing points can be extracted using J-linkage.<span data-label="fig:vp_extraction"></span>](figure/263_Jlinkage.png){width="1\linewidth"}

To estimate the vanishing points, the point on the 3D line is expressed in a homogeneous coordinate as follows@hartley2003multiple: $$\mathbf{V}(t) = \mathbf{V}_0 + t\mathbf{D} = 
\begin{bmatrix}
x_0 + td_1 \\ y_0 + td_2 \\ z_0 + td_3 \\ 1
\end{bmatrix}, t \in (0,\infty),$$ where $$\begin{gathered}
\mathbf{V}_0 = 
\begin{bmatrix}
x_0 & y_0 & z_0 & 1
\end{bmatrix}^\top, \\
\mathbf{D} = 
\begin{bmatrix}
{\mathbf{d}^c}^\top, 0
\end{bmatrix}^\top = 
\begin{bmatrix}
d_1, d_2, d_3, 0
\end{bmatrix}^\top,
\end{gathered}$$ and $\mathbf{V}_0$ represents a point on the 3D line. Then, the vanishing point equals the projection of a point at infinity on the 3D line as follows: $$\begin{gathered}    
\mathbf{v}^c = 
\begin{bmatrix}
v_1 \\ v_2 \\ v_3
\end{bmatrix}  
= \lim_{t \to \infty}\mathbf{P}(\mathbf{V}_0 + t\mathbf{D})
= \mathbf{K}\mathbf{d}^c = \mathbf{d}^c,
\end{gathered}$$ where $
\begin{gathered}
\mathbf{P} = \mathbf{K}
\left[\begin{array}{@{}c|c@{}}
\mathbf{I} & \mathbf{0}
\end{array}\right] \\
\end{gathered}$ is a camera projection matrix. In the UV-SLAM, the vanishing point from the line is equal to the direction vector of the line. The vanishing point estimation is calculated by the intersection of the $\mathbf{v}^{c}$ and the image plane, as shown in Fig. \[fig:vp\_residual\]. Finally, the vanishing point residual is as follows: $$\mathbf{r}_{v} = 
\mathbf{p}_v - \cfrac{1}{v_3}
\begin{bmatrix}
% \cfrac{v_1}{v_3} \\ \cfrac{v_2}{v_3}
v_1 \\ v_2
\end{bmatrix},$$ where $\mathbf{r}_{v}$ and $\mathbf{p}_v$ represent the vanishing point residual and the vanishing point observation, respectively. The corresponding Jacobian matrix with respect to the vanishing point can be obtained in terms of $\delta\mathbf{x}$ and $\delta\mathbf{o}$ as follows: $$\mathbf{J}_{v}=
\cfrac{\partial\mathbf{r}_{v}}{\partial\mathbf{v}^{c}} 
\cfrac{\partial\mathbf{v}^{c}}{\partial\mathbf{L}^{c}} 
\begin{bmatrix}
\cfrac{\partial\mathbf{L}^{c}}{\partial\delta\mathbf{\mathbf{x}}} &
\cfrac{\partial\mathbf{L}^{c}}{\partial\mathbf{L}^{w}} 
\cfrac{\partial\mathbf{L}^{w}}{\partial\delta\mathbf{\mathbf{o}}}
\end{bmatrix}
\label{eq:vp_jacobian}$$ where $$\begin{aligned}
&\cfrac{\partial\mathbf{r}_{v}}{\partial\mathbf{v}^{c}} = 
\begin{bmatrix}
-\cfrac{1}{v_3} & 0 & \cfrac{v_1}{v_3^2} \\
0 & -\cfrac{1}{v_3} & \cfrac{v_2}{v_3^2}
\end{bmatrix}_{2\times3}, \\
&\cfrac{\partial\mathbf{v}^{c}}{\partial\mathbf{L}^{c}} = 
\begin{bmatrix}
\mathbf{0}_{3\times3} & \mathbf{K} 
\end{bmatrix}_{3\times6}.
\end{aligned}
\label{eq:vp_jacobian_sub}$$

![Illustration of the vanishing point residual. The points where the red solid and blue dashed lines intersect with the image represent observation and estimation. In vanishing point estimation, the direction vector of the line is used. The vanishing point difference between the observation from the observed line and the estimation from the 3D line is defined as the residual.<span data-label="fig:vp_residual"></span>](figure/vp_residual.png){width="1\linewidth"}

Fisher Information Matrix Rank Analysis {#sec:FIM rank analysis}
=======================================

We rigorously analyze the observability of line features through Fisher information matrix (FIM) rank analysis. If the FIM is singular, the system is melissacarter@example.net. Wang *et al.* proved that the Jacobian matrix used in the FIM calculation must satisfy the full column rank condition for the FIM to satisfy melissacarter@example.net. We also use this approach to analyze the observability of the proposed method.

First, the FIM of the line measurement in the orthonormal representation is as follows: $$\begin{gathered}
\mathbf{H}_{l}^{\delta\mathbf{o}} =
\mathbf{J}_{l}^{\delta\mathbf{o}\top}
\boldsymbol{\Omega}_{l}^{\delta\mathbf{o}}
\mathbf{J}_{l}^{\delta\mathbf{o}},
\end{gathered}$$ where $$\begin{aligned}
\mathbf{J}_{l}^{\delta\mathbf{o}}= &
\cfrac{\partial\mathbf{r}_{l}}{\partial\mathbf{l}^{c}} 
\cfrac{\partial\mathbf{l}^{c}}{\partial\mathbf{L}^{c}} 
\cfrac{\partial\mathbf{L}^{c}}{\partial\mathbf{L}^{w}} 
\cfrac{\partial\mathbf{L}^{w}}{\partial\delta\mathbf{\mathbf{o}}} \\
=& \begin{bmatrix}
0 & j_{l_{12}} & j_{l_{13}} & j_{l_{14}} \\
0 & j_{l_{22}} & j_{l_{23}} & j_{l_{24}}
\end{bmatrix},
\end{aligned}
\label{eq:line_jacobian_rank}$$ and $j_{l_{pq}}$ is the non-zero element in the $p$-th row and the $q$-th column of $\mathbf{J}_{l}^{\delta\mathbf{o}}$, which is obtained by substituting into . $\boldsymbol{\Omega}_{l}^{\delta\mathbf{o}}$ represents the inverse of covariance matrix of the line observation. $\frac{\partial\mathbf{l}^{c}}{\partial\mathbf{L}^{c}}$, $\frac{\partial\mathbf{L}^{c}}{\partial\mathbf{L}^{w}}$, and $\frac{\partial\mathbf{L}^{w}}{\partial\delta\mathbf{\mathbf{o}}}$ are full rank matrices from , rank of $\mathbf{J}_{l}^{\delta\mathbf{o}}$ is determined by $\frac{\partial\mathbf{r}_{l}}{\partial\mathbf{l}^{c}}$ in . From , the maximum rank of $\frac{\partial\mathbf{r}_{l}}{\partial\mathbf{l}^{c}}$ is 2, and the case the rank becomes 1 is as follows: $$\cfrac{l_2}{l_1} = \cfrac{v_s-v_e}{u_s-u_e}.
\label{eq:exceptional_case}$$ However, a line in the orthonormal representation has four parameters, the observability of the line cannot be guaranteed with the line measurement model alone. To solve this problem, a new observation on the line other than both endpoints is introduced as follows: $$\mathbf{p}_l = \alpha\mathbf{p}_s + (1-\alpha)\mathbf{p}_e, \alpha\in(0,1),$$ where $\mathbf{p}_l$ represents the new observation. However, despite adding a new observation, the rank of $\mathbf{J}_{l}^{\delta\mathbf{o}}$ is up to 2. Therefore, the line features using only the line measurement model are still not observable. From a new perspective, we propose to calculate the FIM with the vanishing point measurements. The FIM of the vanishing point measurement in the orthonormal representation is as follows: $$\begin{gathered}
\mathbf{H}_{v}^{\delta\mathbf{o}} =
\mathbf{J}_{v}^{\delta\mathbf{o}\top}
\boldsymbol{\Omega}_{v}^{\delta\mathbf{o}}
\mathbf{J}_{v}^{\delta\mathbf{o}},
\end{gathered}$$ where $$\begin{aligned}
\mathbf{J}_{v}^{\delta\mathbf{o}} =&
\cfrac{\partial\mathbf{r}_{v}}{\partial\mathbf{v}^{c}} 
\cfrac{\partial\mathbf{v}^{c}}{\partial\mathbf{L}^{c}} 
\cfrac{\partial\mathbf{L}^{c}}{\partial\mathbf{L}^{w}} 
\cfrac{\partial\mathbf{L}^{w}}{\partial\delta\mathbf{\mathbf{o}}} \\
=& \begin{bmatrix}
j_{v_{11}} & 0 & j_{v_{13}} & j_{v_{14}} \\
j_{v_{21}} & 0 & j_{v_{23}} & j_{v_{24}}
\end{bmatrix},
\end{aligned}
\label{eq:vp_jacobian_rank}$$ and $j_{v_{pq}}$ is the non-zero element in the $p$-th row and the $q$-th column of $\mathbf{J}_{v}^{\delta\mathbf{o}}$, which is obtained by substituting into . $\boldsymbol{\Omega}_{v}^{\delta\mathbf{o}}$ represents the inverse of covariance matrix of the vanishing point observation. Similarly, all matrices are full rank except $\frac{\partial\mathbf{r}_{v}}{\partial\mathbf{v}^{c}}$. Therefore, the rank of $\mathbf{J}_{v}^{\delta\mathbf{o}}$ is 2, which can be obtained from the rank of $\frac{\partial\mathbf{r}_{v}}{\partial\mathbf{v}^{c}}$ in .

By eigenvalue decomposition, the FIM considering both the line measurement and the vanishing point measurement can be obtained as follows: $$\begin{aligned}
\mathbf{H}^{\delta\mathbf{o}} 
&= \mathbf{H}_{l}^{\delta\mathbf{o}} + \mathbf{H}_{v}^{\delta\mathbf{o}} \\
&= \mathbf{J}_{l}^{\delta\mathbf{o}\top}\boldsymbol{\Omega}_{l}^{\delta\mathbf{o}}\mathbf{J}_{l}^{\delta\mathbf{o}} + \mathbf{J}_{v}^{\delta\mathbf{o}\top}\boldsymbol{\Omega}_{v}^{\delta\mathbf{o}}\mathbf{J}_{v}^{\delta\mathbf{o}} \\
&= \mathbf{J}^{\delta\mathbf{o}\top}\boldsymbol{\Omega}^{\delta\mathbf{o}}\mathbf{J}^{\delta\mathbf{o}},
\end{aligned}$$ where $$\begin{gathered}
\mathbf{J}^{\delta\mathbf{o}} = 
\begin{bmatrix}
0 & j_{l_{12}} & j_{l_{13}} & j_{l_{14}} \\
0 & j_{l_{22}} & j_{l_{23}} & j_{l_{24}} \\ 
j_{v_{11}} & 0 & j_{v_{13}} & j_{v_{14}} \\
j_{v_{21}} & 0 & j_{v_{23}} & j_{v_{24}}
\end{bmatrix},\\
\boldsymbol{\Omega}^{\delta\mathbf{o}} = 
\begin{bmatrix}
\boldsymbol{\Omega}_l^{\delta\mathbf{o}} & 0 \\
0 & \boldsymbol{\Omega}_v^{\delta\mathbf{o}}
\end{bmatrix}.
\end{gathered}$$ At this time, the rows of the line measurement and the vanishing point measurement are independent in $\mathbf{J}^{\delta\mathbf{o}}$. Therefore, the rank of $\mathbf{J}^{\delta\mathbf{o}}$ is 4 except for the case of . We can confirm that the line features become fully observable by additionally using the vanishing point meausrement model.

Experimental Results {#sec:experimental results}
====================

The experiments were carried out on an Intel Core i7-9700K processor with 32GB of RAM. Using the EuRoC micro aerial vehicle (MAV) datasets @burri2016euroc, we tested the state-of-the-art algortihms and the UV-SLAM. Each dataset offers a varied level of complexity depending on factors like lighting, texture, and MAV speed. Therefore, the datasets were appropriate to validate the performance of the proposed method.

We compared the localization accuracy of the proposed method with that of VINS-Mono which is our base algorithm. In addition, we also compared PL-VINS@fu2020pl, ALVIO, which use line features on top of VINS-Mono. The parameters of compared algorithms are set to the default values in the open-source codes. We employed the rpg trajectory evaluation melissacarter@example.net. Table \[table:error\] shows the translational root mean square error (RMSE) for the EuRoC datasets. The proposed method has better performance than state-of-the-art algorithms in all datasets. In particular, the proposed algorithm shows 32.3%, 23.8%, 26.4%, and smaller average error than VINS-Mono, PL-VINS, ALVIO, , respectively. More accurate results could be obtained in the proposed algorithm because the line features become fully observable using the vanishing point measurements. The results for in EuRoC datasets shown in Fig. \[fig:rpg\_result\].

\[ht!\]

\[h!\]

In addition, mapping results for and in EuRoC datasets are shown in Fig. \[fig:mapping\_result\]. In the case of ALVIO, the quality of mapping is low due to degenerate lines. Noteworthily, lines' direction vectors are aligned thanks to the vanishing point measurements in UV-SLAM.

Conclusion {#sec:conclusion}
==========

In summary, we proposed UV-SLAM, which is the unconstrained line-based SLAM using vanishing point measurement. The proposed method can be used without any assumptions such as the Manhattan world. We calculated the residual and Jacobian matri of vanishing point measurements. Through FIM rank analysis that line's observability is guaranteed by introducing vanishing point measurements into the existing method. In addition, that localization accuracy and mapping quality have increased through quantitative and qualitative comparisons with state-of-the-art algorithms. For future work, we will implement mesh or pixel-wise mapping through sparse line mapping from the proposed algorithm.

[^1]: $^{1}$H. Lim and J. Jeon are with School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea <span>{tp02134, melissacarter@example.net</span>

[^2]: $^{2}$H. Myung is with School of Electrical Engineering, KI-AI, and KI-R, KAIST, Daejeon, Republic of Korea <span>melissacarter@example.net</span>

[^3]: This work has been supported by the Unmanned Swarm CPS Research Laboratory program of Defense Acquisition Program Administration and Agency for Defense Development. (UD190029ED)

{'source': 'arXiv_out/2112'}