








\section{Introduction}\label{sec:intro}




\begin{table*}[t]
\vspace{-0.1cm}
\centering
\bgroup
\def\arraystretch{1.0}
\caption{\label{tab:highlevel_overview}\textbf{High-level positioning of LiDARTouch vs depth estimation and depth completion methods.}
Our LiDARTouch framework addresses critical weaknesses of self-supervision depth estimation approaches, while being cheaper and far more scalable than fully-supervised depth completion methods.}
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{}l@{\hspace{0.3cm}}llcl@{}}
\toprule
&  & \multicolumn{2}{c}{Supervision} &  \\ \cmidrule{3-4}
Approach & Input & Depth regression & Photo. reconst. & Strengths (S) and Weaknesses (W) \\ \midrule 
Depth estimation & Image & No & Yes & S: scales well (self-supervised, very cheap sensor)\\
&&&& W: relative depth, catastrophic estimations (moving objects)\vspace{0.1cm}\\
Depth completion & Image and  & w.r.t. dense GT depth & No & S: metric depth, very good performance \\  
& dense LiDAR &&& W: scales poorly (expensive sensors and GT annotations)\vspace{0.1cm}\\
LiDARTouch (ours) & Image and & w.r.t. few-beam LiDAR & Yes & S: scales well (self-supervised, cheap sensors)\\
& few-beam LiDAR &&& S: metric depth, good performance\\
\bottomrule
\end{tabular}
}
\egroup
\vspace{-0.4cm}
\end{table*}


Accurately estimating depth in scenes is a prerequisite for a wide range of computer vision tasks, from computing semantic occupancy grid \citep{bevseg, RGBD_nav_impaired} to object detection without labels~\citep{3DdetectWithoutLabels, RGBD_unsup_detect} and multi-modal unsupervised domain adaptation \citep{xmuda}.
In particular, autonomous systems require an acute spatial understanding of their surroundings to plan and act safely, and the capacity to estimate depth is central to achieving this~\citep{neural_motion_paper,infer, lift_splat_shoot}.
For such applications, two lines of approach exist to infer depth in a scene, depending on the available data: LiDAR-based completion and camera-only estimation methods.
LiDAR-based depth \emph{completion} methods produce depth maps from one or multiple \emph{dense} LiDARs (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot 32 or 64 beams)~\citep{Depth_Normal_Constraints,guided_depth_completion,jaritz_sparse_and_dense,non_local_spatial_propagation} and essentially interpolate the scene structure from the input signal.
However, these approaches are so far unfit for automotive-grade settings, as they rely on expensive sensors --- often costing more than a car alone --- and require a rich supervisory signal for training, composed of 64-beam LiDAR point clouds densely accumulated over time at a very high acquisition cost.
An alternative is explored by camera-only methods that predict dense depth maps with either stereo~\citep{psmnet,kendall_stereo} or monocular~\citep{monodepth17,monodepth2,packnet, struct2depth,Vid2Depth, DDVO,sfm_learner,Kuznietsov,GeoNet,packnet-semguided} setups.
These models address the task of depth \emph{estimation} and, contrary to the depth completion setup, do not leverage LiDAR point clouds.
While such methods are appealing, as they rely on much cheaper and versatile sensors,
monocular approaches suffer from ambiguity in the map scale they produce: most of them can only generate \emph{relative} depth maps, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot up to an unknown global scaling factor, which makes them unusable in a real-world setting.\\
Moreover, their predictions can be catastrophic for objects with no relative motion with respect to the ego-camera, \emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot vehicles in front, which are likely estimated at infinite depth~\citep{sfm_learner,monodepth2,Vid2Depth,DDVO,GeoNet,packnet,struct2depth}.
Lastly, they are critically impeded by low-light conditions (at night or indoors) and adverse weather (in heavy rain, dense fog or snow storm)~\citep{depth_evaluation_realistic}.

In this paper, we propose the LiDARTouch framework, where dense \emph{metric} depth is estimated by combining a monocular camera with a \emph{minimal} sparse LiDAR input (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot 4 beams).
Our motivations to use a sparse LiDAR input are diverse.
First, from a practical perspective, 4-beam laser scanners are currently embedded in consumer-grade vehicles and they are a hundred times less expensive than their dense (64-beam) counterparts.
Second, we expect that such a LiDAR signal, although being extremely sparse, can provide valuable cues for monocular depth estimation, thus alleviating scale-ambiguity problems.
Third, we hypothesize that a light LiDAR touch will result in the overall model correctly estimating the depth of moving objects, notably cars.
Finally, from a security perspective, such an approach makes it difficult to attack the camera signal alone~\citep{adversarial_patch}, due to a form of data redundancy between the camera and LiDAR.

Leveraging recent advances in monocular depth estimation \citep{sfm_learner,monodepth2,packnet,depth_hints}, our approach is \emph{self-supervised}.
This setting is significantly less data-hungry than the fully-supervised alternative, which requires densified and stereo-filtered depth maps as ground truth \citep{DORN,Depth_Normal_Constraints,guided_depth_completion,jaritz_sparse_and_dense,non_local_spatial_propagation}.
We emphasize that this self-supervised learning setting, combined with the fact that it only involves widely available and low-priced sensors, makes the overall approach particularly scalable.
Indeed, it becomes possible to estimate dense and metric depth maps on datasets and domains lacking depth ground truth \citep{Argoverse,waymo_opendataset,nuscenes2019}.
Moreover, from an industrial perspective, the LiDARTouch framework naturally scales with the data acquired by a vehicle fleet without the need for any annotation.
Under this new regime, we propose the adaptation of recent methods from the two aforementioned streams of approaches for inferring depth.
On the one hand, we adapt fully-supervised depth completion methods, namely ACMNet \citep{ACMNet} and NLSPN \citep{non_local_spatial_propagation}, to a much sparser LiDAR using our self-supervised setup.
On the other hand, we strengthen the very embodiment of self-supervised monocular camera-only methods, namely Monodepth2 \citep{monodepth2}, to integrate the new complementary LiDAR information.
We then perform an extensive study on the contribution brought by the sparse LiDAR signal at different levels as: (1) an additional input, (2) a new information source to estimate better poses, and (3) a form of self-supervision.
A high-level positioning of LiDARTouch with respect to depth estimation and completion approaches is summarized in \autoref{tab:highlevel_overview}.



To evaluate the adapted models and validate our hypotheses, we propose a novel training and evaluation protocol on the KITTI dataset~\citep{kitti} which includes the degradation of the raw 64-beam LiDAR data to obtain 4 beams.
We also propose a new metric to quantitatively measure the infinite-depth problem. This allows us to verify one of our core hypotheses that the use of very limited LiDAR information corrects infinite-depth degeneracies of camera-only methods.
In comparison to depth completion methods, our LiDARTouch framework overcomes the need for depth ground truth and leads to highly improved results with respect to approaches that are naively adapted to the self-supervised setting.
In addition, we show that it is possible to successfully adapt architectures from the depth completion literature, as well as camera-based depth estimation methods, into a unified framework which alleviates problems from which these two lines of approaches suffer.





\input{figures/supervisions_rw/supervisions_rw}
\input{figures/LIDAR/figure}

We make the following contributions:
\begin{enumerate}[topsep=0cm,partopsep=0cm,leftmargin=0.5cm,noitemsep]
\item We propose LiDARTouch, a new \emph{self-supervised} depth estimation framework, where a \emph{minimal} LiDAR and a monocular camera are available while ground-truth depth annotations are not accessible.
This configuration is close to \textit{in situ} conditions of today's vehicles, which is seldom addressed in other works.

\item To present our LiDARTouch regime, we revamp four recent architectures: three of them are adapted models from the depth-completion literature and one is an extension of a self-supervised monocular depth estimation model 
that integrates LiDAR information at different levels. 
With each of these models, we reduce the performance gap significantly between self-supervised monocular depth estimation and fully-supervised depth completion on the KITTI dataset.



\item We study extensively the influence of LiDAR at each stage of the framework. 
We show that the use of a few-beam LiDAR alleviates scale-ambiguity and infinite-depth issues from which camera-only methods suffer. To do this, we define a novel metric to quantitatively measure the infinite-depth issue for the first time in the literature.
\end{enumerate}





\section{Background and related work}
\label{sec:rw}
For the rest of this paper, we refer to a LiDAR as \emph{dense} if it has more than 32 beams, and call it \emph{sparse} or \emph{minimal} otherwise.
Depth ground truth, required by fully-supervised methods, is obtained from a dense LiDAR signal, accumulated over several sweeps. A camera stereo setup is then used to remove trail artifacts from moving objects. We will refer to such densified point-cloud data as \emph{accumulated} LiDAR.
These three density levels are illustrated in \autoref{fig:lidar_density}.
We now detail the two lines of approaches related to our work: camera-only monocular self-supervised methods and LiDAR-based fully-supervised depth completion systems.\newline


\parag{Monocular self-supervised methods.} 
In a fully- or semi-supervised setting, several models estimate depth in a camera-only monocular setup \citep{DORN,Kuznietsov,semidepth}, but acquiring depth ground truth for outdoor environments at scale is challenging and expensive.
To overcome this issue, a few camera-based works~\citep{monodepth17,sfm_learner, struct2depth} propose a \emph{self-supervised} alternative to the use of ground-truth depth. 
Leveraging a set of consecutive frames, this paradigm predicts the depth for one of them and the relative changes in pose across nearby views.
The model is trained by minimizing a photometric reconstruction error defined over these views (\autoref{fig:supervision:estimation}).
Two important issues with such approaches hinder their widespread usage: the scale ambiguity of the produced depth maps and the infinite-depth problem.

The \emph{scale-ambiguity} problem stems from the view synthesis formulation being ill-posed. The formulation is scale ambiguous, as the target view can be correctly reconstructed regardless of the scale of the prediction. As a consequence, estimated depth maps are \emph{relative} --- up to an unknown global scaling factor --- and models thus need additional supervision to accurately estimate a \emph{metric} depth.
Several self-supervised approaches rely on ground-truth LiDAR signal to scale their depth estimation at test time~\citep{sfm_learner, monodepth2, struct2depth, GeoNet, Vid2Depth, DDVO}.
Alternatively, the recent PackNet model~\citep{packnet} proposes to automatically scale estimations with additional constraints imposed by the instantaneous velocity of the ego-vehicle.
Some works have also moved to a stereo setup to disambiguate the scale factor, using additional information, at train time only \citep{monodepth17, depth_adversialreconstruct} or also at run time \citep{psmnet,kendall_stereo, lidarstereonet}, thus abandoning the monocular setup.


The second issue of infinite depth arises when objects move at the same speed as the camera.
In this common situation, a trivial solution for the model is to predict that these objects are infinitely far and big, as they do not change in appearance through time~\citep{sfm_learner,monodepth2,packnet}. 
Recent proposals to address this problem exploit semantic segmentation of classes known to be often dynamic (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot cars, trucks)~\citep{struct2depth, struct2depth_motion}, or automatically prune the dataset by removing these objects~\citep{packnet-semguided}.
The robustness of both these approaches to novel test scenarios, however, remains unclear.



In our work, we build on camera-only methods to additionally integrate LiDAR information and show that: (i)~very few direct depth measures suffice to have a metrically-scaled dense depth estimation, and (ii)~the infinite-depth issue can be partially or completely solved with the use of LiDAR input, depending on its resolution and position, without any additional assumptions.




\parag{Depth completion methods.}
They typically estimate a dense depth map from raw LiDAR measurements.
Current deep-learning based methods for depth completion~\citep{Depth_Normal_Constraints,guided_depth_completion,jaritz_sparse_and_dense,non_local_spatial_propagation, sparse_to_dense, monocular_fisheye_camera,ACMNet} usually learn to regress ground-truth depth maps in a fully-supervised setup (\autoref{fig:supervision:completion}).
Such approaches generally operate over visual and LiDAR inputs.

In~\citet{guided_depth_completion} and~\citet{jaritz_sparse_and_dense}, one encoder per modality is proposed with a multi-scale and a late fusion respectively. Alternatively, \citet{Depth_Normal_Constraints, non_local_spatial_propagation, selfsup_sparse_to_dense} use an early fusion. 
The fusion module of~\citet{guided_depth_completion} only considers the image as a guiding signal for the LiDAR features. This assumes that the LiDAR input is sufficient, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot high-resolution, for estimating depth, and thus unsuitable for our case.
This limits the approach in~\citet{guided_depth_completion} to estimate depth from high-resolution 64-beam LiDAR both at train and run time, making it incomparable to ours as we do not have access to such data.

Alternatively, models like ACMNet~\citep{ACMNet} and NLSPN~\citep{non_local_spatial_propagation} propagate sparse LiDAR features into image features where depth measurements are not available.
The NLPSN architecture~\citep{non_local_spatial_propagation} jointly estimates an initial depth map, a pixel-wise confidence and non-local affinity kernels. This initial depth map is iteratively refined with the input LiDAR features using the predicted confidence map and affinity kernels. 
ACMNet~\citep{ACMNet} uses a multi-scale co-attention-guided graph propagation strategy for depth completion. It propagates the sparse and irregularly distributed LiDAR measurements through a nearest-neighbor encoding. In addition, it uses a symmetric gated fusion strategy to fuse multi-modal contextual information throughout the decoder.

All the aforementioned depth completion methods employ a 64-beam input LiDAR and are trained with accumulated LiDAR as supervision. Here, most of the scene structure is available and the task amounts to color-guided depth interpolation.
This design prevents these works from being easily adapted to new domains.
Indeed, acquisition of ground-truth data is expensive and not scalable, as it is obtained from high-resolution LiDARs and stereo cameras. 
In contrast, our work specifically focuses on minimal 4-beam LiDAR directly, with no densely accumulated LiDAR data as supervision.
We emphasize that in this very sparse 4-beam regime, almost no structural information can be directly extracted for the input signal.
The task we propose is then more akin to depth estimation than depth completion.

A closely related work to ours is the model of \citet{selfsup_sparse_to_dense}, which also uses LiDAR as a \emph{supervisory} signal in a monocular self-supervised setting.
LiDAR and camera signals are merged through an early fusion and the change of pose is estimated by solving a Perspective-$n$-Point problem.
However, their setup is different to ours. Their study focus on the dense depth completion regime, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot with a 64-beam LiDAR, while we work on depth estimation with a minimal 4-beam LiDAR. 
Moreover, they do not compare against other existing architectures in the self-supervised setting.
In contrast, we perform thorough evaluations with existing work by adapting camera-only and depth completion methods to our extremely scarce LiDAR regime.
Additionally, we propose a different supervision scheme and the use of multiple views in photometric reconstruction. These choices lead to a substantial improvement on the KITTI dataset.
Finally, we provide in-depth analyses on the  impact brought by the LiDAR signal at different levels.

\input{figures/overview_lidartouch}
\section{LiDARTouch framework}
\label{sec:learning_system}




This section is organized as three parts, each corresponding to a different and complementary use of the light LiDAR signal.
In \autoref{sec:learning_system:depth_network}, we present the architecture of the depth network, shown in green in \autoref{fig:learning_systems}, which estimates depth by fusing the monocular image with the sparse LiDAR pointcloud. 
In \autoref{sec:learning_system:objectives}, we detail the self-supervision objectives involving a photometric reconstruction along with a LiDAR self-supervision, as illustrated in red in \autoref{fig:learning_systems}.
Lastly, \autoref{sec:learning_system:pose_setups} introduces methods to estimate the relative change of pose between the source and target views, depicted by the orange part of \autoref{fig:learning_systems}.


\input{figures/architectures/architectures_no_input}
\subsection{Depth network}
\label{sec:learning_system:depth_network}


The core of our depth estimation system is a neural network taking the target image $I_{\text{t}}$ coupled with $H_{\text{t}}$, the LiDAR data projected in the image plane, as input, and predicting a depth map $\hat{D}_{\text{t}}$.
Given the multi-modal nature of the input, our depth network employs a fusion strategy, that can be either early or multi-scale.
In this paper, we consider four different architectures that are illustrated in \autoref{fig:architectures_no_input}.
Three of them are from the recent depth-completion literature, namely NLSPN~\citep{non_local_spatial_propagation}, S2D~\citep{selfsup_sparse_to_dense} and ACMNet~\citep{ACMNet}.
The fourth one, we refer to as Monodepth2-L{}, is an  extension of the camera-only model Monodepth2~\citep{monodepth2} to operate over the additional LiDAR input.



The two architectures NLSPN~\citep{non_local_spatial_propagation} and S2D~\citep{selfsup_sparse_to_dense}, illustrated in Figures \ref{fig:architectures:NLSPN} and \ref{fig:architectures:S2D} respectively, employ an early-fusion strategy, combining image and LiDAR features from the start, through concatenation.
Early fusion directly mixes features from both modalities, thus potentially enabling richer interactions across them.
The NLSPN architecture additionally re-injects the LiDAR signal at the end of the processing, as a late refinement strategy to mitigate signal degradation due to normalization layers.

In contrast, Monodepth2-L{} and ACMNet architectures,  represented in Figure \ref{fig:architectures:IL} and \ref{fig:architectures:ACMNet} respectively, use a multi-scale fusion.
They both encode LiDAR and visual data separately so that these modalities are processed differently and their learned features are progressively integrated together.
This design merges modalities more carefully than the early-fusion strategy, which is desirable as visual and LiDAR inputs carry complementary semantics.
The two encoders, based on ResNet-18~\citep{resnet}, are independent and modality-specific features are fused with a series of concatenations.
ACMNet, on the other hand, employs a more sophisticated co-attention strategy to mutually guide the features in the encoders and mix the features in the decoders to finally fuse them into one prediction.


\subsection{Self-supervision objectives}
\label{sec:learning_system:objectives}

Our challenging setting, where depth ground truth is unavailable for training the model, prevents the depth network architecture to be supervised directly.
We address this by training the network under the supervision of two combined objectives.
The first one, photometric reconstruction $L_\text{photo}$, is inspired by recent  advances in self-supervised camera-only monocular depth estimation \citep{sfm_learner,monodepth17,monodepth2}.
However, as discussed in \autoref{sec:rw}, training with this objective alone leads to scale and infinite-depth issues.
Consequently, we leverage a LiDAR self-reconstruction objective, which uses sparse yet complementary LiDAR information to mitigate these issues. 

\parag{Self-supervised photometric reconstruction $L_\text{photo}$.}
We recall that the photometric reconstruction problem is a surrogate task aimed at resynthesizing a target image, given neighboring source images with different viewpoints~\citep{sfm_learner,monodepth2,selfsup_sparse_to_dense}. 
Solutions to this task build on optimization approaches for disparity, motion and depth estimation without learning, based on photo-consistency.
The central idea is to combine pose and depth predictions to project a neighboring source image into the target view. The underlying intuition is that to accurately resynthesize the target view from the source one, both the depth and pose estimation must be accurate.

Formally, the \emph{target} image $I_{\text{t}}$ is considered with a set $S$ of \emph{source} images $I_{\text{s}}$ in its temporal vicinity.
First, the depth network predicts the dense depth map $\hat{D}_{\text{t}}$ for the target image $I_{\text{t}}$.
Second, the relative changes of pose $\hat{P}_{\text{t} \shortrightarrow \text{s}}$ between the target and source views are estimated --- we detail this in~\autoref{sec:learning_system:depth_network}.
One pose transformation ${\hat{P}_{\text{t} \shortrightarrow \text{s}} = \big( \begin{smallmatrix} \hat{R} & \hat{r} \\ 0 & 1 \end{smallmatrix} \big) \in \text{SE}(3)}$%
~is estimated for each source image $I_{\text{s}} \in S$, where $\hat{R}$ is a rotation matrix and $\hat{r}$ the translation component.
Given the estimates of depth and pose, and the camera intrinsics $K$, a source image $I_{\text{s}}$ can be warped via a differentiable geometric transformation into synthetic image $\hat{I}_{\text{s}}$ in the target view. More precisely, for homogeneous coordinates $p_{\text{t}}$ of a pixel in the target image, the projected coordinates $p_{\text{s}}$ in the source image are computed with:
\begin{equation}
\label{eq:projection}
    p_{\text{s}} \simeq K \hat{P}_{\text{t} \shortrightarrow \text{s}}  \hat{D}_{\text{t}}(p_{\text{t}}) K^{-1} p_{\text{t}}\,.
\end{equation}
For a pair $(I_{\text{s}}, I_{\text{t}})$ of source-target images, the reconstructed image $\hat{I}_{\text{s}}$ is enforced to match the target image $I_{\text{t}}$ by a pixel-wise image reconstruction error based on both an $L_1$ intensity loss and a structural similarity (SSIM) loss~\citep{ssim}. Note that this formulation assumes Lambertian surfaces.\\

More formally, at a given pixel location $p$, this loss reads:
\begin{equation}
\label{eq:photo_loss}
    L_\text{photo}(p) = \min_{I_{\text{s}} \in S} \Bigl\{ \frac{\alpha}{2}\big(1 - \mathrm{SSIM}(I_{\text{t}}, \hat{I}_{\text{s}})(p)\big) + (1-\alpha) \big| I_{\text{t}}(p) - \hat{I}_{\text{s}}(p) \big| \Bigr\},
\end{equation}
where $\alpha$ is a hyper-parameter balancing the contributions of the two terms.
Moreover, taking the minimum value over all source images $I_{\text{s}} \in S$ limits the impact of errors resulting from occlusions and disocclusions in the scene due to motion of the ego-car and/or of the other scene elements~\citep{monodepth2}.
To take into account objects with no motion with respect to the ego-car, this loss is only applied to pixels whose appearance between frames varies~\citep{monodepth2}.

\parag{LiDAR self-supervision.}
As detailed in \autoref{sec:rw}, a model solely trained with the photometric reconstruction loss $L_\text{photo}$ suffers from a scale-ambiguity issue and may be affected by the infinite-depth problem. 
In the following, we describe the new role of the low-density input LiDAR as a supervisory signal to mitigate this problem.
We assume that this complementary information source can provide minimal-yet-crucial cues to disambiguate the estimated depth, at a global scale level and especially for moving objects. Furthermore, a sparse depth signal can refine the photometric supervision for small objects, thus improving overall performances~\citep{depth_hints}.
Inspired by the depth completion and the stereo depth estimation literature, we consider three different ways of using LiDAR as a supervisory signal: a straightforward $L_1$ regression along with two refinements that either control the interference with the photometric reconstruction or take into account the inherent noise of the LiDAR signal.

First, we consider a naïve self-supervision scheme, an  $L_1$ loss for all pixels having a LiDAR measurement, in addition to the photometric loss $L_\text{photo}$:
\begin{equation}
    L_\text{naïve}(p)=\begin{cases}
       \lvert\hat{D}_{\text{t}}(p) - H_t(p)\rvert + L_\text{photo}(p) & \text{ if } H_\text{t}(p) > 0,\\
       L_\text{photo}(p) & \text{ otherwise},\\
    \end{cases}
    \label{eq:naive}
\end{equation}  
where $p$ is an index over the pixels, $\hat{D}_{\text{t}}$ the estimated depth and $H_\text{t}$ the input LiDAR projected in the target image plane. The latter being sparse, not all pixels have LiDAR data available; we use the encoding $H_{\text{t}}(p) = 0$ for such pixels.

Second, we consider the \emph{masked} self-supervised objective proposed in~\citet{selfsup_sparse_to_dense}. It makes the LiDAR regression and the photometric loss exclusive by masking-out the photometric loss $L_\text{photo}$ on pixels with a LiDAR measurement. Denoting $L_\text{masked}$ as this loss, it is given by:
\begin{equation}
    L_\text{masked}(p)=\begin{cases}
       \lvert\hat{D}_{\text{t}}(p) - H_\text{t}(p)\rvert & \text{ if } H_\text{t}(p) > 0,\\
       L_\text{photo}(p) & \text{ otherwise.}\\
    \end{cases}
    \label{eq:masked}
\end{equation}  
This loss is similar to $L_\text{naïve}$ but avoids
potential conflicts between
the photometric and LiDAR reconstructions.

Lastly, inspired by \citet{depth_hints}, we also introduce the \emph{hinted} self supervision, $L_\text{hinted}$, that takes into account  the inherent noise of the LiDAR signal.
Despite being a direct depth measurement, raw LiDAR signal is noisy for a number of reasons, including potentially imprecise calibration, approximated projection, and the fact that the camera and LiDAR are not exactly positioned at the same place, which results in objects observable by one but hidden to the other. 
Therefore, the loss $L_\text{hinted}$ integrates the LiDAR self-supervision only where image reconstruction is more precise by using the LiDAR signal instead of the estimated depth.
More precisely, two versions of the photometric contribution of the pixel are computed: the regular pixel-wise photometric loss $L_\text{photo}$, using the estimated depth map $\hat{D}_{\text{t}}$ in \autoref{eq:projection}, and ${L}^H_\text{photo}$ using the input projected LiDAR $H_\text{t}$ instead of $\hat{D}_{\text{t}}$ in \autoref{eq:projection}. Then we only supervise with the LiDAR reconstruction when ${L}^H_\text{photo} < L_\text{photo}$.
The objective is thus:
\begin{equation}
    L_\text{hinted}(p)=\begin{cases}
       \lvert\hat{D}_{\text{t}}(p) - H_t(p)\rvert + L_\text{photo}(p) & \text{ if } L^H_\text{photo}(p) < L_\text{photo}(p)\\
       L_\text{photo}(p) & \text{ otherwise.}\\
    \end{cases}
    \label{eq:hinted}
\end{equation}  


\subsection{Pose estimation}
\label{sec:learning_system:pose_setups}

The formulation of the photometric reconstruction involves the change of pose $\hat{P}_{\text{t} \shortrightarrow \text{s}}$ between the target image $I_{\text{t}}$ and source view $I_{\text{s}}$ for the source image warping.
A first possibility, which is widely used in monocular self-supervised depth estimation~\citep{sfm_learner,monodepth2, packnet,struct2depth}, uses a so-called \emph{pose network} jointly trained with the depth network.
However, due to the monocular ambiguity, this approach can only estimate a relative pose and thus relative depth maps, which then must be rescaled by an unknown factor.
Instead, we explore another way to estimates a metric pose, by leveraging the LiDAR information and solving a Perspective-$n$-Point problem~\citep{EPnP,P3P}. As such, depth estimation should also align to a real-world scaling.

\parag{Perspective-$n$-Point (P$n$P).} %
The P$n$P problem originally seeks the absolute pose of a camera given a set of 3D points and their corresponding 2D image projections.
In our case, we use the P$n$P formulation to estimate the change of pose between the target and source views, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot given the target image $I_\text{t}$ and LiDAR measurements, as well as the source image $I_\text{s}$.

First, pairs of pixels $(p_\text{t}, p_\text{s})$ matching in both views $I_\text{t}$ and $I_\text{s}$ are found using the SIFT descriptor~\citep{sift_Lowe} based on a DoG keypoint detector.
Then, the sole pairs for which $p_\text{t}$ has a LiDAR measurement are considered.
This gives us the pairs of 3D-2D points, where points $p_\text{t}$ 
are complemented with depth measurements and match the 2D points $p_\text{s}$ of the source image $I_\text{s}$.
Given these pairs, we can precisely estimate the \emph{metric}-scaled 6D rigid transformation between the target and source poses by minimizing the cumulative projection error. 

In challenging real-life situations, and especially when dealing with a 4-beam LiDAR, finding matching pixels that have LiDAR measurements can be arduous, making this method prone to errors.
Hence, we follow \citet{selfsup_sparse_to_dense} to remove outliers in the set of point correspondences by using RANSAC in conjunction with the P$n$P solving algorithm.
When this filtering step is insufficient for the algorithm solving the P$n$P problem to converge, we discard the training sample.
\section{Experimental protocol}
\label{sec:protocol}

The first component of our protocol is the dataset used for the experiments, namely KITTI~\citep{kitti}, and our preprocessing to reduce the raw 64-beam LiDAR to a 4-beam one (\autoref{sec:protocol:dataset}). We then introduce baselines in \autoref{sec:protocol:external_baselines}, and in \autoref{sec:protocol:contrib_metric} propose a new metric to assess the occurrence of catastrophically-erroneous depth predictions on key objects.
Additional details are given in the supplementary material. 
\subsection{Dataset and common evaluation metrics}
\label{sec:protocol:dataset}

To train models in our LiDARTouch framework, we need a dataset that provides a camera stream with aligned sparse LiDAR data for training. We also require this dataset to have ground-truth depth data with an associated benchmark to assess and compare our test performances.
We are aware of only one dataset matching both of these requirements, namely KITTI. 
It contains 1.5 hours of recorded driving sessions in urban environment from a video stream synchronized with LiDAR data. Depth ground truth is available: it is derived from dense LiDAR signals accumulated over five sweeps and stereo filtered.
Overall, we use this dataset to train and evaluate the quality of the predictions of our framework, and to compare against baselines and variants.
On the KITTI dataset~\citep{kitti}, we use the so-called Eigen split~\citep{eigen} for train, val and test with a minor modification for the val and test. The ground-truth LiDAR of~\citet{sparsity_invariant_cnns} is not available for some of the frames of the Eigen splits (fewer than 10). Following common practice~\citep{monodepth2,packnet}, we removed them from the val and test splits. Thus, the total number of examples are 22537, 873 and 652 respectively for the train, val and test sets.

The LiDAR data provided in KITTI is obtained with high-end 64-beam sensors, appropriate for \emph{evaluating} our self-supervised models, but much denser than what is expected to \emph{train} our LiDARTouch framework. 
Consequently, we perform a filtering step to extract 4 beams out of the raw 64-beam LiDAR data. To conform with prior works~\citep{selfsup_sparse_to_dense,jaritz_sparse_and_dense, packnet-semisup} and better compare with them, we sample LiDAR beams uniformly: 1 beam is kept every 16.
Note that with such a sampling, while 4 beams are extracted, only three beams effectively project onto the image plane as one beam falls out of the considered visual region.

\parag{Common evaluation metrics.}
Evaluation is conducted against accumulated ground-truth LiDAR obtained following~\citet{sparsity_invariant_cnns}, with the metrics defined in~\citet{eigen}. 
This includes the absolute (Abs Rel) and square (Sq Rel) relative errors, the root mean square error (RMSE), and its log version (RMSE$_{\log}$), as well as precision-under-threshold metrics measuring the percentage of depth predictions $\hat{D}$ close enough to the ground-truth depth $D$, in the sense of the value ${\delta := \text{max}(\frac{\hat{D}}{D}, \frac{D}{\hat{D}})}$ being under a user-defined threshold.
Following~\citet{eigen}, we consider three thresholds: $\delta < 1.25$, $\delta < 1.25^2$ and $\delta < 1.25^3$.

\subsection{Notations, ablations and external baselines}
\label{sec:protocol:external_baselines}

\parag{Notations.}
To refer to the network architecture, independently of the rest of the learning framework, we use Monodepth2, Monodepth2-L, NLSPN, ACMNet and S2D.
When we refer to whole models, i.e., architectures trained under the LiDARTouch framework, we append the `LiDARTouch' prefix. For example, we note `\ssACMNet' when we adapt the ACMNet architecture into the LiDARTouch framework.

For clarity, the inputs and the supervision schemes that are employed by the models are recalled in the tables of the experiments section.
The input of each depth prediction model includes an image (noted `$\mathcal{I}$') and, optionally, a sparse 4-beam LiDAR point cloud (`$\mathcal{L}^4$')
We considered the following supervisions strategies: self-supervised photometric reconstruction (`P') associated to loss \autoref{eq:photo_loss}, supervised LiDAR ground-truth regression with $L_1$ loss (`L$_\text{gt}$'), or LiDAR self-supervision (`L$_{4}$') with one of the three options in Eqs.\,(\ref{eq:naive}), (\ref{eq:masked}), or (\ref{eq:hinted}).

\parag{Ablation: Pose estimation with a Pose Network.}
In \autoref{sec:learning_system:pose_setups}, we presented the P$n$P algorithm, which estimates metric pose changes from source to target views.
To highlight the gains enabled by the use of the extra LiDAR information for computing the pose, we experiment by training a \emph{pose network} instead, a widely used component of monocular depth estimation models~\citep{sfm_learner,monodepth2, packnet,struct2depth}.
For each target-source image pair, the pose network outputs the 6D rigid transformation between views.
It is differentiable and trained jointly with the depth network. 
When only trained with the photometric error (\autoref{eq:photo_loss}), the 6D transformation is estimated up to a scale factor due to the monocular ambiguity. This results in a relative depth estimation requiring to be rescaled by the LiDAR depth ground-truth median value (not available in our case).

A solution is to use data from the IMU/GNSS to supervise the pose estimation scale. In the context of depth estimation, such an approach has been explored by \citet{packnet}.
Formally, we first obtain the approximate change in pose between the source and target views ($P_{\text{t} \shortrightarrow \text{s}}$) from integrated inertial measurements. Then, we extract its translation component $r$ and make the predicted pose translation component $\hat{r}$ regress its magnitude:
\begin{equation}
\label{eq:velocity}
    L_\text{imu} = \Bigl\vert \lVert r\rVert_2  - \lVert\hat{r}\rVert_2 \Bigr\vert.
\end{equation}
As for a given pose there is a unique depth minimizing \autoref{eq:photo_loss}, constraining the pose's magnitude to a metric scale forces the depth estimation to be metric as well.

\parag{Baselines: Monocular methods.}
We compare against state-of-the-art monocular self-supervised approaches such as SfMLearner \citep{sfm_learner}, Vid2Depth \citep{Vid2Depth}, GeoNet \citep{GeoNet}, DDVO \citep{DDVO}, Monodepth2 \citep{monodepth2} and PackNet-SfM \citep{packnet}. Note that these methods can only produce relative depth maps, as they use an unsupervised pose network, so they have to be rescaled using the ground-truth LiDAR. Comparisons with these methods is thus unfair, in their favor.

Additionally, we compare with methods that directly produce metric depth by leveraging additional supervision. This includes (1) DORN~\citep{DORN}, a camera-only method fully-supervised by a dense LiDAR signal, (2) \citet{Kuznietsov}, a semi-supervised method using stereo reconstruction and dense LiDAR supervision, and (3) PackNet-SfM~\citep{packnet} model supervised with IMU prior. 

\parag{Baselines: Depth completion methods.}
We also compare against supervised depth completion methods, namely ACMNet~\citep{ACMNet}, NLSPN~\citep{non_local_spatial_propagation} and S2D~\citep{selfsup_sparse_to_dense}.
However, their original versions are not trained and evaluated on the same splits as monocular methods. We re-train and evaluate them on the Eigen split, in their fully-supervised setting but with only a 4-beam LiDAR input.
Additionally, we also train and evaluate these depth completion methods when the depth ground truth is simply replaced by the 4-beam LiDAR input for supervision signal. We refer to this setting as `Naïve self-sup.'.




\subsection{Catastrophic Distance Rate (CDR) metric}
\label{sec:protocol:contrib_metric}
Monocular image-only depth estimation methods suffer from the infinite-depth problem: vehicles with a motion close to that of the ego vehicle (in other words, with almost no relative motion) can be estimated as being infinitely far away. In the context of autonomous vehicles, such anomalies can lead to potentially dangerous outcomes.
This critical weakness of image-only methods is not well reflected in the commonly-used evaluation metrics, as errors associated with these local flaws are overwhelmed by global scores aggregated at a dataset level.

This problem was qualitatively evaluated in some recent work~\citep{sfm_learner,monodepth2,struct2depth,struct2depth_motion,packnet} but no precise measurement of its severity has yet been proposed.
To address this issue, we define a novel quantitative metric, called the catastrophic distance rate (CDR), to assess the degree to which a model tends to make such disastrous predictions.

CDR measures the percentage of cars whose estimated distance to the ego-car is catastrophically poor in the test set.
To this end, we use instance segmentation masks for all the vehicles of every image of the test set.
With these vehicle instances, CDR is computed in a two-step process:
\begin{enumerate}
    \item Instance mask filtering to keep the ones potentially concerned by the infinite-depth problem;
    \item Computation of the depth error measured on these instance masks. 
\end{enumerate}


\parag{Instance mask filtering.}
For the first step of our CDR metric, we filter out irrelevant masks to only focus on vehicles typically concerned by the infinite-depth problem, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot{} first vehicle in front, unoccluded and not too far. 
As we use a centered frontal camera, we begin by discarding vehicles that are not in the center of the scene. We also remove cars whose instance masks are too small, considered too far from the ego vehicle.
Then, to assess whether a car is occluded or not, we assume that a heavily occluded vehicle generally has a non-convex shape (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot incised by the front vehicle) and that, on the contrary, the mask of a non-occluded car is approximately convex. 
The overall process is illustrated in \autoref{fig:catastrophic_metric} and further details are provided in the supplementary material.

\begin{figure}[]
    \centering
   
    \includegraphics[clip,width=\linewidth]{figures/instance_mask_filtering.pdf}
    \caption{\textbf{Selecting vehicles to compute the CDR metric.} The aim is to extract the individual mask of the first vehicles in front of the ego-car. These are indeed vehicles affected by infinite-depth error due to a small relative motion, leading to potentially catastrophic consequences. The proposed CDR metric computes the rate of such failures over the test set.
    }
    \label{fig:catastrophic_metric}
\end{figure}

\parag{CDR computation.}
CDR estimates the percentage of instances for which the relative depth error is above a manually-defined ``catastrophic'' threshold $\tau$.

Within each segmentation mask $M_k$, indexed by $k \in \mathcal{K}$, we define the set $\mathcal{V}_k$ of pixels that possess a ground-truth LiDAR depth measurement: ${\mathcal{V}_k = \left\{ p \mid M_k(p) > 0 \wedge D_k(p) > 0 \right\}}$. Note that, as with $H_t$, $D(p)=0$ if and only if there is no LiDAR point projecting at $p$.
In the KITTI test set, the average size of $\mathcal{V}_k$ is 543.
The error $R_k$ made by the model on the instance mask $M_k$ is measured by the average signed depth error over $\mathcal{V}_k$: 
\begin{equation}
    R_k =  \frac{1}{|\mathcal{V}_k|} \sum_{p \in \mathcal{V}_k} \frac{\hat{D}_k(p) - D_k(p)}{D_k(p)},
\end{equation}
where $| \mathcal{V}_k |$ is the cardinality of $\mathcal{V}_k$. Please note that no absolute value is involved in the design of $R_k$ as we focus only on the infinite-depth problem, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot $\hat{D}(p) > D(p)$, when a car is predicted catastrophically further away than its true position.

By thresholding the error $R_k$ and aggregating it over instances, we define the ``Catastrophic Distance Rate'' as: 
\begin{equation}
    \text{CDR}(\tau) = \frac{1}{|\mathcal{K}|} \sum_{k \in \mathcal{K}} \llbracket R_k > \tau \rrbracket,
\end{equation}
with $\llbracket \cdot \rrbracket$ the Iverson bracket, $|\mathcal{K}|$ the number of instance masks and $\tau$ a user-defined threshold.
For example, ${\text{CDR}(\tau = 0.5) = 20\%}$ indicates that the distance to front vehicles is over-estimated by more than 50\% of the true distance in 20\% of the cases.





\section{Experiments and discussions}
\label{sec:exp_and_disscusisons}

We
validate the effectiveness of our LIDARTouch framework by
1) evaluating and disentangling the contributions brought by the multiple LiDAR integrations,
2) obtaining state-of-the-art results on self-supervised depth estimation,
3) successfully adapting architectures from the depth completion literature to a self-supervised scheme,
4) studying how LiDAR integration alleviates the infinite-depth problem,
and 5) analysing qualitatively our models.


\subsection{Contributions brought by a touch of LiDAR}
\label{sec:exp_and_disscusisons:lidar_influence}

We begin with an ablation study to assess the contribution brought by the sparse LiDAR at three different levels: as a self-supervision signal, as an input of the depth network, and as additional information for pose estimation.





\parag{Self-supervision with the sparse LiDAR.}
To study the impact of having 4-beam LiDAR as a self-supervisory signal, in addition to the photometric reconstruction, we experiment with the configurations presented in \autoref{sec:learning_system}. We also consider various combinations of LiDAR self-supervision schemes and depth networks.
The results of these experiments are reported in \autoref{tab:lidar_losses}.


These results support that the use of a LiDAR self-supervision improves performance or stays on-par with photometric-only supervision scheme. This is expected as we include additional information from the LiDAR in the supervisory signal.
Indeed, LiDAR self-supervision appears to be especially necessary for Monodepth2{} and Monodepth2-L{}, the photometric self-supervision being not sufficient to learn a meaningful model.
Concerning ACMNet, NLSPN and S2D architectures, they show slightly better performances when trained with the photometric loss only.
However, without any LiDAR self-supervision, they remain badly affected by the infinite-depth phenomenon as we will discuss in \autoref{sec:exp_and_disscusisons:infinite_depth}.


We now go further in studying the design of LiDAR self-supervision. 
In \autoref{tab:lidar_losses}, we compare the loss variants for the LiDAR reconstruction defined in \autoref{sec:learning_system:objectives}, namely the \emph{naive} compound loss \autoref{eq:naive}, the \emph{masked} one \autoref{eq:masked}, which prevents interferences with the photometric error, and the \emph{hinted} loss \autoref{eq:hinted}, which handles the noise of the LiDAR signal.
These experiments are conducted for the four different depth networks considered in \autoref{sec:learning_system:depth_network}.
Overall, averaged over all architectures, the \emph{masked} version the best results, demonstrating the need to reduce interferences between the LiDAR and photometric supervisions.
On the other hand, we observe that the \emph{hinted} loss yields the worst results. The control that it imposes is too strong and discards too many of the already scarce LiDAR measurements.
Hence, \emph{masked} LiDAR self-supervision, being the most effective, is employed in all further experiments.


\input{tables/lidar_losses}

\parag{LiDAR as an input.}
Next, we study the contribution brought by LiDAR when used as an additional input to the model, and discuss the modeling choices to merge LiDAR and image signals in this case. 
In \autoref{tab:lidar_losses}, we compare the Monodepth2{} architecture, whose depth network does not use LiDAR as input, against all other bi-modal architectures leveraging LiDAR input.
We observe that Monodepth2{} is consistently outperformed by these models when they are trained with LiDAR self-supervision, with a relative improvement of at least 13\%. 
This validates the positive influence of integrating the few-beam LiDAR as an input.

When no LiDAR self-supervision is involved, we observe a clear superiority of the early-fusion designs over multi-scale fusion.
Indeed, ACMNet, NLSPN and S2D start leveraging the LiDAR signal at the encoder level, which gives metric-scale depth predictions that are more accurate.
In contrast, Monodepth2{} and Monodepth2-L{}, which are either not using LiDAR at all or integrating it later in the architecture, 
fail to align the scale of their depth predictions with the metric pose coming from the P$n$P solver. In this case the network converges to a degenerate case where depth is uniform.
When LiDAR self-supervision is employed (P+L$_4$), these fusion differences tend to have much less impact on the learning process.
This suggests that the architecture choice is directly linked to the supervision scheme and that LiDAR self-supervision helps in avoiding degenerated local minima.



\input{tables/lidar_influence} 
\parag{Pose estimation with a sparse LiDAR.}
We now show that a precise computation of the change of pose is critical to estimate depth maps that are correctly scaled, and that the use of a touch of LiDAR is beneficial for this purpose. 
To demonstrate this, we experiment by replacing the P$n$P depth estimation with a pose network that does not use any LiDAR information, as detailed in \autoref{sec:exp_and_disscusisons:external_basleines}.
Results of these experiments, conducted with various depth networks and supervision schemes, are reported in \autoref{tab:scaling}.

Overall, we observe very poor performances with the use of the pose network.
First, we note that the use of photometric reconstruction only (P) leads to unsatisfactory results for all networks. Indeed, the pose provided by the pose network is only relative, and the depth estimation is thus unscaled as well.
To enforce a metric scale, we train the pose network with an additional supervision, either the IMU prior explained in \autoref{sec:exp_and_disscusisons:external_basleines} (P+imu), or the LiDAR reconstruction self-supervision (P+L$_4$).
While this helps Monodepth2{} and Monodepth2-L{} to correctly train, the ACMNet, NLSPN and S2D architectures either reach a degenerated local minima or overfit to the LiDAR input.
This signifies that the pose network cannot be a guide consistently to produce accurate depth maps with the supervision schemes we consider.

An alternative to using a pose network is to integrate the LiDAR signal in the pose estimation step. This enables the use of P$n$P methods that produce scaled poses, and thus, the collapse of the depth solutions to the metric depth.
This is consistently verified with the use of the photometric and LiDAR supervisions (P+L$_4$) for each of the five architectures considered, and leads to the best results compared to any other configuration. 


\subsection{Comparison against camera-only methods}
\label{sec:exp_and_disscusisons:external_basleines}

In \autoref{tab:sota}, we report evaluations of the four architectures trained within our LiDARTouch framework against camera-only baselines.

\parag{Self-supervised camera-only methods.}
First, we show that training under our framework outperforms both supervised~\citep{DORN,Kuznietsov} and self-supervised monocular depth estimation methods~\citep{sfm_learner,Vid2Depth,GeoNet,DDVO,monodepth2,packnet}. We note that contrary to other methods, ours uses few-beam LiDAR as input. Furthermore, self-supervised monocular depth estimation approaches only estimate relative depth and thus are rescaled with ground truth before evaluation. With our approach, this unrealistic and impractical rescaling step is no longer needed. 

\parag{Supervised camera-only methods.}
We also obtain better results than monocular depth estimation models trained with ground truth and optional stereo \citep{DORN,Kuznietsov}, while not requiring either of those. While the latter does not use few-beam LiDAR as input, not requiring ground truth at train time makes our method trainable at scale on any domain.
 
Overall, we showed that by integrating few-beam LiDAR in the pipeline, we substantially increase performances on all metrics over other methods not using few-beam LiDAR.


\input{tables/SOTA_comparison}
\input{tables/adapted_depth_completion}

\subsection{Comparison against depth completion methods}
\label{sec:exp_and_disscusisons:adapted_completion}

We compare our LiDARTouch framework against two supervision schemes from the depth completion literature: full-supervision with ground truth (L$_\text{gt}$) and self-supervision (L$_4$-naïve). These results are reported for the three architectures in \autoref{tab:adapted_depth_completion:4beams}.

\parag{Supervised depth completion methods.}
Unsurprisingly, supervising the training of any of the architectures with the privileged ground-truth depth yields better results than our LiDARTouch framework. However, LiDARTouch remain very competitive, \emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot 2.504 vs.\ 2.112 in RMSE for ACMet.
We also investigate the impact of the density of the input LiDAR on these scores in \autoref{fig:frameworks_comparison}. We observe that LiDARTouch is consistently close to the fully-supervised depth completion alternative when the number of layers varies.



\input{figures/frameworks_comparison}

\parag{Self-supervised depth completion method.}
The results in \autoref{tab:adapted_depth_completion:4beams} show that the models trained with naïve 4-beam LIDAR self-supervision are unable to converge to decent results. Architectures cannot generalize from such a sparse LiDAR input as the supervisory signal is not sufficient. 
Moreover, in \autoref{fig:frameworks_comparison}, we remark that the naive self-supervision scheme makes performance plummet when the LiDAR data becomes sparser.




\subsection{Alleviating the infinite-depth problem}
\label{sec:exp_and_disscusisons:infinite_depth}


Next, we quantitatively investigate the infinite-depth problem, i.e., the degree and the frequency to which a model dramatically overestimates the distance to cars in front, using the novel metric introduced in \autoref{sec:protocol:contrib_metric}.
Our intuition is that LiDAR self-supervision is a suitable means to mitigate this problem.



To verify this hypothesis, we study three models:
\begin{itemize}
    \item A model that does not use the LiDAR signal at all, noted `Monodepth w/ IMU supervision', which heavily suffers from the infinite-depth issue;
    \item A model with LiDAR as input and for the P$n$P-estimated pose, but supervised solely with the photometric loss, noted `ACMNet$^\text{P}_{\text{P$n$P}}$';
    \item A model trained within the LiDARTouch framework, using LiDAR for the depth network, pose estimation and self-supervision, noted `\ssACMNet'.
\end{itemize}

\input{figures/quali_inf_depth/quali_inf_depth}
\input{figures/inf_depth_study}

We plot the distribution of the CDR metric against the chosen threshold $\tau$ in \autoref{fig:inf_depth_study}. We observe that the more LiDAR information is integrated, the fewer catastrophic estimations occur. 

Indeed, ACMNet$^\text{P}_{\text{P$n$P}}$, which uses LiDAR both in input and pose, improves over Monodepth2 but is still affected by the infinite-depth issue. We also see a clear improvement of our \ssACMNet{} over the two other models.
For example, for $\tau=0.5$, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot the distance of a car is overestimated by at least half, Monodepth2 has a metric score of 5.02\% while ACMNet$^\text{P}_{\text{P$n$P}}$ has 0.6\% and \ssACMNet{} 0.0\%. Such results show that Monodepth2 predictions cannot be trusted for downstream tasks such as car detection or free space estimation that are both required by functions like automatic emergency braking, keep-lane assist or adaptive cruise control. While ACMNet$^\text{P}_{\text{P$n$P}}$ reduces the likelihood of catastrophic estimation by 8 folds for $\tau=0.5$, 0.6\% is still too high to implemente in a critical system intended for wide commercial use.

Overall, a network trained with our pipeline is significantly less impacted by the infinite-depth problem and we validate our hypothesis that, during training, the LIDAR self-supervision disambiguates cars estimated too far from their real distance. Hence, our models can accurately and safely handle moving objects with no relative motion, typical of cars in fluid traffic.

In \autoref{fig:quali_infinite_depth}, we present three examples along with selected close-ups highlighting the infinite-depth problem.
For example, on the leftmost column, we observe a typical `hole' in the depth map where Monodepth2{} with IMU supervision estimates a vehicle three times as far as its true distance.


\input{figures/qualitative_frameworks/figure}
\subsection{Qualitative analysis}
\label{sec:exp_and_disscusisons:qualitative}

In addition to \autoref{fig:quali_infinite_depth}, we provide some qualitative analyses where we show the depth maps obtained for different frameworks in  \autoref{fig:qualitative_frameworks}. 
First, we observe better overall depth maps with \ssACMNet{} than with Monodepth2{}.
For example, we better estimate the two moving cyclists in \autoref{fig:qualitative_frameworks}a  as well as the fine tree trunks in \autoref{fig:qualitative_frameworks}c.

As expected, the fully-supervised method ACMNet (GT-sup.) delivers the best-qualitative depth maps, as it leverages privileged ground-truth LiDAR depth during training.
However, we observe that self-supervised approaches (Monodepth2 and \ssACMNet{}) better estimate areas near the top of the scene.
This can be explained as LiDAR points are absent from regions above the road, which hinders ACMNet (GT-sup.) prediction in these regions due to the lack of supervisory signal it uses (last row in \autoref{fig:qualitative_frameworks}). 


Despite the successful integration of LiDAR in LiDARTouch, we note that some local depth estimation artifacts still occur, similar to the maps obtained from self-supervised depth estimation methods.
Typically, this concerns distorted, reflective and color-saturated regions because the photometric reconstruction loss assumes Lambertian surfaces (cars in \autoref{fig:qualitative_frameworks}c).
Our model may also produce blurry depth predictions for small or thin objects, such as traffic signs (Figures \ref{fig:qualitative_frameworks}a and \ref{fig:qualitative_frameworks}b).


\section{Conclusion}

In this paper, we introduce LiDARTouch, a novel self-supervised framework for depth estimation with few-beam LiDAR.
While being extremely sparse, we show that the LiDAR signal can be leveraged at three complementary levels of a self-supervised learning scheme. 
Across four different architectures, the LiDARTouch framework can reach competitive performances with respect to fully-supervised depth completion methods while being significantly cheaper and more annotation friendly.
Moreover, we show that the sparse LiDAR signal provides valuable cues to disambiguate monocular depth estimation at a global level as well as for moving objects.
Our method can be trained on any domain with no modification, and it can thus bring accurate and metric depth estimation at a fleet scale.

With our novel LiDARTouch framework, the new CDR metric to measure the infinite-depth problem, and the associate source code of our code, we hope to enable further research on the task of monocular depth prediction with minimal LiDAR input, typical of real-world assisted/automated driving systems.






\paragraph{\bf Acknowledgements} Karteek Alahari was supported in part by the ANR grant AVENUE (ANR-18-CE23-0011). This work was granted access to the HPC resources of IDRIS under the allocation 2021-101766 made by GENCI.

\section*{Authors' biography}
Florent Bartoccioni is a CIFRE PhD candidate in valeo.ai and Inria Grenoble Rhône-Alpes. He received the M.Sc. degree in computer science  from the École Normale Supérieure de Rennes, France, in 2019.
His research interests include dynamic scene forecasting and 3D perception for autonomous robots. 

\'Eloi Zablocki is a research scientist at valeo.ai. He obtained his Ph.D. at Sorbonne University in 2019, on multimodal machine learning with language and vision. His research interests include computer vision for scene understanding, motion forecasting, and explainability of machine learning models.

Patrick Pérez is Scientific Director of valeo.ai. Before joining Valeo, Patrick Pérez has been a researcher at Technicolor (2009-2018), Inria (1993-2000, 2004-2009) and Microsoft Research Cambridge (2000-2004). His research interests include multimodal scene understanding and computational imaging.

Matthieu Cord is a full professor at Sorbonne University. He is also a part-time principal scientist at valeo.ai. His research expertise includes computer vision, machine learning, and artificial intelligence. He is the author of more than 150 publications on image classification, segmentation, deep learning, and multimodal vision and language understanding. He is an honorary member of the Institut Universitaire de France and served from 2015 to 2018 as an AI expert at CNRS and at the French National Research Agency.

Karteek Alahari is a senior researcher at Inria in the Grenoble - Rhône-Alpes center. He was previously a postdoctoral fellow in the Inria WILLOW team at the Department of Computer Science in École Normale Supérieure, after completing his PhD in 2010 in the UK. His current research focuses on learning robust and effective visual representations, when only partially-supervised data is available.

\bibliographystyle{model2-names}









