Meshed Up: Learnt Error Correction in 3D Reconstructions
Michael Tanner Stefan Saftescu Alex Bewley Paul Newman
cs.CV cs.RO


Zonotopes whose cellular strings are all coherent
    Rob Edman, Pakawut Jiradilok, Gaku Liu, Thomas McConville
    December 30, 2023
=============================================================






  Dense reconstructions often contain errors that prior work has so far minimised using high quality sensors and regularising the output. Nevertheless, errors still persist. 
  This paper proposes a machine learning technique to identify errors in 3D meshes. 
  Beyond simply identifying errors, our method quantifies both the magnitude and the direction of depth estimate errors when viewing the scene. This enables us to improve the reconstruction accuracy. 
  
  We train a suitably deep network architecture with two 3D meshes: a high-quality laser reconstruction, and a lower quality stereo image reconstruction. The network predicts the amount of error in the lower quality reconstruction with respect to the high-quality one, having only view the former through its input. We evaluate our approach by correcting 2D inverse-depth images extracted from the 3D model, and show that our method improves the quality of these depth reconstructions by up to a relative 10% RMSE.
  




§ INTRODUCTION








This paper is about detecting and rectifying mess ups in dense reconstructions. Dense reconstruction as a mapping from input images to 3D meshes is a well studied area. However much of that work is feed-forward in the sense that it strives to produce the best mesh in an open loop. In this work we consider how, given an arbitrary generated mesh and the images that were used to create it, one might correct reconstruction errors post-hoc. 

Prior efforts towards improving the accuracy of generated 3D meshes often focused on regularising the camera depth maps <cit.>, using expensive high-quality sensors <cit.>, or regularising the dense-reconstruction output <cit.>. While regularisation based approaches impose structure to smooth local surfaces, gross reconstruction errors still remain. In contrast the focus of this work is to identify error prone areas in 3D reconstructions. The goal is to facilitate the removal or repair of these mesh regions in order to improve overall mapping accuracy. As an example of what we will describe Figure <ref> illustrates how the most inaccurate areas in a reconstruction are identified and corrected in a 2D representation of the scene.








To realise this capability, a mapping is learnt from rasterised geometric and appearance features to the error in the camera-based reconstruction. Accurate laser reconstructions serve as our ground-truth to train a CNN that estimates the quality of depth-map reconstructions.
A simple example of this process is shown in Figure <ref>, which displays the same scene reconstructed by laser and camera input data.
Compared to the laser reconstruction's colour and mesh images, the regions in the camera reconstruction that are likely incorrect can readily be identified –  holes in the road and the “smearing” area between the cars.
Ideally, we would have a similar but automated way of perceiving the mesh outputs to identify erroneous regions.
In this paper, we present our method for training a neural network to recognise and highlight these regions.














A high-level overview of the proposed system is presented in Figure <ref> that shows the framework for building the meshes used in training a CNN. Two dense reconstructions are used: one constructed using camera-only depth-maps and another constructed using more accurate laser data.
Both geometric and appearance features are extracted from each reconstruction and rasterised to a 2D image suitable for training the CNN (Section <ref>).

In Section <ref>, we describe our network architecture and loss function based on related work in the area of monocular depth estimation <cit.> and re-purpose it towards the task of correcting for mesh errors.
In particular, we investigate the potential and value of providing a diverse set of features ( colour, inverse depth, triangle edge ratios and surface normals) to the network which are not commonly available without a mesh representation.
Finally, in Section <ref> we demonstrate the efficacy of the proposed system in identifying and correcting errors on three meshes created using sequences from the KITTI-VO <cit.> data set.
Within this framework, the following contributions are made:

    
  * The novel use of a CNN applied to rasterised mesh features is shown to identify error-prone regions in a low quality mesh 3D reconstruction;
    
  * Using the network predictions to correct for error in the inverse-depth images leads to significant improvements over what is extracted from the original mesh;
    
  * We analyse which of the rasterised mesh features are most informative to the CNN through an ablation study on the trained network.


















§ MESH FEATURES

Much work in the deep learning literature relies upon simple RGB images as the input <cit.><cit.><cit.>.
However, as our problem formulation assumes that a dense 3D model is available, we can extract a much richer set of features from the reconstruction. 
Throughout this work we use the dense mesh reconstruction proposed by Tanner  <cit.> referred to as , although our framework is agnostic to the 3D mesh pipeline.



 §.§ Feature Creation




To extract dense 2D features suitable for a CNN, we begin by using GLSL to “fly” a virtual camera through the 3D models.
Our  shader transforms the 3D model's vertices into the camera reference frame to compute the depth of each vertex and the normalised camera vector.
These camera-frame points are then grouped into triangles and passed along to a  shader that computes the triangle's area, surface normal, and edge lengths.
Finally, the  shader processes each of the preceding feature values and rasterises them into individual pixels in feature images, as shown in Figure <ref>.
This GLSL process enables us to extract six feature images:  two per-pixel features (RGB and depth) and four per-triangle features (area, surface normal, edge length ratio, and surface-to-camera angle).
Intuitively, the mesh-triangle-derived features contain a lot of information; they provide geometric cues that have been accumulated from multiple depth-maps and integrated into a 3D reconstruction by . We look at whether this intuition is verified in Section <ref>.



 §.§ Ground-Truth Data




For a ground-truth reference, we construct a mesh using the same reconstruction pipeline as above. The same virtual camera trajectory is used to create a sequence of feature images from both the depth-map and laser reconstructions. Since the goal is to train a neural network to recognise errors in the depth-map reconstruction, we must first compute ground-truth errors with which to train the network. Rather than directly subtracting depths, we instead elect to use inverse-depth. This way, the network is encouraged to emphasise errors in foreground objects – which are more likely to have complex geometry compared to background surfaces.
Specifically, we compute the per-pixel ground-truth (Δ_gt) as,


    Δ_gt = A/d_c - A/d_l,


where d_c and d_l are pixels in the depth-map features for the camera and laser reconstruction, respectively, with A being the scaling constant.
In disparity images from stereo cameras, A = f_x b where f_x is the camera's x focal length and b is the baseline distance between the left and right image sensors. Because our approach is agnostic of the input sensor, A can be arbitrary, and we set A = 1 when evaluating performance (Section <ref>).




§ RECONSTRUCTION ERROR PREDICTION 







 §.§ Network Architecture

For our network architecture, we based our network on Lania et al.'s Fully Convolutional Residual Network <cit.> with concatenated ReLU activation functions on the intermediate layers <cit.>.
This network was designed to infer per-pixel depth from a single RGB image, a task highly correlated with our aim to compute estimated depth error given a set of feature inputs. In this work, the input layer is generalised to accept F input channels dependant on the number of active features used from the previous section.
Table <ref> provides high-level details of the CNN where a series of residual blocks based on the ResNet-50 architecture <cit.> is followed by up-projection blocks, proposed in <cit.>. Note the fractional strides denote spatial upsampling. The final layer is a simple 3 × 3 convolutional layer which outputs real-valued estimates corresponding to the inverse-depth error at each pixel location.






 §.§ Generalisation Capacity


To be able to deploy a learnt CNN to new meshes created without a high-fidelity laser we need our network generalise to unseen data. To this end, three techniques are employed: cropping, downsampling, and regularisation. 
Firstly, we randomly perturb and crop all input feature images before providing them to the network. After each epoch of training ( the network has viewed all the training images), the next epoch will receive a slightly different cropped region of each input image.  This prevents the network from associating a specific pixel location in the training data with its corresponding output.

Secondly, the feature maps are gradually downsampled via projection blocks (from ResNet-50) creating a bottleneck, thus reducing representational capacity.  This also provides greater context to the convolutional filters in the later layers of the network by increasing the size of their receptive field. 

Thirdly, we implement a L_2 weight regulariser to further constrain the representational capacity by preventing the network from over-relying on the cost function at the expense of generalisation performance. 




 §.§ Loss Function


Several loss functions are widely used in machine learning applications.
The L_2 norm is traditionally popular because it heavily penalises large errors and is smooth.
However, unlike the L_1 norm, it has a near zero gradient for small errors, thus is often unable to drive the error completely to zero.
Over the years, researchers have proposed alternative norms which combine the “best” (based on application) characteristics of each of these norms.
The Huber norm uses L_2 near the origin and L_1 elsewhere, while BerHu uses the L_1 norm near the origin and the L_2 elsewhere <cit.>.
We choose BerHu as our cost function on the networks error prediction since the L_1 term places a higher penalty on small errors (when compared to Huber or L_2) while still providing strong penalties for larger errors. This is in contrast to regularising depth maps where the Huber norm is more favourable to capture high gradient edges around objects <cit.>. These benefits were also observed in the related task of depth prediction <cit.>.













Each convolutional layer performs small convolutions (1 × 1 and 3 × 3) that are added as features for the next layer. These are mostly residual blocks not vanilla conv layers
By using the convolutional layers to add features, rather than directly modify the input feature vector, higher-order feature gradients are better preserved throughout the network, which improves the networks learning rate. Who is this paper meant for? Do we need to explain how representations work in a CNN? This paper if intended for robotist which as Paul puts it are now familiar with the idea of learning/optimisation. However, it gets really confusing when using terms like "regularisation" and "gradients" in a geometric context when they are also an important part of machine learning techniques also used in this paper. I suggest for the geometric versions call them "spatial gradients" and "spatial regularisation".















 §.§ Input Features

As discussed in Section <ref>, our 3D models of the operating environment enable us to extract a much richer feature set than usually available in image-based machine-learning applications. Specifically, five different geometric features are generated from the 3D mesh (inverse depth, triangle area, surface normals, triangle edge length ratio, surface-to-camera angle), and used along with the corresponding colour image from the KITTI-VO data set. Later in Section <ref> we evaluate how informative each of these features are, both qualitatively and empirically.




§ EXPERIMENTS








 §.§ Experimental Setup



  
*Dataset
Three sequences from the KITTI-VO dataset where used as the input to the reconstruction pipeline.
The input of the network consists of features from the camera based mesh, while the ground truth used for both training and evaluation comes from the laser based mesh, as described in Section <ref>.
For all experiments, the KITTI-VO sequences 00, 05 and 06 were used for training and leave-one-out style cross-validation.
All three sequences are predominantly in urban environments with small amounts of visible vegetation.
Using GLSL, we created a virtual camera to project the dense reconstruction into input feature images. The trajectory of the virtual camera follows the original trajectory in the KITTI-VO sequences. This enables the use of real colour images as input alongside the generated features. Both the camera and laser reconstruction meshes were generated with a fixed voxel width of 0.2 m.



  
*Network Training and Inference
The network module was implemented in Python using the TensorFlow framework. 
The network weights were optimised using the ADAM solver <cit.> with a learning rate of 10^-4 for 250 epochs over two of the sequences. At this point, the value of the loss function appears converged with variance coming form the stochasticity of the mini-batch sampling.
To reduce this variance and confirm convergence, the optimisation is run a further 50 epochs with the learning rate reduced to 10^-5.
Other training hyper-parameters include a batch size of 16 and a L_2 weight decay factor of 10^-6 per training step. Inference takes, on average, 72 ms per half-size KITTI image and aassociated features.




  
*Performance Metrics
This work uses two different metrics to measure the resulting performance.
The first metric is the commonly used RMSE which provides a quantitative measure of per pixel error, computed as follows:

    RMSE = √(1/n∑_X(d^* - d_l)^2),

where d_l is the ground truth depth, d^* is the predicted depth, X is the set of valid pixels, n is the cardinality of X. 


Our second metric measures the accuracy of our network’s ability to estimate errors under a given threshold, serving as an indication of how often our estimate is correct.
The thresholded accuracy measure from <cit.> is essentially the expectation that a given pixel in X is less than a threashold thr^k:

    δ_k = 𝔼_X[𝕀(max(d_l/d^*, d^*/d_l) < thr^k)],

where 𝕀(·) represents the indicator function.
As in <cit.>,  thr = 1.25, and k ∈{1, 2, 3}. 



 §.§ Correcting Depth Maps

The central question of this paper is whether we can correct 3D mesh reconstructions using a CNN. 
As a proxy, the mesh surface projected as an inverse-depth image and used to represent the local 3D scene. 
For our baseline reference, we compare the inverse-depth image from the camera-only reconstruction (created using ) with the equivalent inverse-depth from the laser-only reconstruction, thus d^* = d_c. 
To obtain the refined depth-map, we use the network output Δ^* to correct the depth-map reconstruction, giving us d^* = d_c - 1/Δ^*. An example of this process is shown in Figure <ref>. 













Quantitatively, we find our CNN based correction provides a 10% relative RMSE improvement over the baseline generated by . Furthermore, it is consistently more accurate over different thresholds, indicating better depth values both across more pixels.
A qualitative visualisation of the network output is provided in Figure <ref>, along side the ground truth difference in camera to laser inverse depth maps. This visualisation shows that the network is capable of identifying small errors over broad regions (the red offset in the road level potentially caused by imperfect calibration) as well as large but narrow errors commonly found at object boundaries.



 






 §.§ Feature Ablation Study







In the previous experiment, all features were provided to the CNN under the assumption that some features may contain more information than others and that the network can choose to ignore certain features if they contain no information.
This will be reflected by measuring the sensitivity in the network output when a specific feature is withheld from the input.
It is also possible that the full set of features contains redundant information which would result in the network spreading its internal importance weighting across multiple features. Such behaviour is implicitly encouraged through the L_2 weight regularisation placed on the network, meaning that small weights across multiple channels are favoured over large weights on single channels. If the network learns to use redundant features then it should be more robust to the absence of one of the features if the equivalent information could be found elsewhere.

Figure <ref> shows the relative feature importance evaluated by disabling one feature at a time and monitoring the influence on the inverse depth accuracy δ_1. 
Crucially, the inverse-depth from the camera reconstruction was considered to be the most important.
On the other hand, mesh specific features such as triangle area and edge length ratio (Figure <ref> (d)(f)) appear to be least useful.

To test if redundancy is explaining this lack of sensitivity, we keep only RGB, inverse-depth and normals, to fine-tune the network (Table <ref>). Here, the CNN failed to recover the lost performance confirming our intuition that mesh features provide additional information not easily derived from simple (stereo) camera features.
































 



 §.§ Results and Analysis


Key insights:

    
  * Rasterised features from a 3D mesh are useful for improving the inverse depth-map quality.
    
  * Only a subset of the input features are important for the CNN. In particular: inverse depth, normals, and RGB.
    
  * The selected features provide a good geometrical context.




  §.§.§ Rasterised features from a 3D mesh are useful for improving the depth-map quality

Analysis

Refer to Table II and Figure 4. Describe/discuss results in Table. Figure 4 is only the CNN output, not the corrected depth map.






















For our initial experiments, we used RGB images from the KITTI-VO data set plus five generated features: inverse depth, normals, triangle area, triangle edge ratio, and camera-to-surface angle.
We also performed further experiments to determine which of the features from the reconstruction are indeed useful. All the experiments are performed at two scales: a quarter, and half the size of the KITTI-VO data.


While laser reconstructions are of much higher quality, there isn’t as much data as in camera reconstructions. In our evaluation, we use the rasterised laser reconstruction to mask the network output (i.e. we only consider the pixels where d_l is finite).

Example input features, ground-truth data, and predicted errors are presented in Figure <ref>.

The network successfully highlights the high-likelihood error regions—most noticeably the borders around the fence (bottom-left of input image) and car (bottom-right of input image).
More examples of the network's qualitative prediction performance are shown in Figure <ref>.
The trends in the prediction image are similar to those in the ground-truth images.



In the first set of experiments, we wanted to see whether we can use the output of our network to correct the depth-map from the image-only reconstruction. 

For the second set of experiments, we tested whether an additional view would improve the performance of the network. 
We kept a very similar set-up, with the additional view and associated reconstruction features fed as another input to the network. 
  













§ CONCLUSIONS


In this paper we present a supervised learning technique for training a neural network to detect and correct error-prone regions of a dense reconstruction.
Using a GLSL pipeline, we placed a virtual camera at various positions and orientations throughout the reconstruction to generate both input feature data and, by comparing the laser and camera reconstruction features, ground-truth data to train the network. We demonstrated that the error predictions of our model can  be used to make corrections to a mesh projected into an inverse-depth image and provide quantitative evaluation.   




Trained on the reconstructions created in <cit.> for KITTI-VO sequences 00, 05, and 06, our network's performance was quantitatively analysed using leave-one-out cross-validation.
Qualitatively, the network consistently recognised and highlighted the similar problem areas as those found in the ground-truth data.
We leave it to future work to use these network predictions to either remove or fix the error-prone reconstruction regions.




§ ACKNOWLEDGMENT

The authors would like to acknowledge the support of the UK’s Engineering and Physical Sciences Research Council (EPSRC) through the Centre for Doctoral Training in Autonomous Intelligent Machines and Systems (AIMS) Programme Grant EP/L015897/1, Programme Grant EP/M019918/1, and the Doctoral Training Award (DTA).
Additionally, the donation from Nvidia of the Titan Xp GPU used in this work is also gratefully acknowledged.


IEEEtran

  

