A Data-driven Prior on Facet Orientation for Semantic Mesh Labeling
Andrea Romanoni Matteo Matteucci






A Data-driven Prior on Facet Orientation for Semantic Mesh Labeling
    Andrea Romanoni

Politecnico di Milano, Italy

andrea.romanoni@polimi.it
Matteo Matteucci

Politecnico di Milano, Italy

matteo.matteucci@polimi.it

    
========================================================================================================================================================







Mesh labeling is the key problem of classifying the facets of a 3D mesh with a label among a set of possible ones. State-of-the-art methods model  mesh labeling as a Markov Random Field over the facets. These algorithms map image segmentations to the mesh by minimizing an energy function that comprises a data term, a smoothness terms, and class-specific priors. The latter favor a labeling with respect to another depending on the orientation of the facet normals. In this paper we propose a novel energy term that acts as a prior, but does not require any prior knowledge about the scene nor scene-specific relationship among classes. It bootstraps from a coarse mapping of the 2D segmentations on the mesh, and it favors the facets to be labeled according to the statistics of the mesh normals in their neighborhood. We tested our approach against five different datasets and, even if we do not inject prior knowledge, our method adapts to the data and overcomes the state-of-the-art.








§ INTRODUCTION


In the last decades we witnessed a significant evolution in 3D modeling from images: moving from the early point-based approaches, researchers have proposed a wide variety of solutions that output a dense 3D mesh model of the environment.
The most recent trend in computer vision, as well as in robotics and photogrammetry, is to add semantic labels to the 3D reconstruction, in order to estimate a more informative 3D model for robot navigation and localization or visualization and inspection.



On one side volumetric semantic 3D reconstruction <cit.> couples the 3D reconstruction with semantic labeling; this improves the quality of the 3D model recovered  from the images, but the complex minimization techniques involved and the spatial complexity of the volumetric  representation affect negatively  the efficiency and the scalability of this approach.
Recent semantic Multi-View Stereo (MVS) mesh refinement address the problem by estimating a low resolution mesh and refining it exploiting photometric and semantic information <cit.>. As the shape and resolution of the mesh change while the algorithm refines the mesh, the  labels need to be updated accordingly. Therefore a time-crucial step is the semantic mesh labeling.
Moreover, when a big and accurate 3D model, recovered from image-based algorithms, such as <cit.>, is already available, recomputing the entire model through expensive volumetric semantic 3D reconstruction algorithms is inconvenient or, in some cases, impossible due to memory restrictions.
In such cases mesh labeling represents the unique feasible possibility to recover a semantic reconstruction.

Semantic mesh labeling aims at classifying each facet of a 3D mesh model by exploiting available 2D images segmentation, 3D cues, or geometric priors, as in Figure <ref>.
Successful approaches leverage 2D image segmentations, and model the labeling problem as energy minimization  over a Markov Random Field (MRF).
In the MRF, the nodes represent facets whereas the potentials gather semantic information from image segmentations, enforce smoothing, and represent class-specific priors.
For instance, Blaha  <cit.> proposes a unary prior which, for each class, defines a preferred orientation with respect to the ground vector. 

In this paper we propose a novel MRF formulation that does not require any additional knowledge of the environment except the 3D model and the image segmentations. 
We propose to use a novel unary term in conjunction with the usual data term a smoothing term and the discontinuity term proposed in <cit.>. 
We exploit a coarse labeling of the 3D mesh to collect locally the distribution of the normals for each class, it biases the facet classification according to them; instead of defining manually the preferred orientation for each class, as in <cit.> or estimating it globally as in <cit.>.










§ RELATED WORKS


Recovering a semantically annotated 3D model has been addressed by joining the labeling with the 3D reconstruction, or by decoupling the two tasks.

Joint semantic and 3D reconstruction has first been  proposed by Häne  <cit.> and it stems from the idea that scene semantics enables the definition of class-priors to improve the quality and the robustness of a 3D reconstruction algorithm. 
In their work, the authors define a convex minimization problem in the continuum 3D space with a unary and a pairwise potential, then they relax it to work in the discrete voxel space.
The per-voxel unary potential is estimated from 2D image segmentations and the visibility rays from each camera (data term).
The pairwise term is the combination of smoothness and class-specific priors, the latter is learned from manually annotated data, and defines the probability of class changes depending on the normal transition.
This work has been extended in <cit.> to embed higher order ray potentials  into the optimization of the graphical model.
For a more in-depth discussion about semantic volumetric reconstruction we refer the reader to <cit.>.
Conversely, Kundu  <cit.> propose to bootstrap from Structure from Motion (SfM) points and they define a Conditional Markov Random Field on a 3D voxel grid, where they estimate the unary potentials from image segmentations, and they add handcrafted class-specific priors to enforce reconstruction consistency. 
The method is effective for large scale environment, but the 3D labeled model has a low resolution due to the low number of SfM points.
 
Even if joint segmentation and reconstruction algorithms are effective, they strongly rely on volumetric reconstruction and the optimization techniques are usually expensive so that they require a difficult trade off between accuracy scalability and computational speed.
More efficient representations have been proposed, , octrees as in <cit.>, voxel-hashing <cit.>, submaps <cit.>, or multi-grids
<cit.>; however the scalability of multi-label optimization of these methods is still an open issue.


Different approaches decouple the reconstruction and the mesh labeling, so that the complexity required by the 3D reconstruction module is reduced, and the scene segmentation is delegated to a semantic labeling algorithm.
Sengupta  <cit.> build a 3D volumetric reconstruction through the standard Truncated Signed Distance Function (TSDF), then they label the images and fuse them in the 3D model.
Instead, Valentin  <cit.>,  design a CRF whose unary potentials correspond directly to 3D voxels and are trained by taking into account images color, and geometric features of the facets. 
The authors improve the labeling with respect to Sengupta  <cit.>, however the algorithm is computationally expensive and, in our paper, we show that the priors learning phase is not needed. 

Herman  <cit.> and McCormac  <cit.> fuse incrementally the 2D images segmentation into the reconstruction. 
These methods, as well as <cit.>, benefit from the vast literature about 2D segmentation algorithms for 2D image segmentation <cit.>. 
However, the output of <cit.> and <cit.> are not continuous triangular meshes, but, respectively, a point cloud and a surfel model.

Tung  <cit.> proposed an interesting method that estimates the unary potential of a CRF without any classifier, but by comparing the regions of the image computed as in <cit.>, with a database of annotated examples. However they do not use any semantic prior to add robustness of the final outcome. 

Kalogerakis <cit.> segment meshes into semantics categories by learning  different 3D features of the mesh such as shape curvature.
Verdie  <cit.> take as input an existing  3D mesh of a city reconstructed from aerial images. 
They subdivide it into superfacets, which are subareas of the 3D mesh, and they collect geometric attributes, , elevation, planarity and horizontality. 
Then they model the problem with a Markov Random Field with unary class-likelihood and pairwise class-transition terms defined manually.
Even if the results are remarkable, the handcrafted priors do not generalize the method in non urban areas.
For this reason Rouhani  <cit.> and Martinovic  <cit.> extended <cit.> with a classifier trained on both geometrical features and color information from the mesh texture. 
The method improves the results of <cit.> and it avoids using handcrafted priors; however, due to the features taken into account, its scope is still limited to city models from aerial images.
In <cit.> the authors propose a series of classifiers that labels a point cloud but even in this case the algorithm
is domain-specific (restricted to façades) and its application is restricted to building facades.  The main drawback of such methods that rely on just 3D cues to segment the mesh, is that they are not able to exploit the available deep learning methods such as <cit.>, that in the last years have shown impressive performances in image segmentation.
Riemenschneider  <cit.> propose to classify a mesh according to 2D image segmentation, however their method requires a learning stage to estimate which surface orientation has to be labeled from which viewing direction by relying on 3D geometric features.


Blaha  <cit.> and Romanoni  <cit.> propose two different mesh labeling methods, which are the closest to our proposal, and which rely on 2D image segmentation. 
Both <cit.> and <cit.> are two MVS algorithms that extend the photometric refinement of <cit.>.
They bootstrap from a 3D mesh and 2D image segmentations and they jointly refine the former through both semantics and photometric data. Mesh
labeling plays a substantial role here; since the model changes as new refinement iterations are applied, the mesh needs to be relabeled at each iteration. They define a Markov Random Field (MRF) over the mesh facets with unary potentials induced by the 2D image segmentations and pairwise smoothing terms for adjacent facets. 
Blaha  <cit.> add a unary geometric term that encodes handcrafted class-specific priors; Romanoni  <cit.> estimate per-class normal distributions from a coarse mesh labeling and they add a unary term that, if the facets labeled as c show a strong preference for a direction, other facets with similar direction are biased to belong to the same class c. The relevant aspect of these two methods and the method proposed in this paper is that they do not need to learn unary potentials over class labels.

















§ PROPOSED METHOD


The proposed algorithm models the labeling as an energy minimization problem over a Markov Random Field. 
The inputs are: the 3D triangular mesh model of the scene under analysis, and, for each image that captures the scene, the corresponding camera calibration and the output of a semantic classifier in the form of a likelihood distribution for each class c.

Given a label assignment 𝐥, which associates  a label l_f to each facet f∈ℱ, we define the following energy:

    E(𝐥) =  ∑_f∈ℱ E_data(l_f) + μ_1 ∑_f∈ℱE_norm(l_f) + 
    μ_2  ∑_f,h∈ℱ E_disc(l_f,l_h) + μ_3 ∑_f,h∈ℱ E_smooth(l_f)

where E_data and E_smooth are respectively the data unary term proposed in <cit.> and a smoothness term adapted from <cit.>.
The pairwise term E_disc, we named discontinuity prior,  was proposed in <cit.> and biases the class changes between adjacent facets where significant variations between normal orientations occur.






The main contribution of this paper is the unary term E_norm that replaces the priors defined manually as in <cit.> or by means of learning algorithms as in <cit.> or <cit.>.
Assuming a coarse labeling of the mesh is available, through the term E_norm  we label each facet by comparing its normal direction with the normal directions of the facets in its neighborhood. 
For instance, in Figure <ref>(a) we have a toy 3D model with a coarse labeling: the normals directed to the right located on the left and, conversely, the normals directed to the left located on the right likely belongs to blue facets; both on the left and on the right  sides the normals directed upwards likely belong to red facets.
The term  E_norm aims at collecting such likelihoods and extend them locally, so to obtain the result in Figure <ref>(b). To compute the coarse labeling we first apply a simplified version of the MRF that minimizes the energy:

    E_simpl(𝐥) =  ∑_f E_data(l_f) + ∑_f,h E_smooth(l_f)



Figure <ref>  summarizes the pipeline of our algorithm.






 §.§ Data and Smoothness Terms

The term E_data collects the evidences estimated by the image classifier among the different views. 
Since we know the likelihoods I_i(l,x) for label l in the pixel x of the i-th image, we define:

    E_data(l_f) = -log( ∑_i∫_Ω_iI_i(l_f,x)dx)

where Ω_i is the area of the i-th image where the 3D model is projected.

We define the smoothness term between two adjacent facets as:

    E_smooth(l_f,l_h)= 
    
    1.0,    if  l_f ≠ l_h
    
    0.0,    if  l_f = l_h

to penalize frequent class changes among nearby facets.



 §.§ Discontinuity Prior

Given two adjacent facet f_1 and f_2 with the corresponding normals 𝐧_f_1 and 𝐧_f_2, we define:


    E_disc(l_f,l_h) = 
    
    e^-(∠(𝐧_f_1,𝐧_f_2))^2/2*(π/2)^2,    if  l_f ≠ l_h
    
    0.0,    if  l_f = l_h


This term represents the idea that two adjacent objects or two adjacent parts of the scene that belong to two different classes show discontinuities along the the mutual boundary. 
For instance, in the region between a car and the street the facets incident to the wheel-street boundary have significantly different orientations.
On the other hand is likely that adjacent facets with the same labels have similar orientation.
Only in the case of sharp edges the assumption does not hold, however even if we favor a label change in those areas, the data and smoothness terms provide enough evidence to avoid misclassification.






















 §.§ Normal Likelihood

The previous terms are able to map the label from 2D segmentation to the 3D mesh. 
However, in some cases the smoothing term is not able to tackle the noise affecting the segmentations.
In such cases the common approach is to define class specific priors as in <cit.> that favor one class with respect to others depending on the surface orientation.
These priors are effective and they are able to compensate noise and classification errors; however, they require both the knowledge of the gravity vector and to define and tune, for each class, which orientations are more likely.

We replace the prior term, such that we no longer require prior knowledge about the environment, but we still favor a labeling depending on facets orientation. 
The result is a more flexible and general algorithm. 
We follow and extend the intuition reported in <cit.>, that is, given a 3D mesh with a coarse labeling, we can collect, for each label, the distribution of the normals and use them to estimate the likelihood of a facet f with a normal n_f to belong to a class with label l, therefore we turn the prior term into a likelihood term.

For each class, the authors in <cit.> collect the mean normal and its variance: if the facets belonging to a class of label l, , the ground, show a strong tendency towards a single orientation, , the normals have a unimodal distribution with low variance, the method favors the other facets with similar orientation to be labeled as l.
This “global” approach fails in two cases. 
One case is exemplified as follows: if the algorithm biases all the facets oriented upwards to become ground, also the roofs of the cars, or some roofs of buildings are biased to be ground. 
The second case happens if the normals (of the facets) belonging to a class have a multimodal distribution, such as the walls in a street; in this scenario the single mean and variance computed in <cit.> are not expressive enough.

To face these two issues, we propose to locally estimate  the distributions of the normals for each class/label, and to represent and use directly these distributions as a likelihood term in the energy function instead of defining it from the mean and the variance.

First we define a 3D lattice that covers the whole 3D mesh, its cells are cubes of size d_i, , the left part of Figure <ref> exemplifies it in 2D. 
For each cell c_k and each class l we turn the normal of the facets whose centroid is inside of c_i into unit spherical coordinates.
Then, we define two histograms hist_k,l^azim and  hist_k,l^incl to collect respectively the distribution of the normal azimuths and inclinations.

When we evaluate a facet f with the centroid inside the cell c_k, we define the likelihood:

    P_k (f|l) ∼ e^hist_k,l^azim(az(𝐧_f)) * hist_k,l^incl(inc(𝐧_f))

where az(·) and inc(·) convert a vector into its azimuth and inclination.
Therefore the normal likelihood term is:

    E_norm = -log(hist_k,l^azim(az(𝐧_f)) * hist_k,l^incl(inc(𝐧_f)))







  
Normal Likelihood vs. Smoothing
The term E_norm let that, locally, facets with similar orientation takes similar labels. 
Even if this behavior may seems similar to a classical contrast-sensitive smoothing term, its meaning, implementation and result is different.
The two-step approach (coarse labeling with histogram computation + fine labeling) let us to define E_norm as a simple unary term for each facet, instead of a pairwise term, as the usual smoothing term. 
This means that during the optimization procedure E_norm does not change its value for each facet f, similarly to what happens with prior terms differently from the behavior of a smoothness term that spreads the label with current higher support.
In principle, our method could be imitated by a classic smoothing term that adds a pairwise term connecting every facet to all the facets inside the big voxel of the grid. However in this case the dimension of the MRF would quickly become intractable, , in the fountain-P11 dataset we have around 12000 facets for each voxel, therefore this approach would require 12000!/2!(2-12000)!≈72·10^6 pairwise terms for each facet.







































§ EXPERIMENTS


We tested our labeling method against five datasets: fountain-P11, castle-P30 <cit.>, Southbuilding <cit.>, KITTI 95 <cit.> and Dagstuhl.
In Table <ref> we illustrate the statistics of the datasets and the mesh we estimated.  
Even though it is related to the problem we faced in this paper, we did not used the Varcity dataset <cit.>, indeed the classes are all spread along the building facade, and in this case a class-specific normal prior does not have any influence on the labeling stage.
To extract the input mesh we calibrate the cameras with Structure from Motion (SfM) <cit.>, and we estimate the depth maps with the plane sweeping library <cit.>.
We fuse SfM points and the depth maps by means of the volumetric reconstruction algorithm proposed in <cit.>, and we refined the model through photometric refinement <cit.>.
For fountain-P11, castle-P30, Southbuilding and Dagstuhl datasets, we extract the 2D segmentation with the per-pixel likelihood score through the Multiboost classifier <cit.>: we manually labeled few images and we classify all the datasets images into ground, wall, vegetation and other. For the KITTI sequence we used a pre-trained version of <cit.>.
We modeled the MRF through the openGM library <cit.> and we adopted the alpha expansion algorithm <cit.> to infer the labeling.
























We tested our method against the labeling algorithms in <cit.> and <cit.>.
Even if <cit.> and <cit.> iterate between mesh labeling and mesh refinement, and both steps indirectly influence each other, a single labeling step is a single block of the pipeline, and works independently from the refinement.
Since our method focuses on labeling, we compared it against the mesh labeling of <cit.> and <cit.> isolated from the refinement step.  



As a first step, we tuned the three parameters μ_1, μ_2 and μ_3 of the energy function (<ref>) on the dataset fountain-P11.
We assigned different values to each parameter while keeping the other two fixed.
We choose μ_1=0.2, μ_2=0.2 and μ_3=1.0, and we use these same values with the other datasets.
To have a fair comparison we followed the same procedure to tune <cit.> (μ_1=0.4, μ_2=1.0) and <cit.> (μ_1=0.2, μ_2=0.4 and μ_3=0.4).
For <cit.> we also need to manually estimate the gravity vector.

While tuning the size of the cubes in the 3D lattice adopted to estimate the local distributions, we find that the parameter d_i is quite robust to variations, it however depends on the dataset dimension and the labeling granularity and shows best results when it is not too big with respect to the scene, otherwise the results
tends to be similar to the global method proposed in <cit.>. We choose 1.0m, 7m, 3.2m, 4m and 4m respectively for the fountain-P11, castle-P30, Southbuilding, KITTI 95 and Dagstuhl datasets. In Figure <ref> we illustrate the intersection over union for the castle-P30 dataset with varying values of d_i in comparison with the results for <cit.> and <cit.>.



In Table <ref> we show the results of the proposed method against the methods in <cit.> and <cit.> and we evaluate the ablation of the proposed term E_norm (Baseline).
We report different metrics: the average accuracy, recall, F-score and precision are computed as the average metric over all the pixel, while the overall measures are computed as the average over the classes.
To compute those metrics we compare the mesh rendering against 11, 15, 12, 5 and 4 different reference ground truth view respectively for the fountain-P11, the castle-P30, Southbuilding, KITTI 95 and Dagstuhl.
In Figure <ref> we illustrate the segmentation results for a reference view  for the three datasets and in Figure <ref> the labeled mesh.
Even if the image segmentations coming from Multiboost are very noisy (first column of Figure <ref>), the three algorithm managed to well compensate most of the misclassification.
Even if the method we propose does not require to add any prior information, its performance is better than the other methods in most of the metrics for each of the five datasets.
One of the reason is that, since the meshes have high resolution, the per-class distributions of the normals are quite complex, , the ground is not simply flat, but it contains crispier details, therefore the simple prior of <cit.> and the P_norm term of <cit.> are often not sufficiently expressive. Instead, our algorithm explicitly models these distributions through the histograms; therefore the prior-like normal term adapts itself to the scenario. Another reason of the improvement is evident in the fountain-P11 dataset. 
Here both <cit.>, through the handcrafted prior, and <cit.>, through its global prior, favor all the facets perpendicular with respect to the ground to be labeled as wall: this avoids to fill most of the 2D misclassification that happen on the fountain surface. Our approach completely overcome this issue since it collects the statistics locally, and not just because it acts as a strong smoothing term, indeed the small road sign in the KITTI 95 dataset is preserved (see Figure <ref>).

Finally, in addition to the processing shared with <cit.> and <cit.>, our method needs to compute the histogram; however the operation takes only 1.6 to 4.6 seconds. 


In Table <ref> we report a failure case that happened with the sequence 15 of the DTU dataset <cit.>. 
Here the Multiboost algorithm fails to provide a reliable segmentation of the ground (Figure <ref>). 
Therefore our method has not enough support to understand that facets directed upwards likely belongs to the ground.
The ground is better segmented  in <cit.> because they enforce directly this concept by the class-specific prior and even in <cit.> because they collect the normal statistics globally, therefore with a wider support.













§ CONCLUSION AND FUTURE WORKS


In this paper we proposed a novel method to label 3D models with semantic labels. 
In particular we defined a Markov Random Field over the mesh, as in the state-of-the-art, endowed with a new term that biases the  labeling depending on the facet orientation and acts as a class-specific geometric prior, but, instead of learning it or fixing  it manually, we estimate it directly from a coarse labeling of the mesh, collecting locally the distribution of the normals. 
Experiments on different datasets showed that the method is not only able to run without a learning phase or prior knowledge about the geometry of the classes, but it also improves the accuracy of the labeling.

As a future extension, we will investigate the possibility of running the method incrementally and on embedded multicores, coupling it with a real-time 3D reconstruction method <cit.> also accounting the energy constraints imposed by the computing platform  <cit.>.
We also plan to test the method with a wider dataset such as ScanNet (<cit.>), by replacing the Multiboost algorithm, that failed to estimate a reasonable segmentation of ScanNet scenes, with a more sophisticated image segmentation algorithm based on neural networks.


§ ACKNOWLEDGEMENTS

This work has been supported by the “Drones112” project founded by EIT Digital. 

ieee



