epBRM: Improving a Quality of 3D Object Detection using End Point Box Regression Module
Kiwoo Shin Masayoshi Tomizuka




Coupled Cluster Green's function formulations based on the effective  Hamiltonians
    Karol Kowalski
    December 30, 2023
==================================================================================

empty
empty




We present an endpoint box regression module(epBRM), which is designed for predicting precise 3D bounding boxes using raw LiDAR 3D point clouds. The proposed epBRM is built with sequence of small networks and is computationally lightweight. Our approach can improve a 3D object detection performance by predicting more precise 3D bounding box coordinates. The proposed approach requires 40 minutes of training to improve the detection performance. Moreover, epBRM imposes less than 12ms to network inference time for up-to 20 objects.

The proposed approach utilizes a spatial transformation mechanism to simplify the box regression task. Adopting spatial transformation mechanism into epBRM makes it possible to improve the quality of detection with a small sized network.

We conduct in-depth analysis of the effect of various spatial transformation mechanisms applied on raw LiDAR 3D point clouds. We also evaluate the proposed epBRM by applying it to several state-of-the-art 3D object detection systems.

We evaluate our approach on KITTI dataset<cit.>, a standard 3D object detection benchmark for autonomous vehicles. The proposed epBRM enhances the overlaps between ground truth bounding boxes and detected bounding boxes, and improves 3D object detection. Our proposed method evaluated in KITTI test server outperforms current state-of-the-art approaches.







§ INTRODUCTION


3D object detection systems become a core component in most of recent autonomous vehicles system. Autonomous vehicles are required to precisely detect and track the surrounding objects in real-time to achieve safe driving. Among the various sensors used in recent autonomous vehicles system, LiDAR which measures distance between objects and the ego-vehicle become a crucial sensor for 3D object detection system.  

The main goal of this work is to improve quality of bounding box predictions in order to increase recall of 3D object detection from LiDAR 3D point clouds. Despite the close proximity between predicted and ground truth objects, recent 3D object detection systems perform below a specific requirements, thereby increasing false-positive detections.(see Fig. <ref>)








In autonomous driving, the size of search space is usually 80 m(side) X 70 m(forward) from ego vehicle, which is very huge compared to the size of objects, such as car, pedestrian and cyclist. Moreover, the number of object, which can be used as training samples, is also limited. Thus, the majority of search space is occupied by non-object area. During training process, the network is mostly trained to classify object/non-object areas(localization task) and less effort is made for predicting precise 3D bounding boxes(box regression task). Previous works attempt to overcome such challenge by using efficient encoding of 3D bounding box<cit.> and/or by adopting focal loss to dynamically control weight of each task<cit.>, but those approaches have not been very successful.




The main idea behind our proposed detection system is to decompose the whole detection pipeline into a localization task and a box regression task and train each task independently (Fig. <ref>). By decomposing the detection pipeline into two tasks, a box regression network can be trained more efficiently, 1) with a network specifically designed for regression task(<ref>), and 2) using training samples containing information about 3D bounding boxes only.(<ref>).



Because individual network transition slows the detection speed, most previous 3D object detection systems are developed from a unified structure(<cit.>). Therefore, in our approach, box regression network is implemented to accelerate real time object detection. To meet such requirement, we design the epBRM with a small size network, which imposes only 12ms overhead for up to 20 objects(<ref>).





We use current 3D object detection systems as localization module in our detection pipeline. In this work, a localization module is responsible for predicting the location of object and its corresponding confidence score. And epBRM is responsible for predicting all coordinates required for a 3D bounding box, including location, rotation and size of object (Fig. <ref>). 

We evaluate the proposed method by applying to the result of current 3D object detection systems, AVOD(FPN)<cit.>, F-PointNet<cit.> and PointPillars<cit.>. The evaluation result demonstrates that the proposed approach improves the 3D object detection performance immediately after 10k of training iteration, which takes only 40 minutes using one Titan X GPU(not pascal) and i7-6700k CPU.




§ RELATED WORKS






 §.§ Bird's eye view based approaches
 
In early works in 3D object detection system, input feature is represented by projecting LiDAR 3D point clouds onto 2D planes and used CNN-based detection network for prediction<cit.>. A core problem shared by these approaches is a loss of information by projection. Thus, these methods showed low performance in detecting small objects such as pedestrian and cyclist. 



 §.§ Sensor fusion based approaches
 
The density of LiDAR 3D point clouds is sparse when the object is far from the LiDAR. Several sensor fusion based approaches have been proposed to utilize 2D RGB images that can provide higher resolution than LiDAR<cit.>. These methods concatenate features extracted from 2D images and LiDAR point clouds to generate input feature for the region proposal network. These approaches improved detection performance for small objects, but they required additional task for synchronization and calibration between multi-sensors.




 §.§ 3D based approaches
 
One approach in 3D object detection system is to convert Lidar 3D point clouds into voxels and extracts voxelwise features for predicting 3D bounding boxes<cit.>. Several works proposed feature representation based on 3D CNN, but they required high computation<cit.>.

Another line of work in 3D object detection systems can be categorized between sensor fusion based approach and 3D based approach, which is to use 2D object detection results as region proposals and apply PointNet<cit.> to predict 3D bounding boxes<cit.>. One drawback behind these approaches is a slow inference speed due to the need for two sequentially connected pipelines, one for 2D image detection and the other one for 3D point clouds detection.




§ 3D BOX REGRESSION MODULE








 §.§ Role and goal of epBRM



The main purpose of epBRM is not to discover objects that are missed by localization module, but to predict precise 3D bounding box coordinates by using point clouds sampled around the location predicted from the localization module. Therefore, we set a target distance(dist_bound), which represents the maximum distance between a ground truth object and the localization module's predicted location of the object, further refining a 3D bounding box. For objects located more than dist_bound from the ground truth object, we do not aim for such detections to be further regressed.

There are two main factors for deciding the value of dist_bound: 1) performance of the localization module, and 2) the representation ability of epBRM.

First, dist_bound is inversely related to the precision of the localization module. For example, if the localization module is capable of predicting precise location of objects, then epBRM only needs to work well for samples with small error. In this case, we assign small value for dist_bound.

Second, the value of dist_bound is also related to the representation ability of a network used for epBRM. If we increase the value of  dist_bound, it also increases the complexity of task that epBRM is responsible for. If we constrain the representation ability of epBRM for computational efficiency, the value of dist_bound will also decrease.

In <ref>, we evaluate the effect of  dist_bound on various localization modules and its effect on detection performance.



 §.§ epBRM with spatial transformation mechanism
 



Aiming for better computational efficiency and real time inference, we restrict the size of the network for epBRM at the cost of representation ability of the network. This necessitates a need for an additional method to simplify the box regression task in order for a network with low representation ability to be capable of predicting precise 3D bounding boxes. 





To avoid voxelizing<cit.> and/or grouping 3D point clouds<cit.> which require extra processing time, we adopt spatial transformation mechanism<cit.> into epBRM to get a similar effect with pooling based approaches. In essential, our approach first spatially transforms input point clouds and then predicts 3D bounding boxes. We then take the inverse transform from the predicted 3D bounding box to the input point cloud's original coordinate.





In this work, we are focusing on four spatial transformation mechanisms: translation, rotation, scaling, and centering. They take the input point clouds P ∈ℝ^Nx3 with N point clouds, and 3 dimensions for x, y, z coordinates.



Translation and Centering The output of transformation mechanism is defined by (t_x, t_y, t_z) which represents the translation of P by x_0, y_0, z_0 as following equations. 


    x_0   = 2* ( σ(t_x)  - 0.5) * T_x ,
    
    
     y_0   = 2* ( σ(t_y)  - 0.5) * T_y ,
    
    
     z_0   = 2* ( σ(t_z)  - 0.5) * T_z

where σ(·) is the sigmoid activation function. In this work, we set (T_x, T_y, T_z) to the same values as dist_bound. 

The difference between translation mechanism and centering mechanism is determined by whether there exists target transformation parameters to be learned. In translation mechanisms, the network is trained without target transformation parameters to extract most informative feature(<ref>). Unlike to the translation mechanism, centering mechanism is trained to predict the center of object as transformation parameters(<ref>).

Rotation The output of rotation mechanism is defined by t_r which represents a clockwise rotation of P around z axis by rot_z as following equation. 


    rot_z   = 2* ( σ(t_r)  - 0.5) * T_r

We set the value of T_r as π/4 to avoid excessive rotation of input point clouds.

Scaling The output of scaling mechanism is defined by (t_s, xy, t_s, z) which represent the scaling of P on ground plane, scale_xy and on z axis, scale_z as following equation.

    scale_xy   = pow(T_s, xy, 2*( σ(t_s, xy)  - 0.5)),
    
    scale_z   = pow(T_s, z, 2*( σ(t_s, z)  - 0.5))

We set a value of T_s, xy, T_s, z as 2.0, thus constrain scaling factors within [1/2, 2] to avoid distorting input point clouds.


















 §.§ 3D bounding box regression
 

After the transformation is applied to input point clouds, epBRM finally predicts coordinates for the 3D bounding boxes, including location, rotation, and size of objects. We use confidence score predicted from each localization module for scoring our predicted 3D bounding boxes.

Location is predicted by 3 coordinates (t_x, t_y, t_z) for x, y, z direction relative to the origin as following equation:


    x    = 2* ( σ(t_x)  - 0.5) * d_x ,
    
    
     y    = 2* ( σ(t_y)  - 0.5) * d_y ,
    
    
     z    = 2* ( σ(t_z)  - 0.5) * d_z

This has a same form as Equation (1) except for the value of the hyper parameters, (d_x, d_y, d_z).
Aiming for finer prediction of location, we set values for the (d_x, d_y, d_z) as follows:

    [d_x, d_y, d_z]^T    = 0.5* [T_x, T_y, T_z]^T



Rotation is predicted based on hybrid formulation of classification and regression methods. We equally divide [0', 180'] to N_R bins and the rotation angle is predicted by 2xN_R coordinates (t_r_cls(i), t_r_reg(i))_i=1^N_R.

Size is predicted by 3 coordinates, (t_h, t_w, t_l), for height(h), width(w), and length(l). We set anchor of size coordinates, (h_a, w_a, l_a). We use (1.50m, 1.57m, 3.33m) for car, (1.73m, 0.6m, 0.8m) for pedestrian and (1.73m, 0.6m, 1.76m) for cyclist. Then, the size of object is predicted by:


    h = h_a e^t_h,    w = w_a e^t_w,    l = l_a e^t_l







 §.§ Network structure


We design epBRM as sequence of multiple PointNet building blocks. Fig. <ref> shows a structure of the building block based on PointNet . This structure is shared by transformation mechanism and 3D bounding box regression task except for the number of output at the last layer. 



In the case of the transformation mechanism, we transform point clouds using output of network, which is a transformation parameter to be applied to input point clouds(<ref>) and sample point clouds inside sampling region(<ref>).



 §.§ Generating training samples
 
We aim for setting the sampling region as tightly as possible around the object in order to exclude redundant point clouds that do not belong to the object. As we only have location prediction from localization module, we need rotation-invariant sampling region. To satisfy both requirements, we use cylinder shaped sampling region in this work.




























Considering a prediction error from localization module, we set size of sampling region slightly larger than ordinary size of objects. We sample point clouds inside a cylinder defined by radius r and min/max value of point clouds (z_min, z_max) along z axis. (<ref>)





We then describe data augmentation methods for generating training samples. Assume that the 3D bounding box coordinates of the ground truth object are given by loc_gt = (loc_gt,x, loc_gt,y, loc_gt,z), size_gt = (h_gt, w_gt, l_gt) and rot_gt.

First, we sample all point clouds inside sampling region centered at loc_gt. Then, we translate sampled point clouds by subtracting loc_gt from each point and rotate them by -rot_gt to align the object along y axis. 



We change the size of object by multiplying s_x, s_y, s_z for each x, y, z axis of point clouds independently. s_x, s_y, s_z are independently sampled from uniform distribution on the interval [0.9, 1.1]. 











We then rotate the point clouds back to original heading angle plus small angle r_z randomly sampled from uniform distribution on the interval [-pi/8, pi/8]. 











Finally, we sample loc_x, loc_y, loc_z independently from uniform distribution on the interval [-dist_bound, dist_bound].














 §.§ Loss, training and runtime
 
During the training of each network, we optimize the following multi-task loss end-to-end:

    Loss    =  L_loc + L_rot-cls + 1^rot-cls[L_rot-reg] +
    L_size
       +L_loc_center

If we use transformation mechanism such as translation, rotation and scaling, then we use <ref> as a loss function. If we adopt centering mechanism in epBRM, then we use <ref> as a loss function. 



L_loc_center, L_loc, L_rot-reg, and L_size are regression loss for intermediate center location prediction, location, rotation and size of bounding box regression, which are represented as Huber loss. L_rot-cls is classification loss for rotation, which is represented as cross-entropy loss.

For training a network, we use the generated samples described in <ref>. We use batch of 512 samples with fixed learning rate of 5e-4. We do not apply non-maximum suppression on the predictions.

We use one i7-6700k CPU and one Titan X GPU for training and inference. Up to 20 objects, it takes 6.5ms for sampling point clouds at each location predicted from localization module. For actual network inference, it takes 5.5ms with one transformation mechanism adopted in epBRM.






















§ EXPERIMENTS




 §.§ Experiment setup
 

Dataset We use KITTI dataset, the 3D object detection benchmark, to evaluate our approach. It provides synchronized 2D RGB images and 3D LiDAR point clouds, carefully calibrated with annotations on car, pedestrian, and cyclist class. We mainly evaluate our method on the car class which has the most training samples. For pedestrian and cyclist, we discuss in <ref>.



To evaluate our method, we split whole training set into train set of 3,717 frames and val set of 3,769 frames frames. Frames in train set and frames in val set are extracted from different video clips. 3D object detection performance is evaluated at 0.7 IoU threshold for car and 0.5 IoU threshold for pedestrian and cyclist. 


Localization module We utilize previous 3D object detection systems as localization modules. To show the applicability of the proposed method, we select three different 3D object detection systems publicly available as open source: F-PointNet, AVOD(FPN), and PointPillars. These 3D object detection systems use different feature and method for localization.(<ref>). We follow official instruction provided by the authors to train the localization modules and freeze it when we train/evaluate the epBRM network.














Evaluation metric To evaluate our approach, we mainly focus on whether epBRM improves a quality of 3D bounding box prediction. As well as recall and mean average precision(mAP) of detection which are general evaluation metrics for 3D object detection benchmark, we also measure ratio of detected ground truth objects defined as follows:

    ratio    :=  num. of detected GT objects/num. of all GT objects

where the term 'GT' represents ground truth and 'detected' means that the overlap between ground truth 3D bounding box and predicted 3D bounding box is greater than 70%.

This measurement considers all ground truth object regardless of its difficulty level and visible size of object in RGB image plane, thus provides more reliable measurement. 






 §.§ Comparisons of transformation mechanism
 

We first evaluate each spatial transformation mechanism on 3D bounding box regression task. In this experiment, we utilize PointPillars<cit.> as a localization module. We train epBRM with each transformation mechanism for 20k iteration. We use the box regression module which is trained without spatial transformation mechanism as baseline experiment. 



<ref> indicates that spatial transformation mechanisms is essential component of the box regression module to improve the detection performance. Among the various spatial transformation mechanisms, the centering mechanism is the most effective in improving the detection quality.



Fig. <ref> shows the validation result of centering mechanism until 55k training iterations. Note that epBRM outperforms the PointPillars<cit.> after 10k of training iteration. It takes approximately 40 minutes for each 10k iteration of training.





 §.§ Effect of dist-bound
 

In this experiment, we compare effect of dist_bound on several localization modules. The purpose of this experiment is to find optimal value of dist_bound that maximizes the performance of epBRM when epBRM is applied to different localization modules. We again train epBRM with centering mechanism for 20k iteration. 



 


Fig. <ref> visualizes the evaluation results when epBRM is applied to each localization module at different dist_bound. For PointPillars and AVOD(FPN), the value of ratio(<ref>) peaks when dist_bound is 0.15 and then gradually decreases after that. For F-PointNet, the value of ratio peaks when dist_bound is 0.30.



<ref> reports the performance gains when epBRM is applied to each localization module at optimal dist_bound found by this experiment. 






Note that the performance gain from epBRM significantly differs by each localization module. For example, epBRM greatly improves the performance of AVOD(FPN) while less improvement is observed from F-PointNet. This result infers that the box regression of AVOD(FPN) is comparably worse than the F-PointNet. 
























 







 










 §.§ Smaller objects: pedestrian and cyclist
 

Our approach is also applicable for predicting precise 3D bounding boxes for smaller objects such as pedestrian and cyclist. Again, we train epBRM using fixed value of dist_bound as 0.15 for 20k iteration and evaluate the performance by applying it to each localization module. Here, we omit the process of finding optimal value of dist_bound.




 §.§ Evaluation using KITTI test set
 

We also evaluate the proposed approach by using KITTI test set by submitting detection result to KITTI test server. Among several localization modules we evaluate so far, PointPillars<cit.> shows the best performance in all categories, thus we apply epBRM to prediction result from PointPillars. We train epBRM for 40k iterations, which takes approximately 160 minutes.





<ref> reports evaluation result on KITTI test set before/after epBRM is applied. We also reports original PointPillars result() reported by Lang et al.<cit.> for reference. Note that epBRM improves the recall of detection in most cases.











§ DISCUSSION AND FUTURE RESEARCH DIRECTION




In this work, we mainly focused on a box regression task, one of core components in detection pipeline that aims to predict precise 3D bounding box. By adopting spatial transformation mechanism into box regression module, we could build a small network which improves the detection performance within a short time of training.

We summarize the detection failure cases from our experiments. First, epBRM requires that localization modules provide reliable information about the location of objects. If the localization module fails to localize objects precisely, then no further attempt to refine 3D bounding boxes will be made by epBRM. 

Second, given the close proximity between ground truth objects and the predicted locations from localization module, epBRM results in false-positive predictions if the point cloud is sparse. We think that other source of information, such as additional sensor data, can be helpful to overcome such challenge.

The core idea behind the proposed approach is to gradually simplify the box regression task by applying spatial transformation mechanism on raw 3D LiDAR point clouds input. We think that this methodology is generic to other tasks which use raw 3D point clouds and not limited to the box regression task. We will explore the tracking task, optical flow task and instance segmentation task based on the methodology discussed in this work as our future research directions. 







IEEEtran


