Can We Trust You? On Calibration of a Probabilistic Object Detector for Autonomous Driving
Di Feng Lars Rosenbaum Claudius Glaeser Fabian Timm Klaus Dietmayer

Optimal Charging of an Electric Vehicle Battery Pack: A Real-Time Sensitivity-Based MPC approach
    Davide M. Raimondo
    December 30, 2023
================================================================================================




Reliable uncertainty estimation is crucial for perception systems in safe autonomous driving. Recently, many methods have been proposed to model uncertainties in deep learning-based object detectors. However, the estimated probabilities are often uncalibrated, which may lead to severe problems in safety-critical scenarios. In this work, we identify such uncertainty miscalibration problems in a probabilistic LiDAR 3D object detection network, and propose three practical methods to significantly reduce errors in uncertainty calibration. Extensive experiments on several datasets show that our methods produce well-calibrated uncertainties, and generalize well between different datasets.







§ INTRODUCTION


Reliable uncertainty estimation in object detection systems is crucial for safe autonomous driving. Intuitively, a probabilistic object detector should predict uncertainties that match the natural frequency of correct predictions. For example, if the detector makes predictions with 0.9 probability, then 90% of those predictions should be correct. Reliable uncertainty estimation builds trust between a driverless car and its users, as humans have an intuitive notion of probabilities in a frequentist sense <cit.>. Moreover, the uncertainties captured in object detectors can be propagated to other modules, such as tracking and motion planning <cit.>, so that the overall system robustness can be enhanced. 


In recent years, many methods have been proposed to model uncertainties in deep neural networks. Among them, the direct-modeling approach assumes a certain probability distribution over the network outputs (e.g. Gaussian distribution), and uses additional output layers to predict parameters for such a distribution. Due to its simplicity and real-time implementation, this method has been widely applied to object detectors in autonomous driving <cit.>. However, we find that the direct-modeling approach fails to produce reliable probabilities, causing uncertainty miscalibration problems. Using such an unreliable uncertainty estimation in object detectors can lead to wrong decision makings in autonomous driving (e.g. at the planning stage), which may cause fatal accidents, especially in safety-critical scenarios. 


In this study, we identify uncertainty miscalibration problems in a probabilistic LiDAR object detection network (Sec. <ref>) via calibration plots (Sec. <ref>). Then, we propose three practical methods based on recalibration techniques to alleviate such miscalibration (Sec. <ref>), and systematically study their robustness on several datasets. Experimental results show that our methods can significantly reduce the uncertainty calibration errors and improve the detection accuracy (Sec. <ref>). 





§ RELATED WORK






 §.§ Uncertainty Estimation for Object Detection

The methods to model uncertainty in object detection can be categorized into two groups: the ensemble approach and the direct-modeling approach. The former builds an ensemble of object detectors to approximate an output probability distribution with samples, e.g. using Monte-Carlo Dropout <cit.>. This approach has shown to represent the model uncertainty, and has successfully been introduced to tackle open-set object detection challenges <cit.> and active learning <cit.>. The direct-modeling approach uses network output layers to learn and predict the parameters of a pre-defined probability distribution, such as a multi-variate Gaussian distribution <cit.> or mixture of Gaussian <cit.>. It requires only a little additional computation during inference, and can improve the detection accuracy <cit.>. Therefore, we employ the direct-modeling approach to model uncertainty in our LiDAR object detector.




 §.§ Uncertainty Recalibration

Uncertainty recalibration techniques aim to improve the uncertainty estimation of a probabilistic model. Most of them are post-processing steps that directly adjust network probabilistic outputs via a recalibration model. Many models have been developed to calibrate classification uncertainty in deep learning, such as isotonic regression, histogram binning, and temperature scaling <cit.>. Besides, <cit.> introduces isotonic regression to calibrate uncertainties in multiple regression tasks. To the best of our knowledge, there is no previous work focusing on how to calibrate uncertainties for object detections. In this work, we employ isotonic regression and temperature scaling to recalibrate the classification part of our object detector, and extend them to recalibrate uncertainties over the bounding box predictions. Furthermore, we propose a simple loss function to reduce calibration errors during training.



§ PROBABILISTIC LIDAR OBJECT DETECTION






 §.§ Network Architecture

We model uncertainties in PIXOR <cit.>, a state-of-the-art one-stage LiDAR object detection network, with several modifications (Fig. <ref>) <cit.>. PIXOR takes the LiDAR bird's eye view (BEV) feature maps as input, and outputs classification scores and bounding box parameters for each pixel on the feature map. Denote an input sample as 𝐱, the network predicts object classes y_c with softmax score s_𝐱 (for brevity we only consider binary classification “Object" and “Background", i.e. y_c ∈{0,1}). It also regresses the object positions 𝐲_r ∈ℝ^6 including center positional offsets on the horizontal plane (dx and dy), length l, width w, and orientation θ. Following <cit.>, we encode the the final bounding box locations as the row vector 𝐮_𝐱 = [cos(θ), sin(θ), dx, dy, log(l), log(w)]_𝐱 (Fig. <ref>).







 §.§ Uncertainty Estimation

We leverage our previously proposed method <cit.> to model uncertainties in the object detection network. The softmax score is used to estimate the object probability, i.e. p(y_c=1|𝐱)=s_𝐱. We also assume that the network regression outputs follow a multi-variate Gaussian distribution p(𝐲_r|𝐱) = 𝒩(𝐮_𝐱, Σ_𝐱), with its mean being the network's bounding box regression 𝐮_𝐱 = [cos(θ), sin(θ), dx, dy, log(l), log(w)]_𝐱, and its covariance matrix being a diagonal matrix: Σ_𝐱 = diag(σ^2_𝐱), σ^2_𝐱 = [σ^2_cos(θ), σ^2_sin(θ), σ^2_dx, σ^2_dy, σ^2_log(l), σ^2_log(w)]_𝐱. Here, each element in the row vector σ^2_𝐱 represents a variance (or uncertainty) 
to the corresponding element in 𝐮_𝐱. We add additional output layers in our object detector to directly regress σ^2_𝐱. In this way, the network can estimate the probability distribution of the bounding box prediction during test time. We employ the multi-loss function <cit.> to train the regression tasks:

    L_reg = 1/2(𝐲_r-𝐮_𝐱)diag(1/σ^2_𝐱)(𝐲_r-𝐮_𝐱)^T + 1/2log(σ^2_𝐱)1^T,

with 𝐲_r being the regression ground truth. 









§ UNCERTAINTY EVALUATION







 §.§ Definition


Let a labeled dataset {(𝐱^n, y_c^n, 𝐲_r^n)}_n=1^N be the i.i.d. realizations of jointly distributed random variables X, Y_c and Y_r, where X refers to the input data, Y_c the binary classification labels {0,1}, and Y_r the bounding box locations. The marginal distributions of the target variables can be specified as X, Y_c ∼ℙ_c and X, Y_r ∼ℙ_r. Our LiDAR detector predicts the softmax score as the object probability distribution, which can be denoted as F^n_c(y_c=1)=p(y_c=1|𝐱^n)=s_𝐱^n. In the case of bounding box regression, we use the cumulative distribution function (CDF) F^n_r(𝐲_r), with its probability density function being predicted by the detector p(𝐲^n_r|𝐱^n) = 𝒩(𝐮_𝐱^n, Σ_𝐱^n), and F^n^-1_r(p) for its quantile function (inverse cumulative distribution function).


Intuitively, a reliable (or calibrated) uncertainty estimation from the object detector means that a predicted probability should match the natural frequency of correct predictions. For example, if the detector classifies 100 objects as “Car" with a softmax score of 0.9, we expect that 90% of those objects should be correctly classified; In the regression task, if the detector estimates 100 object positions with the 90% confidence interval, 90% of object ground truth positions should fall into such a confidence interval. More formally, a calibrated classification uncertainty for the class label k indicates ℙ_c(Y_c=1|F_c(Y_c=1)=p) = p,  ∀ p ∈ [0,1].
When N is large enough, we have the following approximation <cit.>:

    ℙ_c(Y_c=1|F_c(Y_c=1)=p) ≈∑_n=1^N 𝕀(y^n_c=1, F^n_c(y_c=1)=p)/∑_n=1^N 𝕀(F^n_c(y_c=1)=p).




Similarly, a calibrated regression uncertainty means ℙ_r(Y_r ≤ F^-1_r(p)) = p,  ∀ p ∈ [0,1]. It can be approximated by <cit.>:

    ℙ_r(Y_r ≤ F^-1_r(p)) ≈∑_n=1^N 𝕀(𝐲^n_r ≤ F^n^-1_r(p) )/N.


By using a multi-variate Gaussian distribution with the diagonal covariance matrix, we assume that each element in the regression task is independently distributed. In this regard, we only focus to calibrate the marginal probability for each element separately. In the rest of the paper, we denote y_r as an element in the regression task, and F^n_r(y_r) its CDF. We leave a comprehensive study to calibrate the full probability, where the elements in the regression are dependent, as an interesting future work.



 §.§ Evaluation Tools


The notion of uncertainty calibration can be visualized by the calibration plot (Fig. <ref>), where the horizontal axis represents the predicted probability of the detector, and the vertical axis the empirical probability. In practice, to draw calibration plot for classification, we group the softmax scores into M intervals using the probability thresholds 0 < p_c^1 < ... < p_c^m < ... < 1, and calculate the empirical probability following Eq. <ref> for each interval, denoted as p̂_c^m. In the case of regression, we group predictions into different confidence levels p_r^m, calculated by F_r(𝐲_r), and estimate the corresponding empirical frequency p̂_r^m by Eq. <ref>. Here, we draw a calibration plot for each bounding box regressor separately. A well-calibrated detector produces the diagonal line in the calibration plot (Fig. <ref>(b)). A miscalibrated detector can suffer from over-confident predictions (actual calibration curve is under the diagonal line), or under-confident predictions (above the diagonal line).

Similar to <cit.>, we employ Expected Calibration Error (ECE) as evaluation matric. ECE calculates the weighted error between the actual calibration curve and the diagonal line, i.e. ECE=∑_m=1^M N_m/N|p^m-p̂^m|, with N_m being the number of samples in the mth interval. ECE=0 corresponds to perfectly calibrated predictions.




§ UNCERTAINTY RECALIBRATION



In this section, we introduce in detail our methods to separately recalibrate the marginal probability distribution in each element of the bounding box regression.



 §.§ Isotonic Regression

Recall that p = F_r(y_r) represents the predicted bounding box probability from the network. We train an auxiliary model based on the isotonic regression p ↦ g(p), which is a non-parametric monotonically increasing function, to fit the true probability ℙ_r(Y_r ≤ F^-1_r(p)) <cit.>. During test time, the object detector produces an uncalibrated uncertainty, which will then be corrected by the recalibration model g(·) as the final output. In practice, we build a recalibration dataset from the validation data to learn the isotonic regression model for our pre-trained LiDAR object detector. Specifically, denote {(𝐱^n, 𝐲_r^n)}_n=1^N as the validation dataset; we can build its corresponding recalibration data {( F^n_r(𝐲^n_r), P̂(F_r(𝐲_r)) ) }_n=1^N, where P̂(F_r(𝐲_r)) refers to the empirical probability calculated by Eq. <ref>. 



 §.§ Temperature Scaling

We use different scalars T>0 for each regressor to adjust the variance prediction:
σ̂←σ^2/T,   ∀σ^2 ∈{σ^2_cos(θ), σ^2_sin(θ), σ^2_dx, σ^2_dy, σ^2_log(l), σ^2_log(w)}. When T>1, the adjusted Gaussian distribution becomes sharper, indicating smaller uncertainty. When T<1, the distribution becomes broader, representing larger uncertainty. With T=1, the original probability is unchanged. The optimal T can be found by maximizing the Negative Log Likelihood (NLL) score on the recalibration dataset. 



 §.§ Calibration Loss
 
The object detector learns to predict variances in an unsupervised way (Eq. <ref>), as there is no ground truth for variances in Eq. <ref>. Therefore, the regression loss function is not designed to guarantee calibrated uncertainty. Intuitively, a well-calibrated uncertainty for a Gaussian distribution indicates that for every data sample 𝐱, the predicted variances should match the true differences between predicted and true bounding boxes, i.e. σ^2_𝐱!= (𝐲_r - 𝐮_𝐱) ⊙ (𝐲_r - 𝐮_𝐱), with ⊙ being the element-wise multiplication. In this regard, we design a simple calibration loss L_calib to regularize variances, and train the object detector with a new loss function L_total that adds L_reg and L_calib:

    L_calib = σ^2_𝐱 - (𝐲_r - 𝐮_𝐱) ⊙ (𝐲_r - 𝐮_𝐱) , 
        L_total = L_reg + λ L_calib,

where the hyper-parameter λ is used to control the loss weight.
    


 §.§ Comparison

All three uncertainty recalibration methods can improve the probability estimations (Sec. <ref>). Isotonic regression and temperature scaling are post-processing steps after training the object detector. They do not change mean values 𝐮_𝐱, and thus do not affect the detection accuracy. Furthermore, they are designed to optimize the uncertainty estimation based on the whole recalibration dataset, and do not guarantee that each detection is better-calibrated. Conversely, calibration loss improves the probability estimations when optimizing the object detector. It indirectly improves the detection accuracy by encouraging the network to produce better-calibrated uncertainties for each detection (Sec. <ref>). 

Given enough recalibration data, isotonic regression is guaranteed to produce perfect calibration plots, regardless of the underlying probability distributions <cit.>. However, it changes the probability distribution (in our case a non-Gaussian distribution), making it less interpretable and applicable. On the contrary, temperature scaling and calibration loss recalibrate uncertainties based on the same probability distribution (in our case a Gaussian distribution), which is highly desirable when propagating them to other modules, such as object tracking with Kalman filters. However, if the assumed probability distribution significantly differs from the true distribution, both recalibration methods may fail to achieve well-calibrated uncertainties.





§ EXPERIMENTAL RESULTS










 §.§ Identifying Uncertainty Miscalibration
 
In the first experiment, we use calibration plots to identify the uncertainty miscalibration problem in our probabilistic LiDAR object detector, and then study how such a problem is related to the training process. For the sake of brevity, here we only show the marginal of the dy regression variable, though we observe similar results in other regression variables as well. 



  §.§.§ Experimental Setup

We conduct experiments on the training data of the KITTI object detection benchmark <cit.> using only the “Car" category. We split the data into a train set and a val set with approximately 50:50 ratio <cit.>. The LiDAR detector is trained with the KITTI train set, and its uncertainty estimation quality is evaluated on the KITTI val set. Similar to <cit.>, we pre-train the detector with the normal L_2 loss for 45 epochs, using the SGD optimizer with a learning rate of 0.02. Then, we reduce the learning rate to 0.001 and train the detector following Eq. <ref> for another 100 epochs. Tab.<ref> reports the car detection performance in the Bird's Eye View (BEV), with the Intersection Over Union IOU=0.7 threshold. Our network (“Ours") produces on-par results with the original PIXOR network.



  §.§.§ Calibration Plots

Fig. <ref> shows the calibration plots for classification and regression respectively. From the figures we observe that the probabilistic object detector produces miscalibrated uncertainties. For example, the network is over-confident classifying objects when the predicted softmax scores are smaller than 0.7, and under-confident with softmax scores larger than 0.7 (Fig. <ref>). Conversely, except for the cos(θ) regression, the network makes under-confident predictions at a smaller confidence levels, and over-confident predictions at a higher confidence levels (Fig. <ref>). 






  §.§.§ Training Epochs

We find that achieving more accurate object detection does not guarantee better uncertainty estimation. Fig. <ref> illustrates how the regression and uncertainty estimation errors for dy predictions over the course of training. The horizontal axis represents the training epochs, starting at epoch 45, when we start to model regression uncertainties using Eq. <ref>. The vertical axis represents the expected calibration errors and L_2 loss calculated on the val set. The L_2 loss drops during the training, indicating that the network makes more and more accurate dy predictions. However, the calibration errors tend to increase after the 65th training epoch, showing over-fitting behaviour. A similar phenomenon for classification is found by Guo et al. <cit.>. 













 §.§ Uncertainty Recalibration
 
In this experiment, we evaluate the performance of the proposed uncertainty recalibration methods (“Temp. Scaling", “Isotonic Regr.", and “Calib. Loss"). We first show that the network produces better uncertainties after recalibration. It also achieves a higher detection accuracy when being trained with the calibration loss. Afterwards, we systematically study the robustness of the recalibration methods against different recalibration dataset sizes, and their generalization between different datasets.



  §.§.§ Experimental Setup

The uncertainties predicted by the LiDAR detector in the previous experiment are used as the baseline. They are also used as inputs for the “Temp. Scaling" and “Isotonic Regr". For “Calib. Loss", we train a new LiDAR detector with the calibration loss (Eq. <ref>) using the KITTI train set. Additionally, we combine the calibration loss with the other two recalibration methods as a comparison (“Calib. Loss + Temp. Scaling" and “Calib. Loss + Isotonic Regr."). We equally split the KITTI val set (Sec. <ref>) into the KITTI recal set and the KITTI eval set. The former is used to optimize the recalibration models built by the temperature scaling and isotonic regression, and the latter to evaluate all recalibration methods. 



  §.§.§ Performance

Tab. <ref> shows that the network trained with the calibration loss (“Ours + Calib. Loss") improves the average precision (AP) up to nearly 4% compared to the network without calibration loss (“Ours"). This might because calibration loss serves to regularize σ^2. As a result, the network learns to detect objects more accurately with improved uncertainty estimation. Tab. <ref> compares the Expected Calibration Errors (ECE) between the recalibrated uncertainties and the baseline uncertainties without recalibration. All recalibration methods consistently outperform the baseline with smaller ECE values. Specifically, “Isotonic Regr." performs better than “Temp. Scaling" and “Calib. Loss". This is because the recalibration dataset is large enough to train a well-performed isotonic regression model (cf. Sec. <ref> for more discussion). When combing “Isotonic Regr." and “Calib. Loss", we achieve the best calibrated uncertainties.  







  §.§.§ Qualitative Observations

Fig. <ref> illustrates how recalibration models adjust uncertainty estimation. We colorize each detection according to the softmax score, and draw 95% confidence intervals for dx and dy regressions respectively. We use Isotonic Regr. to recalibrate classification uncertainty, and Temp. Scaling for regression uncertainties (in order to keep the Gaussian distribution assumption). From Fig. <ref> and Fig. <ref> we observe that the recalibration model reduces classification scores for the false positive detections (detections 1, 3, 6, 7), and increases classification scores for the true positive detections (detections 2, 4, 5, 8, 9). This is because the uncalibrated network produces over-confident predictions at small softmax scores, and under-confident predictions at bigger softmax scores (Fig. <ref>). Fig. <ref> shows that the positional confidence intervals do not fully cover the object ground truths (detections 12, 14, 16), as the uncalibrated network produces over-confident position estimations (Fig. <ref>). After recalibration, the confidence intervals are larger such that they fully cover the object positions (Fig. <ref>). Note that obj15 is a false negative in ground truth labels.



  §.§.§ Robustness Testing

We extensively study the robustness of the recalibration methods. Here, we focus on evaluating “Temp. Scaling" and “Isotonic Regr.", as their performance highly depends on the recalibration dataset.  First, we optimize both models with the reduced KITTI recal set and check their performance on the KITTI eval set. Fig. <ref> demonstrates that both methods show good recalibration performance, even with merely 1% of the recalibration data. While “Isotonic Regr." degrades dramatically with 0.4% of data, “Temp. Scaling" performs surprisingly well, showing its high robustness against small recalibration dataset size. 

Next, we evaluate the recalibration generalization capability on new test data. In this regard, we train “Temp. Scaling" and “Isotonic Regr." with the KITTI recal set, and evaluate them on the nuScene dataset <cit.>. Tab. <ref> shows the ECE averaged over all network predictions. The nuScene data significantly differs from the KITTI dataset regarding recording locations, weathers, and sensor setup. Despite that, the recalibration models trained with the KITTI data still halved the averaged ECE in the nuScene data, showing good generalization capability in recalibrating uncertainties. When using only 1% of nuScene data to update the recalibration models, we achieve the best uncertainty recalibration performance.













§ CONCLUSION AND DISCUSSION


In this work, we identify that the direct-modeling method, which is a common method to model uncertainty in deep object detectors, produces miscalibrated uncertainties. Based on a probabilistic LiDAR 3D object detector and calibration plots, we study how the course of training affects the uncertainty miscalibration, especially for the bounding box regression task. Then, we propose three practical uncertainty recalibration methods to alleviate such problems. Experiments on both KITTI and nuScene datasets show that our methods estimate well-calibrated uncertainties, are robust against different recalibration dataset sizes and can generalize to new datasets. 

This work calibrates marginal probabilities in the regression task, with the assumption that each regression variable is independently-distributed. However, we find that the regression variables can be dependent in some scenarios as well. Fig. <ref> illustrates the distribution between the errors of dy predictions (longitudinal position) and l predictions (object length) from the objects which are standing approximately straight in front of the ego-vehicle, and are facing towards or backwards to the ego-vehicle. The errors of dy and l are highly correlated, with a Pearson Correlation Coefficient of more than 0.6. We also show two examplary detections, where the object parts which face towards the ego-vehicle are well-localized, but the backsides have large localization errors. In these scenarios, it is necessary to take dependency within regression variables into consideration (instead of estimating and calibrating marginal probabilities), or propose new bounding box encodings that ensures independency. We leave it as an interesting future work. Furthermore, we intend to model uncertainties in multi-modal fusion networks <cit.> and network quantization <cit.>.







IEEEtran



