One can easily imagine a situation like this, where two models perform similarly based on the test set RMSE, but very differently w.r.t the uncertainty-error distribution. Which model to choose then depends on a consideration of the intended application. If the goal is to get a sample of predictions with as low error as possible, one might choose a model with uncertainties performing similarly to those in Figure even if it shows bad calibration for higher uncertainties. If one is looking for a more "general purpose" uncertainty estimator one might go with something like those showed in Figure .