The ability to assign confidence to predictions is becoming an increasingly important aspect of molecular ML. Therefore, the ability to evaluate whether the uncertainty estimates of an uncertainty quantification (UQ) method are reasonable is required. We present a comparison of the error-based calibration method recently proposed by Levi et al. and three popular methods for the evaluation of uncertainty estimates; Spearman's rank correlation coefficient, the miscalibration area and the negative log likelihood (NLL). We apply the evaluation metrics to uncertainty estimates for two different chemical datasets and three different UQ methods and find the error-based calibration to be the superior choice for UQ evaluation. For the NLL and Spearman's rank correlation coefficient specifically, we found that the introduction of simulated reference values was necessary for these metrics to bear any meaning.